VGA resolution problem

At the moment I hook up the pixelbuffer to the resampler then to the scaler, which is hooked to the alpha blender and then to the controller.

When I remove the scaler I set the resolution of the pixelbuffer to 640x480 but I suppose that's where the culprit is. I can then only write about half of the screen, the bottom half is then a repeat of the top (indicating an overflow with reading the memory perhaps?).

I checked again today through the documentation but it doesn't anywhere mention that I can write to a 640x480 buffer, so I guess no luck with the IP cores.