The reason why the performance is so bad is that the read and write masters are ping ponging for access to the SDRAM. If you increase the arbitration share of each master that should reduce this effect. If you were reading from the memory and writing the data somewhere else this problem would also go away. In the end your VGA output will end up being the bottleneck so as long as you have enough memory bandwidth to keep the video pipeline feed you should be fine.
The only way to get the SDRAM operating higher than 100MHz reliably would be to constrain the interface since Quartus II has no clue what kind of off-chip timing relationships are needed (that's what the constraints are for)