VIP FrameBuffer - How do you think it is accessing the SDRAM?

Are you using burst transfers with your custom frame buffer?

You can't actually read and write to SDRAM at the same time. You are either doing one or the other at any given time. Switching between read and write mode is slow. Therefore you want to perform your reads and writes in bursts to keep the efficiency higher.

Are you sure your FB does not have some other bug causing problems not related to memory bandwidth? Perhaps you could test the system with a much slower pixel clock to try and prove the rest of the logic is correct.