--- Quote Start ---
instead of having this test pattern blended in nicely I see flickering colors at the area of where the test pattern should be. I can also see that frames are dropped by reading the dropped frames register from the frame buffer.
--- Quote End ---
Difficult to say without more information, but "flickering" and "dropped frames" usually sound like clock rate mismatch or memory access issues? To continue brute forcing things, simply try increasing (2x? 4x? 8x?) your FIFO depths across all modules and VFB burst target sizes. Any change in the on screen behavior and you'll get a hint the issue is in this area.