Problem with VIP suite on Cyclone V device

It means your CVO is sinking data faster than your frame buffer is sourcing it. I am guessing your frame buffer Avalon-MM ports are either running too slow or configuration mismatches resulting in inefficient operation of the memory interface?

You'll want to make sure the frame buffer has frame repeating enabled, and then you may want to end up with the Avalon-MM Master ports running on a separate (faster) clock domain than the Avalon-ST ports. Basically, you need to make sure the frame buffer is configured correctly and it has adequate resources to fetch data faster than the CVO can process it. You can do this with either faster clocks or with wider Avalon-MM width. In normal use, the frame buffer should be more than capable of keeping the CVO FIFO nearly full for the entire frame.

From your original post, it sounds like you want CVO clock 27MHz with 16bpp. If your DDR3 is 216MHz, your Avalon-MM is probably half that but double wide (108MHz, 32-bit ? 64-bit?) This is plenty of capacity as long as the frame buffer is using the separate clock domain for the Avalon-MM ports and is using bursts to stream the data. If you've already checked all these things, you can increase the depth of the frame buffer FIFO's and burst lengths and confirm they have an impact on your 17th row underflow.