Problem with VIP suite on Cyclone V device

Without thinking about it too much, "bigger buffer gave better result" so place a Frame Buffer IP directly before the CVO. If that works, then all your parameters regarding video/stream formats are correct and your issue is simply clock/buffer related. As the CVO is underflow, that means it is draining data faster than it is receiving it, but I'm not sure why that would be based on your previous description of your system. Anyway, adding a Frame Buffer should give you some clue.