Video deinterlacing with sopc builder

Are you using synchronize on F1 or synchronize on F0? If yes then I believe this system does not work because the deinterlacer is discarding every other field and uses only F0s (or F1s) to generate progressive frames.

This would mean that the deinterlacer cannot avoid a huge gap between two successives output frames and that your output is underflowing. This would also explain why this system is working with a frame buffer. Switching to synchronize on both may fix the problem.