Latency and corruption on ARM ethernet port

I'm sending an H264 stream out on the ethernet port of the ARM on my Cyclone V SOC chip. In general it sends fine, but occasionally there's a 'blip' and either packets get corrupted or paused for a while and then catch up. Equally, packets that are timed to be at an interval, and I make sure are inserted at 100ms intervals, are often reported as being >500ms apart.

I can receive a stream in without issues, so am wondering why there would be any delay on sending a stream.

If anyone has had similar issues sending video into/out of the ARM I'd be very interested to hear.

Regards,

Simon