video streaming protocol

It's not entirely clear what you're asking here.

If the video stream that you intend to process could change resolution, then you may want to keep the control packets in external memory for your own benefit e.g. to figure out how big the frame you have stored is.

Certainly, you will need to send the type with the SOP when you transmit the processed frame back down an Avalon-ST Video datapath. It is a good idea to to send a control packet before each video frame. In fact, that may become required in a future release.

Aidan