Looking for tips on how to implement variable length data transformation / parsing.

This is what I understood from your text:

You already have the block to receive the packets and remove UDP encapsulation; these blocks give you the UDP payload via a 64 bit wide, Avalon-ST interface.

Now, you need to break that UDP payload into smaller packets and then feed it into another Avalon-ST sink.

Here's my suggestion:

Having to pause the flow while you breakdown the 64 bit words in unavoidable.

But you can do it while keeping up with the throughput.

10Gbit/s over 64 bit means your source will give you, at full throughput, ~156Mword/s.

If you feed the 64 bit words to a FIFO and if the logic that reads and breaks the messages works at more than 156 MHz, then you have extra time to compensate the pauses.

The same kind of goes for the output.

If the breaking logic doesn't have data, it has to pause. But if runs at a higher rate, it can make up for the pauses.

And if needed, you can again use FIFOs to match the rates.