SIV PCIe high performance ref design data can arrive out of order

Your 4k requests are split two times: Once in your CDMA wrapper leading to multiple – say: 8 – read requests, 512 bytes each, respecting the max_read_request_size parameter and hopefully the 4k read request boundary. Then, second, the responses are split by the completer into multiple packets for each request. It’s impossible to tell how many completion packets will be sent by the RC as it only has to adhere to the Read Completion Boundary (RCB), so your read request might end up being serviced as 64 byte chunks, i.e. 8 or 9 completion packets per read request.

While the completion packets for a single read request must be transferred in order to each other, they can overlap in any possible way with completions for the other read requests. This is independent from the relaxed ordering attribute of your read request, and you cannot avoid that behavior of PCIe. Sure, a chipset is not required to chop them up artificially, and it seems your Xeon chipset keeps them in-order, but it is not required to do so, and your MCP7A platform seems to make use of that feature.

So, as soon as you issue multiple PCIe read requests, you have to be prepared to do out-of-order processing for each of the requests – remember, the completions for each request must come in-order. If you have to feed a Fifo internally, be prepared to instantiate some buffer inbetween: Either one fifo per parallel request, or a unified buffer memory block which you write out of order, as the completions arrive, and read in-order.

Keep an eye on completion timeout functionality. If you don’t design it in from the start, you might end up with a rewrite later.

Actually, there is much magic in the CDMA design example, and is not exact where it should be, sometimes obviously buggy. Nothing I would spin off. E.g., look into altpcierd_cpld_rx_buffer.vhd and search for »TODO corner case when simultaneous RX TX«. The example does bring you somewhere, but I like to do my own faults instead of debugging missing support for corner cases. And not all missing/wrong code in the CDMA example documents itself as such …