PCIe w/ Avalon ST: Equivalent of ko_cpl_spc_vc0?

Hi Brendan, I think that there is a misunderstanding about credit accounting on your side.

All credit accounting is done inside the PCIe IP blocks, except for CPLH/CPLD credit, so accounting for that must be done in the application. An application that does not handle CPLH/CPLD credit appropriately will eventually run over the receive buffers in the PCIe IP, lose data and will fail, depending on the recovery options implemented in the application and the PCIe IP. Think of the credit handling as a required add-on for tag handling: You are not allowed to use a tag if a completion with that Transaction ID is still flying around; similarly, you are not allowed to send out a read request if the completion(s) wouldn’t fit the receive buffers at the time of arrival.

IMO there are two reasons for having CPLH/CPLD credit accounting in the application: 1. TLP Transaction Ordering rules, 2. Completion Timeout handling. For the former: If the PCIe IP would just block your read request because of a lack of CPLH/CPLD credit, it would block any other, higher priority outbound traffic as well, like completions to CPU read requests (see cases D3/E3 in the Ordering Rules Summary Table). That would allow deadlocks to occur. For the Completion Timeout aspect: This is an application-specific mechanism that can not be supported universally by the PCIe IP, and that problem is FPGA vendor independent. Sad enough, Altera’s example code (chaining_dma) does not contain any logic for Completion Timeout handling.

You are on the wrong track if you try to get any support for CPLH/CPLD credit accounting from the IP block’s test_in/_out. You have to do it on your own, completely. The only hint you get from the IP block is ko_cpl_spc_vc0 – and in its absence, the equivalent information from the .XML/.HTML or comments in the top-level megawizard file –, which (statically) indicates the maximum credit you have for replies to your read requests.

Based on that credit information, your CPLH/CPLD credit handling logic has to:

• check available CPLH/CPLD credit before sending the read request, eventually delaying the read request until sufficient credit is available – mind PCIe ordering rules and let other outbound traffic pass –,

• debit CPLH/CPLD upon actually sending out a read request,

• charge CPLH/CPLD when a read completion arrives (but charge only to the extent of the actual reply size),

• charge CPLH/CPLD with all remaining credit of that tag when a Completion Timeout occurs (and notify the read requester of the termination of that read request).

In my application I have a block called tagcc that is responsible for tag management and CPLH/CPLD handling. Prior to forming a read request, any read requester block (descriptor fetch, DMA TX) has to ask for a tag from tagcc, indicating the amount of completion credit this read request will require. tagcc will only release a tag if there is a tag available (not all are in-flight) and there is enough credit for the completions not to overrun the buffer on arrival. tagcc remembers three things: (a) this tag is in-flight, (b) the time of issue, (c) the amount of outstanding CPLH/CPLD. On arrival of completions, not only the global CPLH/CPLD counters are charged, but the tag’s outstanding credit is reduced. Once the final completion arrives, the tag is free to be used again and marked not-in-flight. In the background, all in-flight tags are checked for a timeout, and once a Completion Timeout occurs, the tag’s outstanding CPLH/CPLD credit is freed, the tag itself is freed, the read requester is informed of the termination, and an appropriate error is issued on cpl_err.

Additionally, my receive TLP interface is designed to support STREAM_FC. If STREAM_FC (in Xilinx notation) was officially supported by the Altera PCIe Compiler User Guide, I could drop CPLH/CPLD accounting altogether and would be done with just in-flight and Timeout handling.

– Matthias