Forum Discussion
Altera_Forum
Honored Contributor
Honored ContributorPCIe freezing
Hi,
I have strange problem with PCIe IP. I have a custom board with ArriaGX and PCIe instantiated using MegaCore flow with AvalonST interface. The PCIe works (there are some DMAs running, target access, everything works fine), but occasionally the PC freezes when accessing my device. The design is quite complicated, so I started to minimize it and I ended with simple PCIe interface to on-chip memory and SW application that only writes to the device as fast as is possible. In this situation the system freezes after few seconds. Using SignalTap I see folowing: - there are coming writes before the freeze - only rx_xxx, nothing else. - no interrupt is signalled - rx_ready, tx_ready are continuously asserted - tx_credits are nonzero - there are just no transactions coming out of the IP when the system freezes even though my application is ready to receive. So I configured the IP to show me the test_out (512 bit) interface to see more and I found this: - in the time, when the packets stop coming there is bit 313 and 314 set (received erroneous TLP, packet with wrong sequence number) and they are coming repeatedly after that moment - also in the same times there is bit 85 set (that is Data link layer error - TLP error) - in the reaction to bad TLP it seems to me, that the IP sends a NAK to the root port (just few tick after the first erroneous TLP is received) - some ticks after the NAK is sent there is stop in receiving PLP ACKs for some time (I suppose that the IP tries to reset the link) and that's all - I do not see anything more coming. At first I thought that It could be some weird problem with the TLP sequencing, but I discovered, that time from time there is exactly the same situation (bad TLP sequence number, NAK, re-training of the link) but IP recovers from that error without any problem - the packets start coming again. Do you have any ideas, where should I look, what could cause this problem? Maybe I did not find the real moment where the IP freezes. How to track/debug it? Martin23 Replies
Altera_ForumPH/PD changing while receiving write requests is expected, and they do recover. NPH looks good as well. NPD=0 looks bad but might be correct as it’s useless to allocate memory for non-posted data when a native Endpoint does not support non-posted traffic with data. At least I think it doesn’t.
About CPLH/CPLD: Amazing they are zero. Maybe they get positive once non-posted request are issued by the IP.- Altera_Forum
Hi Matthias,
In altera datasheet to the IP core is written that "PCI Express endpoints must offer an infinite number of completion credits", I don't know if it is also rule for root devices, but there is also mentioned, that "When infinite credits are available, the corresponding credit field is all 1's". So all zeroes does really mean zero credits available. Could this be a problem? When I do not issue writes too much fast, the device works with a problem - there are reads/writes and dma read/writes running without any problem - so posted, nonposted transactions and completions are issued on both ends. Just when I issue big bunch of "target writes" from the PC, it freezes... Martin - Altera_Forum
I think it does offer infinite completion credits, but they are not allocated on the RX memory and that’s what this debug output should show – otherwise the debug output would be useless. You could watch out for an increase of CPLH/CPLD values when you issued a non-posted DMA read from the application to verify this behavior.
Generally, I don’t know why large and/or frequent writes should be an issue for PCIe, the IP core or your application. They are the simplest form of transaction, and the system should not be harmed by your application improperly generating TLPs, like violation of max_read_request_size, max_payload_size, byte enables or read completion boundary (RCB). Sorry to repeat the question: Are you handling read requests properly, i.e. either responding with valid data or sending CA? Those requests, once sent to your application, will not show up on the RX buffer usage but affects the root complex’ buffer handling and error reporting. - Altera_Forum
Hi Matthias,
currently I have disabled lot of my application functionality for debugging of this problem, so I do not handle read requests anyhow - now they are just getting timeouted. As a part of debugging I disconnected the whole AST transmitter side from my application (I tied it to ground) to be sure, that some invalid packet from my application is not a cause of this problem. When I issue a read on the PC side, I receive all FFs. When I track changes in credits with the Signal Tap, the NPH stays the same, they are just decremented by one and then incremented back again. I possibly can send CA - I'll only have to find out how to do it as I did not implemented it before, I handled all request from the PCIe. - Altera_Forum
Hi Martin,
If I understand you correctly, the problem appears more frequent if you remove your TX part with all functionality required for read completion and DMA transmission. This makes me think your – missing/wrong – completions migth be the cause for the problems. Sending CA as a Cpl is typically faster than sending a CplD. You always have the problem of managing the reception of non-posted requests while you are still handling the last Cpl/CplD, probably slowed down even more by de-assertion of tx_ready by the Hard/Soft IP. Altera’s approach to this problem by adding rx_st_mask is nice but rather useless as you have to buffer a whoppin’ 14 more read requests in 64-bit AST mode, and you have to do it inside the application which contradicts the idea of a wrapper IP. What you actually could do is play with the max_payload_size and RX buffer space allocation performance parameters, actually try to make the PCIe buffer parameters as default as possible. - Altera_Forum
Hi Matthias,
the problem started to appear on fully functional design. I found, that PC with my device freezes time from time. It was running correctly for hours or days until that freeze - there were running loads of read/write access, DMA read/writes without any problem. After some searching I found that in some situation the control software writes lots of data to my device (reconfiguration of control registers) and that in some cases freezes the PC. So I tried to remove as much as possible from my design to find which part could be responsible for this problem, but until now I did not find any part of my design responsible for freezing of the system - I've ended with design that does not transmit anything and the software only writes to my device... Martin - Altera_Forum
Martin,
even if you don’t have to transmit anything I would suggest that you at least design a detector to see whether any unsupported request is handed over on the rx port. Those requiring a completion should be properly managed by your design, even in the cut-down minimum debugging version. I hope your configuration registers are in a non-prefetchable BAR. Another issue could be the PC driver. Are you sure the hangups are not a driver’s fault? - Altera_Forum
Hi Matthias,
all my BARs are 32-bit non-prefetchable and I doubt that the driver is fauly as I also simplified the driver so it is only simple "mapper" of the BARs to user mode memory and the software only writes bunch of data to that memory. There is no special logic left in the driver now. As for the detector - at least there is some vendor message from the chipset (I suppose) that appears on RX periodically, I think that there is nothing other, but I will create some detector for it. Maybe a dumb question - how can I find that the request requires completion? Just now I have only a list of types of request (from PCIe specification) and I return a completion for every "read" - is there some general bit in the header that tells me that the request is non-posted and needs a completion? Martin - Altera_Forum
Hi Martin,
“Read Requests and Non-Posted Write Requests require Completion.[…]”, from PCIe Section 2.2.9. »Completion Rules«. In fact, all non-posted requests require completion, and they are given, for example, in the »Ordering Rules Summary Table«, Cols 3 & 4: Read Requests, I/O (Read and Write), Configuration Requests (Read and Write) or Trusted Configuration Write Requests. Maybe you can cut down the PC driver+app thing as well? How about writing a driver that just writes some amount of bytes to the device’s configuration register space over and over (e.g. timer-based)? You could write a bottom-half function that does one write cycle and re-schedules itself immediately. - Altera_Forum
Hi Matthias,
right now I have simplified everything that is possible - the FPGA design now consist only of PCIe IP and some virtual pins - rx_ready is '1', rx_mask is '0', there is some reset logic, interrupt and msi are '0', and the tx side is grounded also. PC detects my device and I am able to write to it. The driver is now just a "skeleton driver" that only connects to my device and then writes to to it, nothing else. After the write sequence is started, the PC freezes - sometimes it does it immediatelly, sometimes after a few seconds. I will add some detection logic for detecting non-posted requests, if there is a need to answer to some request, also I will create a detector for the error condition from the very beginning of this post. But after that I am stuck. This is already the simplest design that I can make - there is nothing more left to put away and it still freezes... Do you have any simple design that has only the Soft PCIe and some driver that communicates with it that you could share with me? Or do you know about any that I could use? Maybe there could be some problem with the hardware... Martin
