Forum Discussion
Altera_Forum
Honored Contributor
Honored ContributorPCIe freezing
Hi,
I have strange problem with PCIe IP. I have a custom board with ArriaGX and PCIe instantiated using MegaCore flow with AvalonST interface. The PCIe works (there are some DMAs running, target access, everything works fine), but occasionally the PC freezes when accessing my device. The design is quite complicated, so I started to minimize it and I ended with simple PCIe interface to on-chip memory and SW application that only writes to the device as fast as is possible. In this situation the system freezes after few seconds. Using SignalTap I see folowing: - there are coming writes before the freeze - only rx_xxx, nothing else. - no interrupt is signalled - rx_ready, tx_ready are continuously asserted - tx_credits are nonzero - there are just no transactions coming out of the IP when the system freezes even though my application is ready to receive. So I configured the IP to show me the test_out (512 bit) interface to see more and I found this: - in the time, when the packets stop coming there is bit 313 and 314 set (received erroneous TLP, packet with wrong sequence number) and they are coming repeatedly after that moment - also in the same times there is bit 85 set (that is Data link layer error - TLP error) - in the reaction to bad TLP it seems to me, that the IP sends a NAK to the root port (just few tick after the first erroneous TLP is received) - some ticks after the NAK is sent there is stop in receiving PLP ACKs for some time (I suppose that the IP tries to reset the link) and that's all - I do not see anything more coming. At first I thought that It could be some weird problem with the TLP sequencing, but I discovered, that time from time there is exactly the same situation (bad TLP sequence number, NAK, re-training of the link) but IP recovers from that error without any problem - the packets start coming again. Do you have any ideas, where should I look, what could cause this problem? Maybe I did not find the real moment where the IP freezes. How to track/debug it? Martin23 Replies
Altera_ForumHello, noone has any idea? Any clues?
Martin- Altera_Forum
If it’s a lower-level issue, it should be visible without traffic as well. Maybe you can implement a detector, triggered by the condition on test_out that acts without PCIe traffic. Maybe connect it with a LED to see the rate of such incidents.
In any case you have to make sure that you correctly respond to any non-posted requests sent to you, otherwise the sender will not regain his credits and run out of them quickly. Then his completion timeout mechanism will start to act, and depending on hardware and software configuration, this leads to soft or hard errors, lockups and maybe an immediate reboot. I am not familiar with the soft IP and have never used test_out, so bear with me. - Altera_Forum
Hi Matthias,
I can do a detector, so I can see rate of the "problem", but will that show anything to me? What I don't know is if it is a problem that this happens or it's a normal condition? As a second - I do not issue any non-posted requests, the software only writes to my device, so there should not be any problem with credits, am I right? Martin - Altera_Forum
If the detector shows trouble even in absence of the host/software doing requests, then you know there is something seriously wrong on the lower level. If these errors are just related to the requests or the associated responses to them, it’s hard to tell who is to blame: The PCIe soft IP or the application behind AST.
When writing to your device, there could be problems with credits only if your model behind AST does not fetch the data. Once you fetch data from the buffer (rx_ready/rx_valid asserted), the credit is automatically returned to the Requester by the PCIe IP. If the IP does not return credit for whatever reason, you can get into trouble with write-only requests as well. Remaining issues are: Buggy Soft IP, some synthesis defects – maybe constraints, clocking, clock domain crossing – hardware defects on the FPGA board or the motherboard. - Altera_Forum
Hi Matthias,
when the PC freezes and I have a SignalTap connected, I do not see any problem with rx_ready- it's continuously asserted. The function sends me rx_valids, sops and eops and my logic should handle them. I am issuing rx_st_mask for one tick after each packet starts, but this should not be an issue, am I right? And for the detector - I still don't know if the situation it detects is a problem or if it is some standard situation on the line. Martin - Altera_Forum
Refuse from asserting rx_st_mask at that time. The IP has to tamper with credits when you use it, so for debugging, nail it to idle. You are not handling non-posted requests anyway, so you should throttle posted requests only using rx_ready.
Put the detector idea aside for now. Let’s try to be exact and easy on the AST interface first. - Altera_Forum
Okay, I removed the rx_st_mask logic and tied it to ground. The problem remains with no change... Do you have any other idea?
- Altera_Forum
Do you know any way how to get the credit status of the PCIe root?
- Altera_Forum
Hi Martin,
it looks as if test_out[199:144] contains the flow control information for the receiver. You cannot easily find out how the root is counting credit. Check whether that changes along the process of accessing your device. If credit gets lower over time, there is something wrong with the soft IP or the application interface. If it stays high there might still be something wrong with the soft IP but on a lower level, and it could be another hardware issue together with the root side of the PCIe link. Still, I think it might be an issue with posted requests sent to your IP that get dropped somewhere, most probably in your application. Try to make sure that you don’t just drop such packets but respond accordingly, i.e. with a Completer Abort TLP. Otherwise, the root port’s receive credit will be eaten up, not allowing it to issue new requests until the requests time out. I don’t know precisely how a root port deviates in acting on a timeout event, but it might lead to lockup or reboot. I’m on Arria II GX using hard IP AST (125 MHz, 64 bit), and there is little trouble with handling unsupported requests, though. - Altera_Forum
Hi Matthias,
the test_out at this bits is: PH (7-0) : 50 PD (19-8) : 360 NHP (27-20) : 54 NPD (35-28) : 0 CPLH (43-36) : 0 CPLD (55-44) : 0 the PH and PD credits are changing (+-1) when the PC is not frozen and I send some "writes" to my devices, other stays intact. When the PC freezes, the values are the same as I have written above - so no change. Couldn't be a problem that the NPD, CPLH and CPLD are zeroes? But they are zeroes also when the PC is working... Martin
