Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
15 years ago

PCI Express Tx interface gets stalled

Hi,

I'm still trying to send data (A/D samples) from my PCI Express endpoint (S4 GX Dev. Kit. -> Hard IP) to the root port memory (RAM).

I'm able to receive data from the root port, and also return a completion, but when i try to send a memory write transaction it seems no data is sent at all.

The first packet with a payload of 512 DWORDs can be transfered to the MegaFunction without problem (but it is not sent), but on the second packet (after ~10 clocks) the ready signal of the avalon-st tx interface goes low, and fifo full goes high (at the same time).

There is plenty of time between the two packets so i assume that timing is not an issue. Also the headers and data created/sent seem to be correct. Btw I don't use any of Altera's reference designs, but did the implementation from scratch.

Unfortunately I can't tap into the Hard IP so I have nooooo idea what's going on there.

Does anybody have some suggestions what might be going wrong here, or provide some more detailed information about the MegaFunctions internal operation, pleeeease???

20 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    I don't know that I can tell you what is going on, but it might be a good idea to observe the tx_cred vector given to you from the core. This will let you know if you have run out of posted header or posted data credits. If you are out of credits, this would cause the fifo to backup and assert the fifo full signal.

    tx_cred is documented in Table 5-4 on page 5-13 of the PCI Express Compiler User Guide (v9.1). The fields of the vector are shown in Table 5-5.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Thanks BrettFavre, it seems you are right, I'm running out of posted data credits! Bevore I sent the first packet I had 128 posted data credits (tx_cred=0x080fb0407), and after the packet has been transmitted to the Megafunction the value was zero (tx_cred=0x080fb0007).

    The time between two packets is longer than it takes to transmit the package (512 DW payload) to the MegaFunction, so how long does it take before I can send the next package? How can I get more/new posted data credits?
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    The credits will have to be returned to your PCIe board from the system. The amount of time it takes to get credits back will be system dependent. Just a datapoint...on the system that I am using, I observed a read request to completion latency of ~500ns. So, in my case, credits could not come back faster than 500ns from the time I sent a transmit request.

    Another note, the following thread mentions that the PCIe specification does not require flow control information to be updated when a malformed TLP is sent. I have not personally verified this in the spec, but assuming this to be true, if you sent a malformed request, you may have eaten your credits up and created a lockup.

    http://www.alteraforum.com/forum/showthread.php?t=18801&highlight=pcie+lockup

    Unfortunately, we are starting to get into PCIe system behavior, which I am a lilttle sketchy on...still learning myself.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    that sounds plausible (the thing about the lockup). On the other hand I double checked the header and data I send to the MegaFunction. Also the PCI Express User Guide says on page 4–57 that malformed TLPs are deleted by the MegaFunction (and thus shouldn't account for data credits?).

    But now I know at least what to look at... I'll keep you informed about results (if any).
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    OK. Keep me posted.

    I think the malformed TLP text you were reading is referring to requests from the system to the PCIe core, not the other way around. What this text (on 4-57) is saying is that your application h/w will never see any packet which the PCIe core has received and determined to be malformed.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Hi,

    I still couldn't fix the problem.

    I've now changed the payload size to 129 DW (since I'm using a 3 DW Header, because the addr. I'm writing to is 32 Bit, and thus a 4 DW header mustn't be used). The strange thing is that now the Tx ready signal from the Megafunction goes low although there are still posted data credits left. But I never get any credits back which indicates that there is still something wrong. I checked the generated messages, and they should be okay (not malformed).

    How can I determine if/for what reason the root port dropped a packet (unfortunately I don't have a PCI Express protocol analyzer handy)?

    I'm using Jungo WinDriver to develop the driver for the host PC.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Maybe this isn't what you meant, but your post seemed to indicate that you were accounting for the header in your payload length. The header should not be included in the payload length field. Sorry if this isn't what you meant...I just wanted to be sure.

    Here are a few other things that could potentially affect core behavior (and them impact system behavior as a result)l:

    (1) Are you properly aligning the payload data on your tx_st_data bus as described in the "Mapping of Avalon-ST Packets to PCI Express" section starting on page 5-15 of the v9.1 SP PCIe Compiler User Guide? If the payload alignment doesn't line up with the address you send in the packet header, I don't know what the core will do.

    (2) If you are using a 128-bit core, are you setting the tx_st_empty signal if the upper 64-bits of your last AV-ST word do not contain valid data?
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    I'm aware that the payload length field in the header contains just the amount of payload DWORDs, and not also the header.

    I Just wanted to point out that I'm using a 3 DW header (because my destination memory addr. is 32 bits), and this results in an odd number of payload DWORDs (since the 2nd QWORD [I'm using Avalon-St interface with 64 bits] contains the 3rd DWORD of the header and also the 1st DWORD of the payload).

    @ (1): the alignment of my payload data looks exactly like in figure 5-14 on page 5-15 of the user guide

    @ (2): I'm using 64 bit Avalon-St interface

    Is it possible to determine the root ports status from the endpoing e.g. by reading the "PCI Express Advanced Error Reporting Capability Structure"?

    edit: uploaded the waveforms captured with signaltap to

    http://rapidshare.com/files/376204701/datagrabber.vwf
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    One quick follow up to the question about data alignment. Since Data0 is in DW3 (and header is in DW0-2), then the 3 lsbs of your address (including the two reserved bits must be: 2b'100. If the 3 lsbs of your address was 2b'000, then your data alignment would look like Figure 5-6 on page 5-9.

    You should be able to use lspci on linux to read the PCIe capability registers. If you are using windows, there must be some utility analogous to lspci that you can use. I don't know how it would be possible to query the root port status the way you propose (I am not saying it isn't possible...this is just outside the bounds of my knowledge)
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    I'm sorry, but I don't get it. Why do my 3 addr. LSBs need to be 100b for the waveform in figure 5-14? I couldn"t find any note regarding this in the user guide right now. The two LSBs have to be zero for sure since the address is a byte address, and we're talking about DWORDs...