Forum Discussion

Dexter22's avatar
Dexter22
Icon for New Contributor rankNew Contributor
5 days ago

Agilex 7 R-Tile PIPE Direct endpoint receives Set_Slot_Power_Limit, but no Config TLPs

Hello,
I am developing a custom PCIe endpoint using the Agilex 7 R-Tile PIPE Direct interface. The PHY is provided by the R-Tile IP, while the LTSSM, Data Link Layer, and TLP/DLLP handling are implemented in soft RTL.

Environment

  • FPGA: Agilex 7 AGIB027R29A
  • Quartus Prime Pro 24.3.0 Build 212
  • R-Tile PIPE Direct mode
  • PCIe Gen3 x8
  • Custom soft LTSSM and Data Link Layer
  • Linux host with a commercial PC/root complex
  • The FPGA is configured before a host cold boot
  • The endpoint is not currently visible in lspci

In a successful Signal Tap capture:

  1. The LTSSM reaches Gen3 L0.
  2. The Data Link Layer reaches DL_ACTIVE.
  3. InitFC1 and InitFC2 for Posted, Non-Posted, and Completion traffic are exchanged.
  4. We observe valid incoming TLP traffic with a correct LCRC.
  5. The host sends a Message TLP that appears to decode as Set_Slot_Power_Limit.
  6. No subsequent Configuration Type 0 Read TLP or other expected enumeration TLP is observed.
  7. The link later returns from L0 to Recovery.RcvrLock.

Our understanding is that Set_Slot_Power_Limit is a posted Message TLP and should not require a Completion TLP from the endpoint. The endpoint should only acknowledge successful reception at the Data Link Layer. Please correct us if this interpretation is wrong.

Questions

  1. Is receiving Set_Slot_Power_Limit before the first Configuration Read a normal root-port enumeration sequence?
  2. Does the endpoint need to respond to this Message TLP in any way other than sending a Data Link Layer ACK?
  3. If the ACK DLLP has an incorrect sequence number, CRC, or timing, could the root port stop sending further TLPs and eventually force the link into Recovery?
  4. Are there any R-Tile PIPE Direct-specific status or control signals that must be handled before the host starts Configuration transactions?
  5. Are there recommended R-Tile debug signals for determining why the root port stops after this message?
  6. Could an incorrect Flow Control advertisement—especially Completion Header/Data credits—cause this behavior even if InitFC2 completes?
  7. Are there known PERST#, cold-boot, or FPGA-configuration timing requirements for enumeration when using PIPE Direct mode?

We are currently capturing the following signals:

  • LTSSM state
  • Data Link Layer state
  • InitFC1/InitFC2 and UpdateFC status
  • Received TLP Fmt/Type and complete header
  • Received TLP sequence number
  • LCRC result
  • ACK/NAK request and transmitted ACK/NAK sequence number
  • Replay timer/replay status
  • RX block lock, CDR lock, and lane numbers
  • Raw 1024-bit PIPE RX block data

Any guidance on the expected transaction sequence after Set_Slot_Power_Limit, or additional signals that should be captured, would be greatly appreciated.

Thank you.

Board Debug
Fpga Devkit

1 Reply

  • Wincent_Altera's avatar
    Wincent_Altera
    Icon for Regular Contributor rankRegular Contributor
    1 hour ago

    Hi Dexter22 ,

    Some of your question, you may get the answer from PCI specification. 
    But I can try my best understanding to help you answer, hope that can be a good reference point for you.

    1. IF refer to PCI Express Base Specification Revision 6.2, Section 2.9.2, “Transaction Layer Behavior in DL_Up Status”. 

    • The guide mentioned : “When transitioning from a non-DL_Up status to a DL_Up status and the Auto Slot Power Limit Disable bit is Clear in the Slot Control Register, the Port must initiate the transmission of a Set_Slot_Power_Limit Message to the other component on the Link to convey the value programmed in the Slot Power Limit Scale and Slot Power Limit Value fields of the Slot Capabilities Register.” 
    • Hence I would said the spec ties this message to the DL_Up transition, not to completion of Configuration Reads, so seeing Set_Slot_Power_Limit before the first CfgRd0 is normal.

    2. IF refer to PCI Express Base Specification Rev. 4.0 Version 1.0, Section 2.2.8, “Message Request Rules”.

    • “Message Requests are posted and do not require Completion.” 
    • “The component on the other side of the Link (with Endpoint, Switch, or Bridge Functions) that receives Set_Slot_Power_Limit Message must copy the values in the data payload into the Device Capabilities register associated with the component's Upstream Port.” 
    • So your endpoint should not send Cpl/CplD for this message. The required behavior is normal DLL reliability handling plus, functionally, capturing the payload into the appropriate slot-power-limit fields if your implementation exposes that capability behavior.

    3. I never try such customization before, but ideally it shall be yes , because PCIe DLL reliability depends on valid ACK/NAK, sequence tracking, replay buffering, and timeout/replay behavior; if ACKs are unusable, forward progress can stall and link recovery behavior can follow.

     

    4. If refer to R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express User Guide, Section 2.4, “PIPE Direct Mode”.

    •  “When implementing a Soft IP Controller using the R-tile IP for PCI Express in PIPE Direct mode, the FPGA device must be fully configured in order to start the link training sequence since the Controller is in the FPGA fabric” 
    • “The soft IP controller must release the per-lane lnX_pipe_direct_pld_pcs_rst_n_i signal out of reset after the per-lane lnX_pipe_direct_tx_transfer_en_o signal is asserted.”
    •  “There is only one PERST# pin ( pin_perst_n) on the R-tile. Therefore, toggling pin_perst_n will affect the entire R-tile. This pin also resets the EMIB interface.”

    5. Since you already capture LTSSM, DLL, replay, LCRC, and raw PIPE data, the most useful additional captures are the tx_transfer_en_o to lane-reset relationship, unused-lane powerdown_i handling, and PERST# / EMIB / refclk sequencing, because those are the guide-called-out controls most likely to block clean bring-up before Config traffic ever becomes stable.

    6. based on my understanding , due to PCIe flow control is separate by traffic type and separate for header/data, so a bad Completion-credit advertisement can block progress even though the InitFC1/2 exchange itself finished. But that is just my assumption based on my own experience, please treat that as reference purpose.

    7. You may refer to below chart, those are proven sequence of Pin Perst , timing of other signal to be work well for PIPE direct mode

    Regards,

    Wincent_Altera