PCIe Enumeration Failure for CXL IP
When attempting to validate the Agilex 7 R-Tile Compute Express Link (CXL) 1.1/2.0 IP (Type 2 and Type 3) using a CXL compatible host server, the host server is unable to complete PCIe bus enumeration. The host server stalls while attempting to complete PCIe bus enumeration. The stall never resolves after boot, and access to to the host is never granted. Depiction of the stall and its status code from the host server's perspective is provided as an attached PNG file titled: "pcie_enumeration_stall". Debugging Information: A PCIe Gen 5.0 reference design using the Altera R-Tile Avalon Streaming IP For PCI Express was used to validate that PCIe enumeration could complete fully without failure, and that the host server could exchange data with the FPGA. While running the CXL example design, the Quartus System Console's Link Logger indicates that the LTSSM state is in the "UP_L0" before the PCIe bus enumeration stall. The state may sometimes change when attempting to "Refresh" the status during the PCIe bus enumeration stall. The state may briefly enter recovery (UP_L0 -> REC_IDLE -> REC_RCVRCFG -> REC_RCSVLOCK -> REC_COMPLETE -> UP_L0). Depiction of the Quartus System Console's Link Logger when this occurs is provided as an attached PNG file titled: "ltssm_link_logger". While running the CXL example design, the Quartus System Console's Link Logger indicates that the advertised and negotiated link speeds and widths are both 32.0 GT and x16. Depiction of a CXL Type 3 Quartus System Console's Overview is provided as an attached PNG file titled: "cxl_ip_systemconsole_overview". Instead of generating the example design, the pre-compiled binary offered by Altera for Type 2 and Type 3 CXL IP designs was used and resulted in the exact same failures as shown above. CXL.mem transaction registers (M2S and S2M) are 0x00, indicating that the host server never progresses far enough to begin sending/receiving transactions/requests. Between the PCIe build that functions and the CXL build that does not function (stalls at enumeration), no server UEFI settings were changed. A CXL enable function was enabled for all tests. Several PCIe UEFI settings were changed in an attempt to resolve the enumeration stall, but none changed the outcome. Attempting to disable the CXL Compliance 2.0 and the HDM decoder registers both did not resolve the issue. The FPGA was powered and programmed before the server was launched. Two different CXL servers were tested and both resulted in the same behavior. The relevant PCIe and CXL settings from BIOS is provided as an attached PNG file titled: "cxl_server_settings". The CXL REFCLK was tested as both COMMON and SRIS/SRNS. Each test changed SW3 to use relevant onboard and connected based clocks. IP Settings: CXL IP settings are uploaded as PNG files titled: "cxl_ip_settings_N". The settings tested are the default provided settings as well as a version with a 300 MHz PLD clock (SRIS). Hardware Details: FPGA is connected to host server via PCIe Gen 5.0 x16 slot on Tile 14C. FPGA device is the Altera Agilex 7 FPGA I-Series Development Kit (Production 2x R-Tile & 1x F-Tile) (AGIB027R29A1E1VB) The DIMM provided with the development kit is slotted into DIMM Slot A. SW1 is set to 1000 (PCIe PRSNT x16). SW3 is set to 0110 for designs using the CXL/PCIe common clock and 0000 for designs using the CXL/PCIe onboard REFCLK (SRIS). Software Details: Quartus Prime Pro Edition v25.1 was used to generate the designs. R-Tile Altera FPGA IP for Compute Express Link (CXL) was generated with version 1.17.0. FPGA Design: The FPGA design is generated using the example design with the IP settings given above. A pre-compiled binary provided by Altera was also used to test instead of a generated design. Server details: SMC AS-1126HS-TN (CXL 2.0 via 4x PCIe gen5 x16 slots) CPU: 2x AMD EPYC 9135 (CXL 2.0) RAM: 4x Micron 64GB @ 6000 MT/s UEFI: AMI 1.7a 10/30/2025 Attachments: The system console debug register outputs are saved to CSV files attached to this post. These CSV files are taken from a CXL Type 3 reference design with PLD REFCLK at 300 MHz (SRIS). Questions: Can you provide guidance on how to obtain more information on the enumeration status other than the LTSSM register? Can you provide the UEFI/BIOS settings for PCIe/CXL that was used to test this IP as reference? Could the configuration space registers (DVSEC/HDM) or the TLP handling implemented in the CXL example design RTL create this PCIe enumeration failure? Can you provide guidance on what debug/status registers the CXL IP provides that could be relevant to this issue?
Agilex 7 R-Tile CXL Type-2 IP Hang with Incomplete CXL.cache Operations
Device: DK-DEV-AGI027RBES (Power Solution 2) Software Version: Quartus Prime Pro 24.3 IP Core: CXL Type 2 Hard IP Issue Description: We observed that the CXL Type-2 IP can hang when CXL.cache transactions remain incomplete under heavy workloads. Our design is based on the CXL Type-2 design example, with a delay unit inserted between the CXL IP and the DDR controller. We ran Intel MLC bandwidth tests on the CXL device while monitoring host reachability using continuous ping. In the first experiment (Figure 1), the delay unit inserted a 10,000-cycle delay for each CXL.mem request, with no CXL.cache operations involved. In this case, ping latency increased from approximately 0.2 ms to 45 ms, but the system remained stable. In the second experiment (Figure 2), we replaced the 10,000-cycle delay with a CXL.cache operation, which typically completes in around 300 cycles. Under this configuration, the system hung and ping indicated that the host became unreachable. We observed that the CXL.cache request was issued but never received a response, leading to the hang. We would like to know if there is a known issue or recommended solution for handling incomplete CXL.cache operations in this scenario.
CXL ARB/MUX initialization debug
Hi, I am currently implementing the CXL controller using the FPGA. (connected to Intel Xeon Sapphire Rapids Server) (I am not sure I am writing to the appropriate board. If I need to move this post, please let me know ) Actually, I am not using the Intel CXL IP and implementing myself based on PCIE IP. and I am on the road to implement the FPGA that want to be connected to CPU as CXL.io & CXL.cache enabled(CXL type 1 device). But the problem is I am stuck at ARB/MUX layer initialization flow. (I already successfully done connecting CPU and FPGA as CXL.io only enabled . As CXL.io only enabled, ARB/MUX layer is set to bypassed so that ARB/MUX layer initialization flow is not required at this situation. ) In this above picture(Fig. 5-12) from the CXL specification, CPU(Left side) and Endpoint device(Right side) exchange the ALMPs(ARB/MUX link management packets) and finishes the state transition to active state. but the problem is that CPU is not responding the State_Status_Active_ALMP that notifies the CPU's ARB/MUX layer initialization is done even if previous ALMPs sent well from the endpoint device and received well to CPU. Can anybody help me with these problems ? Any similar circumstances or advices would be welcome Best,
CXL Type 3 + AXI Reg slice causing system deadlock
Hi all, I'm currently using the CXL Type 3 IP, and am encountering the following issue. Whenever I hook up the CXLIP directly to the MC, things work fine. However, when I place an AXI register slice between the CXLIP and the MC (which in theory should just add a one cycle latency), my system deadlocks when attempting a memory-intensive workload. I've used a couple of different AXI Reg slice implementations, including some online, some of my own, and some manual FIFO instantiations. I've narrowed it down to the AW, W, and AR channels (any manager-initiated ones); putting a FIFO on any of these paths causes deadlocks. I'm pulling my hair out, since I've confirmed that everything is compliant to AXI handshake specifications. Any ideas for next steps would be much appreciated.Inquiry about Device Compatibility and Version Support for AGIB027R29A1E2VR3 in Quartus Prime Pro V
During the New Project stage, the device model AGIB027R29A1E2VR3 is not available, but the Development Kit DK-DEV-AGI027R1BES is present. However, in the device model list, AGIB027R29A1E2VRC is found. Are VRC and VR3 the same device? Or is it that version 24.2 does not support the device AGIB027R29A1E2VR3? OS: Windows11 Quartus version:24.2
CXL IP Debug Toolkit
Hello, An Intel development kit DK-DEV-AGI027R1BES with the CXL Type 3 Example Design image causes an AMD Siena architecture system to reboot when a single write is issued. The information extracted by the Debug Toolkit seems to point to failures, but the documentation does not give details on the description of the registers. The most notable entries are: Local Retry State Machine,0x8c00 Num Local CRC Detected,0x2 Local FSM State Status,0x3 Viral Log,0x4 Link Received Viral,0x1 BBS Idle Status,0x0 BBS Error Status,0x1 BBS CXL Status Register Slice0,0xc0000000 BBS Error Status Register,0x12 Device Protocol Table Error,0x1 M2S Viral Received,0x1 BBS Error Status First Register,0x10 The counters show some interesting results. Even though a single Byte RwD was requested by the application, a Req also happened, and apparently only the Req was responded with DRS, whereas the RwD didn't trigger NDR to be sent: Counter of M2SReq Operations,0x1 M2SReq Counter,0x1 Counter of M2SRwD Operations,0x1 M2SRwD Counter,0x1 Counter of S2MDRS Operations,0x1 S2MDRS Counter,0x1 Counter of S2MNDR Operations,0x0 S2MNDR Counter,0x0 Is there more information available on the meaning of the registers for the CXL IP Debug Toolkit? Thank you, Ricardo PS: The complete dump of registers from the Debug Toolkit can be found attached.
sof file for CXL Type3 ED for DK-DEV-AGI027-RA?
Is there a pre-built CXL Type3 example design for DK-DEV-AGI027-RA with CXL IO on the PCIe fingers? I have built one myself, but I run into cyclic boots under Linux. I was wondering if there is a pre-built and verified sof for this design available to rule out problems with my own build. Right now I don't know if the problem is with my build or with BIOS settings required by the design. BTW I have a PCIe build working at Gen5 x16. I also have ASIC based CXL attached memory working on the same host.
Error in Xcelium simulation for CXL Type 3 example design
Hi, I am trying to simulate the CXL Type 3 example design using Xcelium simulator. I am getting error "error within protected source code". We are getting four types of Xcelium errors. xmvlog MIMPST xmvlog SVVMAP xmvlog DUPIDN xmvlog SVNOTY The error log is given below: file: /intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_ast_2331/sim/soft_wrapper/rnr_cxl_soft_rx_tx_wrapper.sv xmvlog: *E,SVVMAP (/intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_ast_2331/sim/soft_wrapper/rnr_cxl_soft_rx_tx_wrapper.sv): error within protected source code. errors: 1, warnings: 0 file: /intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_ast_2331/sim/soft_wrapper/rnr_cxl_soft_tx_wrapper.sv xmvlog: *E,SVVMAP (/intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_ast_2331/sim/soft_wrapper/rnr_cxl_soft_tx_wrapper.sv): error within protected source code. xmvlog: *E,SVVMAP (/intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_ast_2331/sim/soft_wrapper/rnr_cxl_soft_tx_wrapper.sv): error within protected source code. errors: 2, warnings: 0 file: /intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_ast_2331/sim/soft_wrapper/rnr_cxl_reset_ctrl.sv xmvlog: *E,DUPIDN (/intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_ast_2331/sim/soft_wrapper/rnr_cxl_reset_ctrl.sv): error within protected source code. errors: 1, warnings: 0 file: /intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_top_1130/sim/cxl_io_cmb_pf1_regs_2_rdl.sv xmvlog: *E,MIMPST (intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_top_1130/sim/cxl_io_cmb_pf1_struct_ports.vh.iv): error within protected source code. xmvlog: *E,SVNOTY (intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_top_1130/sim/cxl_io_cmb_pf1_struct_ports.vh.iv): error within protected source code. xmvlog: *E,MIMPST (intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_top_1130/sim/cxl_io_cmb_pf1_struct_ports.vh.iv): error within protected source code. xmvlog: *E,SVNOTY (intel_rtile_cxl_top_cxltyp3_ed/intel_rtile_cxl_top_1130/sim/cxl_io_cmb_pf1_struct_ports.vh.iv): error within protected source code. . . libncprotect: *W,ENVDEPRREN: Environment Variable (NCPROTECT_KEYDB) is deprecated. Use (XMPROTECT_KEYDB) instead. errors: 208, warnings: 0
