Why does the NVMe device report errors after an HPS warm reset on Agilex® 5 ES devices when FPGA reconfiguration is skipped?

Description

Due to a problem in the Agilex® 5 ES devices (A5Ex065BxxxxExxR0 and A5Ex065AxxxxExxR0), the FPGA-to-HPS (F2H) bridge can fail after an HPS warm or cold reset, or after a subsequent FPGA reconfiguration, when the System Memory Management Unit (SMMU) is enabled in HPS Boot First mode. To avoid this, some PCIe* Root Port designs skip FPGA reconfiguration in U-Boot after an HPS warm reset and keep the FPGA configuration from the previous boot.

When FPGA reconfiguration is skipped, the PCIe Root Port in the FPGA fabric and the attached Non-Volatile Memory Express (NVMe) endpoint are not reset. They keep stale state from the previous boot. When Linux enumerates the PCIe bus and loads the NVMe driver, the stale state might cause the NVMe admin command to time out and the NVMe driver probe to fail. The kernel log shows messages similar to:

nvme nvme0: pci function 0000:01:00.0
nvme nvme0: I/O tag 24 (0018) QID 0 timeout, disable controller
nvme nvme0: Device not ready; aborting shutdown, CSTS=0x5 
nvme nvme0: Identify Controller failed (-4)
nvme 0000:01:00.0: probe with driver nvme failed with error -5

The NVMe device still enumerates and its BAR is assigned, but /dev/nvme0n1 is not created.

Refer to the section "FPGA-to-HPS (F2H) bridge, HPS warm/cold reset and FPGA reconfiguration issues with SMMU enabled" in the Agilex® 5 ES Device Errata and User Guidelines.

Resolution

To work around this problem on Agilex 5 ES devices, reset the PCIe Root Port secondary bus and re-enumerate the PCIe devices from Linux after an HPS warm reset. Run these commands as root:

1. Assert a secondary bus reset by setting bit 6 of the Bridge Control register on the Root Port: setpci -s 00:00.0 BRIDGE_CONTROL=40:40
2. Wait, then release the reset by clearing bit 6: sleep 1 setpci -s 00:00.0 BRIDGE_CONTROL=00:40
3. Wait for the link to retrain, then remove the Root Port device: sleep 1 echo 1 > /sys/bus/pci/devices/0000:00:00.0/remove
4. Rescan all PCIe buses: echo 1 > /sys/bus/pci/rescan

After the rescan, Linux re-enumerates the Root Port and the NVMe device with clean state.

This problem is fixed in Agilex 5 production devices (A5Ex065BxxxxExx and A5Ex065AxxxxExx).

Additional Information

• 00:00.0 is the Root Port bus/device/function in the Agilex 5 PCIe Root Port system example design. Run lspci to confirm the Root Port address in your system and replace 00:00.0 if it differs.
• The setpci value 40:40 uses the value:mask format, so it changes bit 6 only and leaves the other Bridge Control bits unchanged.
• The PCI Express® Base Specification requires the secondary bus reset to be held for at least 1 ms, and software to wait at least 100 ms after releasing it before sending configuration requests. The sleep 1 commands above meet both requirements.
• An FPGA power cycle (cold boot) also resets the Root Port and the NVMe device.