Bug! Quartus Pro 20.1.1, Cyclone V, utilizing PCIe example from 16.1.

It's almost completely working. There is one problem left. First let me summarize something that people may not know.

With Cyclone V pcie controller, there is no way to specify dma-ranges with the 5.4 kernel. It does nothing. However, on kernel 5.11 dma-ranges can be set and if you set it, the pci device driver will crash.

There is a problem with read errors from pcie devices. I fixed this by adding this to the top of my board device tree dts file:

(Long comment above it, code after)

	// Reserve the first 64KB cause the pcie driver doesn't know better?
	// Assumption: the pcie driver overwrites the vector table.
	// If this is not present, the nvme has read errors.
	// The kernel complains about the reservation, but does it:
	// "ERROR: reserving fdt memory region failed (addr=0 size=10000)"
	// You can see it reserved the memory in iomem:
	// root@cyclone5:~# cat /proc/iomem 
	// 00010000-001fffff : System RAM
	// 00200000-3fffffff : System RAM
	//   00d00000-00daf47f : Kernel data
	// ff700000-ff701fff : ff700000.ethernet ethernet@ff700000
	// ff702000-ff703fff : ff702000.ethernet ethernet@ff702000
	// ff704000-ff704fff : ff704000.dwmmc0 dwmmc0@ff704000
	// ff705000-ff705fff : ff705000.spi spi@ff705000
	// ff706000-ff706fff : ff706000.fpgamgr fpgamgr@ff706000
	// ff709000-ff709fff : ff709000.gpio gpio@ff709000
	// ffa00000-ffa00fff : ff705000.spi spi@ff705000
	// ffb90000-ffb90003 : ff706000.fpgamgr fpgamgr@ff706000
	// ffc02000-ffc0201f : serial
	// ffc03000-ffc0301f : serial
	// ffc04000-ffc04fff : ffc04000.i2c i2c@ffc04000
	// ffc05000-ffc05fff : ffc05000.i2c i2c@ffc05000
	// ffd02000-ffd02fff : ffd02000.watchdog watchdog@ffd02000
	// ffd05000-ffd05fff : rstmgr
	// ffd08140-ffd08143 : ffd08140.l2-ecc
	// ffe01000-ffe01fff : pdma@ffe01000
	// ffff0000-ffffffff : ffff0000.sram
	
	reserved-memory {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		pcidma1@0 {
			reg = <0x00000000 0x00010000>;
			no-map;
		};
	};

Note: The 5.4 kernel reserves exactly 64K for the vector table and stops the read errors. The 5.11 kernel reserves 2 MegaBytes and also stops the read errors. I do not understand this issue, but this is the work around.

I believe this only applies to Cyclone V which has an ARMv5 whose vector table is not movable. I suspect the kernel is not protecting the vector table from PCIe writes, unless the dts specifically reserves the memory. But this is just a guess. A kernel guru needs to be consulted to be sure.

I have a very simplified design that works well, almost. There are interrupt issues which I haven't been able to get past. I have the help of a linux driver expert and he suspects the hardware is misbehaving.

The failure:

When doing extensive writes to the NVMe, the following messages are generated and if the timeout is set to 30 seconds, each of them represents a 30 second delay.

root@cyclone5:~# ./test_nvme.sh 
[   75.836736] nvme nvme0: I/O 904 QID 1 timeout, completion polled
[   75.981635] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled.
Starting Trace...
[   82.826885] nvme nvme0: I/O 896 QID 1 timeout, completion polled
[   85.116846] nvme nvme0: I/O 928 QID 1 timeout, completion polled
[   86.156823] nvme nvme0: I/O 998 QID 1 timeout, completion polled
[   87.196816] nvme nvme0: I/O 928 QID 1 timeout, completion polled
[   88.236805] nvme nvme0: I/O 929 QID 1 timeout, completion polled
[   89.276824] nvme nvme0: I/O 928 QID 1 timeout, completion polled
root@cyclone5:~# grep timeout nvme_
nvme_full_2021_03_23_debug.log   nvme_full_2021_03_25_debug.log   nvme_full_2021_03_26_debug.log   nvme_trace_2021_0323.log
nvme_full_2021_03_24_debug.log   nvme_full_2021_03_25b_debug.log  nvme_trace.log                   
root@cyclone5:~# grep timeout nvme_full_2021_03_26_debug.log 
    kworker/0:2H-76      [000] ....    27.676716: nvme_timeout: qid: 1, tag: 98, completed: 4
    kworker/0:2H-76      [000] ....    75.836728: nvme_timeout: qid: 1, tag: 904, completed: 1
    kworker/1:1H-50      [001] ....    82.826875: nvme_timeout: qid: 1, tag: 896, completed: 65
    kworker/0:2H-76      [000] ....    85.116834: nvme_timeout: qid: 1, tag: 928, completed: 65
    kworker/0:2H-76      [000] ....    86.156812: nvme_timeout: qid: 1, tag: 998, completed: -961
    kworker/0:2H-76      [000] ....    87.196806: nvme_timeout: qid: 1, tag: 928, completed: 64
    kworker/0:2H-76      [000] ....    88.236793: nvme_timeout: qid: 1, tag: 929, completed: 65
    kworker/0:2H-76      [000] ....    89.276815: nvme_timeout: qid: 1, tag: 928, completed: 57

The trace is generated by patches applied to drivers within the kernel.

Some NVMe's have less issue with this than others.

I have noticed that the MSI and MSI-X interrupts are not being used and I suspect this is related to the issue.

This is lspci -v for my laptops nvme controller:

02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
        Flags: bus master, fast devsel, latency 0, IRQ 31, NUMA node 0
        Memory at d1800000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158] Power Budgeting <?>
        Capabilities: [168] Secondary PCI Express
        Capabilities: [188] Latency Tolerance Reporting
        Capabilities: [190] L1 PM Substates
        Kernel driver in use: nvme

Notice MSI-X Enable has a '+' sign after it. Here is the output of lspci -v for the Cyclone V

00:00.0 PCI bridge: Altera Corporation Device e000 (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff [size=4K]
        Memory behind bridge: 00000000-000fffff [size=1M]
        Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
        Capabilities: [68] MSI-X: Enable- Count=513 Masked-
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [200] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
lspci: Unable to load libkmod resources: error -12

01:00.0 Non-Volatile memory controller: Sandisk Corp WD Black 2018/PC SN520 NVMe SSD (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Sandisk Corp WD Black 2018/PC SN520 NVMe SSD
        Flags: bus master, fast devsel, latency 0, IRQ 57
        [virtual] Memory at c0000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [b0] MSI-X: Enable- Count=17 Masked-
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [1b8] Latency Tolerance Reporting
        Capabilities: [300] Secondary PCI Express <?>
        Capabilities: [900] L1 PM Substates
        Kernel driver in use: nvme

Notice that MSI Enable and MSI-X Enable have '-' minus signs after them.

I thought that maybe I have to enable the msi driver in the kernel, but I think the pcie-altera-msi driver is not used for the Avalon-MM PCIe controller because it is designed for the stand alone MSI to GIC module.

How do I get MSI / MSI-X to enable with the pcie-altera driver?

Thanks for your support.

Brian