Forum Discussion

BrianM's avatar
BrianM
Icon for Occasional Contributor rankOccasional Contributor
5 years ago
Solved

Bug! Quartus Pro 20.1.1, Cyclone V, utilizing PCIe example from 16.1.

Problem Details Error: Internal Error: Sub-system: VPR20KMAIN, File: /quartus/fitter/vpr20k/altera_arch_common/altera_arch_place_anneal.c, Line: 2744 Internal Error Stack Trace: 0xdce7a: ...
  • BrianM's avatar
    BrianM
    5 years ago

    I got it working.

    The last hurdle was the MSI interface. I'm not sure why it wasn't working, but restarting the design from scratch with my recently acquired knowledge got everything working.

    I've attached my qsys file and socfpga.dtsi. Hopefully it will help others get a jump on things so they don't have to learn everything the hard way as I did.

    This design is not optimized for speed, nor is it optimized for space.

    NVMe read speed is around 80 MB/s.

    NVMe write speed is around 50 MB/s.

    Faster drives will do a little better, but even the over a gig per second on PC 4 lane part I have doesn't do much better than 110 MB/s read. Bandwidth is limited by the ARM memory interface and the fact that bursting logic at 125MHz causes timing violations. A burst length of one on the Txs interface has got to slow things down.

    Performance is quoted with the 5.11 kernel, the 5.4 kernel is not quite as fast. Still fast enough for an embedded system though.

    Don't forget to enable the fpga, pcie and msi modules in your top level board dts file.

    And don't forget to reserve the first 64K of DRAM as stated above. If you don't you will get read errors.

    There are probably better configurations, and eventually I'll probably try to optimize for size since my logic will need to get bigger on the next project. But for now. It finally works.

    Good luck to you all.

BrianM's avatar
BrianM
Icon for Occasional Contributor rankOccasional Contributor
5 years ago

Let's keep this open until I figure it out. I'll update the thread with the solution.

BrianM's avatar
BrianM
Icon for Occasional Contributor rankOccasional Contributor
5 years ago

I've made significant progress. Let me state some things I've learned so people don't have to go through the same stuff.

1. You must regenerate u-boot every time you change Platform Designer's qsys file. This took me two weeks to realize. Nothing worked because I simply hadn't understood that there is a huge amount of hardware configuration in u-boot that the kernel cannot do because once the DRAM is active, the registers can't be changed.

2. In order to run at full speed, 125 MHz, you must use clock-crossing and pipeline bridges. The "auto constructor" software is not generally good enough to build connections that will run at speed.

3. All addresses listed in Platform Designer must be consistent across MM Masters. Period. The software assumes that and there is no way to work around it.

4. It does not appear (although I still have to confirm this) that the MSGDMA and embedded SRAM modules are used at all by the linux kernel. There is no point in adding them.

5. Bandwidth from a two lane NVMe is roughly 132 MB / second read. But my config is not working yet, so this will need to be confirmed as well.

I think I'm stuck in the same way as the posters in this thread: https://forum.rocketboards.org/t/altera-pcie-driver-issue-with-ssd-devices/545

There is an issue with the way the Altera PCIe linux driver programs the Root Port to write DRAM. I have not been able to get it to work. There are two proposed (and working versions from 15.1 to 18.1) in that thread. I have not tried the one that requires editing of the verilog generated by Platform Designer. Creating a flow based on code hacks is not appealing to me. I have tried the second fix, but either I have misinterpreted how they got it to work, or more recent software does not allow the hack. I have tried dozens of configurations to try to get it to work, but all have failed. If I tamper with the offset address of Txs, Root Port DMA dead locks every time, I assume because the data does not go where it is expected to be.

As documented in the thread, the issue occurs when reading from the PCIe device. One group was reading from a PCIe SATA controller and the other group from an NVMe like me. Reads fail. Zero data is returned instead of actual data.

According to the thread, the Root Port is trashing ram, writing over Kernel space or something like that. I don't fully understand the claims. One solution suggested involves adding an address span extender between the h2f HPS interface and the Txs port and adjusting the offset to the Txs by 64 MB and also reserving the second 64MB of RAM for Root Port DMA space. The offset is magically added to the DRAM destination address, thus preventing the Root Port from destroying DRAM. The other solutio shifts the Txs the Root Port DMA out of DRAM altogether and hacking the generated code to add an offset of 0x50000000 to the Txs address. I don't see how that can work, since bit 30 is not utilized on the h2f interface and the Root Port needs to write to physical DRAM to bring data in from the PCIe device.

In my case I get errors such as this:

root@cyclone5:~# ./hdparm -tT /dev/nvme0n1p1

/dev/nvme0n1p1:
Timing cached reads: 940 MB in 2.00 seconds = 469.80 MB/sec
Timing buffered disk reads: [ 204.434383] blk_update_request: critical medium error, dev nvme0n1, sector 281088 op 0x0:(READ) flags 0x80700 phys_seg 1 prio c
lass 0
[ 204.449791] blk_update_request: critical medium error, dev nvme0n1, sector 281096 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 204.461410] Buffer I/O error on dev nvme0n1p1, logical block 34881, async page read
read(2097152) returned 266240 bytes

After that I can do an fsck.ext4 /dev/nvme0n1p1 and repair the drive, but the next read will fail again.

Sometimes the read hangs the CPU and watchdog triggers. Which genuinely sounds like code space is being corrupted.

I have studied the 'ranges' and the 'dma-ranges' properties. I have dug through the code to try to understand how I can control the addresses used by the Root Port. I have not found anything that seems to work. The only thing I've succeeded in doing is causing the pcie driver to not load because it can't find the Txs port and the nvme driver to hang.

I feel I am very close to getting this to work, but I need a better understanding of how the Root Port should be configured to ensure "DMA" works from the Root Port to DRAM.

Thank you for your patience and any assistance you can offer.

  • BrianM's avatar
    BrianM
    Icon for Occasional Contributor rankOccasional Contributor
    5 years ago

    Another week, another bit of progress.

    First the observations:

    1. I proved that the MSGDMA and 256KB of SRAM are not used by removing them and getting the same performance. This saves a lot of space and performance is good enough with 140MB/s read.

    2. The most important thing that I missed, which I forgot to mention last week, was the third and forth FPGA Bridges in the dts file. Here they are so others can add them: (note this has been added to the latest kernel, you just have to activate them. I found them in the latest kernel yesterday after spending weeks trying to figure out why DMA didn't work and adding them myself last week.)

    fpga_bridge0: fpga_bridge@ff400000 {
    compatible = "altr,socfpga-lwhps2fpga-bridge";
    reg = <0xff400000 0x100000>;
    resets = <&rst LWHPS2FPGA_RESET>;
    clocks = <&l4_main_clk>;
    };

    fpga_bridge1: fpga_bridge@ff500000 {
    compatible = "altr,socfpga-hps2fpga-bridge";
    reg = <0xff500000 0x10000>;
    resets = <&rst HPS2FPGA_RESET>;
    clocks = <&l4_main_clk>;
    };

    fpga_bridge2: fpga-bridge@ff600000 {
    compatible = "altr,socfpga-fpga2hps-bridge";
    reg = <0xff600000 0x100000>;
    resets = <&rst FPGA2HPS_RESET>;
    clocks = <&l4_main_clk>;
    status = "disabled";
    };

    fpga_bridge3: fpga-bridge@ffc25080 {
    compatible = "altr,socfpga-fpga2sdram-bridge";
    reg = <0xffc25080 0x4>;
    status = "disabled";
    };

    fpgamgr0: fpgamgr@ff706000 {
    compatible = "altr,socfpga-fpga-mgr";
    reg = <0xff706000 0x1000
    0xffb90000 0x4>;
    interrupts = <0 175 4>;
    };

    3. The evil hack I came up with to fix the read errors is documented here. It seems to work, but as with all hacks, YMMV. (No I never understood how the DMA destination address is supposed to work, I'm not even positive I understand why this works).

    Add this to your dts top: (sorry no formatting)

    reserved-memory {
    #address-cells = <1>;
    #size-cells = <1>;
    ranges;

    // 2 MiB reserved for PCI Express DMA
    pcidma1@0 {
    reg = <0x00000000 0x00200000>;
    // Note: this is the maximum you can reserve with kernel defaults and relocating the kernel doesn't seem to work with ARM ARCH.
    no-map;
    };
    };

    This reserves the first 2MB of DRAM. The kernel complains:

    ERROR: reserving fdt memory region failed (addr=0 size=200000)

    however, it still works, as shown in 'cat /proc/iomem':

    root@cyclone5:/mnt/test# cat /proc/iomem
    00200000-3fffffff : System RAM <- Notice first 2MB is not used...
    00c00000-00cabe7f : Kernel data
    c0000000-c01fffff : pcie@000000000
    c0000000-c00fffff : PCI Bus 0000:01
    c0000000-c0003fff : 0000:01:00.0
    c0000000-c0003fff : nvme
    ff200000-ff20007f : ff200080.msi vector_slave
    ff200080-ff20008f : ff200080.msi csr
    ff220000-ff223fff : c0000000.pcie Cra
    ff700000-ff701fff : ff700000.ethernet ethernet@ff700000
    ff702000-ff703fff : ff702000.ethernet ethernet@ff702000
    ff704000-ff704fff : ff704000.dwmmc0 dwmmc0@ff704000
    ff705000-ff705fff : ff705000.spi spi@ff705000
    ff706000-ff706fff : ff706000.fpgamgr fpgamgr@ff706000
    ff709000-ff709fff : ff709000.gpio gpio@ff709000
    ffa00000-ffa00fff : ff705000.spi spi@ff705000
    ffb90000-ffb90003 : ff706000.fpgamgr fpgamgr@ff706000
    ffc02000-ffc0201f : serial
    ffc03000-ffc0301f : serial
    ffc04000-ffc04fff : ffc04000.i2c i2c@ffc04000
    ffd02000-ffd02fff : ffd02000.watchdog watchdog@ffd02000
    ffd05000-ffd05fff : rstmgr
    ffe01000-ffe01fff : pdma@ffe01000
    ffff0000-ffffffff : ffff0000.sram

    Next, make the Txs bus on the Avalon-MM PCIe hard macro, 2MB big by setting it to 32 bits wide with a 1MB address width and two address pages. (This might work with 1MB in 64 bit mode, I will verify that later to shrink the reserved memory area once I figure out my latest issue)

    On to the latest issue.

    Interrupts are being dropped by hardware or software:

    In this example I copy a 463MB file from the SDCARD to the NVMe drive: (It seems to happen more on writes than reads).

    root@cyclone5:/mnt/test# cp /home/root/463MB.bin 3MB_Copy.bin
    [ 226.418525] nvme nvme0: I/O 128 QID 1 timeout, completion polled
    [ 257.138480] nvme nvme0: I/O 160 QID 1 timeout, completion polled
    [ 290.408531] nvme nvme0: I/O 192 QID 1 timeout, completion polled
    [ 320.498581] nvme nvme0: I/O 260 QID 1 timeout, completion polled
    [ 350.569413] nvme nvme0: I/O 288 QID 1 timeout, completion polled
    [ 380.648522] nvme nvme0: I/O 288 QID 1 timeout, completion polled
    [ 410.728549] nvme nvme0: I/O 288 QID 1 timeout, completion polled
    [ 440.808519] nvme nvme0: I/O 288 QID 1 timeout, completion polled
    [ 470.888467] nvme nvme0: I/O 320 QID 1 timeout, completion polled
    [ 500.968466] nvme nvme0: I/O 329 QID 1 timeout, completion polled
    [ 534.898518] nvme nvme0: I/O 323 QID 1 timeout, completion polled
    root@cyclone5:/mnt/test#
    root@cyclone5:/mnt/test# [ 544.193059] systemd-journald[64]: Sent WATCHDOG=1 notification.
    [ 564.968528] nvme nvme0: I/O 352 QID 1 timeout, completion polled
    [ 595.048526] nvme nvme0: I/O 384 QID 1 timeout, completion polled
    [ 625.128505] nvme nvme0: I/O 418 QID 1 timeout, completion polled
    [ 655.848455] nvme nvme0: I/O 469 QID 1 timeout, completion polled

    You can see that many many interrupts are being dropped. I'm expecting to have to outfit the altera-msi.c code with printk's to try to figure this out. So far, all attempts to improve it by adjusting hardware have not made any difference. I'm tempted to try an old kernel, but that might open a bigger can of worms.

    Again, if anyone knows anything about this issue, I'm all ears.

    • BrianM's avatar
      BrianM
      Icon for Occasional Contributor rankOccasional Contributor
      5 years ago

      I think the issue I'm having relates to the fact that the only place the Root Port can write data in main memory is at location 0x00000000. I Have managed to move the target address (address behind):

      [ 165.301799] altera-pcie c4000000.pcie: host bridge /soc/bridge@c0000000/pcie@000000000 ranges:
      [ 165.310496] altera-pcie c4000000.pcie: Parsing ranges property...
      [ 165.316659] altera-pcie c4000000.pcie: MEM 0x00c4000000..0x00c40fffff -> 0x0004000000
      [ 165.324966] altera-pcie c4000000.pcie: Parsing dma-ranges property...
      [ 165.331416] altera-pcie c4000000.pcie: IB MEM 0x0004000000..0x00040fffff -> 0x0004000000
      [ 165.339966] altera-pcie c4000000.pcie: PCI host bridge to bus 0000:00

      The NVMe driver locks up on a read to the device.

      With the address behind the device at 0x00000000, the Root Port can overwrite the vector tables and crash the system.

      It seems to me that the altera-pcie driver is immature and can't work for most applications. I don't have the time to work on this. My project is already two months behind because of this.

      I'm going to assume that the SDCard is fast enough for now and work on the rest of the hardware bring up / debug and let the software guys determine if the SDCard will work for them. If it won't, I will revisit this thread and the other threads at Rocketboards that I have participated in.

      It seems to me that a single knowledgeable kernel software Engineer could resolve these issues in a few weeks. Three weeks of work within intel to save months of work outside intel seems like an obvious choice. But as you have stated that intel cannot help me with linux or u-boot drivers, I'm disappointed in the level of support I've received from intel.

      • BrianM's avatar
        BrianM
        Icon for Occasional Contributor rankOccasional Contributor
        5 years ago

        It's almost completely working. There is one problem left. First let me summarize something that people may not know.

        With Cyclone V pcie controller, there is no way to specify dma-ranges with the 5.4 kernel. It does nothing. However, on kernel 5.11 dma-ranges can be set and if you set it, the pci device driver will crash.

        There is a problem with read errors from pcie devices. I fixed this by adding this to the top of my board device tree dts file:

        (Long comment above it, code after)

        	// Reserve the first 64KB cause the pcie driver doesn't know better?
	// Assumption: the pcie driver overwrites the vector table.
	// If this is not present, the nvme has read errors.
	// The kernel complains about the reservation, but does it:
	// "ERROR: reserving fdt memory region failed (addr=0 size=10000)"
	// You can see it reserved the memory in iomem:
	// root@cyclone5:~# cat /proc/iomem 
	// 00010000-001fffff : System RAM
	// 00200000-3fffffff : System RAM
	//   00d00000-00daf47f : Kernel data
	// ff700000-ff701fff : ff700000.ethernet ethernet@ff700000
	// ff702000-ff703fff : ff702000.ethernet ethernet@ff702000
	// ff704000-ff704fff : ff704000.dwmmc0 dwmmc0@ff704000
	// ff705000-ff705fff : ff705000.spi spi@ff705000
	// ff706000-ff706fff : ff706000.fpgamgr fpgamgr@ff706000
	// ff709000-ff709fff : ff709000.gpio gpio@ff709000
	// ffa00000-ffa00fff : ff705000.spi spi@ff705000
	// ffb90000-ffb90003 : ff706000.fpgamgr fpgamgr@ff706000
	// ffc02000-ffc0201f : serial
	// ffc03000-ffc0301f : serial
	// ffc04000-ffc04fff : ffc04000.i2c i2c@ffc04000
	// ffc05000-ffc05fff : ffc05000.i2c i2c@ffc05000
	// ffd02000-ffd02fff : ffd02000.watchdog watchdog@ffd02000
	// ffd05000-ffd05fff : rstmgr
	// ffd08140-ffd08143 : ffd08140.l2-ecc
	// ffe01000-ffe01fff : pdma@ffe01000
	// ffff0000-ffffffff : ffff0000.sram
	
	reserved-memory {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		pcidma1@0 {
			reg = <0x00000000 0x00010000>;
			no-map;
		};
	};

        Note: The 5.4 kernel reserves exactly 64K for the vector table and stops the read errors. The 5.11 kernel reserves 2 MegaBytes and also stops the read errors. I do not understand this issue, but this is the work around.

        I believe this only applies to Cyclone V which has an ARMv5 whose vector table is not movable. I suspect the kernel is not protecting the vector table from PCIe writes, unless the dts specifically reserves the memory. But this is just a guess. A kernel guru needs to be consulted to be sure.

        I have a very simplified design that works well, almost. There are interrupt issues which I haven't been able to get past. I have the help of a linux driver expert and he suspects the hardware is misbehaving.

        The failure:

        When doing extensive writes to the NVMe, the following messages are generated and if the timeout is set to 30 seconds, each of them represents a 30 second delay.

        root@cyclone5:~# ./test_nvme.sh 
[   75.836736] nvme nvme0: I/O 904 QID 1 timeout, completion polled
[   75.981635] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled.
Starting Trace...
[   82.826885] nvme nvme0: I/O 896 QID 1 timeout, completion polled
[   85.116846] nvme nvme0: I/O 928 QID 1 timeout, completion polled
[   86.156823] nvme nvme0: I/O 998 QID 1 timeout, completion polled
[   87.196816] nvme nvme0: I/O 928 QID 1 timeout, completion polled
[   88.236805] nvme nvme0: I/O 929 QID 1 timeout, completion polled
[   89.276824] nvme nvme0: I/O 928 QID 1 timeout, completion polled
root@cyclone5:~# grep timeout nvme_
nvme_full_2021_03_23_debug.log   nvme_full_2021_03_25_debug.log   nvme_full_2021_03_26_debug.log   nvme_trace_2021_0323.log
nvme_full_2021_03_24_debug.log   nvme_full_2021_03_25b_debug.log  nvme_trace.log                   
root@cyclone5:~# grep timeout nvme_full_2021_03_26_debug.log 
    kworker/0:2H-76      [000] ....    27.676716: nvme_timeout: qid: 1, tag: 98, completed: 4
    kworker/0:2H-76      [000] ....    75.836728: nvme_timeout: qid: 1, tag: 904, completed: 1
    kworker/1:1H-50      [001] ....    82.826875: nvme_timeout: qid: 1, tag: 896, completed: 65
    kworker/0:2H-76      [000] ....    85.116834: nvme_timeout: qid: 1, tag: 928, completed: 65
    kworker/0:2H-76      [000] ....    86.156812: nvme_timeout: qid: 1, tag: 998, completed: -961
    kworker/0:2H-76      [000] ....    87.196806: nvme_timeout: qid: 1, tag: 928, completed: 64
    kworker/0:2H-76      [000] ....    88.236793: nvme_timeout: qid: 1, tag: 929, completed: 65
    kworker/0:2H-76      [000] ....    89.276815: nvme_timeout: qid: 1, tag: 928, completed: 57

        The trace is generated by patches applied to drivers within the kernel.

        Some NVMe's have less issue with this than others.

        I have noticed that the MSI and MSI-X interrupts are not being used and I suspect this is related to the issue.

        This is lspci -v for my laptops nvme controller:

        02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
        Flags: bus master, fast devsel, latency 0, IRQ 31, NUMA node 0
        Memory at d1800000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158] Power Budgeting <?>
        Capabilities: [168] Secondary PCI Express
        Capabilities: [188] Latency Tolerance Reporting
        Capabilities: [190] L1 PM Substates
        Kernel driver in use: nvme

        Notice MSI-X Enable has a '+' sign after it. Here is the output of lspci -v for the Cyclone V

        00:00.0 PCI bridge: Altera Corporation Device e000 (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff [size=4K]
        Memory behind bridge: 00000000-000fffff [size=1M]
        Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
        Capabilities: [68] MSI-X: Enable- Count=513 Masked-
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [200] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
lspci: Unable to load libkmod resources: error -12

01:00.0 Non-Volatile memory controller: Sandisk Corp WD Black 2018/PC SN520 NVMe SSD (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Sandisk Corp WD Black 2018/PC SN520 NVMe SSD
        Flags: bus master, fast devsel, latency 0, IRQ 57
        [virtual] Memory at c0000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [b0] MSI-X: Enable- Count=17 Masked-
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [1b8] Latency Tolerance Reporting
        Capabilities: [300] Secondary PCI Express <?>
        Capabilities: [900] L1 PM Substates
        Kernel driver in use: nvme

        Notice that MSI Enable and MSI-X Enable have '-' minus signs after them.

        I thought that maybe I have to enable the msi driver in the kernel, but I think the pcie-altera-msi driver is not used for the Avalon-MM PCIe controller because it is designed for the stand alone MSI to GIC module.

        How do I get MSI / MSI-X to enable with the pcie-altera driver?

        Thanks for your support.

        Brian