Forum Discussion

BrianM's avatar
BrianM
Icon for Occasional Contributor rankOccasional Contributor
5 years ago
Solved

Bug! Quartus Pro 20.1.1, Cyclone V, utilizing PCIe example from 16.1.

Problem Details Error: Internal Error: Sub-system: VPR20KMAIN, File: /quartus/fitter/vpr20k/altera_arch_common/altera_arch_place_anneal.c, Line: 2744 Internal Error Stack Trace: 0xdce7a: ...
  • BrianM's avatar
    BrianM
    5 years ago

    I got it working.

    The last hurdle was the MSI interface. I'm not sure why it wasn't working, but restarting the design from scratch with my recently acquired knowledge got everything working.

    I've attached my qsys file and socfpga.dtsi. Hopefully it will help others get a jump on things so they don't have to learn everything the hard way as I did.

    This design is not optimized for speed, nor is it optimized for space.

    NVMe read speed is around 80 MB/s.

    NVMe write speed is around 50 MB/s.

    Faster drives will do a little better, but even the over a gig per second on PC 4 lane part I have doesn't do much better than 110 MB/s read. Bandwidth is limited by the ARM memory interface and the fact that bursting logic at 125MHz causes timing violations. A burst length of one on the Txs interface has got to slow things down.

    Performance is quoted with the 5.11 kernel, the 5.4 kernel is not quite as fast. Still fast enough for an embedded system though.

    Don't forget to enable the fpga, pcie and msi modules in your top level board dts file.

    And don't forget to reserve the first 64K of DRAM as stated above. If you don't you will get read errors.

    There are probably better configurations, and eventually I'll probably try to optimize for size since my logic will need to get bigger on the next project. But for now. It finally works.

    Good luck to you all.

SengKok_L_Intel's avatar
SengKok_L_Intel
Icon for Regular Contributor rankRegular Contributor
5 years ago

If further support is needed in this thread, please post a response within 15 days. After 15 days, this thread will be transitioned to community support. The community users will be able to help you with your follow-up questions.


  • BrianM's avatar
    BrianM
    Icon for Occasional Contributor rankOccasional Contributor
    5 years ago

    Let's keep this open until I figure it out. I'll update the thread with the solution.

    • BrianM's avatar
      BrianM
      Icon for Occasional Contributor rankOccasional Contributor
      5 years ago

      I've made significant progress. Let me state some things I've learned so people don't have to go through the same stuff.

      1. You must regenerate u-boot every time you change Platform Designer's qsys file. This took me two weeks to realize. Nothing worked because I simply hadn't understood that there is a huge amount of hardware configuration in u-boot that the kernel cannot do because once the DRAM is active, the registers can't be changed.

      2. In order to run at full speed, 125 MHz, you must use clock-crossing and pipeline bridges. The "auto constructor" software is not generally good enough to build connections that will run at speed.

      3. All addresses listed in Platform Designer must be consistent across MM Masters. Period. The software assumes that and there is no way to work around it.

      4. It does not appear (although I still have to confirm this) that the MSGDMA and embedded SRAM modules are used at all by the linux kernel. There is no point in adding them.

      5. Bandwidth from a two lane NVMe is roughly 132 MB / second read. But my config is not working yet, so this will need to be confirmed as well.

      I think I'm stuck in the same way as the posters in this thread: https://forum.rocketboards.org/t/altera-pcie-driver-issue-with-ssd-devices/545

      There is an issue with the way the Altera PCIe linux driver programs the Root Port to write DRAM. I have not been able to get it to work. There are two proposed (and working versions from 15.1 to 18.1) in that thread. I have not tried the one that requires editing of the verilog generated by Platform Designer. Creating a flow based on code hacks is not appealing to me. I have tried the second fix, but either I have misinterpreted how they got it to work, or more recent software does not allow the hack. I have tried dozens of configurations to try to get it to work, but all have failed. If I tamper with the offset address of Txs, Root Port DMA dead locks every time, I assume because the data does not go where it is expected to be.

      As documented in the thread, the issue occurs when reading from the PCIe device. One group was reading from a PCIe SATA controller and the other group from an NVMe like me. Reads fail. Zero data is returned instead of actual data.

      According to the thread, the Root Port is trashing ram, writing over Kernel space or something like that. I don't fully understand the claims. One solution suggested involves adding an address span extender between the h2f HPS interface and the Txs port and adjusting the offset to the Txs by 64 MB and also reserving the second 64MB of RAM for Root Port DMA space. The offset is magically added to the DRAM destination address, thus preventing the Root Port from destroying DRAM. The other solutio shifts the Txs the Root Port DMA out of DRAM altogether and hacking the generated code to add an offset of 0x50000000 to the Txs address. I don't see how that can work, since bit 30 is not utilized on the h2f interface and the Root Port needs to write to physical DRAM to bring data in from the PCIe device.

      In my case I get errors such as this:

      root@cyclone5:~# ./hdparm -tT /dev/nvme0n1p1

      /dev/nvme0n1p1:
      Timing cached reads: 940 MB in 2.00 seconds = 469.80 MB/sec
      Timing buffered disk reads: [ 204.434383] blk_update_request: critical medium error, dev nvme0n1, sector 281088 op 0x0:(READ) flags 0x80700 phys_seg 1 prio c
      lass 0
      [ 204.449791] blk_update_request: critical medium error, dev nvme0n1, sector 281096 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
      [ 204.461410] Buffer I/O error on dev nvme0n1p1, logical block 34881, async page read
      read(2097152) returned 266240 bytes

      After that I can do an fsck.ext4 /dev/nvme0n1p1 and repair the drive, but the next read will fail again.

      Sometimes the read hangs the CPU and watchdog triggers. Which genuinely sounds like code space is being corrupted.

      I have studied the 'ranges' and the 'dma-ranges' properties. I have dug through the code to try to understand how I can control the addresses used by the Root Port. I have not found anything that seems to work. The only thing I've succeeded in doing is causing the pcie driver to not load because it can't find the Txs port and the nvme driver to hang.

      I feel I am very close to getting this to work, but I need a better understanding of how the Root Port should be configured to ensure "DMA" works from the Root Port to DRAM.

      Thank you for your patience and any assistance you can offer.

      • BrianM's avatar
        BrianM
        Icon for Occasional Contributor rankOccasional Contributor
        5 years ago

        Another week, another bit of progress.

        First the observations:

        1. I proved that the MSGDMA and 256KB of SRAM are not used by removing them and getting the same performance. This saves a lot of space and performance is good enough with 140MB/s read.

        2. The most important thing that I missed, which I forgot to mention last week, was the third and forth FPGA Bridges in the dts file. Here they are so others can add them: (note this has been added to the latest kernel, you just have to activate them. I found them in the latest kernel yesterday after spending weeks trying to figure out why DMA didn't work and adding them myself last week.)

        fpga_bridge0: fpga_bridge@ff400000 {
        compatible = "altr,socfpga-lwhps2fpga-bridge";
        reg = <0xff400000 0x100000>;
        resets = <&rst LWHPS2FPGA_RESET>;
        clocks = <&l4_main_clk>;
        };

        fpga_bridge1: fpga_bridge@ff500000 {
        compatible = "altr,socfpga-hps2fpga-bridge";
        reg = <0xff500000 0x10000>;
        resets = <&rst HPS2FPGA_RESET>;
        clocks = <&l4_main_clk>;
        };

        fpga_bridge2: fpga-bridge@ff600000 {
        compatible = "altr,socfpga-fpga2hps-bridge";
        reg = <0xff600000 0x100000>;
        resets = <&rst FPGA2HPS_RESET>;
        clocks = <&l4_main_clk>;
        status = "disabled";
        };

        fpga_bridge3: fpga-bridge@ffc25080 {
        compatible = "altr,socfpga-fpga2sdram-bridge";
        reg = <0xffc25080 0x4>;
        status = "disabled";
        };

        fpgamgr0: fpgamgr@ff706000 {
        compatible = "altr,socfpga-fpga-mgr";
        reg = <0xff706000 0x1000
        0xffb90000 0x4>;
        interrupts = <0 175 4>;
        };

        3. The evil hack I came up with to fix the read errors is documented here. It seems to work, but as with all hacks, YMMV. (No I never understood how the DMA destination address is supposed to work, I'm not even positive I understand why this works).

        Add this to your dts top: (sorry no formatting)

        reserved-memory {
        #address-cells = <1>;
        #size-cells = <1>;
        ranges;

        // 2 MiB reserved for PCI Express DMA
        pcidma1@0 {
        reg = <0x00000000 0x00200000>;
        // Note: this is the maximum you can reserve with kernel defaults and relocating the kernel doesn't seem to work with ARM ARCH.
        no-map;
        };
        };

        This reserves the first 2MB of DRAM. The kernel complains:

        ERROR: reserving fdt memory region failed (addr=0 size=200000)

        however, it still works, as shown in 'cat /proc/iomem':

        root@cyclone5:/mnt/test# cat /proc/iomem
        00200000-3fffffff : System RAM <- Notice first 2MB is not used...
        00c00000-00cabe7f : Kernel data
        c0000000-c01fffff : pcie@000000000
        c0000000-c00fffff : PCI Bus 0000:01
        c0000000-c0003fff : 0000:01:00.0
        c0000000-c0003fff : nvme
        ff200000-ff20007f : ff200080.msi vector_slave
        ff200080-ff20008f : ff200080.msi csr
        ff220000-ff223fff : c0000000.pcie Cra
        ff700000-ff701fff : ff700000.ethernet ethernet@ff700000
        ff702000-ff703fff : ff702000.ethernet ethernet@ff702000
        ff704000-ff704fff : ff704000.dwmmc0 dwmmc0@ff704000
        ff705000-ff705fff : ff705000.spi spi@ff705000
        ff706000-ff706fff : ff706000.fpgamgr fpgamgr@ff706000
        ff709000-ff709fff : ff709000.gpio gpio@ff709000
        ffa00000-ffa00fff : ff705000.spi spi@ff705000
        ffb90000-ffb90003 : ff706000.fpgamgr fpgamgr@ff706000
        ffc02000-ffc0201f : serial
        ffc03000-ffc0301f : serial
        ffc04000-ffc04fff : ffc04000.i2c i2c@ffc04000
        ffd02000-ffd02fff : ffd02000.watchdog watchdog@ffd02000
        ffd05000-ffd05fff : rstmgr
        ffe01000-ffe01fff : pdma@ffe01000
        ffff0000-ffffffff : ffff0000.sram

        Next, make the Txs bus on the Avalon-MM PCIe hard macro, 2MB big by setting it to 32 bits wide with a 1MB address width and two address pages. (This might work with 1MB in 64 bit mode, I will verify that later to shrink the reserved memory area once I figure out my latest issue)

        On to the latest issue.

        Interrupts are being dropped by hardware or software:

        In this example I copy a 463MB file from the SDCARD to the NVMe drive: (It seems to happen more on writes than reads).

        root@cyclone5:/mnt/test# cp /home/root/463MB.bin 3MB_Copy.bin
        [ 226.418525] nvme nvme0: I/O 128 QID 1 timeout, completion polled
        [ 257.138480] nvme nvme0: I/O 160 QID 1 timeout, completion polled
        [ 290.408531] nvme nvme0: I/O 192 QID 1 timeout, completion polled
        [ 320.498581] nvme nvme0: I/O 260 QID 1 timeout, completion polled
        [ 350.569413] nvme nvme0: I/O 288 QID 1 timeout, completion polled
        [ 380.648522] nvme nvme0: I/O 288 QID 1 timeout, completion polled
        [ 410.728549] nvme nvme0: I/O 288 QID 1 timeout, completion polled
        [ 440.808519] nvme nvme0: I/O 288 QID 1 timeout, completion polled
        [ 470.888467] nvme nvme0: I/O 320 QID 1 timeout, completion polled
        [ 500.968466] nvme nvme0: I/O 329 QID 1 timeout, completion polled
        [ 534.898518] nvme nvme0: I/O 323 QID 1 timeout, completion polled
        root@cyclone5:/mnt/test#
        root@cyclone5:/mnt/test# [ 544.193059] systemd-journald[64]: Sent WATCHDOG=1 notification.
        [ 564.968528] nvme nvme0: I/O 352 QID 1 timeout, completion polled
        [ 595.048526] nvme nvme0: I/O 384 QID 1 timeout, completion polled
        [ 625.128505] nvme nvme0: I/O 418 QID 1 timeout, completion polled
        [ 655.848455] nvme nvme0: I/O 469 QID 1 timeout, completion polled

        You can see that many many interrupts are being dropped. I'm expecting to have to outfit the altera-msi.c code with printk's to try to figure this out. So far, all attempts to improve it by adjusting hardware have not made any difference. I'm tempted to try an old kernel, but that might open a bigger can of worms.

        Again, if anyone knows anything about this issue, I'm all ears.