Forum Discussion

jjb169's avatar
jjb169
Icon for New Contributor rankNew Contributor
4 months ago

Agilex 7 I-Series "aocl diagnose acl0" error following OFS

Hello,

I've been working through the Open FPGA Stack (OFS) guides to set up my Agilex 7 I-Series development kit for use with oneAPI. I've worked through prior SystemVerilog issues encountered by switching the generated FPGA Interface Manager (FIM) from a 1x16 PCIe configuration to a 2x8 configuration (although 1x16 would be more preferred).

I am now on the final step of wrapping the FIM into a BSP and validating it for use with oneAPI by running the "aocl diagnose acl0" command. I should note that performing just "aocl diagnose" works fine. When I add "acl0" and execute, however, I find that all attempts to communicate between the host and FPGA via DMA fail (although we do see a single VTP L2 4KB hit). The exact output from the diagnose command is in the text file attached. I have tried using both a minimal FIM generated via command provided in the OFS guides, as well as pre-builts from the Github page.

Why might this error be occurring, and how can I fix it? Any help is greatly appreciated, thank you!

James

36 Replies

  • jchecmu's avatar
    jchecmu
    Icon for New Contributor rankNew Contributor

    NOTE: THIS SOLUTION DOES NOT WORK. CONTINUE READING THE THREAD FOR AN UPDATED CONCLUSION

    Hi,

    I can confirm that AGIB027R29A1E1VB works with oneAPI support when configured for PCIe Gen4 1x16 and a 2x8 memory subsystem.

    oneAPI does not work with either of the PCIe Gen 5 configurations even though I can successfully build OFS FIMs without oneAPI support for both Gen 5 1x16 and Bifurcated Gen 5 2x8 PCIe. When I try to build the oneAPI ASP with PCIe Gen 5, I encounter one of the following errors:

    1. For Gen 5 1x16, the build fails due to a Verilog error in ofs-platform-afu-bbb
    2. For Bifurcated Gen 5 2x8, the build succeeds but aocl diagnose acl0 times out (DMA fail)

     

    Here are the changes that I made to get OFS and oneAPI to work with my card:

    1. Follow the instructions to migrate to the AGIB027R29A1E1VB card: PCIe Attach I-Series (2xR-Tile, F-Tile) - OFS
    2. Remove the two South I/O Rows. Follow steps 1-4 from PCIe Attach I-Series (2xR-Tile, F-Tile) - OFS and then follow the removal steps from PCIe Attach F-Series (P-Tile/E-Tile) - OFS. The second set of instructions are for removing the HPS, which is not present in the mem_ss ip, but the steps for removing the two North banks are the same. I attached my preset for your reference.
    3. Configure PCIe for Gen4 1x16 by specifying pcie_gen = 4 under [settings] in pcie_host_1pf_1vf.ofss. I attached my ofss for your reference.
    4. (Optional) Refloorplan the PR region to recover resources that were used by the southern memory banks by following the steps here PCIe Attach I-Series (2xR-Tile, F-Tile) - OFS. I attached my pr_assignments.tcl for your reference.
    5. Build the FIM with ./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_1pf_1vf.ofss iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_iseries-dk_minimal_fim
    6. Add a device model for AGIB027R29A1E1VB to $INTELFPGAOCLSDKROOT/share/models/dm. I attached mine for your reference.
    7. Modify oneAPI ASP board_spec.xml for a 2x8 memory subsystem by following the instructions here: oneAPI Accelerator Support Package(ASP) Reference Manual - OFS. Make sure the number of memory banks is 2, the maximum theoretical bandwidth is 42656 ((1333MHz x 2 x 64 bits / 8 bits) x 2 banks), the Quartus project settings points to the correct device model, and the FPGA resources for kernel region matches your PR region size.  I attached my preset file for your reference. 
    8. Set $OFS_ASP_ROOT appropriately and build the ASP by following the instructions here: oneAPI ASP Getting Started User Guide - OFS
    9. Program the FIM following the instructions here: OFS for Agilex® 7 FPGA I-Series Development Kit (2xR-Tile,1xF-Tile) - OFS. Do not skip the sudo pci_device <PCIe BDF> unplug step, otherwise your PCIe link speed may be downgraded. 
    10. Initialize the board and run aocl diagnose acl0 following oneAPI ASP Getting Started User Guide - OFS

     

    I am a student unaffiliated with Altera. Let me know if this works for you.

    Best,
    Jason

    • jchecmu's avatar
      jchecmu
      Icon for New Contributor rankNew Contributor

      It seems like not all my attachments got posted. Let me know if you would like them or if I should share my entire repo.

      • jjb169's avatar
        jjb169
        Icon for New Contributor rankNew Contributor

        Hello,

        Amazing, thank you so much!! I'll try following these steps next time I am at my workstation, as I am currently barred by lots of snow.

        If you would be able to share your board model file, or just the entire repo as you offer for reference, that would be phenomenal. I am not deeply experienced with OFS, and much of this process is new to me, so I greatly appreciate your help.

        Best,
        James

  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor

    Hi,


    From the log, it looks like the DMA is stuck. There might be some implmentation issue on your PIM or BSP. Are you using creating your own bsp or this is the BSP release from github?


  • Anonymous's avatar
    Anonymous

    Are you using creating your own bsp or this is the BSP release from github? Can you share how you implement it?

    • jjb169's avatar
      jjb169
      Icon for New Contributor rankNew Contributor

      Hello,

      Thank you for your response! I've been keeping an eye on when the forums would come back online, but must've missed the launch day.

      I have tried both my own, minimal FIM and BSP, as well as those released on the Github page.

      For my own BSP, I've tried both the 1x16 1PF/1VF minimal FIM (which resulted in SystemVerilog errors previously mentioned), as well as a 2x8 1PF/1VF minimal FIM (resulting in the aocl diagnose acl0 error). The command for generating the 2x8 FIM is as follows, and is taken from the OFS guides:

      ./ofs-common/scripts/common/syn/build_top.sh -p --ofss tools/ofss_config/pcie/pcie_host_2link_1pf_1vf.ofss iseries-dk:null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg,no_hssi work_iseries-dk_minimal_fim

      As for the pre-built FIMs I have tried wrapping into a BSP for use, I've downloaded them from the assets section of the 2024.2-1 OFS Github at the below link.
      https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1 

      I'll go through all of the available pre-built packages once more to ensure I did not miss anything, but none of them have worked for me yet.

      I look forward to hearing from you!

      Thanks,

      James

      • Anonymous's avatar
        Anonymous

        Hi,

        May I know which OS are you using? Have you try to use the pre-compile design directly?

         

  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor

    Hi,

     

    Can you check which driver was installed for the FPGA board? Can you share the "lspci -vv" for that specific board?

     

    Thanks.

    • jjb169's avatar
      jjb169
      Icon for New Contributor rankNew Contributor

      Hello,

      Sure thing, attached to this post is the output of "lspci -vv" as you requested. There are two devices listed: 0000:01:00.0 and 0000:01:00.1, as I followed the tutorial for creating and binding the PF and VF. Below are also some outputs that may be relevant here:

      [root@localhost scripts]# ls -lt /dev/vfio
      total 0
      crw-rw----. 1 jbick root 234,   0 Dec  2 14:54 15
      crw-rw-rw-. 1 root  root  10, 196 Dec  2 14:54 vfio


      [root@localhost scripts]# sudo opae.io ls
      [0000:01:00.0] (0x8086:0xbcce 0x8086:0x0001) Intel Acceleration JTAG PCI Development Kit (Driver: dfl-pci)
      [0000:01:00.1] (0x8086:0xbccf 0x8086:0x0001) Intel Acceleration JTAG PCI Development Kit (Driver: vfio-pci)

      Thank you!

      James

      • JohnT_Altera's avatar
        JohnT_Altera
        Icon for Regular Contributor rankRegular Contributor

        Hi,

         

        Can you run "aocl list-devices" to understand the current status of the FPGA? 

         

        Thanks

  • jchecmu's avatar
    jchecmu
    Icon for New Contributor rankNew Contributor

    Following this thread. I am encountering the same issues shown in James's screenshots with the AGIB027R29A1E1VB card