Forum Discussion

Arnab1's avatar
Arnab1
Icon for New Contributor rankNew Contributor
2 years ago

Stratix 10 MCDMA host: Queue reset failed

Hello,

I have been trying to get an MCDMA example design working on a Stratix 10 MX device (1SM21BEU2F55E2VG). You can consider me a beginner with the Intel FPGAs. The project details are as follows.

Device side

-----------------
Project:

Platform designer project with H-tile PCIe MCDMA with AVMM interface.

PCIe configuration - Gen3x16 512-bit interface, 250 MHz

No of PFs - 1

Everything else is set to default, as this is a slightly modified version of the basic example design generated by Quartus.

refclk - connected to differential PCIe clock PCIE_REFCLK_P and PCIE_REFCLK_N

pin_perst - connected to PCIe PERST#

npor - tied to 1 in the top level wrapper Verilog module

xcvr - PCIe interface

The top-level wrapper only instantiates this, sets npor to 1, and passes every other signals.
The hip_ctrl and hip_pipe are left unconnected in the wrapper.

The design compiles properly, and the sof file can be programmed without error.

-----------------------

Host side

-----------------------

On the host side, I followed the steps mentioned in the MCDMA example user guide.
https://www.intel.com/content/www/us/en/docs/programmable/683517/23-4/introduction.html

The device is listed the lspci output and shows up the BAR regions.

However, when I try to run any of the test programs in the software/user/cli, nothing runs successfully (this driver and user utilities/examples are also generated as part of the example MCDMA project). Most of the time the error is "Queue reset failed" or the programs hang without any output. Reading device memory via the supplied devmem utility always returns 0xFFFFFFFF. My guess is something wrong with the reset logic/process, however I am unable to fix it so far and it feels like I am missing something obvious. Any lead in this regard would be helpful. Let me know if you need more information in this context.

Thanks and regards,
Arnab

Board Debug
Configuration
Fpga Devkit
Generic Io
Other Ips

6 Replies

  • VenT_Altera's avatar
    VenT_Altera
    Icon for Frequent Contributor rankFrequent Contributor
    2 years ago

    Hi @Arnab1,


    Thanks for reaching out to us.


    May I know which MCDMA Avalon-MM DMA Design Example and driver you are using? With SRIOV support enabled? There are multiple of MCDMA Design Examples, and each variant of the MCDMA design example is supported by specific driver only. Please make sure that the driver used is appropriate.


    Can you please make sure the prerequisites and software setup for the selected driver are performed according to the steps outlined in the User Guide?


    Next, what are the test programs you performed from the User Guide? It would be helpful if you could screenshot the commands you entered and errors you encountered to help us better narrow down the issue.


    Regarding the host side (lspci_output.png), the link speed appears to be downgraded from (Gen 3) 8 GT/s to 2.5 GT/s. Can you please check if the host PCIe settings are able to support Gen 3?


    Thanks.

    Best Regards,

    VenTingT


    • Arnab1's avatar
      Arnab1
      Icon for New Contributor rankNew Contributor
      2 years ago

      Dear VenTingT,

      Thank you for your reply. I am providing the information below that you have requested for. I have also attached the screenshots of relevant configuration/outputs that could provide more insight into the issue.

      I am using "H-Tile Multichannel DMA Intel FPGA IP for PCI Express" on Stratix 10 MX device (1SM21BEU2F55E2VG). I generated the example design simply by clicking on the "Generate Example Design" button, which generated the Quartus project files (named pcie_ed.qar) and the software codes containing the kernel and user codes. You can find all IP configuration details in the screenshots.

      SRIOV was disabled. I wanted to have a bare minimum working MCDMA - AVMM design.

      I understand. In this case, I was using the kernel+user code that was generated along with the Quartus example design files (present in the software directory).

      I followed the steps that are mentioned here - https://www.intel.com/content/www/us/en/docs/programmable/683517/23-4/quick-start-guide.html

      I followed this for the BIOS configuration + setting up hugepages + loading uio + compiling and loading ifc_uio driver process, and I did not see any error here.

      For the tests, I started with the devmem utility and simple_app examples. I have attached the screenshots. However, not much debug info/error info is available, and there is nothing in the kernel log too. Same goes for the examples too.

      I noticed that, but chose to continue despite that. The server supports PCIe Gen3 (and Gen4 too), and the link configuration is set to automatic. If needed, I can fix that to Gen3; let me know.

      Let me know if you need any further information in this regard. Thanks!

      Kind regards,
      Arnab

      • VenT_Altera's avatar
        VenT_Altera
        Icon for Frequent Contributor rankFrequent Contributor
        2 years ago

        Hi @Arnab1,

        Thank you for attaching the screenshots.

        May I know are you using the MCDMA custom driver?

        Besides, for the devmem utility, are you trying to perform the IP reset as in the Step 4. of Section 3.5.2.3.3.? The command you entered seems different from the one specified in the IP reset step.

        It is suggested to change the host PCIe support to Gen3 which follows the design. Could you try to perform the tests again after the changes made?

        Thanks.

        Best Regards,

        VenTingT

  • VenT_Altera's avatar
    VenT_Altera
    Icon for Frequent Contributor rankFrequent Contributor
    2 years ago

    Hi @Arnab1,


    We have not received any response from you on the previous reply that we provided. Please login to ‘https://supporttickets.intel.com’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you.


    After 15 days, this thread will be transitioned to community support.

    The community users will be able to help you with your follow-up questions.


    If you feel your support experience was less than a 9 or 10, please allow me to correct it before closing, or please let me know the cause so that I may improve your future support experience.


    Thanks.

    Best Regards,

    VenTingT


  • Arnab1's avatar
    Arnab1
    Icon for New Contributor rankNew Contributor
    2 years ago

    Hi @ventt ,

    Sorry, I was away and busy with other things a bit. I understand the policy on responding to issues on this forum.
    I have provided responses to your queries below, and updates on the issue which might be useful other readers.

    Q. May I know are you using the MCDMA custom driver?
    R. Yes, I am using the MCDMA custom driver.

    Q. Besides, for the devmem utility, are you trying to perform the IP reset as in the Step 4. of Section 3.5.2.3.3.? The command you entered seems different from the one specified in the IP reset step.
    R. Yes, I did try to perform IP reset as mentioned in Section 3.5.2.3.3.4 of the documentation.

    Q. It is suggested to change the host PCIe support to Gen3 which follows the design. Could you try to perform the tests again after the changes made?
    R. The host PCIe version support is set to auto. I can retry after setting it to Gen3 explicitly.

    Update: I created a custom design with MCDMA + Dual Port On-chip RAM (target memory) + custom PIO-access block in Quartus 24.1 , and that seems to work on a different tower workstation specifically set up for testing the FPGA cards/designs. I could run my own host-side DMA applications, and can access the PIO interface directly using sysfs. The kernel version used is 5.15.0-88-generic. The design does not work with kernel version 5.15 as recommended in Section 3.5.2.1 of the MCDMA user guide. However, we have dedicated rack mounted 1U servers for these FPGA cards, and we would like to finally move to those servers as the FPGA hosts. These 1U servers have the same kernel and driver configuration as set up on the tower workstation. In few instances, the same design seems to work on the 1U servers, but the "Queue reset failed" error occasionally pops up. It would be great if you could comment on the possible reasons and fixes for this error.

    Kind regards,
    Arnab