Forum Discussion

MehmetFide's avatar
MehmetFide
Icon for New Contributor rankNew Contributor
3 years ago

Arria 10 GX: 64 bit atomic access to Avalon Bus through PCIe?

Hi,

We have an external processor Arm Cortex A53 with Linux 5.4 aarch64 connected to an Altera Arria 10 GX FPGA through a PCIe bus.

On the FPGA side, we have an Avalon bus, all addresses are set as 64bit. All implementation is done using Quartus 22.4 and QSys.

And I have no problem to access to the avalon bus though PCIe (DMA works, MSI interrupt works) on the Linux side as long as I use 8/16 or 32 bit accesses. Whenever I use 64 bit access for example using ioread64(), it returns 0xffffffffffffffff and I observe that PCIe Config page on the FPGA side gets corrupted. After that I have to restart FPGA to make it functional again.

My original question is here:

https://stackoverflow.com/questions/75097169/any-known-issue-with-ioread64-iowrite64-on-a-pcie-bus

Any idea that if Avalon bus and PCIe hard core in the Arria 10 GX together supports 64 bit io read and write operations?

Thank you.

11 Replies

  • Wincent_Altera's avatar
    Wincent_Altera
    Icon for Regular Contributor rankRegular Contributor
    3 years ago

    Hi,

    Can I know which design example that you are using ?
    AVST ? AVMM ? MCDMA ?

    Regards,

    Wincent_Intel

  • MehmetFide's avatar
    MehmetFide
    Icon for New Contributor rankNew Contributor
    3 years ago

    Hello,

    Our design is based on Arria 10 - AN690: PCIe Gen3x8 AVMM DMA ( here )

    Regards,

    Fide.

  • Wincent_Altera's avatar
    Wincent_Altera
    Icon for Regular Contributor rankRegular Contributor
    3 years ago

    Hi,


    I am not familiar with the platform and code you are using.

    Is that feature inside Quartus ? or it is from the driver ?


    Why you need to change the IOread ?


    Regards,

    Wincent_Intel


  • Wincent_Altera's avatar
    Wincent_Altera
    Icon for Regular Contributor rankRegular Contributor
    3 years ago

    Hi,

    I wish to follow up with you about this case.

    Do you have any further questions on this matter ?

    ​​​​​​​Else I would like to have your permission to close this forum ticket

    Regards,

    Wincent_Intel


    • MehmetFide's avatar
      MehmetFide
      Icon for New Contributor rankNew Contributor
      3 years ago

      Hi @Wincent_Altera,

      All BAR registers definition and implementations are inside the Quartus.

      When we try to access the memory area in FPGA from Linux, there are couple of options like Altera doing it here

      Those are

      ioread8 -> Works fine

      ioread16 -> Works fine

      ioread32 -> Works fine

      ioread64 -> Doesn't work and causes FPGA PCIe to be crashed

      Avalon DMA -> Works fine

      I'm trying to understand what is special with 64 bit direct access to make it non functional.

      Thank you.

  • Wincent_Altera's avatar
    Wincent_Altera
    Icon for Regular Contributor rankRegular Contributor
    3 years ago

    Hi,

    I wish to follow up with you about this case.

    Do you have any further questions on this matter ?

    ​​​​​​​Else I would like to have your permission to close this forum ticket

    Regards,

    Wincent_Intel


  • MehmetFide's avatar
    MehmetFide
    Icon for New Contributor rankNew Contributor
    3 years ago

    Hi @Wincent_Altera,

    As you suggested, we have tested it with both internal RAM and external DDR ram without DMA.

    The problem exists at both conditions. To be more specific, accessing bars as 8/16/32 bit data work fine. But 64 bit accesses (especially read attempt) corrupts the PCIe config page in the FPGA and it fully stops functioning. It only recovers when we reconfigure the FPGA.

    My question is that anyone on Intel ever tested the PCIe hard core in the Arria 10 GX to do 64 bit data access?

    Thank you.

    • Wincent_Altera's avatar
      Wincent_Altera
      Icon for Regular Contributor rankRegular Contributor
      3 years ago

      Hi,

      I check previous record, we had tested and it able to run.
      I lay down some suggestion so that we can narrow down this further.

      The issue might be caused by data packing limitation in interconnect on AXI/Avalon boundary. If your AXI is 32bit wide and PCIe Avalon interface is 64bit wide. In such situation multiple two 32bit transactions will be used to implement 64bit transfer.

      If you want to further analyze, Singal TAP traces showing transactions on PCIe Avalon interface and HPS AXI should show what is going on.

      Please try

      1. can you please try disabling the "Enable Reordering" option (under the Controller tab)?
      2. At the same time, please try running the Calibrate Termination command in the EMIF Toolkit (Tools menu -> System Debugging Tools -> External Memory Interface Toolkit) to verify the optimal ODT settings.

      Regards,

      Wincent_Intel

  • Wincent_Altera's avatar
    Wincent_Altera
    Icon for Regular Contributor rankRegular Contributor
    3 years ago

    Hi,

    I wish to follow up with you about this case.

    Do you have any further questions on this matter ?


    Regards,

    Wincent_Intel


  • Wincent_Altera's avatar
    Wincent_Altera
    Icon for Regular Contributor rankRegular Contributor
    3 years ago

    Hi

    We have not hear from you and this Case is idling. It is not recommended to idle for too long.

    Therefore following our support policy, I have to put this case in close status. My apologies if any inconvenience cause

    Hence, This thread will be transitioned to community support.

    If you have a new question, feel free to open a new thread to get support from Intel experts.

    Otherwise, the community users will continue to help you on this thread. Thank you

    If you feel your support experience was less than a 9 or 10,

    please allow me to correct it before closing or let me know the cause so that I may improve your future support experience.

    Regards,

    Wincent_Intel