Forum Discussion

danield17's avatar
danield17
Icon for New Contributor rankNew Contributor
7 months ago

Over 4GB PCIe BAR Size Issue

Hi,

I created a design that utilizes the Arria 10 Hard IP and an EMIF.

On my board, I have 8GB of DDR memory, and I wanted to connect one of the PCIe BARs to the EMIF so I could read from and write to the DDR.

I encountered an issue where the OS (I tried both Windows and Linux) detects the BAR size but cannot use it. I was unable to read from or write to this BAR.

When I used the Address Span Extender to reduce the BAR size to 2GB, I was able to read and write. However, when I increased it to 4GB or above, it no longer worked.

I also tried enabling "Above 4G Decoding" and "Resizable BAR" in the BIOS, but it made no difference.

Has anyone else encountered this issue?

Memory
Pcie

11 Replies

  • AdzimZM_Altera's avatar
    AdzimZM_Altera
    Icon for Regular Contributor rankRegular Contributor
    5 months ago

    As we do not receive any response from you on the previous reply that we have provided, I now transition this thread to community support. Please login to ‘https://supporttickets.intel.com’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.


  • VenT_Altera's avatar
    VenT_Altera
    Icon for Frequent Contributor rankFrequent Contributor
    6 months ago

    Hi Daniel,


    My apologies for the delayed response.


    Please address the following questions for us to better understand the situation.


    1. Could you please share the steps of your testing when using the BAR2?

    2. Could you elaborate more on the failure situation where you were unable to read from or write to this BAR2? What are the outputs when you tried to read/write the BAR2?

    3. From your description, my understanding is that when you checked the Enable burst capability for RXM BAR2 port, you observed that read/write to the BAR2 was good regardless of the BAR size 2/4GB. On the other hand, when this parameter is unchecked, it doesn't work properly. Is my understanding correct?

    4. When I used the Address Span Extender to reduce the BAR size to 2GB, I was able to read and write. However, when I increased it to 4GB or above, it no longer worked.

    Does this occur when the Enable burst capability for RXM BAR2 port is unchecked, and only on BAR2? Have you tried other BARs?


    Thanks.

    Best Regards,

    Ven


    • danield17's avatar
      danield17
      Icon for New Contributor rankNew Contributor
      6 months ago

      Hi Ven,

      1. I'm using my own application, which I'm confident is functioning correctly. Through it, I read from BAR2 at the EMIF offset.

      2. In the failure scenario, the values I read were all 0xFFs. Even after attempting to write to the same offset, I continued to read only 0xFFs.

      3. When the parameter was unchecked, I could only read/write successfully if the BAR size was set to 2GB.

      4. Read and write operations only worked when burst capability was enabled. I also tried using other BARs and encountered the same issue as with BAR2 when burst was disabled.

      Regards,
      Daniel

      • Wincent_Altera's avatar
        Wincent_Altera
        Icon for Regular Contributor rankRegular Contributor
        6 months ago

        Hi Daniel,

        The DMA master interface is also connected to the EMIF, and with DMA, both reads and writes work as expected. But with the BAR2 master, it didn’t work.
        >> Do you have any log or .stp showing failure vs passing scenario ?
        >> With that, we can better understand how the fail behave.

        I'm using my own application, which I'm confident is functioning correctly. Through it, I read from BAR2 at the EMIF offset.
        >> do you ever try out the design example that we provided https://www.intel.com/content/www/us/en/design-example/714464/intel-arria-10-fpga-an708-pcie-3-0-x8-avalon-memory-mapped-direct-memory-access-dma-with-external-memory-design-example.html

        >> If not, maybe you can try that and see if the same scenario happen (ONLY if you have time, else we can continue debug your design). BUT , trying that will help to narrow down where the fail coming from and help to accelerate our debug progress - I left the decision back to you.

        Read and write operations only worked when burst capability was enabled. I also tried using other BARs and encountered the same issue as with BAR2 when burst was disabled.
        >> is it compulsory for your design to disable the burst mode ?

        Additional information
        >> Based on my experience IF the Avalon bus implementation does not use the burst. When the memory controller receives a read transaction but the number of write data beats does not match what has been indicated by the busrtcount signal then I assume it will wait for more write data before processing the read.
        >> That could be the reason why you seeing those delay.

        Regards,
        Wincent_Altera

  • VenT_Altera's avatar
    VenT_Altera
    Icon for Frequent Contributor rankFrequent Contributor
    6 months ago

    Hi Daniel,

    I will look into your issue with the PCIe IP.

    Could you please share the pcie.ip file used in your design? I would like to understand further the parameters and settings you have configured.

    Thanks.
    Best Regards,
    Ven

    • danield17's avatar
      danield17
      Icon for New Contributor rankNew Contributor
      6 months ago

      Hi Ven,

      Sorry for the delay in getting back to you.

      I’ve attached the PCIe IP file along with the Quartus project in case you’d like to take a look.
      As I mentioned earlier, everything worked fine when I was using the burst master.
      The issue only came up when I used the BAR2 master without bursting.
      The DMA master interface is also connected to the EMIF, and with DMA, both reads and writes work as expected. But with the BAR2 master, it didn’t work.

      Best Regards,

      Daniel

  • AdzimZM_Altera's avatar
    AdzimZM_Altera
    Icon for Regular Contributor rankRegular Contributor
    6 months ago

    Hi Daniel,


    "Do you still want snapshots and the DDR datasheet? If so, how should I provide the snapshots—images?"

    • Yes, you can send the EMIF IP setting in the PDF and DDR datasheet in PDF.
    • I know the operation is running okay, but let start to debug at EMIF IP first.
    • Then, can check the other.


    For the burst mode questions, the PCIE expert will comment later.


    Regards,

    Adzim


  • AdzimZM_Altera's avatar
    AdzimZM_Altera
    Icon for Regular Contributor rankRegular Contributor
    6 months ago

    Hi Daniel,


    I'm Adzim from Altera. I will assist you in this case.


    Have you verified that you are able to access 8GB DDR before? -maybe use difference design such as EMIF example design.


    We need to make sure the EMIF IP setting is set correctly for the DDR that you used.

    Can you provide the EMIF IP setting (can share the snapshot of the EMIF IP setting) and the DDR/memory datasheet?


    Regards,

    Adzim


    • danield17's avatar
      danield17
      Icon for New Contributor rankNew Contributor
      6 months ago

      Hi Adzim,

      I’ve already created a design that uses DMA and the Address Span Extender, with my BAR set to 2GB, and everything worked fine.

      I haven’t tried the EMIF example design, but I’ve confirmed that I can read and write to the DDR using both the DMA and the BAR master.

      After investigating the issue, I found that read and write operations only work for me when the PCIe Hard IP is configured with burst enabled for the BAR2 master.

      Is there a reason it only works with the burst master? Could it be something internal to the PCIe IP?

      Did I miss something that would cause it to work only in burst mode?

      Do you still want snapshots and the DDR datasheet? If so, how should I provide the snapshots—images?

      Regards,

      Daniel