Occasional Contributor
Occasional ContributorHi Pramod,
we did a similar architecture where we had multiple masters connected to a single HBM2 channel + PCIe DMA access, too.
Here the HBM2 burst controller seems to have a bug in some specific Quartus versions (Q20.4 and lower).
We succeeded with a solution provided by Intel to patch the IP files after IP generation (yes, you won't be able to use the common tool flow anymore sadly). With Quartus 21.1 you will have this fix provided in the IP again.
https://www.intel.com/content/www/us/en/support/programmable/articles/000086781.html
For us it was a long way to debug this. Hopefully this will help you.
FYI:
We currently didn't switch to 21.1 because with 20.4 we got no timing violations in our design and with 21.1 the retiming process is not working properly again. With this we got tremendous timing violations and there we did not get a clear information from Intel support team why this is happening with a version upgrade.
Let's see if this is fixed in further versions...
Kind regards,
Michael
New ContributorHi Michael,
Thanks for the reply.
I am using Quartus 21.3 version. That should have solved the burst count issue, but still I am seeing the same issue.
I will try replacing the auto-generated file from the link you sent and check.
In 21.3, I am getting some timing violations, but most of them are false paths (signaltap related).
Did you use AVMM or AVST for PCIe interface ?
Best Regards,
Pramod
Occasional ContributorHi Pramod,
ok I would assume that should be resolved already.
Could you share the configuration of the HBM2 controller IP file? Then I could compare it with our IP file.
Maybe there are some further settings necessary to enable the burst support.
We are using the PCIe AVMM (DMA) connected to the HBM2 (Stratix 10 MX).
Do you use any module in between PCIe <-> HBM2?
We faced some issue with the AXI bridge in between and just connected the AVMM of the PCIe directly to the HBM2 AXI ports.
Kind regards,
Michael
New ContributorHi Michael,
There is an option in HBM Controller, which says enable burstcount greater than 2 for AXI interface. I have enabled it and entered a value of 128. Ideally, it should have solved the issue.
HBM2 Controller IP file -- you mean XML file which contains the configuration, right ?
I have attached the XML file here.
I started with the design example PCIe + DDR + HBM2 :
https://fpgacloud.intel.com/devstore/platform/19.1.0/Pro/an881-pcie-avmm-dma-gen3x16-ddr4-and-hbm2/
I upgraded the design from 19.2 to 21.3. I couldnt get the DMA software working in our Ubuntu 18.04 setup.
Problem was PIO operation was working fine, but DMA operation was always failing. Some of the functions in driver provided requires older linux kernel version, while Ubuntu 18.04 has newer linux kernel version.
If you can provide some pointers on your setup or how to make it working, it would be helpful.
Hence I replaced the AVMM PCIe IP with MCDMA IP, whose software is working in Ubuntu 18.04.
Example design has HBM to AXI verilog wrapper file, I have not modified it.
I am able to do PIO operations, DMA operations where payload is limited to 64 Bytes at a time.
Best Regards,
Pramod
Best Regards,
Pramod
Occasional ContributorHi Pramod,
here a screenshot of our current controller settings in the HBM2 controller IP core (Quartus 20.4 / IP version 19.6.1):
HBM2 controller settings
Unfortunately we did not test the example design. Furthermore we wrote our own driver for the PCIe DMA access which I'm allowed to share due to confidential information.
We implemented our own converter (confidential, too) because the Intel AXI Bridge could not achieved the promised data rate.
We had several discussion/debug sessions with Intel Premier support over weeks but all provided possibilities with Intel IP only were not successful.
We added a signal tap in the mm_interconnect (which is autogenerated by QSYS) and AXI Bridge IP core and saw wrong AXI transactions ongoing which blocked/decreased the throughput tremendously. We proved that in our simulations, too.
Further, the data rate between PCIe DMA and HBM2 wasn't our main topic. We had some further AXI busses connected to the HBM2 which are providing the data (here we had a high data rate requirement).
The readout of the data using PCIe DMA wasn't used to have a specific data rate.
For the AXI connection to the AXI HBM2 we implemented our own fabric, too, instead of using the autogenerated mm_interconnect.
Kind regards,
Michael
