Forum Discussion
Occasional ContributorLWH2F Throughput
Dear all,
I'm facing the issue that the throughput on the Agilex 5 LWH2F interface appears to be lower compared to Cyclone 5.
- I have two custom boards, one is based on Cyclone 5, the other on Agilex 5.
- I use a linux OS. The code runs in a linux kernel driver. The code for the driver is the same for both devices.
- The memory is mapped using ioremap(), i.e. it is mapped as device memory.
- I measure performance by taking a timestamp with ktime_get_ns(), then read/write 10000 uint32 values, then take another timestamp.
- I've measured the follwing values
- Cyclone5 board read: 47.2 MB/s (million bytes per second or 11.8 million words à 4 byte per second)
- Cyclone5 board write: 73.1 MB/s
- Agilex5 board read: 24.8 MB/s
- Agilex5 board write: 18.8 MB/s
- I've noticed that performance varies, depending on the CPU that the process is running on (when measuring in userspace I can explicitly set the cpu affinity; For the kernel driver I've noticed that it is sometimes slower than above, presumably because it's running on a different cpu).
- There is a slight difference in the QSYS design:
- The Cyclone5 based board:
- drives the AXI bus with a 64 MHz clock.
- uses the Avalon MM Slave Translator.
- The Agilex5 based board:
- drives the AXI bus with a 200 MHz clock.
- uses the Avalon Memory Mapped Pipeline Bridge Intel FPGA IP.
- Our FPGA takes one 64/200 MHz cycle to process the read (readdatavalid). For a write our FPGA doesn't generate a writeresponsevalid, this is handled by the IP block.
- I'm using Quartus 25.1.1 for the Agilex 5 design.
I'm aware that the LWH2F interface is not intended for high throughput. Also, since the memory is mapped as "Device Memory", every load/store is processed separately and we're not taking advantage of AXI bursts, etc.
I'm aware that we could improve performance by using the H2F interface and mapping the memory as normal memory.
That said, we have a prooven design and are reluctant to change it unless absolutely neccessary.
So I have the following questions:
- Is a higher latency expected on Agilex5? (E.g. due to a different architecture of the interconnect)
- Have you measured the performance of the LWH2F interface? Can you give me a number on how many transactions per second we can expect?
Kind Regards,
Eric Opitz
10 Replies
EricOpitzHi,
For completeness I'm adding some additional measurements (I've already sent them to Altera via E-Mail).
The following tests were made using the LWH2F or H2F bridge with a clock rate of 200 MHz. The width of the H2F bridge is 128 bit. The tests are run from CPU 2 (A76). I've measured the write throughput.
lwh2f kernel driver ioremap *ptr=i 18.5 MB/s
lwh2f kernel driver ioremap memcpy 69.9 MB/s
lwh2f kernel driver ioremap_wc memcpy 205.1 MB/s
lwh2f kernel driver ioremap_cache memcpy [kernel panic]
lwh2f userspace (O_SYNC) memcpy 68.2 MB/s
lwh2f userspace (no O_SYNC) memcpy 68.3 MB/sh2f kernel driver ioremap memcpy 72.2 MB/s
h2f kernel driver ioremap_wc memcpy 258.0 MB/s
h2f kernel driver ioremap_cache memcpy [kernel panic]
h2f userspace (no O_SYNC) memcpy 73.6 MB/sThe ioremap_* variants should result in the following flags:
cached arm64 flags
ioremap no device memory, nGnRE
ioremap_wc no normal memory
ioremap_cache yes normal memory- ioremap_wc is much faster than ioremap(). The performance improvements is due to longer bursts (4 cycle bursts, see below). Still, there is a long latency between requests (around 250ns), the requests are not pipelined, and the bursts aren't as long as they could be (According to the TRM max burst length is 16 for LWH2F and 8 for H2F).
- The theoretical throughput is 800 MB/s for LWH2f and 3200 MB/s for H2F. Above measurements show significantly lower throughput.Kind Regards,
Eric Opitz----
Write CPU2 (A76) H2F ioremap_wc
NDoeNew Contributor
Hi, we are also seeing similar issues with both the lwhps2fpga and hps2fpga interfaces.
The board is the Terasic DE25-Nano, the device is A5EB013BB23BE4SR1 with Quartus version 25.3.1 patch 1.02.
The bus cycle completes quickly in the time expected, but repeated bus cycles stall for a very long time between consecutive reads or consecutive writes. In my case I am seeing a delay of 400ns between cycle end and the next cycle start with all bus clocks connected to a single 100MHz source.
Best regards,
Nigel Doe
- EricOpitz
Hi tehjingy_Altera,
I took some measurements with the scope that I want to share with you.
I've modified the Agilex 5 QSYS image:
- In the "Avalon Memory Mapped Pipeline Bridge Inte FPGA IP" I've deactivated the options "Pipeline command signals" and "Pipeline response signals". This has improved performance slightly.
- I want to monitor the AXI signals directly. I have exported the AXI signals, output them on pins, and reimport them in QSYS using an AXI bridge. The bridge appears to add a small delay, degrading the performance slightly.The read/write performance is:
read: 24.8 MB/s
write: 17.8 MB/sNote that write is slightly slower compared to my first post.
Above performance measurements and the measurements from my first post were taken using the A76 (taskset -c 2). When I run the memory test on A55 (taskset -c 0) I get:
read: 16.3 MB/s
write: 17.8 MB/s
Surprisingly, the read performance is a lot worse compared to A76.Attached you find corresponding scope images.
From the scope images you can see that the AXI write is finished (bvalid) in <50ns. The majority of the time is spent waiting for the next request from the CPU.Do you see a similar behavior on your systems?
Kind Regards,
Eric Opitz----
legend:
axi0 = agilex is master
axi1 = AXI-To-AvalonMM converter is master
mm_test = After the AXI-To-AvalonMM converter
D[ 0] = axi0_awvalid;
D[ 1] = axi1_awready;
D[ 2] = axi0_wvalid;
D[ 3] = axi0_wlast;
D[ 4] = axi1_wready;
D[ 6] = axi1_bvalid;
D[ 7] = axi0_bready;
D[ 8] = axi0_arvalid;
D[ 9] = axi1_arready;
D[10] = axi1_rvalid;
D[11] = axi0_rready;
D[12] = mm_test_write;
D[13] = mm_test_read;
D[14] = mm_test_readdatavalid;Read CPU 0 (A55)
Read CPU 2 (A76)
Write CPU 0 (A55)
Write CPU 2 (A76)
tehjingy_AlteraRegular Contributor
Hi EricOpitz
Did you mean the changes did not improve the performance?
Could you share what is the current R/W speed of the LWH2F after having the workaround?
- EricOpitz
Hi tehjingy_Altera,
The measurements in my first post were taken with the workaround (cache size = 1MB).
Kind Regards,
Eric Opitz
SueC_AlteraContributor
Hi Eric,
We are looking into this. We'll get back to you when we have more information.
Thanks,
Sue
- EricOpitz
Our FPGA design has the workaround activated already.
I've also tried deactivating it ("Override default cache size" = false). It did not degrade the performance in my particular case.
Kind Regards,
Eric Opitz - EricOpitz
Hi tehjingy_Altera,
Yes our device is affected by the errata. The device number is A5ED013BB32AE4S. The errata suggests the workaround "Reduce L3 cache from 2MB to 1MB." We have configured this workaround in QSYS ("Override default cache size", MPU L3 Cache Size = 1MB).
Should the workaround eliminate the performance issues entirely?
Kind Regards,
Eric Opitz- tehjingy_Altera
Hi EricOpitz
This would not eliminate the performance issue but it would help improve the performance of the device.
- tehjingy_Altera
Hi EricOpitz
For your reference, there is an errata that discusses potential HPS performance considerations related to the L3 cache, which you can review here:
https://docs.altera.com/r/docs/825514/current/agilextm-5-es-device-errata-and-user-guidelines/degraded-hps-emif-performance-with-2mb-l3-cacheTo better understand whether this may be applicable in your case, could you please confirm if your device part number is included in the list described?
If possible, kindly share the exact device part number so we can help review it more accurately.
