AI Suite System Throughput Issue
When using AI Suite, we are seeing a significant gap between IP throughput and achieved system throughput on Agilex 5. I am using the following: Hardware: Agilex™ 5 FPGA and SoC E-Series Modular Development Kit (ES silicon) Software: Quartus Prime Pro + AI Suite 25.3.1 SD Image: agx5_soc_s2m coredla-image-agilex5_mk_a5e065bb32aes1.wic Architecture and Bitstream: AGX5_Performance Using MobileNetV2 (Open Model Zoo 2024.6.0) compiled using AGX5_Performance architecture gives the following results through dla_benchmark IP throughput per instance: ~151 FPS Estimated throughput (200 MHz): ~178 FPS System throughput: nireq=1 → 41 FPS nireq=4 → 54 FPS Why is there such a big delta between IP Performance and System Throughput and how can we improve the system throughput? For more details please see the append log showing the commands that I run to do the benchmark Any pointers or help would be highly appreciated. Thanks ===================================================================== 1. Using mobilenet v2 from model zoo ===================================================================== Commands used to download and compile model: git clone https://github.com/openvinotoolkit/open_model_zoo.git cd open_model_zoo git checkout 2024.6.0 omz_downloader --list omz_downloader --name mobilenet-v2-pytorch --output_dir $COREDLA_WORK/demo/models/ omz_converter --name mobilenet-v2-pytorch --download_dir ../demo/models/ --output_dir ../demo/models/ cd $COREDLA_WORK/demo/models/public/mobilenet-v2-pytorch/FP32 dla_compiler --march $COREDLA_ROOT/example_architectures/AGX5_Performance.arch --network-file ./mobilenet-v2-pytorch.xml --foutput-format=open_vino_hetero --o $COREDLA_WORK/demo/mobilenet-v2-pytorch_dla.bin --batch-size=1 --fanalyze-performance --fassumed-fmax-core 200 Executing performance estimate ---------------------------------------------------------------- main_graph_0 reported throughput: 178.617 fps TOTAL DDR SPACE REQUIRED = 16.9756 MB DDR INPUT & OUTPUT BUFFER SIZE = 0.781738 MB DDR CONFIG BUFFER SIZE = 0.0986328 MB DDR FILTER BUFFER SIZE = 15.3296 MB DDR INTERMEDIATE BUFFER SIZE = 0.765625 MB NOTE: THIS ESTIMATE ASSUMES 1x I/O BUFFER. THE COREDLA RUNTIME DEFAULTS TO 5 TOTAL DDR TRANSFERS REQUIRED = 18.7003 MB DDR FILTER READS REQUIRED = 16.2124 MB DDR FEATURE READS REQUIRED = 1.62164 MB DDR FEATURE WRITES REQUIRED = 0.767578 MB NUMBER OF DDR FEATURE READS = 9 MINIMUM AVERAGE DDR BANDWIDTH REQUIRED = 3340.19 MB/s ASSUMED DDR BANDWIDTH PER IP INSTANCE = 6400 MB/s ---------------------------------------------------------------- Performance Estimator Throughput Breakdown Arch: kvec64xcvec32_i12x1_fp12agx_sb32768_xbark32_actk32_poolk4 Number of DLA instances = 1 Number of DDR Banks per DLA instance = 1 CoreDLA Target Fmax = 200 MHz PE Target Fmax = 200 MHz Batch Size = 1 PE-only Conv Throughput No DDR = 186 fps PE-only Conv Throughput = 185 fps Overall Throughput Inf PE Buf Depth (zero MPBW) = 185 fps Overall Throughput Zero PE Buf Depth (zero MPBW) = 183 fps Overall Throughput Inf PE Buf Depth = 184 fps Overall Throughput Zero PE Buf Depth = 182 fps ---------------------------------------------------------------- FINAL THROUGHPUT = 178.617 fps FINAL THROUGHPUT PER FMAX (CoreDLA) = 0.893086 fps/MHz FINAL THROUGHPUT PER FMAX (PE) = 0.893086 fps/MHz Running the model on dev kit: ./dla_benchmark -b=1 -cm $compiled_model -d=HETERO:FPGA,CPU -i $imgdir -niter=8 -plugins ./plugins.xml -arch_file $archfile -api=async -groundtruth_loc $imgdir/ground_truth.txt -perf_est -nireq=1 -bgr -nthreads=1 [Step 11/12] Dumping statistics report count: 8 iterations system duration: 191.3784 ms IP duration: 52.7551 ms latency: 23.4076 ms system throughput: 41.8020 FPS number of hardware instances: 1 number of network instances: 1 IP throughput per instance: 151.6441 FPS IP throughput per fmax per instance: 0.7582 FPS/MHz IP clock frequency measurement: 200.0000 MHz estimated IP throughput per instance: 178.6172 FPS (200 MHz assumed) estimated IP throughput per fmax per instance: 0.8931 FPS/MHz ./dla_benchmark -b=1 -cm $compiled_model -d=HETERO:FPGA,CPU -i $imgdir -niter=8 -plugins ./plugins.xml -arch_file $archfile -api=async -groundtruth_loc $imgdir/ground_truth.txt -perf_est -nireq=4 -bgr -nthreads=4 [Step 11/12] Dumping statistics report count: 8 iterations system duration: 147.8426 ms IP duration: 52.7619 ms latency: 69.8254 ms system throughput: 54.1116 FPS number of hardware instances: 1 number of network instances: 1 IP throughput per instance: 151.6246 FPS IP throughput per fmax per instance: 0.7581 FPS/MHz IP clock frequency measurement: 200.0000 MHz estimated IP throughput per instance: 178.6172 FPS (200 MHz assumed) estimated IP throughput per fmax per instance: 0.8931 FPS/MHz