How to reliabily program multiple FPGAs using quartus_pgm?

Hello Farabi, thanks for your reply. I believe that I don't have a signal integrity problem on the USB chain due to the following experiements that did not involve changing the USB wiring:

1. Disable some USB ports in software, effectively removing some FPGAs visible to jtagd. The remaining FPGAs could then be repeatedly programmed successfully without and reported errors.
2. Run eight Docker containers each with one jtagd that can only see one FPGA. The FPGAs can then be repeatedly programmed reliably.

Do you know if USB-to-JTAG programming of at least eight FPGAs connected to one PC has been tested?

Thanks for confirming that you have also seen difficulties with multiple programmers connected to one PC.

I did try the GUI for programming, but this didn't improve programming reliability; I see the same behaviour. I am only programming one FPGA at a time to avoid any issues with jtagd struggling with parallel clients.

Hello Fakhrul,

Thanks for investigating.

The JTAG Multi-Device Configuration documentation appears to only cover multiple FPGAs on the same JTAG chain. For my setup I have a USB-to-JTAG adapter per FPGA (on the Terasic DE10Pro board) rather than chaining many FPGAs on the same JTAG chain. Simularly for Debugging Suggestions the bug isn't with multiple FPGAs on the same JTAG chain.

I'm pretty certain the bug is in software, most likely in the jtagd demon but possibly in quartus_pgm.

I have the eight FPGAs connected via two 4-port USB hubs that have per-port controllable power on/off. If I use uhubctl (under Linux) to turn off all ports exept for one FPGA and restart jtagd so that rescans USB ports, then I'm able to program that one FPGA reliably. If I enable any two FPGAs, then I'm able to program them. But if I enable all eight FPGAs the programming usually fails.

- Simon

Hello Fakhrul,

I'm using Quartus 21.3Pro at present. I have 8 x Terasic DE10Pro cards that contain a Stratix 10 SX part and an on-board USB Blaster II that I'm using to program them in FPGA mode only (i.e. I'm not using the HPS at present). The details of the FPGA image don't appear to matter. In fact the specific evaluation card doesn't appear to matter either: we've seen similar problems programming several Stratix V boards, e.g. the Terasic DE5.

The DE10Pro boards are in a GPU-style server (a dual-socket AMD Epyc running Ubuntu 20.04.4 LTS) plugged into PCIe slots. The USB tree is as follows: 2 x native USB3 boards on the motherboard each connected to a four-port USB hub. Four boards are plugged into one hub and four into the other.

I'm programming the boards from the command-line, e.g.:

quartus_pgm -m jtag -c "DE10-Pro [5-2.3.1]" -o "p;../output_files/DE10_Pro.sof@1"

I've since done a bit of Python scripting (attached, with .py replaced with .txt so that it woudl upload) to monitor the output of quartus_pgm and retry programming on any errors, which at least gives me a good way forward. When using this script to reprogram all 8 boards 100 times I'm seeing around 10 to 20 retries, which isn't too bad. An example error report from quartus_pgm is also attached (quartus_pgm_error.txt). While this is probably now good enough that we can use the systems for automated regression test, etc., it is still curious that there are some failures.

I also notice that quartus_pgm checked for licenses but doesn't require a license to run. Making LM_LICENSE_FILE environment variable to an empty string causes quartus_pgm to not check for a license, which speeds things up.

- Simon

Hello Fakhrul,

Thanks for continuing to look into this. I'm still very interested to know about the general case rather that the specifics of my setup. It would be interesting to know if programming eight or more FPGA evaluation cards connected to one PC with a USB-to-JTAG adapter for each board has been tested regardless of the particular FPGA baord. That said, programming large FPGAs like the Stratix 10 with a full design is far more likely to provoke the issues reported.

Our primary FPGA board at present is the DE10Pro board from Teraisc:
https://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=164&No=1144#contents

The USB-to-JTAG interface is embedded on this board.

I don't believe the USB hub we are using is a factor since we've seen the same problems using a range of different USB host controllers and USB hubs.

Thanks,

Simon