swm
New ContributorJoined 3 years ago
User Profile
New Contributor
User Widgets
Contributions
Re: How to reliabily program multiple FPGAs using quartus_pgm?
Hello Fakhrul, As I previously mentioned, signal integrity issues and USB hubs are very unlikely to be the problem given the experiement I've undertaken using Docker contains. In each Docker container I put one jtagd and ensured it had access to just one of the USB ports. So I end up with eight Docker containers, one for each FPGA board. At that point porogramming is reliable. Therefore it seems improbably that the issue is signal integrity related. I'm pretty certain the issue is with jtagd not being able to reliably handle multiple FPGAs where each FPGA board has its own USB-to-JTAG. Ideally I'd like to talk with somebody who maintains jtagd. Might this be possible? It might be worth setting up a call. My contact details can be found here: <link removed - you can be contacted by PM> - Simon722Views0likes0CommentsRe: How to reliabily program multiple FPGAs using quartus_pgm?
Thanks for investigating this issue, but your answer is technically incorrect. I only have one FPGA on each JTAG chain not several. So there are no signal integrity issues with the JTAG. The problem is that I have multiple USB connections, one for each FPGA, and the jtagd driver does not cope well with multiple FPGAs each on their own USB-to-JTAG adapter. I'm certain that this is a software issue and not a hardware issue as previsouly explained: Disable some USB ports in software, effectively removing some FPGAs visible to jtagd. The remaining FPGAs could then be repeatedly programmed successfully without and reported errors. Run eight Docker containers each with one jtagd that can only see one FPGA. The FPGAs can then be repeatedly programmed reliably. I would still like somebody at Intel to test jtagd when at least four evaluation cards are connected to the same PC using a USB-to-JTAG link for each board. The bug manifests itself most clearly if the FPGAs and the bit image for the FPGAs is large. Given that jtagd is designed to support multiple FPGA evaluation cards, I am looking for confirmation that there is a regression test that it does indeed work correctly when several cards are connected to the same machine, e.g. that jtagd can in fact reliabily handle multiple FPGA evaluation cards connected. This seems like a reasonable request.5.9KViews0likes0CommentsRe: How to reliabily program multiple FPGAs using quartus_pgm?
Hello Fakhrul, Thanks for continuing to look into this. I'm still very interested to know about the general case rather that the specifics of my setup. It would be interesting to know if programming eight or more FPGA evaluation cards connected to one PC with a USB-to-JTAG adapter for each board has been tested regardless of the particular FPGA baord. That said, programming large FPGAs like the Stratix 10 with a full design is far more likely to provoke the issues reported. Our primary FPGA board at present is the DE10Pro board from Teraisc: https://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=164&No=1144#contents The USB-to-JTAG interface is embedded on this board. I don't believe the USB hub we are using is a factor since we've seen the same problems using a range of different USB host controllers and USB hubs. Thanks, Simon6.1KViews0likes0CommentsRe: How to reliabily program multiple FPGAs using quartus_pgm?
Hello Fakhrul, I'm using Quartus 21.3Pro at present. I have 8 x Terasic DE10Pro cards that contain a Stratix 10 SX part and an on-board USB Blaster II that I'm using to program them in FPGA mode only (i.e. I'm not using the HPS at present). The details of the FPGA image don't appear to matter. In fact the specific evaluation card doesn't appear to matter either: we've seen similar problems programming several Stratix V boards, e.g. the Terasic DE5. The DE10Pro boards are in a GPU-style server (a dual-socket AMD Epyc running Ubuntu 20.04.4 LTS) plugged into PCIe slots. The USB tree is as follows: 2 x native USB3 boards on the motherboard each connected to a four-port USB hub. Four boards are plugged into one hub and four into the other. I'm programming the boards from the command-line, e.g.: quartus_pgm -m jtag -c "DE10-Pro [5-2.3.1]" -o "p;../output_files/DE10_Pro.sof@1" I've since done a bit of Python scripting (attached, with .py replaced with .txt so that it woudl upload) to monitor the output of quartus_pgm and retry programming on any errors, which at least gives me a good way forward. When using this script to reprogram all 8 boards 100 times I'm seeing around 10 to 20 retries, which isn't too bad. An example error report from quartus_pgm is also attached (quartus_pgm_error.txt). While this is probably now good enough that we can use the systems for automated regression test, etc., it is still curious that there are some failures. I also notice that quartus_pgm checked for licenses but doesn't require a license to run. Making LM_LICENSE_FILE environment variable to an empty string causes quartus_pgm to not check for a license, which speeds things up. - Simon6.2KViews0likes0CommentsRe: How to reliabily program multiple FPGAs using quartus_pgm?
Hello Fakhrul, Thanks for investigating. The JTAG Multi-Device Configuration documentation appears to only cover multiple FPGAs on the same JTAG chain. For my setup I have a USB-to-JTAG adapter per FPGA (on the Terasic DE10Pro board) rather than chaining many FPGAs on the same JTAG chain. Simularly for Debugging Suggestions the bug isn't with multiple FPGAs on the same JTAG chain. I'm pretty certain the bug is in software, most likely in the jtagd demon but possibly in quartus_pgm. I have the eight FPGAs connected via two 4-port USB hubs that have per-port controllable power on/off. If I use uhubctl (under Linux) to turn off all ports exept for one FPGA and restart jtagd so that rescans USB ports, then I'm able to program that one FPGA reliably. If I enable any two FPGAs, then I'm able to program them. But if I enable all eight FPGAs the programming usually fails. - Simon6.3KViews0likes0CommentsRe: How to reliabily program multiple FPGAs using quartus_pgm?
Thanks for confirming that you have also seen difficulties with multiple programmers connected to one PC. I did try the GUI for programming, but this didn't improve programming reliability; I see the same behaviour. I am only programming one FPGA at a time to avoid any issues with jtagd struggling with parallel clients.6.3KViews0likes0CommentsRe: How to reliabily program multiple FPGAs using quartus_pgm?
Hello Farabi, thanks for your reply. I believe that I don't have a signal integrity problem on the USB chain due to the following experiements that did not involve changing the USB wiring: Disable some USB ports in software, effectively removing some FPGAs visible to jtagd. The remaining FPGAs could then be repeatedly programmed successfully without and reported errors. Run eight Docker containers each with one jtagd that can only see one FPGA. The FPGAs can then be repeatedly programmed reliably. Do you know if USB-to-JTAG programming of at least eight FPGAs connected to one PC has been tested?6.4KViews0likes0CommentsHow to reliabily program multiple FPGAs using quartus_pgm?
I have a server with 8 x Stratix 10 SX boards (Terasic DE10Pro), which I would like to be able to program over usb-to-jtag. When I have one or two baords installed, I can reliably program a board, e.g. using: quartus_pgm -c 1 -m jtag -o "p;output_files/DE10_Pro.sof@2" However, once all eight FPGA baords are connected, the programming process times out: Info (18942): Configuring device index 2 Error (18939): Unexpected error in JTAG server: Timeout Error (18939): Unexpected error in JTAG server: Timeout Error (18952): Error status: Synchronization failed Error (209012): Operation failed Is quartus_pgm known to work with eight or more boards? - Simon7.1KViews0likes21Comments