Crash when programming .sof file

This is an Arria 10 device.

We've set up an automated test-server (to test incoming pull request) and we're seeing this failure occasionally on that server.

That this is not stable is problematic as it reduces the throughput of that test-server.

Nothing else was running at the time. This is an automated script, so I know nothing else is going on on that machine. It is a Jenkins slave that's dedicated to regression teseting our FPGA bit file. The Arria 10 PC is a PCIe card in that machine and we're programming it using a loopback USB cable(we can't use CvP as we need to change the clock frequency of the design).

I need more data on the frequency, but I guesstimate the chances of failure to be 1:10.

The loopback cable means that the the Arria 10 PCIe eval card is in the PC and that I have connected the USB Blaster of that eval card to the same PC.

The command I used to program the FPGA is the below. The second time I ran it, using the same .sof file, just 30 minutes afterwards without any other change that I'm aware of, it worked.

Hmm.... come to think of it... We've upgraded to Quartus 20.1 elsewhere, but the niso2-configure-sof command is still from Quartus 17.3 (as I recall).

# program FPGA

nios2-configure-sof top.sof

The output is:

Searching for SOF file:

in .

top.sof

*** Fatal Error: Segment Violation at 0x22f700000076

Module: quartus_pgm

Stack Trace:

0xf8ea: rml::internal::internalPoolFree(rml::internal::MemoryPool*, void*) + 0x4a (tbbmalloc.so.2)

0x1729: operator delete(void*) + 0x9 (tbbmalloc_proxy.so.2)

0x2083e: S4_SII_BIT::~S4_SII_BIT() + 0x3e (pgm_build_npcf)

0x20901: S4_SII::~S4_SII() + 0x5b (pgm_build_npcf)

0x209de: NPCF_DEVICE_INFO::set_s4_s2_list(std::vector<S4_SII*, std::allocator<S4_SII*> >*, bool, bool) + 0x4c (pgm_build_npcf)

0x24655: NPCF_DEVICE_INFO::~NPCF_DEVICE_INFO() + 0x169 (pgm_build_npcf)

0x246e7: NPCF_INFO::~NPCF_INFO() + 0x41 (pgm_build_npcf)

0xe183c: FBGEN_GEN::~FBGEN_GEN() + 0x5c (pgm_fbgen)

0xe1e19: FBGEN_GEN::~FBGEN_GEN() + 0x9 (pgm_fbgen)

0x420be3: PGMIO_FBGEN_PROXY::~PGMIO_FBGEN_PROXY() + 0x13 (pgm_pgmio)

0x2c2d5c: PGMIO_F2P::create_bitstream(PGM_CHAIN_ELEMENT*, std::vector<std::string, std::allocator<std::string> >*, PGMIO_CCF*) + 0x1dc (pgm_pgmio)

0x284c14: PGM_CHAIN_ELEMENT::generate_bv_list(bool) + 0x104 (pgm_pgmio)

0x28868d: PGM_CHAIN_ELEMENT::create_chain_element(PGM_CHAIN_ELEMENT*, bool, FIO_PATH*, bool, PGMIO_CONFIG_SCHEME, bool, bool) + 0xd2d (pgm_pgmio)

0x232d9: PGME_PROGRAMMER::lookup_device(PGM_CHAIN_ELEMENT*, PGMIO_CONFIG_SCHEME, bool, bool, bool) + 0x29 (pgm_pgme)

0x2175d: QPGM_FRAMEWORK::create_element(std::string, std::string, unsigned int, unsigned int) + 0x601 (quartus_pgm)

0x23b91: QPGM_FRAMEWORK::process_operation(std::string*) + 0x1e93 (quartus_pgm)

0x24cde: QPGM_FRAMEWORK::post_check_arguments() + 0x2d6 (quartus_pgm)

0x1c08f: qexe_standard_main(QEXE_FRAMEWORK*, QEXE_OPTION_DEFINITION const**, int, char const**) + 0x1bc (comp_qexe)

0x1fd97: qpgm_main(int, char const**) + 0x5e (quartus_pgm)

0x40720: msg_main_thread(void*) + 0x10 (ccl_msg)

0x602c: thr_final_wrapper + 0xc (ccl_thr)

0x407df: msg_thread_wrapper(void* (*)(void*), void*) + 0x62 (ccl_msg)

0xa559: mem_thread_wrapper(void* (*)(void*), void*) + 0x99 (ccl_mem)

0x8f92: err_thread_wrapper(void* (*)(void*), void*) + 0x27 (ccl_err)

0x63f2: thr_thread_wrapper + 0x15 (ccl_thr)

0x427e2: msg_exe_main(int, char const**, int (*)(int, char const**)) + 0xa3 (ccl_msg)

0x1fe21: main + 0x26 (quartus_pgm)

0x270b3: __libc_start_main + 0xf3 (c.so.6)

End-trace

Failed to program top.sof, erasing possibly corrupt top.sof

I'll see if I can do that. A good suggestion. Though... if you look at the stack trace, you'll find that the crash is in the quartus_pgm process, so nios2-configure-sof must be launching quartus_pgm.

I'll also upgrade to the latest quartus_pgm.