Forum Discussion

HKim27's avatar
HKim27
Icon for Occasional Contributor rankOccasional Contributor
6 years ago
Solved

Quartus crash with "Internal Error: Sub-system: THR"

Hi, all.

I use Intel® Acceleration Stack Version 1.2 with Intel® Programmable Acceleration Card with Intel Arria® 10 GX FPGA on CentOS Linux release 7.2.1511.

When I compile my design, Quartus keeps crashing with the following error:

Internal Error: Sub-system: THR, File: /quartus/ccl/thr/thr_interprocess_mutex.cpp, Line: 122
err == EACCES || err == EAGAIN
Stack Trace:
     0x5db7: thr_internal_error + 0x1e (ccl_thr)
     0x9d09: thr_ip_mutex_lock_helper(THR_IP_MUTEX_STRUCT*, bool) + 0x81 (ccl_thr)
     0x9d5c: thr_ip_mutex_lock + 0xe (ccl_thr)
    0x89eb1: RDB_CONCURRENT_ACCESS::lock(std::string const&, RDB_REPORT_TYPE) + 0x2bf (db_rdb)
    0x438a1: FIT2_REPORTING::initialize_rdb(CMP_FACADE&, QHD_ENUM_IMPL<QHD_STATE::TYPE_BASE>, bool) + 0xb3 (comp_fit2)
    0x3fa25: FIT2_DATABASE_EXPERT_QDB::load(FIT2_ENV::STAGE, std::string, unsigned int const*, bool, bool, bool) + 0x679 (comp_fit2)
    0x253f4: fit2_create_fitter_netlist + 0x1154 (comp_fit2)
    0x4c977: TclNRRunCallbacks + 0x47 (tcl8.6)
    0x4e1ca: TclEvalEx + 0x96a (tcl8.6)
    0x4e4d6: Tcl_EvalEx + 0x16 (tcl8.6)
    0x4e4fd: Tcl_Eval + 0x1d (tcl8.6)
    0x1c9e1: atcl_tcl_eval(Tcl_Interp*, std::string const&) + 0x12d (ccl_atcl)
    0x44bba: fit2_run_create_fitter_netlist(Tcl_Interp*, std::string const&, ATCL_ARGS&, int&) + 0x5cd (comp_fit2)
    0x26663: fit2_fit_plan_init + 0xd3 (comp_fit2)
    0x4c977: TclNRRunCallbacks + 0x47 (tcl8.6)
    0x14973: fit2_fit_plan + 0x2ff (comp_fit2)
    0x4c977: TclNRRunCallbacks + 0x47 (tcl8.6)
    0x4e1ca: TclEvalEx + 0x96a (tcl8.6)
    0xf68c6: Tcl_FSEvalFileEx + 0x266 (tcl8.6)
    0xf69ce: Tcl_EvalFile + 0x2e (tcl8.6)
    0x11d8b: qexe_evaluate_tcl_script(std::string const&) + 0x390 (comp_qexe)
    0x1611d: qexe_do_tcl(QEXE_FRAMEWORK*, std::string const&, std::string const&, std::list<std::string, std::allocator<std::string> > const&, bool, bool) + 0x6c0 (comp_qexe)
    0x1720e: qexe_run_tcl_option(QEXE_FRAMEWORK*, char const*, std::list<std::string, std::allocator<std::string> >*, bool) + 0x5df (comp_qexe)
    0x39069: qcu_run_tcl_option(QCU_FRAMEWORK*, char const*, std::list<std::string, std::allocator<std::string> >*, bool) + 0xf20 (comp_qcu)
    0x1ab65: qexe_standard_main(QEXE_FRAMEWORK*, QEXE_OPTION_DEFINITION const**, int, char const**) + 0x486 (comp_qexe)
   0x403630: qfit2_main(int, char const**) + 0xd0 (quartus_fit)
    0x3f160: msg_main_thread(void*) + 0x10 (ccl_msg)
     0x5b4c: thr_final_wrapper + 0xc (ccl_thr)
    0x3f21f: msg_thread_wrapper(void* (*)(void*), void*) + 0x62 (ccl_msg)
     0xac5c: mem_thread_wrapper(void* (*)(void*), void*) + 0x5c (ccl_mem)
     0x8b49: err_thread_wrapper(void* (*)(void*), void*) + 0x27 (ccl_err)
     0x5b8f: thr_thread_wrapper + 0x15 (ccl_thr)
    0x41b11: msg_exe_main(int, char const**, int (*)(int, char const**)) + 0xb2 (ccl_msg)
    0x21b15: __libc_start_main + 0xf5 (c)
End-trace

I also attach the full log.

Reinstalling the stack did not solve the problem.

Does anyone have experience with this problem?

  • HKim27's avatar
    HKim27
    6 years ago

    Hi, I actually solved the problem yesterday.

    My compile system uses Lustre filesystem. The filesystem setup was a bit broken, and it was returning ENOSYS for fcntl(..., F_SETLK, ...) call.

    After fixing the filesystem, compilation works perfectly.

    Here is the script I used to debug:

    #include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
 
int main() {
  int fd = open("a.txt", O_RDWR);
  if (fd == -1) {
    printf("open error (errno=%d)\n", errno);
    exit(1);
  }
 
  struct flock lock;
  lock.l_type = F_WRLCK;
  lock.l_whence = SEEK_SET;
  lock.l_start = 0;
  lock.l_len = 1;
  int ret = fcntl(fd, F_SETLK, &lock);
  if (ret == -1) {
    printf("fcntl error (errno=%d)\n", errno);
    exit(1);
  }
 
  printf("no error\n");
 
  return 0;
}

    If someone get "fcntl error (errno=38)", that's the problem.

10 Replies

  • MEIYAN_L_Intel's avatar
    MEIYAN_L_Intel
    Icon for Frequent Contributor rankFrequent Contributor
    6 years ago

    Hi,

    May I know that do you trying to open project which is created in a different version of Quartus(example if the project is created in sdt version and trying to open in pro version)?

    May you try to open the project with an appropriate auto upgrade?(if all the IP are supported in the Quartus version, Project will be opened without and any error)

    Thanks

    • HKim27's avatar
      HKim27
      Icon for Occasional Contributor rankOccasional Contributor
      6 years ago

      Hi, @MeiYanL_Intel​ .

      I use OPAE included in Intel Acceleration Stack Version 1.2, so the project is created and compiled automatically with the following script.

      afu_synth_setup --source hw/rtl/sources.txt --platform discrete_pcie3 build_synth
cd build_synth
$(OPAE_PLATFORM_ROOT)/bin/run.sh

      So I guess it is not version mismatch problem. (will be created and compiled in Quartus Prime Pro 17.1.1)

      I opened the project with Quartus, and all IPs seem fine.

    • HKim27's avatar
      HKim27
      Icon for Occasional Contributor rankOccasional Contributor
      6 years ago

      Hi, @MeiYanL_Intel​ ​

      How do you know the issues come from Intel FPGA BBB?

      Because I should have some evidences that the issues come from Intel FPGA BBB, to report issues on their side :)

      Thanks in advance.

  • MEIYAN_L_Intel's avatar
    MEIYAN_L_Intel
    Icon for Frequent Contributor rankFrequent Contributor
    6 years ago

    Hi,

    Can you help to provide all the neccessary file (i.e Kernal *.cl files and etc) that were able to duplicate the error above? We will try in our side to see if this can be duplicated. From there, we will try to get a workaround for you if they are.

    Thanks

    • HKim27's avatar
      HKim27
      Icon for Occasional Contributor rankOccasional Contributor
      6 years ago

      Hi, I sent you a message.

      You can compile with "make" command inside __all___workspace directory.

      Note that Intel® Acceleration Stack Version 1.2 should be installed and environmental variable FPGA_BBB_CCI_SRC should point to intel-fpga-bbb. (e.g., /home/heehoon/intel-fpga-bbb)

  • MEIYAN_L_Intel's avatar
    MEIYAN_L_Intel
    Icon for Frequent Contributor rankFrequent Contributor
    6 years ago

    Hi,

    I had received and downloaded the file. I would like to have sometime to setup the environment.

    Thanks

    • HKim27's avatar
      HKim27
      Icon for Occasional Contributor rankOccasional Contributor
      6 years ago

      Hi, I actually solved the problem yesterday.

      My compile system uses Lustre filesystem. The filesystem setup was a bit broken, and it was returning ENOSYS for fcntl(..., F_SETLK, ...) call.

      After fixing the filesystem, compilation works perfectly.

      Here is the script I used to debug:

      #include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
 
int main() {
  int fd = open("a.txt", O_RDWR);
  if (fd == -1) {
    printf("open error (errno=%d)\n", errno);
    exit(1);
  }
 
  struct flock lock;
  lock.l_type = F_WRLCK;
  lock.l_whence = SEEK_SET;
  lock.l_start = 0;
  lock.l_len = 1;
  int ret = fcntl(fd, F_SETLK, &lock);
  if (ret == -1) {
    printf("fcntl error (errno=%d)\n", errno);
    exit(1);
  }
 
  printf("no error\n");
 
  return 0;
}

      If someone get "fcntl error (errno=38)", that's the problem.