I run a Questa simulation for an IP core from the command line on a Linux server. Usually, the simulation finishs successfully. But every now and then the simulation hangs.When it hangs, I can see that the CPU load is 100% but I can't break the simulation by hitting Ctrl-C. The key-press is detected ("# Break key hit") but the simulation won't halt.Since I use the same seed for random number generation in each run, this seems like nondeterministic behavior.I have no clue, where this behavior is comming from. Are there any command line options that I can use to get closer to the problem? How can I go one with debugging in such a case?

I tried to run all processes with valgrind. This causes the simulation to terminate with:"Process terminating with default action of signal 41 (SIGRT9)"and"Trouble with Simulation Kernel."Of course, this might be a side effect of valgrind.

I tried a testbench from another project with valgrind and it works fine. So it seems there is really something wrong with the simulator.

As it worked fine after you changed the project, the error may related to the project and testbench. It may be that your project got stuck during simulation, causing excessive CPU resource usage.You may check your testbench to see if there are any operations that cause an infinite loop.

Thanks for the reply.Questa is able to detect zero delay loops, so I should see an error about this. Furthermore, most of the time the tests run just fine. I have to run a single test a few hundred times to cause this behavior. That means it is nondeterministic behavior. If there is an infinite zero delay loop I should see the hang up in every simulation run.Besides that, I should be able to break/pause simulation in such a loop, which does not work either.

Questa simulation hangs (nondeterministic)

37 Replies

c-thaler
Occasional Contributor
2 years ago
I tried to run all processes with valgrind. This causes the simulation to terminate with:
"Process terminating with default action of signal 41 (SIGRT9)"
and
"Trouble with Simulation Kernel."
Of course, this might be a side effect of valgrind.
- c-thaler
  Occasional Contributor
  2 years ago
  I tried a testbench from another project with valgrind and it works fine. So it seems there is really something wrong with the simulator.

c-thaler

Occasional Contributor

2 years ago

I was able to reproduce the hang up on a Windows system.
In contrast to the Linux system I was able to stop the process with Ctrl-C leading to the following output:

vish lost connection to vsim process.

                                                                                                                                                                                    
simulator: Unexpected EOF on RPC channel sock0000000007A71780
Kernel lost connection to front end process.
** Fatal: Exiting VSIM license process.

TingJiangT_Intel
Contributor
2 years ago
As it worked fine after you changed the project, the error may related to the project and testbench. It may be that your project got stuck during simulation, causing excessive CPU resource usage.
You may check your testbench to see if there are any operations that cause an infinite loop.
- c-thaler
  Occasional Contributor
  2 years ago
  Thanks for the reply.
  
  Questa is able to detect zero delay loops, so I should see an error about this. Furthermore, most of the time the tests run just fine. I have to run a single test a few hundred times to cause this behavior. That means it is nondeterministic behavior. If there is an infinite zero delay loop I should see the hang up in every simulation run.
  Besides that, I should be able to break/pause simulation in such a loop, which does not work either.
- c-thaler
  Occasional Contributor
  2 years ago
  I was able to reproduce with the other project, too. It took a few thousand simulation runs over the weekend. But finally it got stuck with 100% CPU load.
TingJiangT_Intel
Contributor
2 years ago
I see, can you upload your project and tell me your detail operation steps to help me reproduce the issue?
- c-thaler
  Occasional Contributor
  2 years ago
  The whole project is quite huge. I will try to reproduce the issue with a smaller example, e.g. a module test. This may take a few days.
TingJiangT_Intel
Contributor
2 years ago
Ok, really appreciate for this.
- c-thaler
  Occasional Contributor
  2 years ago
  I'm finished with the smaller example. How can I upload the shipment without making it available to the public?
TingJiangT_Intel
Contributor
2 years ago
Hi there, I have sent you an email, you can upload your project via it.
TingJiangT_Intel
Contributor
2 years ago
Hi there, I noticed that your email has been blocked. Seems it contains some files that is not allowed.
Please check the attachments. You can upload qar file after archived.
- c-thaler
  Occasional Contributor
  2 years ago
  Thanks for he response. I'll try it again.
  
  Unfortunately, I cannot build an qar file since it is not an Quartus project. All Questa call are made from a Shell Script. The mail was probably blocked because of this script, since it is executable.
TingJiangT_Intel
Contributor
2 years ago
Hi there, is there any updates on this issue?
- c-thaler
  Occasional Contributor
  2 years ago
  Hi,
  I've sent you another mail on 07-19 with a reproducible example attached. Was it blocked again?
  - c-thaler
    Occasional Contributor
    2 years ago
    I've sent you another mail with a link to our file exchange server, where I've just uploaded the reproducible example.
TingJiangT_Intel
Contributor
2 years ago
Hi there, I checked your script. It seems that you are not using the officially generated scripts. Using non-official scripts can lead to unpredictable behavior during simulation. It is recommended to use the simulation scripts generated by Quartus. For more details, please refer to the following link:
https://www.intel.com/content/www/us/en/docs/programmable/730191/24-1/generating-a-msim-setup-tcl-simulation.html
- c-thaler
  Occasional Contributor
  2 years ago
  In my case, there are no generated scripts since I'm not using any Intel IP in the simulation. It is just a testbench for a simple module.
TingJiangT_Intel
Contributor
2 years ago
Yes, but seems you write many scripts yourself, we can't make sure if there is any issue related to these flow. With the generated script it will ensure all the necessary file is included and the flow was verified.
- c-thaler
  Occasional Contributor
  2 years ago
  Well, the script and the flow work 99.9% of the time. But every now and then I see a hang up of the simulator, without doing any changes to the VHDL files or the flow at all.
  How can this behavior be caused by missing files or a bad flow? I mean, simulation is deterministic. I'd expect the same outcome on every run.
  - c-thaler
    Occasional Contributor
    2 years ago
    I'm now trying to create a Quartus project to run the simulation.
    Analysis and synthesis works fine. But it seems that Quartus does not respect the correct compile order for the files when generating the Questa "*_run_msim_rtl_vhdl.do" script. The script tries to compile the architecture of a module before its entity which results in a compiler error.
    (I moved the entity compilation call manually before the architecture compilation and it worked then.)
    The file order in the QSF file is correct and the same as in "Settings -> Files". How can I force Quartus to keep this order when generating the Questa DO script?

Forum Discussion

Questa simulation hangs (nondeterministic)

37 Replies

Recent Discussions

EDA_MAINTAIN_DESIGN_HIERARCHY obsolete?

Known issues in Quartus Standard 18.0

Setup slack violations?

I do not get an eMail with the generated license file

serv_req_info