Inconsistent HW behavior

--- Quote Start ---

originally posted by sulrich@Mar 22 2006, 11:12 AM

hello,

i'm seeing odd/inconsistent behavior in my system that i'm developing. operations (hardware and software) that were working on the board stop working when unrelated vhdl code is changed and recompiled.

the board is a custom board with a cyclone 1c20, a custom asic, and various usb and interface ics. i am using the niosii processor and c-code to develop some calculation/ algorithms for the system. due to speed and throughput requirements, the calculations must eventually reside in hardware. once i determine the algorithm, i migrate the calculations to a hardware-based module in vhdl using some altera megawizard lpms and some custom vhdl code. the new module is then integrated into the main vhdl.

specifically, my problem is that as i tweaked the algorithm's vhdl (invert a signal, register another, etc.) i started seeing weird behavior in the rest of my system. for example, the nios processor initializes other chips on the board (interface, asic). functions, chips, etc. that were functioning beforehand stop working. most of these functions are not related to the vhdl code being developed. i have found that i can comment out a line of c-code or change another line in the vhdl and some functions will start functioning again.f

This problem is maddening and has halted development. Has anyone seen this behavior before? [/b]

I have seen something vaguely similar on a previous product several years ago using an Altera 20K100 part and strictly VHDL code (no NIOS). I resolved the problem by loading the design into Altera OEM versions of Leonardo Spectrum and FPGA Express which evidently had better error checking routines within the compiler. Once the syntax errors were identified (not seen in Quartus), I made the changes to the design and recompiled the design in Quartus and everything worked as designed.

I recently upgraded to Q5.1, SOPC, and IDE (and had using Q5.0 seeing the same problem)and have been seeing inconsistent behavior in my system.

I would seek help from Altera Support and may eventually do that; however, without a specific indentifiable and reproducable problem that is not specific to my hardware, I doubt if they would be able to help.

Any suggestions greatly appreciated.

thanks.

<div align='right'><{post_snapback}> (index.php?act=findpost&pid=13689)

--- Quote End ---

[/b]

--- Quote End ---

Hi Sulrich,

I&#39;m not sure what may be happening in your system, but is the VHDL code you are modifying an SOPC Builder component or some VHDL code that is outside the system? If the VHDL code is part of an SOPC Builder component be sure you are modifying the SOPC builder component source files in the component dir&#39;s HDL directory NOT the "copy" of the source files in the Quartus project directory.

Also since you are using a NIOS processor in your design, are you aware of this critical issues patch that inpacts all IP cores (including the Nios processor?)

http://www.altera.com/support/software/dow...ical-issue.html (http://www.altera.com/support/software/download/service_packs/quartus/dnl-q51-critical-issue.html)

It is also fixed in service pack 1 for Quartus 5.1.

It sort of sounds like what you have been encountereing with odd functional failures.

If you still have odd functional failure do what I do, deleate the db directory and re-compile the design again. It is possible that the design database (db) got corrupted. I know in 5.x Quartus II introduced incremenal recompilation, so that the software could do partial updates of a design without rebuilding the whole design (saves compilation time).

BUT, before you start deleting the db dir, do a whole ZIP of your project directory first so that you have a snapshot of the problem. In that way you can send a "good" and "bad" version of your project to Altera Support - this will help them isolate the software problems and get a fix out in the next release. Altera support can be a great help if they have a test case showing the problem clearly so that they can file bug reports and fix the problem with a known test case. If the problem is a corrupt design database then your two zip versions of your should show a difference in the design database - it is a lot of low level debuging so it may take some time to determine the cause.

I hope this help.

Regards,

-ATJ

Hi ATJ,

Thanks for taking the time to respond. The VHDL code I am modifying is not part of a SOPC component. In my search for the problem I have modified NIOS components but have deleted the component within the NIOS, added the modified component, re-generated NIOS, and then recompiled my design. That approach should be clean.

Regarding the IP critical issue, I have seen this notice and downloaded the patch. I "upgraded" from 5.0 earlier this week because of a critical issue seen with altsyncram LPMs. My design uses several comonents with altsyncram LPMs and even though I&#39;m not using them in the way described in THAT critical issue, I decided to update to 5.1 plus SP1.

Regarding the db directory, I forgot about this approach. I have used it on several occasions, kind of a "Hail Mary" approach. http://forum.niosforum.com/work2/style_emoticons/<#EMO_DIR#>/tongue.gif I have not tried it here but may resort to that. I don&#39;t believe I&#39;m doing incremental compilation but should check my project settings just to be sure.

I will certainly back my project up before doing anything drastic. As you re-iterated, the difficulty in getting support is demonstrating the problem without having MY hardware present.

I did have a bit of good news last night. After fighting this problem for over 10 days I saw a glimmer of light just before I shut down for the evening. I noticed that some of an external chip&#39;s registers which were being read back after initialization were not correct. I know I had checked this before and everything was OK (there&#39;s that inconsistent behavior again). My suspicion is that my modifications, no matter how small, are affecting the P&R and changing overall timing. Perhaps everything "appears" to be executing fine but in actuality there&#39;s a slight timing change which violates the external parts&#39; timing constraints. My plan is to:

1) look at prop delays, timing, etc. in my compilation report

2) look into Timing constrained compilation

3) look into Logic Lock (I believe?) to freeze sections of logic which are working.... assuming I can get back to "working". http://forum.niosforum.com/work2/style_emoticons/<#EMO_DIR#>/laugh.gif http://forum.niosforum.com/work2/style_emoticons/<#EMO_DIR#>/laugh.gif http://forum.niosforum.com/work2/style_emoticons/<#EMO_DIR#>/laugh.gif

Once again, thanks for you insight. I&#39;ll post a response when (there will be a when) I resolve this problem.

thanks,

sulrich

--- Quote Start ---

originally posted by atjung@Mar 22 2006, 06:30 PM

hi sulrich,

i&#39;m not sure what may be happening in your system, but is the vhdl code you are modifying an sopc builder component or some vhdl code that is outside the system? if the vhdl code is part of an sopc builder component be sure you are modifying the sopc builder component source files in the component dir&#39;s hdl directory not the "copy" of the source files in the quartus project directory.

also since you are using a nios processor in your design, are you aware of this critical issues patch that inpacts all ip cores (including the nios processor?)

http://www.altera.com/support/software/dow...ical-issue.html (http://www.altera.com/support/software/download/service_packs/quartus/dnl-q51-critical-issue.html)

it is also fixed in service pack 1 for quartus 5.1.

it sort of sounds like what you have been encountereing with odd functional failures.

if you still have odd functional failure do what i do, deleate the db directory and re-compile the design again. it is possible that the design database (db) got corrupted. i know in 5.x quartus ii introduced incremenal recompilation, so that the software could do partial updates of a design without rebuilding the whole design (saves compilation time).

but, before you start deleting the db dir, do a whole zip of your project directory first so that you have a snapshot of the problem. in that way you can send a "good" and "bad" version of your project to altera support - this will help them isolate the software problems and get a fix out in the next release. altera support can be a great help if they have a test case showing the problem clearly so that they can file bug reports and fix the problem with a known test case. if the problem is a corrupt design database then your two zip versions of your should show a difference in the design database - it is a lot of low level debuging so it may take some time to determine the cause.

i hope this help.

regards,

-atj

<div align='right'><{post_snapback}> (index.php?act=findpost&pid=13703)

--- quote end ---

--- Quote End ---

Hello,

I fear that your problem is due to some asynchronous signal in your design. So your behaviour is very fitting dependant ( a fit may work fine while a second fit without changing any line code may crash !).

I experimented such erratic phenomena a few year ago (when i started FPGA design).

Some things I learned (golden rules now for me) :

1) always make your design full synchronous . It is not always possible but in 99% it IS !

2) ensure that you treated metastability on asynchronous input signals.

Hope this will help you.

http://forum.niosforum.com/work2/style_emoticons/<#EMO_DIR#>/wink.gif

--- Quote Start ---

originally posted by sulrich@Mar 22 2006, 01:12 PM

hello,

i&#39;m seeing odd/inconsistent behavior in my system that i&#39;m developing. operations (hardware and software) that were working on the board stop working when unrelated vhdl code is changed and recompiled.

the board is a custom board with a cyclone 1c20, a custom asic, and various usb and interface ics. i am using the niosii processor and c-code to develop some calculation/ algorithms for the system. due to speed and throughput requirements, the calculations must eventually reside in hardware. once i determine the algorithm, i migrate the calculations to a hardware-based module in vhdl using some altera megawizard lpms and some custom vhdl code. the new module is then integrated into the main vhdl.

specifically, my problem is that as i tweaked the algorithm&#39;s vhdl (invert a signal, register another, etc.) i started seeing weird behavior in the rest of my system. for example, the nios processor initializes other chips on the board (interface, asic). functions, chips, etc. that were functioning beforehand stop working. most of these functions are not related to the vhdl code being developed. i have found that i can comment out a line of c-code or change another line in the vhdl and some functions will start functioning again.f

This problem is maddening and has halted development. Has anyone seen this behavior before? [/b]

I have seen something vaguely similar on a previous product several years ago using an Altera 20K100 part and strictly VHDL code (no NIOS). I resolved the problem by loading the design into Altera OEM versions of Leonardo Spectrum and FPGA Express which evidently had better error checking routines within the compiler. Once the syntax errors were identified (not seen in Quartus), I made the changes to the design and recompiled the design in Quartus and everything worked as designed.

I recently upgraded to Q5.1, SOPC, and IDE (and had using Q5.0 seeing the same problem)and have been seeing inconsistent behavior in my system.

I would seek help from Altera Support and may eventually do that; however, without a specific indentifiable and reproducable problem that is not specific to my hardware, I doubt if they would be able to help.

Any suggestions greatly appreciated.

thanks.

<div align='right'><{post_snapback}> (index.php?act=findpost&pid=13689)

--- Quote End ---

[/b]

--- Quote End ---

Hey bigboss25,

Thanks for your input. I, too, have been burned with using async signals in an synchronous (FPGA) design. As a matter of fact, early in this project&#39;s design, I re-used some simple, "known working" code, and it behaved erratically. After an evening of thought I realized I was being thumped on the nose by this "async signal in a synchrous land" problem. http://forum.niosforum.com/work2/style_emoticons/<#EMO_DIR#>/biggrin.gif

The key players in my present problem are: 1)NiosII processor, 2)ASIC chip, 3)VHDL code consisting of a pipelined calculation (multipliers, adders) and 4)VHDL state machine sequencing/feeding data from ASIC to the calculator. The# 1 Nios and# 4 FSM both must talk to# 2 ASIC although never at the same time. The Nios "engages" the hardware calculation mode and sequencer and does not intervene until it wants to stop the flow of data. My problem is that the ASIC is behaving oddly. It appears that it is getting errant initialization data. I&#39;m in the process of trying to add timing constraints to the compilation to see if that improves things. In reference to your suggestions, the Nios and the FSM are synchronous with no async signals involved, at least no obvious ones. There is certainly the possibility that I&#39;ve glanced over one/some.

The power of the abstraction that VHDL gives the designer is wonderful and it is very very easy to forget good engineering practices. I certainly don&#39;t miss doing Karnaugh maps; however, a working knowledge of the "old school" rules (e.g., async/sync) is still important.

Still scratching my head. I&#39;ll keep everyone posted. thanks for posting!

steve

--- Quote Start ---

originally posted by bigboss25@Mar 24 2006, 06:48 AM

hello,

i fear that your problem is due to some asynchronous signal in your design. so your behaviour is very fitting dependant ( a fit may work fine while a second fit without changing any line code may crash !).

i experimented such erratic phenomena a few year ago (when i started fpga design).

some things i learned (golden rules now for me) :

1) always make your design full synchronous . it is not always possible but in 99% it is !

2) ensure that you treated metastability on asynchronous input signals.

hope this will help you.

http://forum.niosforum.com/work2/style_emoticons/<#emo_dir#>/wink.gif

<div align='right'><{post_snapback}> (index.php?act=findpost&pid=13756)

--- quote end ---

--- Quote End ---