Forum Discussion
Cyclone VGT Dev Kit boards - some new boards failing to boot from NOR Flash
We've been using these boards for years, having certified them for use in one of our products.
In the last few months we have now received 5 of these boards and they fail to configure from NOR flash. These are all the new Rev B CVGT Dev Kit edition.
Not all of the new RevB fail, but the fail rate is high, getting close to 50%. Yes, we know they changed to a Micron NOR flash for Rev B and rerouted some data lines, we are using the new RevB MAX5 files and have updated the Cyclone V NOR flash pins to match as well.
I made some diagnostic changes to the MAX5 boot source (I set the PGM leds to count retries) and discovered that on the bad boards, the boot process goes through multiple configuration retries and eventually the watchdog timer fires, turns on the ERR (D5) red LED and stops.
With the factory image they come with, there are also dozens of retries, then sometimes the boards fail, sometimes they boot up.
With the slow speed that the NOR flash configuration runs at, there is no reason that it should ever fail and retry, and indeed on good boards they configure the first time every time with no retries.
The first four we were able to send back to Altera (Via digikey where we bought them).
We just got another bad one yesterday, this one from Mouser.
Has anyone else seen this issue, and/or heard from Altera about this?
Board link:
25 Replies
- GordWait
Occasional Contributor
That's rather disappointing to hear, actually. The Max V design is Altera's design. One of the boards fails to boot with multiple retries even with the factory binary.
If I remove the factory_user input inversion (a simple NOT statement) then the factory Max V design fails on 5 out of say about 20 boards, since Rev B boards came out. Why? I am not convinced that these new boards are identical to the working ones, sorry.
I suspect the "pass/fail" OQC for these boards includes checking that they configure without error, but under the hood, some of them are failing and retrying configuration multiple times. At the performance of the parts on board, a single configuration failure is a board failure. I suggest the factory bootloader be reconfigured to fail at one single retry, then you will notice the board issues before shipping them out.
This Max V design is Altera's loaded up to 99% utilization. Not my doing.
In short, I have a new build that "works" with the new boards, but no evidence as to why. None of the timing reports on a "failing" design are showing up with errors, they are just mildly different from the "working" design.
I am not pleased with this outcome at all, sorry to say.
- LeonardoG
New Contributor
Hello,
We are seeing the exact same issue on our Cyclone V GT Dev Kit boards. We run our own custom FPGA firmware, which is known-good and has been in use for quite some time, and we have already adapted to the rev. A to rev. B flash changes, so that is not a factor.
Over the last few weeks, several brand-new Dev Kit boards have started behaving exactly as GordWait described: the board retries configuration multiple times, with nStatus and nConfig toggling, until it hits the timeout. This happens with both our custom firmware and the factory image. It was intermittent at first and has gotten progressively worse, and we also noticed a dependence on board temperature, which could be relevant for a timing issue.
The failing board we have on hands now are the 5CPCIE00100214 and 5CPCIE00100263, both with the QS3861. We also have an older board that works 100% of the time (5CPCIE00100041) which uses the QS3VH800.
As OP suggested we briefly tried changing the MAX 5 design to improve timing and the situation improved greatly. Still, this is a disappointing situation with a lot of time lost chasing ghosts.
Thanks.
- FakhrulA_altera
Regular Contributor
Hi Leonardo,
Thank you for sharing your findings.
As this thread is focused on GordWait's investigation, could you please open a new thread for your boards? This will help us track your case separately and avoid mixing debug information from different systems.
Please include the board serial numbers, revision, U29 component, and observed behavior in the new post.
Thank you for your understanding.
- LeonardoG
New Contributor
Hi Fakhrul,
I created a new post: Cyclone V GT Dev Kit - new units fail to boot from FPP | Altera Community - 355525
- GordWait
Occasional Contributor
We just received 5 new boards from Terasic.
Terminology:
"Rev B good boards" - these are Rev B, and work fine with our previous default release that has been stable since the start of the rev B release.
"New marginal boards" - these only work with the "timing improved" rebuild of the Altera Max V bootloader code, but do not work with our previous default release.
"Bad new boards" - these do not work with any of our custom Max V bootloaders, any edition. They also either do not work at all with the factory Max V binary, or will boot up eventually, just making it under the timeout limit of the factory bootloader (which seems to be about 6 seconds). These boards (when investigated) run dozens of "fail then retry" cycles.
Of the five new ones we ordered from Terasic, four are the same as the "Rev B good boards".
One is a "Bad new board" and will not boot even with the factory installed firmware.
We will request an RMA/replacement for this one, and will do so for any more that we receive in the "Bad new board" state.
For anyone receiving these boards, first run the quartus programmer and examine the Max V part (5M2210Z) to see what the checksum is. The factory max5.pof seems to be: Checksum 008D0C68, Usercode 008DOC49.
See how long it takes to boot out of the box. If it's not instantaneous (say well under 100 ms?) then it's probably ok. If it takes seconds to boot, or turns on the red ERROR LED, I'd send it back for replacement.
This is extremely frustrating and tedious. - GordWait
Occasional Contributor
One more update. I spent the last few days creating a very dumb basic bootloader, so that I could: 1. simulate the RTL code in Questasim, and 2. Sample internal nets of the same code using the ISSP function. I ran it on the "very bad" boards I have and on one "mostly bad" board (it fails 1/5 even with my new slightly better Max5 bootloader). I also tried it on fully good known boards. What I found is that the bad boards CVGT FPGA are triggering a "Framing/CRC" error on the bit stream, at RANDOM offsets in the NOR flash memory. IE every reset of the Max5 CPLD fails but fails at a different address in NOR flash. I also had ISSP capture some of the NOR data bytes that the CPLD was about to send to the FPGA config port, around a failure event. The data seemed stable, although not enough samples were captured to guarantee this statement. In short, there is something wrong with the hardware on the bad boards. Note also this dumb basic bootloader runs at 25 mhz, lots of time for the NOR flash output to settle. Altera continues to send out a mix of good/notsogood/bad boards at this point. We will be sending back any that fail normal bootloading for refund/replacement.
- FakhrulA_altera
Regular Contributor
Hi GordWait,
Thank you for the additional findings and the serial numbers.
We would like to clarify a couple of points to ensure we correctly understand the latest results.
In an earlier update, you mentioned that the factory max5.pof was able to boot the marginal boards, whereas your custom build did not. In the latest update, you noted that some boards do not boot reliably even with the factory image.
Could you please help confirm:
For SN 5CPCIE00100119, 5CPCIE00100129, and 5CPCIE00100262, what is the behavior when using the unmodified factory max5.pof?
- Always boots
- Boots intermittently
- Never boots
For each of the above boards, could you share the MAX V timing report from the corresponding build that was programmed onto the board?
- In particular, the recovery/setup timing around the critical clk_config path that you identified earlier.
This will help us determine whether the latest observations are still consistent with the timing-margin hypothesis or indicate a different issue.
Thanks.
- GordWait
Occasional Contributor
Using the Altera unmodified factory max5.pof :
The two "Very Bad" boards never boot.
The mostly bad board boots sometimes.Since the errors are random, and the Altera factory image is smallish, AND the factory max5 has retries enabled, the mostly bad boards can sometimes boot the Altera firmware after several retries. This is of course a complete failure. If Altera's OQC is just - did the board eventually boot? - then the marginal boards can pass even though they have real hardware issues.
The two very bad boards fail for us completely. It is hard to understand how they passed OQC at Altera.
For the timing, I do not have the timing report for the Altera factory max5.pof as someone at Altera compiled it.
BUT since I was able to get some of the new "marginal boards" (not these three bad ones) to boot with our modified standard bootloader (based on the factory build) by improving the timing, AND these new "marginal boards" also boot with my "from scratch" bootloader that runs very conservatively (not based on the Altera PFL core), it does seem like some sort of time critical issue on the part of the latest batches of the boards from Altera.
- GordWait
Occasional Contributor
I too have discovered that the Factory Max5 bootloader source code has inadequate or missing timing constraints.
Note now we have two useless CVGT dev kits on hand, and we still need RMAs for them. We are about to order 7 more boards - we only need 5 working ones but we assume 2 out of 7 will fail in the same way.
This is a lot of cash we have tied up in bad boards.
Here are the details of my timing investigations of the factory build:## Summary list of unconstrained I/O pads (as-shipped SDC)
There are two distinct categories of "unconstrained" here, and the difference matters: one is caught by Quartus's own built-in `report_ucp` (Report Unconstrained Paths) command, the other is invisible to it entirely.
**Category 1 — truly and completely unconstrained.** No `set_input_delay`/`set_output_delay`/`set_false_path`/max-skew assignment of any kind exists for these ports. This is exactly what `report_ucp` is designed to catch, and does catch, against the as-shipped SDC:
Pad | Direction | Function | Quartus's own flag |
| `factory_user` | Input | Physical 2.5 V DIP switch selecting Factory vs. User image at power-up; feeds directly into the `p2` megafunction's own page-select logic (`pfl_pagesel_int[0]`) and `image_sel2` | `report_ucp`: "No input delay, min/max delays, false-path exceptions, or max skew assignments found" |
| `max_as_conf` | Output | Driven by `pfl_control`'s internal `msel_reset_n` | `report_ucp`: "No output delay, min/max delays, false-path exceptions, or max skew assignments found" |
`factory_user` is a slow, manually-operated switch, so the missing constraint is plausibly benign in practice — but nothing in the SDC records that as a deliberate decision (contrast with `factory_load`, `pgm_sel`, `cpu_resetn`, and several other genuinely-async inputs elsewhere in the same SDC, which *do* have explicit `set_false_path` entries marking them as intentionally excluded). `max_as_conf` has no such treatment either way.
**Category 2 — nominally constrained, but the constraint is dead.** These ports *do* have a `set_output_delay` assignment in the SDC, so Quartus considers them "constrained" and **`report_ucp` does not flag them at all** — but the assignment references the wrong clock and is silently voided by the `clk_config`↔`clkin_50` false-path exception described above. This category is the one worth emphasizing: it is invisible to the standard unconstrained-paths audit, and would only surface by manually checking that every `-clock` argument matches the signal's actual driving domain.
| Pad | Direction | Nominal constraint | Why it's actually dead |
| `flash_oen` | Output | `set_output_delay -clock clkin_50 ...` | Driven by `clk_config`; false-pathed against `clkin_50` |
| `flash_wen` | Output | `set_output_delay -clock clkin_50 ...` | Same |
| `flash_cen` | Output | `set_output_delay -clock clkin_50 ... {flash_cen[*]}` | Same wrong-clock bug, **plus** the bracket never matches (`flash_cen` is a scalar port) — doubly unconstrained |
| `flash_resetn` | Output | `set_output_delay -clock clkin_50 ...` | Same wrong-clock bug (a separate, deliberate `set_false_path -to flash_resetn` also exists, so this one is arguably intentionally excluded from setup/hold checking anyway — flagged here for completeness) |
| `fpga_config_d[0]` … `fpga_config_d[7]` (8 pins) | Output | `set_output_delay -clock clkin_50 ... {fpga_config_d[*]}` | Same wrong-clock bug |
That's **14 physical pads total** with no effective timing verification in the as-shipped design: 2 in Category 1 (`factory_user`, `max_as_conf`) and 12 in Category 2 (`flash_oen`, `flash_wen`, `flash_cen`, `flash_resetn`, and the 8 bits of `fpga_config_d`). Of those, `flash_resetn` is arguably intentionally exempt (it has its own deliberate `set_false_path`, separate from the wrong-clock bug) — so 11 pads have a genuinely missing, non-deliberate setup/hold check. The `flash_oen`/`fpga_config_d[*]` group is the one directly implicated in the real, measured Slow-model setup violation reported below.
- FakhrulA_altera
Regular Contributor
Hi,
Thanks for the detailed information and for outlining the debug steps taken.
From our side, there are known changes in the RevB board (including NOR flash device and routing), and there is a related knowledge base article that may be relevant to what you are seeing:
- Why can’t I access the EPCQ flash device on the Cyclone V GT FPGA Development Kit (DK-DEV-5CGTD9N-B)?
https://community.altera.com/kb/knowledge-base/why-cant-i-access-the-epcq-flash-device-on-the-cyclone®-v-gt-fpga-development-ki/344352
Before concluding this is the same root cause, could you help confirm a few points:
- Are all failing boards confirmed to be RevB, and using the same NOR flash device (Micron)?
- Does the issue persist even with the factory image without any modification?
- Are you able to consistently program and verify the NOR flash contents via JTAG on failing boards?
- Is the retry behavior reproducible across power cycles, or intermittent?
This will help us determine if your case aligns with the known issue or if something else is contributing.
Looking forward to your feedback.
Regards,
Fakhrul- GordWait
Occasional Contributor
Hi, thanks for your comments.
Some answers
We don't use the EPCQ flash, just the NOR flash.
These are all Rev B with Micron Flash
Yes even the factory image is failing and retrying. On a good board the factory image boot is almost instant. On a bad board it takes seconds to "succeed" but sometimes fails. I ran tests with a modified bootloader and found that it is retrying dozens of times.
Programming the NOR Flash via JTAG tools works with no errors, including the CRC readback tests, which verifies the data path from the flash to the MaxV pins is ok.
The error is definitely repeatable across power cycles, and repeatable via pressing the config button.
One new weird data point - if we load the factory binary maxv.pof image, (and have to switch the factory/user button to user) then 3 of the boards seem to boot ok with our image. The other 2 are still bad.
I looked at the source, for some reason Altera inverted the factory_user switch signal in the v22 source code.
BUT if I compile that same factory image from the V22 source, my version of it fails to work. I tried using the exact same Quartus edition - no better.
All the compilations I make work fine on the good Rev B boards we have. Very mysterious. - FakhrulA_altera
Regular Contributor
Hi,
Thanks for the detailed update, this is very helpful.
Based on your findings, a few observations:
- Since JTAG programming and CRC verification are clean, the data path from NOR flash to MAX V appears valid.
- The repeated configuration retries (especially dozens of attempts) are not expected and indicate instability during configuration.
- The difference between the factory .pof and self-compiled version may point to differences in fitter/timing results.
There is a related KDB which may be relevant to RevB boards with updated device changes:
https://community.altera.com/kb/knowledge-base/why-cant-i-access-the-epcq-flash-device-on-the-cyclone®-v-gt-fpga-development-ki/344352Although it refers to EPCQ, it highlights a known incompatibility on RevB (U29), which may also affect configuration robustness.
To help narrow this down further, could you also share the following:
- For the 3 boards that work with the factory .pof, do they complete configuration in a single attempt or still show retries?
- For the 2 boards that still fail, is the retry behavior identical across all images?
- Have you compared settings between your compiled .pof and the factory .pof (e.g. compression, configuration options)?
- Could you share the serial numbers of the affected boards? There may be a specific affected range for certain revisions.
- For the silk screen difference, if possible, please share board markings or photos for comparison.
This will help us check if your boards fall within any known affected batch or if this is a new variation.
Thanks again for the detailed debugging effort.
Regards,
Fakhrul
- Why can’t I access the EPCQ flash device on the Cyclone V GT FPGA Development Kit (DK-DEV-5CGTD9N-B)?
- GordWait
Occasional Contributor
Some more updates and summary info:
Update 11 June 2026 at 1:32 PM- We have 5 CVGT Dev Kit boards that will not boot the FPGA with our released production firmware. We do have Rev B boards that work fine with our firmware, so it is not all boards. Our production released firmware has been working for the old Rev A boards, and the new Rev B boards except for these bad 5.
- I have diagnosed that the bad boards retry the FPGA configuration multiple times, then ultimately fail, as a watchdog timer in the MaxV design times out and turns on the ERR led on the CVGT board. On a rare occasion a couple of the boards may actually manage to fully boot, but then the next power cycle it will fail again.
- For our production, we make a minor edit to the factory source - setting the final image to a read only version, and we use a write enabled version of that in our programming sequence:
- Load the read/write edition MaxV image
- Program the FPGA using Quartus tools
- Load the read only MaxV image.
- Three of the “bad” boards - if we load the factory max5.pof from Altera and switch the Factory Image over to the “User” setting, then they boot up and appear to be fine. The other two bad boards still fail in this mode.
- If I just recompile the Factory image - as is - using Quartus 23.1 Std 1, that compiled image does not work with ANY of the 5 bad boards. It does work with known good boards.
- Some of the bad boards fail to boot randomly when loaded with the “restore to factory” firmware, but they all seem to retry many times before they pass or fail. Retrying just once is an indication that something is wrong with the board.
- We notice one of the new bad boards has a different silk screen (it does not say UK/CA on it). Why are there two variants of the Rev B dev kit? Not sure if this is relevant but it seems odd.
Questions:
- Why did Altera invert the Factory/User switch state in the V22 CVGT Dev Kit Release?
- Why does my compiled version of the V22 MaxV device act differently (behave worse) than the binary max5.pof that came with the V22 edition source code? It seems like the source code is not able to build the same design as the Altera binary.
- Is anyone else having issues with some of the CVGT Dev Kits booting from NOR flash?
- GordWait
Occasional Contributor
Hi Fakhrul, I looked into your questions.
I have 4 of the bad boards in hand, so I can verify the following serial numbers and the U29 part. All four have the same part that the schematic calls for:
5CPCIE 00100129 - has IDT QS3861
5CPCIE 00100142 - has IDT QS3861
5CPCIE 00100147 - has IDT QS3861
5CPCIE 00100148 - has IDT QS3861An older Rev B I have here is a good board, it has an IDT Q53VH8COQG device on board.
You mention : " The difference between the factory .pof and self-compiled version may point to differences in fitter/timing results." - since I am using the same provided source files, and the same Quartus version, should it not create the exact same max5.pof binary?
More answers:
- For the 3 boards that work with the factory .pof, do they complete configuration in a single attempt or still show retries? - it seems like the configure in a single attempt, based only on the boot up speed.
- For the 2 boards that still fail, is the retry behavior identical across all images? - yes, random N number of retries until the ERR led turns on.
- Have you compared settings between your compiled .pof and the factory .pof (e.g. compression, configuration options)? - I used the exact factory source files with the factory QSF and constraints files, I get a different checksum, and my compiled factory image does not work at all.
- Could you share the serial numbers of the affected boards? There may be a specific affected range for certain revisions. - yes, see above. (missing one, I don't have it at the moment)
- For the silk screen difference, if possible, please share board markings or photos for comparison.
For my older good working board, there is a silk screen label UK/CA that is in the open to the "East" of the USB jack. For the new bad boards, the UK/CA label is there partly covered by the FPGA fan, further "East".
This will help us check if your boards fall within any known affected batch or if this is a new variation.
- FakhrulA_altera
Regular Contributor
Hi GordWait,
Thanks for the detailed follow-up and the serial number information, this helps a lot.
A couple of clarifications from our side:
- The serial numbers you shared (5CPCIE00100129 and above) are outside the known affected range documented in the KDB (5CPCIE00100011 to 5CPCIE00100115).
- Based on our internal data, the impacted component is expected to be QS3VH800QG, which can affect the AS configuration path.
Regarding the U29 marking you mentioned (Q53VH8COQG), we suspect this may be a marking interpretation issue. The assembled part for this location is QS3VH800QG, and depending on the top marking, it can sometimes be misread (e.g. “QS3VH800” appearing similar to “Q53VH8CO”).
Could you help double confirm the marking again (or share a close-up photo if convenient)? This will help ensure we are referencing the exact component correctly.
At the same time, there is still a mismatch between:
- Expected impacted component vs what you’re seeing behaving as “good”
- Your boards being outside the known affected SN range
So your case may not fully align with the documented issue. We are checking this internally and will update once we have more clarity.
Thanks again for the very thorough debug work.
Regards,
Fakhrul- GordWait
Occasional Contributor
I was finally able to get the factory source to compile and create the exact same binary as the factory binary (max5.pof). (I used Quartus 18.1 and did not allow it to update the library parts. From the files it seems like Quartus 15.1 was used to build the factory max5.pof found in the v22 package. )
Here is our updated S/N list with notes:
5CPCIE 00100129 - has QS3861 - will boot sometimes with factory binary max5.pof (Digikey)
5CPCIE 00100142 - has QS3861 - will boot with factory binary max5.pof. (Digikey)
5CPCIE 00100147 - has QS3861 - will boot with factory binary max5.pof. (Digikey)
5CPCIE 00100148 - has QS3861 - will not boot at all. (Mouser)
5CPCIE 00100134 - not checked yet - will boot with factory binary max5.pof. (Digikey)
5CPCIE 00100144 - not checked yet - will boot with factory binary max5.pof. (Mouser)That binary works with both known good boards and our (4) "marginal" boards. The marginal boards only work with the factory binary max5.pof but not our released production max5.pof with our own edits).
I then just removed the factory_user inversion and restored it to the way it was in previous releases. This build did NOT work with the 3 marginal boards but does work with the good boards.I noticed that the max5 design is 99% full. Any minor change affects timing:
Critical Difference: Slow Model Recovery on `clk_config`
| Check | works | inverted switch | delta |
| Slow Model Recovery `clk_config` (worst path) | **0.361 ns** | **0.122 ns** | −0.239 ns |
| Slow Model Setup `clk_config` | 0.675 ns | 0.973 ns | (inverted switch better) |
| Fast Model Hold `clk_config` | 0.280 ns | 0.280 ns | 0 |
I was able to restore the timing and finally build a working version. Since we don't use the FPGA access port or the I2C controller in the Max5, I removed those to reduce the logic in the Max5, and now my build also works.
Something on the 4 marginal boards is slightly different, so this minor timing change causes them to fail to boot properly.
I suspect the max5.pof timing has always been very close to failing, and something has now changed that makes our production Ikomed max5.pof fail all the time.
Any idea if Altera moved the MaxV CPLD to a different silicon fab line, or perhaps Micron did with the NOR flash?
My next goal is to understand the flash reset timing in more detail and perhaps adjust our Ikomed copy of the max5 design to have far better margin. A timing shift of 0.239 ns is very small.
- For the 3 boards that work with the factory .pof, do they complete configuration in a single attempt or still show retries? - it seems like the configure in a single attempt, based only on the boot up speed.
- GordWait
Occasional Contributor
Here is the photo of the different U29 from my (dusty but working fine) engineering test CVGT:
- GordWait
Occasional Contributor
Our 3 known bad boards are SN:
5CPCIE00100119
5CPCIE00100129
5CPCIE00100262