Forum Discussion

GordWait's avatar
GordWait
Icon for Occasional Contributor rankOccasional Contributor
2 months ago

Cyclone VGT Dev Kit boards - some new boards failing to boot from NOR Flash

We've been using these boards for years, having certified them for use in one of our products.
In the last few months we have now received 5 of these boards and they fail to configure from NOR flash. These are all the new Rev B CVGT Dev Kit edition. 

Not all of the new RevB fail, but the fail rate is high, getting close to 50%. Yes, we know they changed to a Micron NOR flash for Rev B and rerouted some data lines, we are using the new RevB MAX5 files and have updated the Cyclone V NOR flash pins to match as well. 

I made some diagnostic changes to the MAX5 boot source (I set the PGM leds to count retries)  and discovered that on the bad boards, the boot process goes through multiple configuration retries and eventually the watchdog timer fires, turns on the ERR (D5) red LED and stops. 

With the factory image they come with, there are also dozens of retries, then sometimes the boards fail, sometimes they boot up. 

With the slow speed that the NOR flash configuration runs at, there is no reason that it should ever fail and retry, and indeed on good boards they configure the first time every time with no retries. 

The first four we were able to send back to Altera (Via digikey where we bought them).
We just got another bad one yesterday, this one from Mouser. 

Has anyone else seen this issue, and/or heard from Altera about this?

Board link: 

https://docs.altera.com/r/docs/792833/current/cyclone-v-gt-fpga-development-kit-user-guide/kit-features

 

25 Replies

  • GordWait's avatar
    GordWait
    Icon for Occasional Contributor rankOccasional Contributor

    Hi Fakhrul, I looked into your questions. 

    I have 4 of the bad boards in hand, so I can verify the following serial numbers and the U29 part. All four have the same part that the schematic calls for: 

    5CPCIE 00100129 - has IDT QS3861
    5CPCIE 00100142 - has IDT QS3861
    5CPCIE 00100147 - has IDT QS3861
    5CPCIE 00100148 - has IDT QS3861

    An older Rev B I have here is a good board, it has an IDT Q53VH8COQG device on board. 

    You mention : " The difference between the factory .pof and self-compiled version may point to differences in fitter/timing results." - since I am using the same provided source files, and the same Quartus version, should it not create the exact same max5.pof binary? 

    More answers:

    1. For the 3 boards that work with the factory .pof, do they complete configuration in a single attempt or still show retries? - it seems like the configure in a single attempt, based only on the boot up speed. 

    2. For the 2 boards that still fail, is the retry behavior identical across all images? - yes, random N number of retries until the ERR led turns on. 

    3. Have you compared settings between your compiled .pof and the factory .pof (e.g. compression, configuration options)?  - I used the exact factory source files with the factory QSF and constraints files, I get a different checksum, and my compiled factory image does not work at all. 

    4. Could you share the serial numbers of the affected boards? There may be a specific affected range for certain revisions. - yes, see above.  (missing one, I don't have it at the moment)

    5. For the silk screen difference, if possible, please share board markings or photos for comparison.
      For my older good working board, there is a silk screen label UK/CA that is in the open to the "East" of the USB jack. For the new bad boards, the UK/CA label is there partly covered by the FPGA fan, further "East". 

    This will help us check if your boards fall within any known affected batch or if this is a new variation.

    • FakhrulA_altera's avatar
      FakhrulA_altera
      Icon for Regular Contributor rankRegular Contributor

      Hi GordWait,

      Thanks for the detailed follow-up and the serial number information, this helps a lot.

      A couple of clarifications from our side:

      • The serial numbers you shared (5CPCIE00100129 and above) are outside the known affected range documented in the KDB (5CPCIE00100011 to 5CPCIE00100115).
      • Based on our internal data, the impacted component is expected to be QS3VH800QG, which can affect the AS configuration path.

      Regarding the U29 marking you mentioned (Q53VH8COQG), we suspect this may be a marking interpretation issue. The assembled part for this location is QS3VH800QG, and depending on the top marking, it can sometimes be misread (e.g. “QS3VH800” appearing similar to “Q53VH8CO”).

      Could you help double confirm the marking again (or share a close-up photo if convenient)? This will help ensure we are referencing the exact component correctly.

      At the same time, there is still a mismatch between:

      • Expected impacted component vs what you’re seeing behaving as “good”
      • Your boards being outside the known affected SN range

      So your case may not fully align with the documented issue. We are checking this internally and will update once we have more clarity.

      Thanks again for the very thorough debug work.

      Regards,
      Fakhrul

      • GordWait's avatar
        GordWait
        Icon for Occasional Contributor rankOccasional Contributor

        I was finally able to get the factory source to compile and create the exact same binary as the factory binary (max5.pof).  (I used Quartus 18.1 and did not allow it to update the library parts. From the files it seems like Quartus 15.1 was used to build the factory max5.pof found in the v22 package. )

        Here is our updated S/N list with notes:

        5CPCIE 00100129 - has QS3861 - will boot sometimes with factory binary max5.pof (Digikey)
        5CPCIE 00100142 - has QS3861 - will boot with factory binary max5.pof. (Digikey)
        5CPCIE 00100147 - has QS3861 - will boot with factory binary max5.pof. (Digikey)
        5CPCIE 00100148 - has QS3861 - will not boot at all. (Mouser)
        5CPCIE 00100134 - not checked yet - will boot with factory binary max5.pof. (Digikey)
        5CPCIE 00100144 - not checked yet - will boot with factory binary max5.pof. (Mouser)

        That binary works with both known good boards and our (4) "marginal" boards. The marginal boards only work with the factory binary max5.pof but not our released production max5.pof with our own edits). 

        I then just removed the factory_user inversion and restored it to the way it was in previous releases. This build did NOT work with the 3 marginal boards but does work with the good boards. 

        I noticed that the max5 design is 99% full. Any minor change affects timing:

        Critical Difference: Slow Model Recovery on `clk_config`

        | Check | works | inverted switch | delta |

        | Slow Model Recovery `clk_config` (worst path) | **0.361 ns** | **0.122 ns** | −0.239 ns |

        | Slow Model Setup `clk_config` | 0.675 ns | 0.973 ns | (inverted switch better) |

        | Fast Model Hold `clk_config` | 0.280 ns | 0.280 ns | 0 |

        I was able to restore the timing and finally build a working version. Since we don't use the FPGA access port or the I2C controller in the Max5, I removed those to reduce the logic in the Max5, and now my build also works. 

        Something on the 4  marginal boards is slightly different, so this minor timing change causes them to fail to boot properly. 

        I suspect the max5.pof timing has always been very close to failing, and something has now changed that makes our production Ikomed max5.pof fail all the time. 

        Any idea if Altera moved the MaxV CPLD to a different silicon fab line, or perhaps Micron did with the NOR flash?

        My next goal is to understand the flash reset timing in more detail and perhaps adjust our Ikomed copy of the max5 design to have far better margin. A timing shift of 0.239 ns is very small. 

  • GordWait's avatar
    GordWait
    Icon for Occasional Contributor rankOccasional Contributor

    Some more updates and summary info:
    Update 11 June 2026 at 1:32 PM

    1. We have 5 CVGT Dev Kit boards that will not boot the FPGA with our released production firmware. We do have Rev B boards that work fine with our firmware, so it is not all boards. Our production released firmware has been working for the old Rev A boards, and the new Rev B boards except for these bad 5.
    2. I have diagnosed that the bad boards retry the FPGA configuration multiple times, then ultimately fail, as a watchdog timer in the MaxV design times out and turns on the ERR led on the CVGT board. On a rare occasion a couple of the boards may actually manage to fully boot, but then the next power cycle it will fail again.
    3. For our production, we make a minor edit to the factory source - setting the final image to a read only version, and we use a write enabled version of that in our programming sequence:
      1. Load the read/write edition MaxV image
      2. Program the FPGA using Quartus tools
      3. Load the read only MaxV image.
    4. Three of the “bad” boards - if we load the factory max5.pof from Altera and switch the Factory Image over to the “User” setting, then they boot up and appear to be fine. The other two bad boards still fail in this mode. 
    5. If I just recompile the Factory image - as is - using Quartus 23.1 Std 1, that compiled image does not work with ANY of the 5 bad boards. It does work with known good boards. 
    6. Some of the bad boards fail to boot randomly when loaded with the “restore to factory” firmware, but they all seem to retry many times before they pass or fail. Retrying just once is an indication that something is wrong with the board. 
    7. We notice one of the new bad boards has a different silk screen (it does not say UK/CA on it). Why are there two variants of the Rev B dev kit? Not sure if this is relevant but it seems odd.

    Questions: 

    1. Why did Altera invert the Factory/User switch state in the V22 CVGT Dev Kit Release?
    2. Why does my compiled version of the V22 MaxV device act differently (behave worse) than the binary max5.pof that came with the V22 edition source code? It seems like the source code is not able to build the same design as the Altera binary. 
    3. Is anyone else having issues with some of the CVGT Dev Kits booting from NOR flash? 
  • FakhrulA_altera's avatar
    FakhrulA_altera
    Icon for Regular Contributor rankRegular Contributor

    Hi,

    Thanks for the detailed information and for outlining the debug steps taken.

    From our side, there are known changes in the RevB board (including NOR flash device and routing), and there is a related knowledge base article that may be relevant to what you are seeing:

    • Why can’t I access the EPCQ flash device on the Cyclone V GT FPGA Development Kit (DK-DEV-5CGTD9N-B)?
      https://community.altera.com/kb/knowledge-base/why-cant-i-access-the-epcq-flash-device-on-the-cyclone®-v-gt-fpga-development-ki/344352

    Before concluding this is the same root cause, could you help confirm a few points:

    1. Are all failing boards confirmed to be RevB, and using the same NOR flash device (Micron)?
    2. Does the issue persist even with the factory image without any modification?
    3. Are you able to consistently program and verify the NOR flash contents via JTAG on failing boards?
    4. Is the retry behavior reproducible across power cycles, or intermittent?

    This will help us determine if your case aligns with the known issue or if something else is contributing.

    Looking forward to your feedback.

    Regards,
    Fakhrul

    • FakhrulA_altera's avatar
      FakhrulA_altera
      Icon for Regular Contributor rankRegular Contributor

       

      Hi,

      Thanks for the detailed update, this is very helpful.

      Based on your findings, a few observations:

      • Since JTAG programming and CRC verification are clean, the data path from NOR flash to MAX V appears valid.
      • The repeated configuration retries (especially dozens of attempts) are not expected and indicate instability during configuration.
      • The difference between the factory .pof and self-compiled version may point to differences in fitter/timing results.

      There is a related KDB which may be relevant to RevB boards with updated device changes:
      https://community.altera.com/kb/knowledge-base/why-cant-i-access-the-epcq-flash-device-on-the-cyclone®-v-gt-fpga-development-ki/344352

      Although it refers to EPCQ, it highlights a known incompatibility on RevB (U29), which may also affect configuration robustness.

      To help narrow this down further, could you also share the following:

      1. For the 3 boards that work with the factory .pof, do they complete configuration in a single attempt or still show retries?
      2. For the 2 boards that still fail, is the retry behavior identical across all images?
      3. Have you compared settings between your compiled .pof and the factory .pof (e.g. compression, configuration options)?
      4. Could you share the serial numbers of the affected boards? There may be a specific affected range for certain revisions.
      5. For the silk screen difference, if possible, please share board markings or photos for comparison.

      This will help us check if your boards fall within any known affected batch or if this is a new variation.

      Thanks again for the detailed debugging effort.

      Regards,
      Fakhrul

    • GordWait's avatar
      GordWait
      Icon for Occasional Contributor rankOccasional Contributor

      Hi, thanks for your comments. 

      Some answers
      We don't use the EPCQ flash, just the NOR flash.
      These are all Rev B with Micron Flash
      Yes even the factory image is failing and retrying. On a good board the factory image boot is almost instant. On a bad board it takes seconds to "succeed" but sometimes fails. I ran tests with a modified bootloader and found that it is retrying dozens of times. 
      Programming the NOR Flash via JTAG tools works with no errors, including the CRC readback tests, which verifies the data path from the flash to the MaxV pins is ok. 
      The error is definitely repeatable across power cycles, and repeatable via pressing the config button. 

      One new weird data point - if we load the factory binary maxv.pof image, (and have to switch the factory/user button to user) then 3 of the boards seem to boot ok with our image. The other 2 are still bad. 
      I looked at the source, for some reason Altera inverted the factory_user switch signal in the v22 source code. 
      BUT if I compile that same factory image from the V22 source, my version of it fails to work. I tried using the exact same Quartus edition - no better. 
      All the compilations I make work fine on the good Rev B boards we have. Very mysterious.