Forum Discussion
Occasional ContributorCyclone V GT Dev Kit - new units fail to boot from FPP
We have been using this development kit internally for many years, and on our last order 2 out of 3 units fail to boot using the Fast Passive Parallel (FPP) configuration scheme through the MAX V.
We normally use our own custom MAX V bitstream, but the issue is present even with the factory MAX V image and the factory FPGA image/flash content. The MAX V appears to try loading the image (LED D6 flashing or statically lit, depending on the MAX V firmware version) for about 6 seconds before giving up and signaling an error (LED D5).
On reception, one of the two faulty boards seemed to work, it was just taking a bit longer to boot than usual (presumably it took a few retries). It then got progressively worse: the boot started failing intermittently, and now it is almost impossible to get it to boot through FPP. The boards can still boot through Active Serial (AS), and the FPGA can properly read and write the main flash, so the issue seems to affect only the MAX V flash access / FPGA configuration.
Following this thread, which reports the same issue: Cyclone VGT Dev Kit boards - some new boards failing to boot from NOR Flash | Altera Community - 353984 we tried reducing the MAX V utilization to improve timing. With this change one of the boards managed to boot a few times, but very rarely and not reliably.
The affected boards are all rev. B and have serials:
- 5CPCIE00100214
- 5CPCIE00100263
While debugging, I also noticed that the timing constraints do not follow the Parallel Flash Loader IP User Guide. For example, fpga_dclk should be a generated clock and used as the reference for the other constraints, and the set_output_delay on fpga_data does not respect the tDSU from the Cyclone V datasheet. That said, correcting these constraints did not improve the boot behavior.
We are now stuck with two boards that can only boot from AS, which is a concern for future orders. Between this and the flash revision change to rev. B (where the PCB is actually labelled rev. A and the example designs are mixed and not clearly labelled), these boards are becoming difficult to work with.
Since there is at least one other thread reporting the same behavior it looks less like an isolated fault and more like a recurring issue on recent production. Could someone from Altera confirm whether this is being tracked, and what the recommended action is for affected boards?
An update/fix if someone else encounters the same issue.
GordWait an I kept investigating the problem and sharing our findings daily, until we got all our boards working 100% of the time on hundreds of reboots. While we ended up with slightly different solutions the root cause is the same, and I will share the core of what helped me for future reference.The root of the problem is presumably some signal integrity/timing issue across the fpga_config_d/fpga_dclk lines, which seems to be present only on newer boards/chips.
The greatest improvement came from reducing the drive strength of these lines, by adding in the QSF:
set_instance_assignment -name CURRENT_STRENGTH_NEW 7MA -to fpga_dclk
set_instance_assignment -name CURRENT_STRENGTH_NEW 7MA -to fpga_config_d[*]While this solution helped on most of the affected boards it did not fix them all.
To better improve the situation the fpga_dclk can be effectively reduced to 50 MHz, with the easiest way being changing the pfl_x8_inst's "Ratio between input clock and DCLK output" to 2. By doing this the fpga_config_d changes on the fpga_dclk's falling edge, thus improving the hold slack.
To better suit this fix there are some constraints to add in the .sdc.
- fpga_dclk should be a generated one:
create_generated_clock -name {fpga_dclk} -source {clk_config} -divide_by 2 [get_ports {fpga_dclk}]- The data output delay should be based on this clock and respect the Cyclone V datasheet tsu:
set_output_delay -add_delay -max -clock [get_clocks {fpga_dclk}] 5.500 [get_ports {fpga_config_d[*]}]
set_output_delay -add_delay -min -clock [get_clocks {fpga_dclk}] -1.000 [get_ports {fpga_config_d[*]}]- If using the "Ratio" option a multicycle path should actually be defined:
set_multicycle_path -setup -start 1 -from [get_registers {*|alt_pfl_cfg_fpga:*|fpga_data_reg[*]}] -to [get_ports {fpga_config_d[*]}]
set_multicycle_path -hold -start 1 -from [get_registers {*|alt_pfl_cfg_fpga:*|fpga_data_reg[*]}] -to [get_ports {fpga_config_d[*]}]Note that these fixes don't consider the usage of the FPPovJ, for which better constrains should be designed.
Moreover, the FPP boot time is largely limited by the flash access time, so even halving the DCLK won't affect the boot time much. Even with the 50 MHz clock it still boots comfortably within our system's PCIE enumeration window.All of this was tested on a limited number of boards, so I can't assure this works 100% of the time, but until Altera finds a proper solution this can be a good starting point for anyone facing the same problem as we did.
Leonardo
13 Replies
GordWaitI just discovered the same thing on the factory design constraints. The FPGA config pins are entirely unconstrained. The Max5 part can have siginificant internal delays so this is not good..
FakhrulA_alteraRegular Contributor
Hi Leonardo,
Thank you for sharing the detailed observation.
We understand your concern, especially since the issue is seen even with the factory MAX V image and factory FPGA/flash content. Based on your description, the behavior appears to be specific to FPP configuration through the MAX V path, while AS configuration is still working.
We will need to further review this internally. Could you please help provide the following details?
1. Board revision and serial numbers of the affected and working units
2. MAX V image version used
3. Factory design package version used
4. Any observed difference in boot behavior between factory image and custom MAX V image
We will check this further and update once we have more information.
Regards,
Fakhrul
LeonardoGHi Fakhrul,
Correct, AS works as expected while the FPGA can't be programmed by the MAX V through FPP.
Here are the info you requested:
1. We have two affected boards with S/N: 5CPCIE00100214 and 5CPCIE00100263, and one working board with S/N 5CPCIE00100041. All three boards are revision B (the PCB is actually labelled rev. A, but the mismatch between the PCB rev. and the actual rev. has always been there). In the past we had other rev. B boards which were all working, but I don't have the S/N available.
2. We used both the MAX V v6 image included in the factory design "Installer Package for Windows (v22.1std)", and a MAX V v5 image edited with the correct flash pinout according to the revision change. We also used some other custom images based on both v5 and v6 with different timing constraints and/or utilization.
3. The factory design package used is the one included in "Installer Package for Windows (v22.1std)".
4. There is no significant difference in behavior between the MAX V images. With a very lightweight MAX V image to reduce utilization, one of the two affected boards actually boots once every ~50 power cycles, but I wouldn't consider it significant enough.
Thank you,
Leonardo
- FakhrulA_altera
Hi Leonardo,
Thank you for providing the detailed results.
We have identified a related case that is currently under internal review, and we are assessing whether these boards have the same failure mechanism. Since both boards fail with the factory image, AS operation and flash read/write remain functional, and timing changes did not provide a reliable recovery, the symptoms indicate a possible board-level issue or insufficient timing margin in the FPP configuration path.
At this stage, further changes to the MAX V constraints are unlikely to provide a reliable solution. If test equipment is available, it would be helpful to compare a working and affected board by capturing DCLK, nCONFIG, nSTATUS, CONF_DONE, and several FPP data signals during boot, while also checking the configuration power rails and reset sequencing. Please share the waveforms through the support case if possible. However, this debug should not delay warranty verification and RMA or repair evaluation for S/N 5CPCIE00100214 and 5CPCIE00100263.
To proceed with the RMA, please contact the distributor from whom the boards were purchased. The distributor can verify the warranty and submit an Altera Premier Support (APS) RMA case on your behalf.
We apologize for the inconvenience and appreciate your cooperation.
Regards,
Fakhrul- LeonardoG
Hi Fakhrul,
I have some new information which might be helpful.
I was probing the signals you asked about with an oscilloscope. As I previously observed on the affected boards, nCONFIG and nSTATUS pulse many times over the 6 seconds before timing out. The pulses are also random between power cycles, so the issue is not deterministic. The FPP data and clock look reasonable and consistent with this behavior, with no surprises.
On the other hand, I observed an interesting behavior on the reference working board we have been using. Normally everything works just fine, but if I probe FPGA_DCLK (pin 16 of U33), the FPGA programming through the MAX V starts to fail, requiring many retries and sometimes even timing out at 6 seconds. This was quite repeatable: without the probe connected the FPGA is programmed straight away, while with the probe touching the pin the MAX V struggles to program the FPGA.
To me this suggests a signal integrity / EMI issue on this line specifically. Looking at the schematic, FPGA_DCLK also routes to a few different places spread around the board, which I guess makes it more sensitive to such issues.
For this reason, in the original MAX V firmware I changed the QSF to reduce the drive strength of the FPGA-related lines (fpga_dclk and fpga_config_d[*]). I set the current strength of these lines to 7 mA, and now the previously non-working boards seem to work without any issues.
I will try to investigate this a bit further, but this is the most promising result I have got so far.
Regards,
Leonardo- FakhrulA_altera
Hi Leonardo,
Thank you. This is a very useful finding. The repeatable sensitivity to probing FPGA_DCLK, together with reliable boot after reducing the MAX V drive strength to 7 mA, strongly suggests marginal signal integrity on the FPP path rather than a flash-content issue. Please note that probe capacitance can itself affect DCLK, so we should treat this as strong diagnostic evidence rather than a confirmed root cause.
Could you please validate the 7 mA setting on both affected boards over at least 100 cold power cycles and share:-The exact QSF assignments and Quartus version
-Results for both affected boards and the working board
-Any DCLK/nSTATUS waveforms before and after the change
Thank you again for the detailed investigation.
Regards,
Fakhrul
- LeonardoG
An update/fix if someone else encounters the same issue.
GordWait an I kept investigating the problem and sharing our findings daily, until we got all our boards working 100% of the time on hundreds of reboots. While we ended up with slightly different solutions the root cause is the same, and I will share the core of what helped me for future reference.The root of the problem is presumably some signal integrity/timing issue across the fpga_config_d/fpga_dclk lines, which seems to be present only on newer boards/chips.
The greatest improvement came from reducing the drive strength of these lines, by adding in the QSF:
set_instance_assignment -name CURRENT_STRENGTH_NEW 7MA -to fpga_dclk
set_instance_assignment -name CURRENT_STRENGTH_NEW 7MA -to fpga_config_d[*]While this solution helped on most of the affected boards it did not fix them all.
To better improve the situation the fpga_dclk can be effectively reduced to 50 MHz, with the easiest way being changing the pfl_x8_inst's "Ratio between input clock and DCLK output" to 2. By doing this the fpga_config_d changes on the fpga_dclk's falling edge, thus improving the hold slack.
To better suit this fix there are some constraints to add in the .sdc.
- fpga_dclk should be a generated one:
create_generated_clock -name {fpga_dclk} -source {clk_config} -divide_by 2 [get_ports {fpga_dclk}]- The data output delay should be based on this clock and respect the Cyclone V datasheet tsu:
set_output_delay -add_delay -max -clock [get_clocks {fpga_dclk}] 5.500 [get_ports {fpga_config_d[*]}]
set_output_delay -add_delay -min -clock [get_clocks {fpga_dclk}] -1.000 [get_ports {fpga_config_d[*]}]- If using the "Ratio" option a multicycle path should actually be defined:
set_multicycle_path -setup -start 1 -from [get_registers {*|alt_pfl_cfg_fpga:*|fpga_data_reg[*]}] -to [get_ports {fpga_config_d[*]}]
set_multicycle_path -hold -start 1 -from [get_registers {*|alt_pfl_cfg_fpga:*|fpga_data_reg[*]}] -to [get_ports {fpga_config_d[*]}]Note that these fixes don't consider the usage of the FPPovJ, for which better constrains should be designed.
Moreover, the FPP boot time is largely limited by the flash access time, so even halving the DCLK won't affect the boot time much. Even with the 50 MHz clock it still boots comfortably within our system's PCIE enumeration window.All of this was tested on a limited number of boards, so I can't assure this works 100% of the time, but until Altera finds a proper solution this can be a good starting point for anyone facing the same problem as we did.
Leonardo
