Forum Discussion
Occasional ContributorCyclone V GT Dev Kit - new units fail to boot from FPP
An update/fix if someone else encounters the same issue.
GordWait an I kept investigating the problem and sharing our findings daily, until we got all our boards working 100% of the time on hundreds of reboots. While we ended up with slightly different solutions the root cause is the same, and I will share the core of what helped me for future reference.The root of the problem is presumably some signal integrity/timing issue across the fpga_config_d/fpga_dclk lines, which seems to be present only on newer boards/chips.
The greatest improvement came from reducing the drive strength of these lines, by adding in the QSF:
set_instance_assignment -name CURRENT_STRENGTH_NEW 7MA -to fpga_dclk
set_instance_assignment -name CURRENT_STRENGTH_NEW 7MA -to fpga_config_d[*]While this solution helped on most of the affected boards it did not fix them all.
To better improve the situation the fpga_dclk can be effectively reduced to 50 MHz, with the easiest way being changing the pfl_x8_inst's "Ratio between input clock and DCLK output" to 2. By doing this the fpga_config_d changes on the fpga_dclk's falling edge, thus improving the hold slack.
To better suit this fix there are some constraints to add in the .sdc.
- fpga_dclk should be a generated one:
create_generated_clock -name {fpga_dclk} -source {clk_config} -divide_by 2 [get_ports {fpga_dclk}]- The data output delay should be based on this clock and respect the Cyclone V datasheet tsu:
set_output_delay -add_delay -max -clock [get_clocks {fpga_dclk}] 5.500 [get_ports {fpga_config_d[*]}]
set_output_delay -add_delay -min -clock [get_clocks {fpga_dclk}] -1.000 [get_ports {fpga_config_d[*]}]- If using the "Ratio" option a multicycle path should actually be defined:
set_multicycle_path -setup -start 1 -from [get_registers {*|alt_pfl_cfg_fpga:*|fpga_data_reg[*]}] -to [get_ports {fpga_config_d[*]}]
set_multicycle_path -hold -start 1 -from [get_registers {*|alt_pfl_cfg_fpga:*|fpga_data_reg[*]}] -to [get_ports {fpga_config_d[*]}]Note that these fixes don't consider the usage of the FPPovJ, for which better constrains should be designed.
Moreover, the FPP boot time is largely limited by the flash access time, so even halving the DCLK won't affect the boot time much. Even with the 50 MHz clock it still boots comfortably within our system's PCIE enumeration window.All of this was tested on a limited number of boards, so I can't assure this works 100% of the time, but until Altera finds a proper solution this can be a good starting point for anyone facing the same problem as we did.
Leonardo
Occasional ContributorHi Fakhrul,
I have some new information which might be helpful.
I was probing the signals you asked about with an oscilloscope. As I previously observed on the affected boards, nCONFIG and nSTATUS pulse many times over the 6 seconds before timing out. The pulses are also random between power cycles, so the issue is not deterministic. The FPP data and clock look reasonable and consistent with this behavior, with no surprises.
On the other hand, I observed an interesting behavior on the reference working board we have been using. Normally everything works just fine, but if I probe FPGA_DCLK (pin 16 of U33), the FPGA programming through the MAX V starts to fail, requiring many retries and sometimes even timing out at 6 seconds. This was quite repeatable: without the probe connected the FPGA is programmed straight away, while with the probe touching the pin the MAX V struggles to program the FPGA.
To me this suggests a signal integrity / EMI issue on this line specifically. Looking at the schematic, FPGA_DCLK also routes to a few different places spread around the board, which I guess makes it more sensitive to such issues.
For this reason, in the original MAX V firmware I changed the QSF to reduce the drive strength of the FPGA-related lines (fpga_dclk and fpga_config_d[*]). I set the current strength of these lines to 7 mA, and now the previously non-working boards seem to work without any issues.
I will try to investigate this a bit further, but this is the most promising result I have got so far.
Regards,
Leonardo
Regular ContributorHi Leonardo,
Thank you. This is a very useful finding. The repeatable sensitivity to probing FPGA_DCLK, together with reliable boot after reducing the MAX V drive strength to 7 mA, strongly suggests marginal signal integrity on the FPP path rather than a flash-content issue. Please note that probe capacitance can itself affect DCLK, so we should treat this as strong diagnostic evidence rather than a confirmed root cause.
Could you please validate the 7 mA setting on both affected boards over at least 100 cold power cycles and share:
-The exact QSF assignments and Quartus version
-Results for both affected boards and the working board
-Any DCLK/nSTATUS waveforms before and after the change
Thank you again for the detailed investigation.
Regards,
Fakhrul
- LeonardoG
Hi Fakhrul
- The QSF assignments that fixed the issue for me on MAX 5 image are:
set_instance_assignment -name CURRENT_STRENGTH_NEW 7MA -to fpga_dclk
set_instance_assignment -name CURRENT_STRENGTH_NEW 7MA -to fpga_config_d[*]
To be honest, in the MAX 5 image I'm currently using I also changed the logic to run fpga_dclk at 50 MHz, but even in the tests I've done at 100 MHz it was working fine (though the testing was not so extensive).- I just tested the 2 affected boards and 1 working board with the new MAX image for around 100 power cycles. They worked 100% of the time. Obviously my sample size is quite limited, and this was only tested at room temperature.
- Regarding the waveforms, you can find the ones regarding dclk and config_d below. On the left is the one obtained with the new MAX image (also with the reduction to 50 MHz), and on the right the original image. While I reckon that the new MAX image is a bit too slow for my liking, the original one has quite a bad overshoot/undershoot.
Below you can find the acquisitions on nConfig and nStatus with the original firmware on a not working board. It is not particularly exciting but you can see the FPGA configuration retrying many times until the 6 seconds timeout. Also note that the time interval between the retries is not repeatable across different power cycles, confirming that the issue has some randomness to it.
Do you have any update on the "related case that is currently under internal review" you mentioned in a previous message? Can you confirm whether you were able to replicate/verify the issue on your side?
Best regards,
Leonardo
GordWaitAltera, you need to confirm WHY these changes are suddenly required. We have tried similar updates to our Max 5 image and see promising results, but I need root cause so I can document why the solution is good and is not a risk for any production systems in the field. Leonardo has done excellent work debugging an issue that Altera should have been working on themselves.
- FakhrulA_altera
Hi Leonardo,
Thank you for the detailed results and waveforms. Achieving 100 successful power cycles on all three boards is encouraging. The original overshoot/undershoot and the improvement with reduced drive strength support a marginal signal-integrity or timing-margin issue in the FPP path.
However, since the modified image changes both the drive strength and DCLK frequency, we cannot yet confirm the root cause or treat this as an official production fix. If practical, could you perform one final comparison at 100 MHz, changing only the drive strength to 7 mA, and share the Quartus version used?
The related internal case remains under engineering review; we do not yet have a confirmed internal reproduction or root cause. We will add your findings and request review of the DCLK routing and factory MAX V settings. Please treat the 7 mA setting as an experimental workaround. You may proceed with the distributor RMA process if you prefer, while we continue the engineering assessment.
Thank you again for your valuable investigation.
Regards,
Fakhrul- LeonardoG
Hi Fakhrul,
Sorry, I forgot the Quartus version in my last message. For the MAX V image I'm using Quartus 14.1.
I did a final comparison at 100 MHz, changing only the drive strength to 7 mA. Even in this case the 2 "affected" boards always booted correctly over around 100 retries each.
One caveat is that in this situation, when probing DCLK with the oscilloscope, the boot failed and I could clearly see the waveform not reaching 2.5 V. Unfortunately I don't have an active probe available at the moment to scope it better. But even a "working" board with the default image sometimes failed to boot when being probed.
I guess the 7 mA clearly helps, but I can't really say if it is 100% reliable. With the 50 MHz DCLK I feel confident enough.
Regards,
Leonardo
