Unstable fpga programming using HPS(Agilex3)

Hi,

Thanks for the detailed investigation, the comparison between using fpga_clk_100 directly and via the PLL is very helpful.

Based on your results, this points to a clock/reset sequencing issue during FPGA reconfiguration, rather than a problem with the Linux overlay commands themselves.

What is happening?

You are using HPS Boot First with runtime reconfiguration (Linux overlay). In this mode, the HPS is running while the FPGA fabric is reconfigured dynamically by the SDM. During this process, the fabric is cleared and reloaded in phases, so resources such as PLLs are reset and rebuilt and are not stable during this window.

A key difference between your two cases:

• fpga_clk_100 (pin clock) is a free-running board reference and remains stable across reconfiguration
• fpga_clk_100 routed through a fabric PLL depends on fabric resources, so it is reset and relocked on each reconfiguration

Without proper reset control, logic can start running before the device is fully in user mode, leading to intermittent illegal states.

The downstream effects you observed (SMMU CMD_SYNC timeout, Ethernet watchdog, system reset) are consistent with the system hanging during this transition.

Boot mode context:

Your flow requires HPS Boot First. In this mode, the HPS boots before the FPGA core is programmed (Phase 1 vs Phase 2). Fabric-derived clocks (PLL outputs and f2s clocks) are not present at HPS boot and are recreated during Phase 2. On each reconfiguration, these clocks are reset and can glitch or stop temporarily.

Only board-level clocks such as fpga_clk_100, HPS_OSC_CLK, and pll_ref_clk are guaranteed to remain stable across reconfiguration.

Likely root cause ->

The behavior strongly indicates that PLL and/or fabric logic are being released before the FPGA reaches full user mode (nINIT_DONE) during reconfiguration.

This can lead to:

• PLL output not yet stable or briefly glitching
• Logic starting in an undefined state
• AXI transactions stalling, eventually triggering system reset

This also explains why the issue is intermittent (for example, failing on the third attempt), depending on timing alignment during reconfiguration.

Recommended fix

For Agilex 3 designs using runtime FPGA reconfiguration:

• Instantiate Reset Release IP
• Gate PLL reset and all fabric logic reset using nINIT_DONE
• Ensure logic is released only after:
  • nINIT_DONE indicates the device is fully in user mode
  • and PLL lock is stable (do not rely on PLL locked alone)

In practice:

• Hold PLL in reset until nINIT_DONE is deasserted
• Hold all fabric logic in reset until clocks are valid and stable

Additional notes

• The “memory leak” warnings during overlay removal are known kernel messages and are not related to the reset issue
• In HPS Boot First with runtime reconfiguration, fabric clocks (PLL outputs, f2s clocks) are not free-running and should not be used without proper reset gating during reconfiguration

Summary

The issue is caused by fabric PLL and logic being released too early during SDM reconfiguration, leading to unstable clocking and intermittent system hangs. Proper nINIT_DONE-based reset gating resolves this type of issue.

Regards,
Tien Fong