Unstable fpga programming using HPS(Agilex3)

In the original design everything run on a 100 MHz clock input fpga_clk_100, when I use this the programming is stable and work every time, but I need more than one clock in my design so I connected the fpga_clk_100 to a pll and used the output clock from the pll in the design, then the programming became unstable. Does anybody understand why? The output clock from the pll is also 100MHz

root@agilex3:~# rmdir /sys/kernel/config/device-tree/overlays/0 ; sleep 1 ; mkdir /sys/kernel/config/device-tree/overlays/0 ; cd /lib/firmware/ ; sleep 1; echo overlay.dtb >/sys/kernel/config/device-tree/overlays/0/path
rmdir: '/sys/kernel/config/device-tree/overlays/0': No such file or directory
[   36.922294] fpga_manager fpga0: writing overlay.rbf to Stratix10 SOC FPGA Manager
[   39.120291] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-region/firmware-name
[   39.130686] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-region/config-complete-timeout-us
root@agilex3:/lib/firmware# rmdir /sys/kernel/config/device-tree/overlays/0 ; sleep 1 ; mkdir /sys/kernel/config/device-tree/overlays/0 ; cd /lib/firmware/ ; sleep 1; echo overlay.dtb >/sys/kernel/config/device-tree/overlays/0/path
[   45.737387] fpga_manager fpga0: writing overlay.rbf to Stratix10 SOC FPGA Manager
[   47.872751] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-region/firmware-name
[   47.886499] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-region/config-complete-timeout-us
root@agilex3:/lib/firmware# rmdir /sys/kernel/config/device-tree/overlays/0 ; sleep 1 ; mkdir /sys/kernel/config/device-tree/overlays/0 ; cd /lib/firmware/ ; sleep 1; echo overlay.dtb >/sys/kernel/config/device-tree/overlays/0/path
[   56.264190] fpga_manager fpga0: writing overlay.rbf to Stratix10 SOC FPGA Manager
[   59.386742] arm-smmu-v3 16000000.iommu: CMD_SYNC timeout at 0x00004449 [hwprod 0x0000444a, hwcons 0x00004449]
[   60.461430] arm-smmu-v3 16000000.iommu: CMD_SYNC timeout at 0x0000444b [hwprod 0x0000444c, hwcons 0x00004449]
[   61.536055] arm-smmu-v3 16000000.iommu: CMD_SYNC timeout at 0x0000444d [hwprod 0x0000444e, hwcons 0x00004449]
[   62.610600] arm-smmu-v3 16000000.iommu: CMD_SYNC timeout at 0x0000444f [hwprod 0x00004450, hwcons 0x00004449]
[   63.685144] arm-smmu-v3 16000000.iommu: CMD_SYNC timeout at 0x00004451 [hwprod 0x00004452, hwcons 0x00004449]
[   64.759689] arm-smmu-v3 16000000.iommu: CMD_SYNC timeout at 0x00004453 [hwprod 0x00004454, hwcons 0x00004449]
[   64.769798] socfpga-dwmac 10830000.ethernet eth0: NETDEV WATCHDOG: CPU: 1: transmit queue 7 timed out 6452 ms
[   65.844646] arm-smmu-v3 16000000.iommu: CMD_SYNC timeout at 0x00004455 [hwprod 0x00004456, hwcons 0x00004449]
[   66.919264] arm-smmu-v3 16000000.iommu: CMD_SYNC timeout at 0x00004457 [hwprod 0x00004458, hwcons 0x00004449]
[   66.934688] socfpga-dwmac 10830000.ethernet eth0: Reset adapter.

U-Boot SPL 2025.10 (May 18 2026 - 08:34:29 +0000)
Reset state: Cold
MPU           800000 kHz
L4 Main       400000 kHz
L4 sys free   100000 kHz
L4 MP         200000 kHz
L4 SP         100000 kHz
SDMMC         200000 kHz
init_mem_cal: Initial DDR calibration IO96B_0 succeed
DDR: Calibration success
is_mailbox_spec_compatible: IOSSM mailbox version: 1
LPDDR4: 1792 MiB
ecc_interrupt_status: ECC error number detected on IO96B_0: 0
SDRAM-ECC: Initialized success

Hi,

Thank you for reaching out and bringing this to our attention.

We have received your inquiry and are currently reviewing the details. We will get back to you with an update as soon as possible.

Appreciate your patience in the meantime.

Best regards,

Tien Fong

Hi,

Thanks for the detailed investigation, the comparison between using fpga_clk_100 directly and via the PLL is very helpful.

Based on your results, this points to a clock/reset sequencing issue during FPGA reconfiguration, rather than a problem with the Linux overlay commands themselves.

What is happening?

You are using HPS Boot First with runtime reconfiguration (Linux overlay). In this mode, the HPS is running while the FPGA fabric is reconfigured dynamically by the SDM. During this process, the fabric is cleared and reloaded in phases, so resources such as PLLs are reset and rebuilt and are not stable during this window.

A key difference between your two cases:

• fpga_clk_100 (pin clock) is a free-running board reference and remains stable across reconfiguration
• fpga_clk_100 routed through a fabric PLL depends on fabric resources, so it is reset and relocked on each reconfiguration

Without proper reset control, logic can start running before the device is fully in user mode, leading to intermittent illegal states.

The downstream effects you observed (SMMU CMD_SYNC timeout, Ethernet watchdog, system reset) are consistent with the system hanging during this transition.

Boot mode context:

Your flow requires HPS Boot First. In this mode, the HPS boots before the FPGA core is programmed (Phase 1 vs Phase 2). Fabric-derived clocks (PLL outputs and f2s clocks) are not present at HPS boot and are recreated during Phase 2. On each reconfiguration, these clocks are reset and can glitch or stop temporarily.

Only board-level clocks such as fpga_clk_100, HPS_OSC_CLK, and pll_ref_clk are guaranteed to remain stable across reconfiguration.

Likely root cause ->

The behavior strongly indicates that PLL and/or fabric logic are being released before the FPGA reaches full user mode (nINIT_DONE) during reconfiguration.

This can lead to:

• PLL output not yet stable or briefly glitching
• Logic starting in an undefined state
• AXI transactions stalling, eventually triggering system reset

This also explains why the issue is intermittent (for example, failing on the third attempt), depending on timing alignment during reconfiguration.

Recommended fix

For Agilex 3 designs using runtime FPGA reconfiguration:

• Instantiate Reset Release IP
• Gate PLL reset and all fabric logic reset using nINIT_DONE
• Ensure logic is released only after:
  • nINIT_DONE indicates the device is fully in user mode
  • and PLL lock is stable (do not rely on PLL locked alone)

In practice:

• Hold PLL in reset until nINIT_DONE is deasserted
• Hold all fabric logic in reset until clocks are valid and stable

Additional notes

• The “memory leak” warnings during overlay removal are known kernel messages and are not related to the reset issue
• In HPS Boot First with runtime reconfiguration, fabric clocks (PLL outputs, f2s clocks) are not free-running and should not be used without proper reset gating during reconfiguration

Summary

The issue is caused by fabric PLL and logic being released too early during SDM reconfiguration, leading to unstable clocking and intermittent system hangs. Proper nINIT_DONE-based reset gating resolves this type of issue.

Regards,
Tien Fong

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

entity legacy_baseline_top is
    port (
        -- Clock and Reset
        fpga_reset_n                        : in    std_logic;
        fpga_clk_100                        : in    std_logic;
    );
end entity legacy_baseline_top;

architecture rtl of legacy_baseline_top is

    -- Component declarations
    component altera_std_synchronizer is
        generic (
            depth : integer := 3
        );
        port (
            clk     : in  std_logic;
            reset_n : in  std_logic;
            din     : in  std_logic;
            dout    : out std_logic
        );
    end component;

     component qsys_top is
        port (
            clk_100_clk                           : in    std_logic                     := 'X';             -- clk
	   clock_bridge_0_out_clk_clk            : out   std_logic;                                        -- clk
            o_pma_cu_clk_clk                      : out   std_logic_vector(0 downto 0);                     -- clk
	    iopll_0_locked_export                 : out   std_logic;                                        -- export
            reset_bridge_0_in_reset_reset_n       : in    std_logic                     := 'X';             -- reset_n
            reset_reset_n                         : in    std_logic                     := 'X';             -- reset_n
	    reset_bridge_pll_in_reset_reset_n     : in    std_logic                     := 'X';             -- reset_n
            ninit_done_ninit_done                 : out   std_logic;                                        -- ninit_done
            h2f_reset_reset                       : out   std_logic                                        -- reset
        );
    end component qsys_top;

    -- Internal signals
    signal system_clk_100                      : std_logic;
    signal pll_clk_100                         : std_logic;
    signal pll_locked                          : std_logic;
    signal pll_locked_stable                   : std_logic;
    signal ninit_done                          : std_logic;
    signal combined_reset_n                    : std_logic;
    signal system_reset_n                      : std_logic;
    signal pll_reset_n                         : std_logic;
    signal pll_reset_sync_n                    : std_logic;
    signal h2f_reset                           : std_logic;

begin

    -- Concurrent signal assignments
    combined_reset_n <= fpga_reset_n and (not h2f_reset) and (not ninit_done);
    pll_reset_n <= fpga_reset_n and (not ninit_done);
    system_clk_100   <= fpga_clk_100;

    -- Reset synchroniser
    fpga_reset_n_sync : altera_std_synchronizer
        generic map (
            depth => 3
        )
        port map (
            clk     => system_clk_100,
            reset_n => combined_reset_n,
            din     => '1',
            dout    => system_reset_n
        );
		  
	  -- Reset synchroniser
    pll_reset_n_sync : altera_std_synchronizer
        generic map (
            depth => 3
        )
        port map (
            clk     => system_clk_100,
            reset_n => pll_reset_n,
            din     => '1',
            dout    => pll_reset_sync_n
        );
	
pll_stable_filter : block
	     constant CLK_FREQ_HZ  : positive := 100_000_000;
        constant STABLE_CYCLES : positive := CLK_FREQ_HZ;

        signal counter  : natural range 0 to STABLE_CYCLES := 0;
        signal stable_r : std_logic := '0';
    begin
 
        p_filter : process (system_clk_100, pll_reset_sync_n) is
        begin
            if pll_reset_sync_n = '0' then
                counter  <= 0;
                stable_r <= '0';
 
            elsif rising_edge(system_clk_100) then
                if pll_locked = '0' then
                    counter  <= 0;
                    stable_r <= '0';
 
                elsif counter = STABLE_CYCLES - 1 then
                    stable_r <= '1';
                else
                    counter  <= counter + 1;
                    stable_r <= '0';
                end if;
            end if;
        end process p_filter;
 
        pll_locked_stable <= stable_r;
 
    end block pll_stable_filter;

soc_inst : qsys_top
    port map (
        clk_100_clk                           => system_clk_100,
        clock_bridge_0_out_clk_clk            => pll_clk_100, --clock out from pll
        reset_reset_n                         => system_reset_n,
        iopll_0_locked_export				=> pll_locked,
	reset_bridge_pll_in_reset_reset_n     => pll_reset_n, --Reset to pll
	reset_bridge_0_in_reset_reset_n       => pll_locked_stable, --This goes into a reset bridge in nios with deassert synced to output clock from pll
        ninit_done_ninit_done                 => ninit_done, 
        h2f_reset_reset                       => h2f_reset
    );

Hi, thank you for answering, I have included your suggestions in my code.

Here is the top file of my project, I have removed everything not related to clk/reset.

Everything inside qsys including all HPS bridges is clocked from the pll_clk_100 and reset using the pll_locked_stable after it has been syncronized to pll_clk_100 so it has a syncronized deassert.
Do you see anything wrong with it?
How will the h2f_reset behave during a reconfiguration, do I need to include it in the reset of the pll?
How long do I need to wait for stable locked from pll?, I currently wait 1 second

I also wonder what will happen if I have any in flight transactions on any of the hps bridges when I start programming the FPGA, cause I use the f2sdram as a communication channel between niosV on the fpga and the HPS, do I need to shut this off manually before fpga programming or will the fpga-manager handle this automatically?

Hi ,

Just to follow up as the case been idling for sometime, may I ask the status for this case?

Thanks

Regards

Kian