Agilex 5 GTS to HPS CDC/STA issue: help request!

I am trying to bring-up the GTS component on the A5.

The general idea is: Oscillator -> GTS -> FIFO -> DMA -> HPS -> Application.

The Qsys diagram looks as followed (HPS section omitted for brevity):

It's also worth noting that the Osc -> FIFO -> DMA-> HPS design has been tested and works ok.

Furthermore, the only changes I have made to the GTS and FIFO IP are as followed:

-----------------

1: Reduced test

-----------------

Clock domain A: Oscillator -> GTS -x

Clock domain B: 32'b1 -> FIFO -> DMA -> HPS

The linking HDL in the top level (HPS section omitted for brevity):

module top(...);
...
// GTS signals
wire gts_i_rx_cdr_refclk;
wire gts_i_tx_pll_refclk;
reg gts_i_tx_reset, gts_i_rx_reset;
wire gts_o_tx_reset_ack, gts_o_rx_reset_ack;
wire gts_i_tx_coreclkin, gts_i_rx_coreclkin;
wire gts_o_tx_clkout, gts_o_rx_clkout;
wire gts_o_tx_serial_data, gts_o_tx_serial_data_n;
wire gts_i_rx_serial_data, gts_i_rx_serial_data_n;
wire [79:0] gts_i_tx_parallel_data;
wire [79:0] gts_o_rx_parallel_data;

wire osc_clk, raw2s_clk, fifo_clkin;
wire osc_rst, raw2s_rst, fifo_reset;
wire [31:0] osc_pattern_o;
wire [31:0] raw2s_data_i;

// reset
always @(posedge system_clk_100) begin
      if (system_reset) begin
            gts_i_tx_reset <= 1'b1;
            gts_i_rx_reset <= 1'b1;
      end else begin
            if((gts_o_tx_reset_ack==1'b1) && (gts_o_rx_reset_ack==1'b1)) begin
                  gts_i_tx_reset <= 1'b0;
                  gts_i_rx_reset <= 1'b0;
            end
      end
end

// GTS internal clocks
assign gts_i_tx_coreclkin = gts_o_tx_clkout;
assign gts_i_rx_coreclkin = gts_o_rx_clkout;

// serial data
assign gts_i_rx_serial_data = gts_o_tx_serial_data;
assign gts_i_rx_serial_data_n = gts_o_tx_serial_data_n;

// GTS to DMA
assign fifo_clkin = system_clk_100;
assign fifo_reset = ~system_reset;
assign raw2s_clk = system_clk_100;
assign raw2s_rst = ~system_reset;
assign raw2s_data_i = 32'b1;

// Oscillator to GTS
assign osc_clk = gts_o_tx_clkout;
assign osc_rst = gts_i_tx_reset;
assign gts_i_tx_parallel_data[31:0] = osc_pattern_o;

// Qsys Top module
qsys_top soc_inst (
.clk_100_clk                               (system_clk_100),
.ninit_done_ninit_done                     (ninit_done),
.intel_directphy_gts_0_i_rx_cdr_refclk_p_clk                     (REFCLK),
.intel_directphy_gts_0_i_tx_pll_refclk_p_clk                     (REFCLK),
.intel_directphy_gts_0_i_tx_reset_tx_reset                       (gts_i_tx_reset),
.intel_directphy_gts_0_i_rx_reset_rx_reset                       (gts_i_rx_reset),
.intel_directphy_gts_0_o_tx_reset_ack_tx_reset_ack               (gts_o_tx_reset_ack),
.intel_directphy_gts_0_o_rx_reset_ack_rx_reset_ack               (gts_o_rx_reset_ack),
.intel_directphy_gts_0_i_tx_coreclkin_clk                        (gts_i_tx_coreclkin),
.intel_directphy_gts_0_i_rx_coreclkin_clk                        (gts_i_rx_coreclkin),
.intel_directphy_gts_0_o_tx_clkout_clk                           (gts_o_tx_clkout),
.intel_directphy_gts_0_o_rx_clkout_clk                           (gts_o_rx_clkout),
.intel_directphy_gts_0_o_tx_serial_data_o_tx_serial_data         (gts_o_tx_serial_data),
.intel_directphy_gts_0_o_tx_serial_data_n_o_tx_serial_data_n     (gts_o_tx_serial_data_n),
.intel_directphy_gts_0_i_rx_serial_data_i_rx_serial_data         (gts_i_rx_serial_data),
.intel_directphy_gts_0_i_rx_serial_data_n_i_rx_serial_data_n     (gts_i_rx_serial_data_n),
.intel_directphy_gts_0_i_tx_parallel_data_i_tx_parallel_data     (gts_i_tx_parallel_data),
.intel_directphy_gts_0_o_rx_parallel_data_o_rx_parallel_data     (gts_o_rx_parallel_data),
.intel_srcss_gts_0_i_src_rs_priority_src_rs_priority             (1'b1),
.oscillator_0_clk_clk                                            (osc_clk),
.oscillator_0_rst_reset                                          (osc_rst),
.oscillator_0_oscillator_pattern_o_new_signal                    (osc_pattern_o),
.raw_to_stream_0_conduit_end_1_new_signal                        (raw2s_data_i),
.raw_to_stream_0_rst_reset                                       (raw2s_rst),
.raw_to_stream_0_clk_clk                                         (raw2s_clk),
.st_dc_fifo_0_in_clk_clk                                         (fifo_clkin),
.st_dc_fifo_0_in_clk_reset_reset_n                               (fifo_reset),
...
);
...
endmodule

The STA output:

And the system boot (SUCCESS):

Downloading FPGA bitstream...
Using ethernet@10830000 device
TFTP from server **bleep**.**bleep**.**bleep**.**bleep**; our IP address is **bleep**.**bleep**.**bleep**.**bleep**
Filename 'fpga/soc_system_base.rbf'.
Load address: 0x90000000
Loading: #################################################################
         #################################################################
         ######
         2 MiB/s
done
Bytes transferred = 1994752 (1e7000 hex)
..FPGA reconfiguration OK!

------------

2: Full test

------------

Clock Domain A+B: Oscillator -> GTS -> FIFO -> DMA -> HPS

The changes made to the linking HDL:

module top();
...
// GTS to DMA
assign fifo_clkin = gts_o_rx_clkout;
assign fifo_reset = gts_i_rx_reset;
assign raw2s_clk = gts_o_rx_clkout;
assign raw2s_rst = gts_i_rx_reset;
assign raw2s_data_i = gts_o_rx_parallel_data[31:0];
...
endmodule;

Note that the reason for connecting the tx_out and rx_out can be seen in this comment

The STA output:

The system boot (HANG):

Downloading FPGA bitstream...
Using ethernet@10830000 device
TFTP from server **bleep**.**bleep**.xx.x; our IP address is **bleep**.**bleep**.xx.x
Filename 'fpga/soc_system_base.rbf'.
Load address: 0x90000000
Loading: #################################################################
         #################################################################
         #######
         2 MiB/s
done
Bytes transferred = 2011136 (1eb000 hex)
........
FPGA reconfiguration failed!Command 'load' failed: Error -110
FPGA not ready. Bridge reset aborted!

Conclusion:

=========

The above has led me to believe that the issue with the design occurs at the CDC between the GTS and the HPS.

I am not experienced with these sort of issues where the STA files seem to hold the key to debugging, and so would appreciate some help navigating how to approach this kind of problem!

Many thanks