Forum Discussion

BeB's avatar
BeB
Icon for Occasional Contributor rankOccasional Contributor
5 years ago

Stratix 10 Native PHY 10GBASE-R serial loopback: Rx is not always mirroring Tx

Hello,

I am using the native PHY configured for 10GBASE-R. I enabled the serial loopback feature to troubleshoot some issues with the Tx path, and found in the process that although the serial loopback to Rx works in the large majority of the time, i.e. the Rx XGMII data is a carbon copy of the Tx XGMII data, sometimes that is not the case, i.e. some bits are corrupted. If the loopback is clean at power up, it seems to stay clean, but sometimes the loopback shows bit corruption. When that happens, as far as I can tell, all status signals look fine, i.e. tx pll is locked, rx locked to data is true, the PHY is not in reset, the transfer ready signals are high, etc. so there is no obvious way to tell that something is off at the application level in the FPGA.

Without an obvious smoking gun I do not know if that is an issue with the serial loopback itself, or if it reveals something fundamentally wrong with the way the PHY is setup, whether it is an issue with the Tx path or the Rx path or both.

The way I setup the PHY is the following:

- ATX pll provides 5156.25MHz to the TX serial clock

- Max 10 provides 156.25MHz to the MAC and to the CDR ref clock

- tx_clkout loops back to tx_coreclkin

- rx clkout loops back to rx_coreclkin

Any hint would be greatly appreciated.

Many thanks in advance,

BeB

7 Replies

  • CheepinC_altera's avatar
    CheepinC_altera
    Icon for Regular Contributor rankRegular Contributor
    5 years ago

    Hi,


    As I understand it, you observe bit error issue when using serial loopback with the 10GBaseR mode Native PHY. This occurs intermittently and depending on the power up. For example, at one power up, it will stay clean. Another power up, it may observe bit error.


    For your information, if you are using serial loopback, we could isolate the signal integrity problem from the root cause.


    Based on your observation that it is power up dependent, this is trending towards potential calibration related issue. For your information, to ensure a successful power up calibration, you need to provide free-running and stable clock sources to TXPLL and CDR ref clock. These clocks should be directly sourced from on-board oscillator. I notice that the CDR refclk is coming from Max10 device. Can you try with on-board oscillator?


    Also, you should feed free-running and stable clock to the OSC_CLK_1 as well.


    Please let me know if there is any concern. Thank you.



    • BeB's avatar
      BeB
      Icon for Occasional Contributor rankOccasional Contributor
      5 years ago

      Hi,

      Thank you for your answer.

      I misrepresented the situation: the 156.25MHz is coming from a programmable clock source (Si5341) controlled by a Max10. It is coming in via a differential pair of dedicated clock pins.

      I have to say that I am a little confused. I thought that the best clock input option for the Tx PLL was to use a dedicated reference clock pin. Doesn't that imply that the reference clock comes from outside the Stratix 10?

      Sorry for the what is probably a basic question, but how should I setup an on-board oscillator for the CDR ref clock?
      - The native PHY IP implies that a wide range of frequencies are supported for the CDR ref clock, including 156.25MHz, but what would be the best choice? For example if I pick 10GBASE-R for the native PHY config, the default CDR ref clock is set to 644.53125MHz.
      - If I look at the 3 types of PLL for the transceivers:
      -- CMU PLL doesn't accept 156.25MHz as an input
      -- fPLL accepts 156.25MHz, but what should be the target output frequency? 644.53125MHz?
      -- ATX PLL can also accept 156.25MHz and deliver 644.53125MHz for example

      Thank you again, and sorry for all the questions. I am having a tough time navigating the documentation.

      BeB

  • CheepinC_altera's avatar
    CheepinC_altera
    Icon for Regular Contributor rankRegular Contributor
    5 years ago

    Hi,


    Thanks for your clarification that the refclks are coming directly from on-board oscillators. Initially I was confused that they clock might be coming from the Max10 output.


    Yes, you are right, it is recommended to use dedicated refclk pin for the TX PLL. In addition to that, you would need to ensure also the clock source is stable and free-running. Just for your information, in some cases, an oscillator is controlled by FPGA which means that FPGA needs to be powered up then only it programs the oscillator. In this case, the clock source is not available during XCVR power up.


    Regarding the clock configuration, for CDR, if you select 644.53125MHz refclk frequency, then you need to connect a clock with this 644.53125MHz frequency.


    As for the ATX PLL, assuming you are using the same clock source, its configuration should have refclk frequency = 644.53125MHz and output frequency = 5156.25MHz (for 10GBase-R data rate).


    Please let me know if there is any concern. Thank you.


    • BeB's avatar
      BeB
      Icon for Occasional Contributor rankOccasional Contributor
      5 years ago

      Hello,

      Thank you for the information. I will have to look in more details about clock stability, but if I understand well the design should work equally well with either 156.25MHz or 644.53125MHz for the reference clock for CDR and ATX PLL? And yes, the output frequency of the latter is 5156.25MHz.

      Thanks again,

      BeB

  • CheepinC_altera's avatar
    CheepinC_altera
    Icon for Regular Contributor rankRegular Contributor
    5 years ago

    Hi,


    Just my own experience, generally the higher refclk frequency you use, the better the jitter performance. This is because lesser multiplying factor is required internally.


  • CheepinC_altera's avatar
    CheepinC_altera
    Icon for Regular Contributor rankRegular Contributor
    5 years ago

    Hi,


    As I understand it, it has been some time since I last heard from you. This thread will be transitioned to community support. If you have a new question, feel free to open a new thread to get the support from Intel experts. Otherwise, the community users will continue to help you on this thread. Thank you.


    • BeB's avatar
      BeB
      Icon for Occasional Contributor rankOccasional Contributor
      5 years ago

      Hello,

      I spent a bit of time on this issue in the last few weeks, and I can confirm that

      - the clocks are definitely stable once coming out of reset

      - the same issue arises with 156.25MHz or 644.53125MHz for the reference clock for CDR and ATX PLL (the clocks are provided by a programmable on board oscillator)

      So I am back to where I started from: the Rx path does not always mirror the Tx path when the serial loopback is enabled, but all status signals are green.

      BeB