Forum Discussion

FabianL's avatar
FabianL
Icon for Occasional Contributor rankOccasional Contributor
10 months ago

Arria 10: Remote Update may brick FPGA and Factory Fallback won't work

Hello,

we have observed some critical failures when doing tests with various potential error scenarios concerning a remote Update of the FPGA bitstream in the attached SPI Flash device.

We could repeatedly trigger cases, when the FPGA internal fallbeck mechanism to the factory load does not work. We do not use any bitstream encryption.

Test scenarios:

  1. Erased flash & partially programmed application load image --> Fallback mechanism works as expected
  2. Invalid application load image location, i.e. start of application load is shifted by1-10 Byte (Manually induced error scenario) --> The reprogramming sequence starts but never completes and no fallback to the factory load is performed. => The FPGA is completely unresponsive unless programmed via JTAG

It is obvious, that the 2nd scenario might be a more exotic error scenario, however we require a robust setup and have to make sure, that the FPGA remains accessible under any circumstances, so we need the Factory Fallback mechanism to work reliable!

As a best guess I could assume it might be related to this Note in 1.3.1. Remote System Configuration Mode that the factory fallback mechanism won't work for Arria 10 FPGAs if the last 576 Bytes of the bitstream are corrupted.

Note: The fallback to the factory image does not work under the following conditions: If the last 576 bytes of an unencrypted application image bitstream are corrupted. 

Intel recommends that you examine the last 576 bytes of the unencrypted application image before triggering the application image configuration.

But I have noticed that the binary images of the FPGA bitstream vary in size. So there is no way to check explicit memory locations for these 576 Bytes. Is there any way to identify this section?

My Questions:

  1. Why is the factory configuration fallback mechanism not working in the above described scenario? The Factory load image is valid!
  2. What method does intel recommend to reliable make the factory fallback mechanism work?
  3. How can I examine/validate a FPGA bitstream in flash memory before executing it?

Thanks a lot for any help

Best regards

Fabian

Configuration

26 Replies

Show More
  • FabianL's avatar
    FabianL
    Icon for Occasional Contributor rankOccasional Contributor
    7 months ago

    Hi Xiaoyan,

    are there any news on this topic?

    best regards

    Fabian

  • FabianL's avatar
    FabianL
    Icon for Occasional Contributor rankOccasional Contributor
    6 months ago

    Hello,

    it's been already 2 months without any news to this topic.

    The fact that the factory fallback mechanism is not reliably and is not fully understood is still a major problem for us.

    Are there any news from your side?

    best regards

    Fabian

  • Farabi's avatar
    Farabi
    Icon for Regular Contributor rankRegular Contributor
    5 months ago

    Hello Fabian,


    Sorry to keep you waiting. Few steps to check for debug:


    Could you please check below?

    1- confirm factory image is correctly programmed at boot address.

    2- ensure application image is programmed at known valid address.

    3- verify the .jic file start address matches the RU_PAGE_SELECT value.


    Application Image validation.

    1- Confirm application image can be loaded successfully when normal condition

    2- Confirm FPGA able to enter user mode (CONF_DONE-> HIGH)

    3- Confirm watchdog is not enabled - no writes to RU_RESET_TIMER


    Factory Image validation.

    1- Power up and confirm it boot into factory image.

    2- read RU_RECONFIG_TRIGGER_CONDITIONS to confirm power up state (Bit0 = 0)

    3- Factory image set to below parameters:

    a. RU_PAGE_SELECT = application image address

    b. RU_CONFIGURATION_MODE = 1 (Application Mode)

    c. RU_WATCHDOG_TIMEOUT = set the correct value

    d. RU_WATCHDOC_ENABLE = 1

    4- trigger reconfiguration by writing 1 to RU_RECONFIG


    Test fallback mechanism.

    1-Manually corrupt the application image (erase some part or misalign)

    2- Check if FPGA fail to enter user mode

    3- check if FPGA fallback to factory image

    4- read the RU_RECONFIG_TRIGGER_CONDITIONS reflecting the correct cause(Bit1 = watchdog timeout)


    If you have followed all above and still you observed the fallback mechanism doesn't kick in. We will need your design for escalation.


    regards,

    Farabi


  • Farabi's avatar
    Farabi
    Icon for Regular Contributor rankRegular Contributor
    5 months ago

    Hello,


    Do you have further question?


    regards,

    Farabi


  • FabianL's avatar
    FabianL
    Icon for Occasional Contributor rankOccasional Contributor
    5 months ago

    Hello Farabi,

    thanks for the reply. As mentioned before, if I enable the watchdog as described in your post, the fallback mechanism kicks in.

    But this leaves me with 2 critical questions:

    1. When we do not use the watchdog (RU_WATCHDOC_ENABLE = 0). The factory fallback does not work when the application image is misaligned. Why does the factory fallback not happen in this case? I expect a misaligned application load to trigger a CRC error.
    2. Our application design is not actively serving the watchdog. We are using the Avalon IP "Remote Update Intel FPGA IP" Version 19.1.0. We do not set any of the Watchdog Registers in the application image, so I would have expected that the watchdog would trigger and cause a factory image fallback. But his is not happening. The documentation indicates, that a watchdog timeout may occur after entering application user mode. Hence I would expect that an application image, that does not service the watchdog would trigger a factory fallback. Why is this not the case?

    Having these questions open gives the whole fallback mechanism an unreliable touch. So I would be very thankful if this behavior could be clarified.

    Thanks.

    kind regards

    Fabian

  • FabianL's avatar
    FabianL
    Icon for Occasional Contributor rankOccasional Contributor
    5 months ago

    Hello Farabi,

    is there any news concerning my 2 questions?

    best regards

  • Farabi's avatar
    Farabi
    Icon for Regular Contributor rankRegular Contributor
    5 months ago

    Hello Fabian,


    Sorry to take sometime to answer your question. I just transfer this case to myself so I can monitor this case individually.


    1-Why factory fallback doesn't occur when application image is misaligned and watchdog disabled

    <ANS> Misaligned image might not trigger CRC error, because FPGA may not recognize the bitstream header correctly. If the image is corrupted in such a way that will prevent configuration from starting, the CRC logic never got invoked, resulting the fallback mechanism does not activated. Please check your bitstream if the last 576 bytes are corrupted or not. If corrupted, this will prevent fallback to factory image as well. During this failure, can you check the nSTATUS signal?


    2- Why doesn't the watchdog trigger fallback when not served in the application image?

    <ANS> From Remote Update IP document: watchdog timer only starts counting after FPGA enters usermode. If the application image is invalid- FPGA never reach usermode, watchdog is never activated. Watchdog must be explicitly enabled via RU_WATCHDOG_ENABLE in the factory image. If this is not done, the watchdog is by default- disabled.


    In your setup:

    1- You correctly enabled the watchdog in the factory image.

    2- The application image does not serve the watchdog (no writes to RU_RESET_TIMER) which should trigger fallback only if the image enters user mode.

    3- If the image is misaligned and fails to enter user mode, watchdog never starts-> fallback will never occurs.


    regards,

    Farabi



    • FabianL's avatar
      FabianL
      Icon for Occasional Contributor rankOccasional Contributor
      5 months ago

      Hello Farabi,

      Thanks very much for the reply.

      1. Thanks for the explanation about the missing CRC error. So I guess the only way to safely deal with this is to enable the watchdog. That is fine for us.
      2. I'm sorry, but I don't fully under stand your answer. We have to scenarios (see also here
        1. Misaligned Image:
          1. Enable Watchdog in Factory Image
          2. trigger reconfiguration (write 1 to RU_RECONFIGURATION_MODE & RU_RECONFIG)
          3. Reconfiguration fails due to misaligned image --> Watchdog triggers
          4. Fallback to factory mode
          5. ==> This case is working as expected. Good Case!
        2. Aligned valid Image
          1. Enable Watchdog in Factory Image
          2. trigger reconfiguration (write 1 to RU_RECONFIGURATION_MODE & RU_RECONFIG)
          3. Application Image starts. Application Image does not serve or actively disable the watchdog!
          4. Since the application image does not serve the watchdog, I would expect a factory fallback due to watchdog triggering. NOTE: We do not talk about further reconfiguration triggered from within application image. We only do reconfiguration from within the factory load.
          5. ==> This is not happening. And I don't understand why. Or is the watchdog automatically disabled once a valid application image is loaded?

      best regards

      Fabian

  • Farabi's avatar
    Farabi
    Icon for Regular Contributor rankRegular Contributor
    5 months ago

    You should see nSTATUS like below:

    regards,

    Farabi

  • Farabi's avatar
    Farabi
    Icon for Regular Contributor rankRegular Contributor
    4 months ago

    Hello,


    Thanks for your update. I am not sure what happened to your design, but few things to check below:


    1- Can you make sure : RU_WATCHDOG_ENABLE = 1 is written before triggering reconfiguration, and make sure the setting is persist during configuration. Some reconfig reset settings.


    2- Please make sure the watchdog timeout not too. eg. Dont set RU_WATCHDOG_TIMEOUT = 0xFFF (this is too long)


    3- Please confirm the application image does not contain any logic that somehow triggers/modify RU_RESET_TIMER register.


    regards,

    Farabi


    • FabianL's avatar
      FabianL
      Icon for Occasional Contributor rankOccasional Contributor
      4 months ago

      Hello Farabi,

      1. Yes I have double checked the RU_WATCHDOG_ENABLE = 1 is set before reconfiguration. This was also proved by the test with the misaligned bitstream image. Without the watchdog enabled, the misalinged image reconfiguration gets stuck, with the watchdog enabled it works as expected and triggers a factory fallback.
      2. Why is this too long? I have actually set the watchdog timeout to 0xFFF.
        1. But it works in case of misaligned application loads, so why should this a problem in normal operation?
        2. What is the unit of the watchdog timeout register? is it clock cycles? if yes which clock cycles?
      3. The application image is not setting any registers in the Remote Update Intel FPGA IP Core. It only reads RU_RECONFIG_TRIGGER_CONDITIONS (register 0)

      best regards

      Fabian

  • Farabi's avatar
    Farabi
    Icon for Regular Contributor rankRegular Contributor
    4 months ago

    Hello,


    Do you have further request?


    regards,

    Farabi