Forum Discussion

New Contributor

5 years ago

Solved

Issues with updating Arria10 PAC for AFU

Hello, Platform info: Arria 10 GX PAC Host System: Ubuntu 18.04 ( 4.15.0 kernel), Xeon Gold 6226R CPU dual-socket server We have two Arria10 PAC cards that we are trying to run the AFU Getting ...

jyoung

5 years ago

I was finally able to update this using super-rsu after completely shutting off power to the server (cold reboot):

[]$ super-rsu --log-level trace /usr/share/opae/a10-gx-pac/super-rsu/base/rsu-09c4.json
[2020-12-28 16:33:37,086] [DEBUG   ] [MainThread  ] - found fpga objects: ['/sys/class/fpga/intel-fpga-dev.0']
[2020-12-28 16:33:37,088] [DEBUG   ] [MainThread  ] - found device at 0000:3d:00.0 -tree is
 [pci_address(0000:3a:00.0), pci_id(0x8086, 0x2030)]
    [pci_address(0000:3b:00.0), pci_id(0x10b5, 0x8747)]
        [pci_address(0000:3c:08.0), pci_id(0x10b5, 0x8747)]
            [pci_address(0000:3d:00.0), pci_id(0x8086, 0x09c4)]
        [pci_address(0000:3c:10.0), pci_id(0x10b5, 0x8747)]
            [pci_address(0000:3e:00.0), pci_id(0x198a, 0x385c)]

[2020-12-28 16:33:37,096] [WARNING ] [MainThread  ] - Update starting. Please do not interrupt.
[2020-12-28 16:33:37,097] [DEBUG   ] [MainThread  ] - [3d:00.0] version (0x0124000200000367) up to date for sr
[2020-12-28 16:33:37,098] [DEBUG   ] [MainThread  ] - bmc_fw is being force flashed
[2020-12-28 16:33:37,098] [DEBUG   ] [MainThread  ] - bmc_fw versions not equal (system:0x0000000000026889 != manifest:0x0000000000026895)
[2020-12-28 16:33:37,098] [DEBUG   ] [MainThread  ] - bmc_fw versions not equal (system:0x0000000000026889 != manifest:0x0000000000026895)
[2020-12-28 16:33:37,099] [DEBUG   ] [MainThread  ] - [3d:00.0] update timeout set to: 1200.0
[2020-12-28 16:33:37,099] [DEBUG   ] [3d:00.0     ] - update of board at [pci_address(0000:3d:00.0), pci_id(0x8086, 0x09c4)] started
[2020-12-28 16:33:37,099] [DEBUG   ] [MainThread  ] - max timeout set to: 0:20:00
[2020-12-28 16:33:37,100] [DEBUG   ] [3d:00.0     ] - starting task: fpgasupdate /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4_bootloader-26895-fw_Release.bin 0000:3d:00.0
[2020-12-28 16:33:37,222] [WARNING ] Update starting. Please do not interrupt.
[2020-12-28 16:33:37,223] [INFO    ] updating from file /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4_bootloader-26895-fw_Release.bin with size 38016
[2020-12-28 16:33:37,331] [INFO    ] writing to staging area
[2020-12-28 16:34:36,173] [DEBUG   ] [MainThread  ] - waiting (0:19:00.927721) for threads: 3d:00.0
[2020-12-28 16:34:36,674] [DEBUG   ] [MainThread  ] - waiting (0:19:00.426487) for threads: 3d:00.0
(100%) [____________________] [38016/38016 bytes][Time:0:01:34.404933]
[2020-12-28 16:35:11,747] [INFO    ] applying update to 0000:3d:00.0
(100%) [____________________][Time:0:00:08.010363]
[2020-12-28 16:35:19,757] [INFO    ] update of 0000:3d:00.0 complete
[2020-12-28 16:35:19,758] [INFO    ] Secure update OK
[2020-12-28 16:35:19,758] [INFO    ] Total time: 0:01:42.536032
[2020-12-28 16:35:19,809] [DEBUG   ] [3d:00.0     ] - task completed in 0:01:42.707920
[2020-12-28 16:35:19,809] [DEBUG   ] [3d:00.0     ] - starting task: fpgasupdate /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4-26895-fw_Release.bin 0000:3d:00.0
[2020-12-28 16:35:19,932] [WARNING ] Update starting. Please do not interrupt.
[2020-12-28 16:35:19,934] [INFO    ] updating from file /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4-26895-fw_Release.bin with size 244864
[2020-12-28 16:35:20,039] [INFO    ] writing to staging area
(100%) [____________________] [244864/244864 bytes][Time:0:00:01.575939]
[2020-12-28 16:35:21,626] [INFO    ] applying update to 0000:3d:00.0
[2020-12-28 16:35:36,247] [DEBUG   ] [MainThread  ] - waiting (0:18:00.853465) for threads: 3d:00.0
[2020-12-28 16:35:36,748] [DEBUG   ] [MainThread  ] - waiting (0:18:00.352268) for threads: 3d:00.0
(100%) [____________________][Time:0:00:43.055355]
[2020-12-28 16:36:04,681] [INFO    ] update of 0000:3d:00.0 complete
[2020-12-28 16:36:04,682] [INFO    ] Secure update OK
[2020-12-28 16:36:04,682] [INFO    ] Total time: 0:00:44.749368
[2020-12-28 16:36:04,702] [DEBUG   ] [3d:00.0     ] - task completed in 0:00:44.892688
[2020-12-28 16:36:05,283] [INFO    ] [MainThread  ] - 1 board updated. A power-cycle is required.
[2020-12-28 16:36:05,284] [INFO    ] [MainThread  ] - super_rsu.pyc completed in: 0:02:28.187105
[2020-12-28 16:36:05,284] [INFO    ] [MainThread  ] - super-rsu exiting with code '0'

#Check the fme with fpgainfo to make sure it is updated
[]$ fpgainfo fme
Board Management Controller, microcontroller FW version 26895
Last Power Down Cause: POK_CORE
Last Reset Cause: None
//****** FME ******//
Object Id                     : 0xEB00000
PCIe s:b:d:f                  : 0000:3D:00:0
Device Id                     : 0x09C4
Socket Id                     : 0x00
Ports Num                     : 01
Bitstream Id                  : 0x124000200000367
Bitstream Version             : 1.2.4
Pr Interface Id               : 38d782e3-b612-5343-b934-2433e348ac4c
Boot Page                     : user

I'm not totally sure why the fpga-otsu command would not originally complete (and then finally completed), but my best guess is that using a slightly different kernel minor version or that restarting the server with cold reboot (powering off the server) helps to reinitialize the FPGA state and devices under /sys/. Note that warm reboots (normal power-cycle) may be causing some weirdness with the FPGA device initialization, which is why I've recommended cold reboots (turning off power completely for 20-30 seconds).

For the original issue with super-rsu my suggested solution is the following:

1) Follow the instructions to run fpga-otsu on pg. 40 of the AFU Quick Start Guide. If it fails, power the server off completely for ~30 seconds (cold reboot), power on, initialize the AFU devstack (`. /opt/inteldevstack/init_env.sh`) and rerun the command until it succeeds.

2) Once fpga-otsu completes, perform a cold reboot again. This command should probably replace Step 2 on pg. 41 that originally suggests to "2. Power cycle the server." which can mean either a warm or cold reboot.

3) Check that the ifpga_sec_mgr module is properly loaded correctly and that the ifpga_sec_mgr device exists. If it does not exist, try a cold reboot and check each time after initializing the AFU devstack.

`ls /sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec_mgr/
ifpga_sec0`

4) If this device exists, then the super-rsu command should complete successfully (or at least fail elsewhere).

EricMunYew_C_Intel

Frequent Contributor

5 years ago

Hi, Young

Can you try to upgrading from 1.2 to 1.2.1 (according to page 39), and follow page 40 for upgrading the FIM and BMC. And you can follow page 46 to change the permission permanently.

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug-qs-ias-v1-2-1.pdf

jyoung

New Contributor

5 years ago

@EricMunYew_C_Intel I tried the fpgaotsu command, and this finished correctly this time. However, I'm back to the initial super-rsu error from my original post.

FPGAOTSU finishes correctly - the only change is the * as far as I can tell so not sure what changed. I am using the exact same AFU stack as previously.

fpgaotsu /usr/share/opae/a10*/fpgaotsu/base/otsu-09C4.json
[2020-12-28 14:40:06,356] [INFO    ] [MainThread] Intel PAC with Intel Arria 10 GX FPGA 0000:3d:00.0 is not secure.
[2020-12-28 14:40:06,368] [WARNING ] [MainThread] Update starting. Please do not interrupt.

...
[2020-12-28 14:49:05,510] [INFO    ] [0000:3d:00.0] Writing flash@0x3000000 for 71200768 bytes (dcp_1_2_1_rot_reversed.rpd)
(100%) [____________________] [71200768/71200768 bytes][Time:0:01:32.556300]
[2020-12-28 14:50:38,162] [INFO    ] [0000:3d:00.0] Reading flash@0x3000000 for 71200768 bytes for verification
(100%) [____________________] [71200768/71200768 bytes][Time:0:01:36.341767]
[2020-12-28 14:52:14,553] [INFO    ] [0000:3d:00.0] Verified flash@0x3000000 for 71200768 bytes (dcp_1_2_1_rot_reversed.rpd)
[2020-12-28 14:52:14,562] [INFO    ] [MainThread] Total time: 0:12:08.193506
[2020-12-28 14:52:14,563] [INFO    ] [MainThread] One-Time Secure Update OK

However, super-rsu does not finish correctly:

super-rsu --log-level trace /usr/share/opae/a10-gx-pac/super-rsu/base/rsu-09c4.json
[2020-12-28 16:14:13,362] [DEBUG   ] [MainThread  ] - found fpga objects: ['/sys/class/fpga/intel-fpga-dev.0']
[2020-12-28 16:14:13,363] [DEBUG   ] [MainThread  ] - found device at 0000:3d:00.0 -tree is
 [pci_address(0000:3a:00.0), pci_id(0x8086, 0x2030)]
    [pci_address(0000:3b:00.0), pci_id(0x10b5, 0x8747)]
        [pci_address(0000:3c:08.0), pci_id(0x10b5, 0x8747)]
            [pci_address(0000:3d:00.0), pci_id(0x8086, 0x09c4)]
        [pci_address(0000:3c:10.0), pci_id(0x10b5, 0x8747)]
            [pci_address(0000:3e:00.0), pci_id(0x198a, 0x385c)]

[2020-12-28 16:14:13,363] [DEBUG   ] [MainThread  ] - could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec_mgr/ifpga_sec*"
[2020-12-28 16:14:13,364] [WARNING ] [MainThread  ] - [0000:3d:00.0] does not support secure update
[2020-12-28 16:14:13,364] [ERROR   ] [MainThread  ] - missing one or more items required by rsu config
[2020-12-28 16:14:13,364] [INFO    ] [MainThread  ] - super-rsu exiting with code '78'

Looking at the working card/server it seems that the modules are the same - ifpga_sec_mgr module is loaded correctly on both systems, but the ifpga_sec_mgr device folder does not exist on the "non-working" server. I don't see any messages in dmesg that indicate any particular errors.

[]$ lsmod | grep ifpga
ifpga_sec_mgr 16384 1 intel_fpga_fme

#Check to see if the ifpga security manager device is loaded.
[]$ ls -all /sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec*
ls: cannot access '/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec*': No such file or directory

I don't see anything that sticks out in dmesg or /var/log/messages with respect to the security manager module.

jyoung

New Contributor

5 years ago

I was finally able to update this using super-rsu after completely shutting off power to the server (cold reboot):

[]$ super-rsu --log-level trace /usr/share/opae/a10-gx-pac/super-rsu/base/rsu-09c4.json
[2020-12-28 16:33:37,086] [DEBUG   ] [MainThread  ] - found fpga objects: ['/sys/class/fpga/intel-fpga-dev.0']
[2020-12-28 16:33:37,088] [DEBUG   ] [MainThread  ] - found device at 0000:3d:00.0 -tree is
 [pci_address(0000:3a:00.0), pci_id(0x8086, 0x2030)]
    [pci_address(0000:3b:00.0), pci_id(0x10b5, 0x8747)]
        [pci_address(0000:3c:08.0), pci_id(0x10b5, 0x8747)]
            [pci_address(0000:3d:00.0), pci_id(0x8086, 0x09c4)]
        [pci_address(0000:3c:10.0), pci_id(0x10b5, 0x8747)]
            [pci_address(0000:3e:00.0), pci_id(0x198a, 0x385c)]

[2020-12-28 16:33:37,096] [WARNING ] [MainThread  ] - Update starting. Please do not interrupt.
[2020-12-28 16:33:37,097] [DEBUG   ] [MainThread  ] - [3d:00.0] version (0x0124000200000367) up to date for sr
[2020-12-28 16:33:37,098] [DEBUG   ] [MainThread  ] - bmc_fw is being force flashed
[2020-12-28 16:33:37,098] [DEBUG   ] [MainThread  ] - bmc_fw versions not equal (system:0x0000000000026889 != manifest:0x0000000000026895)
[2020-12-28 16:33:37,098] [DEBUG   ] [MainThread  ] - bmc_fw versions not equal (system:0x0000000000026889 != manifest:0x0000000000026895)
[2020-12-28 16:33:37,099] [DEBUG   ] [MainThread  ] - [3d:00.0] update timeout set to: 1200.0
[2020-12-28 16:33:37,099] [DEBUG   ] [3d:00.0     ] - update of board at [pci_address(0000:3d:00.0), pci_id(0x8086, 0x09c4)] started
[2020-12-28 16:33:37,099] [DEBUG   ] [MainThread  ] - max timeout set to: 0:20:00
[2020-12-28 16:33:37,100] [DEBUG   ] [3d:00.0     ] - starting task: fpgasupdate /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4_bootloader-26895-fw_Release.bin 0000:3d:00.0
[2020-12-28 16:33:37,222] [WARNING ] Update starting. Please do not interrupt.
[2020-12-28 16:33:37,223] [INFO    ] updating from file /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4_bootloader-26895-fw_Release.bin with size 38016
[2020-12-28 16:33:37,331] [INFO    ] writing to staging area
[2020-12-28 16:34:36,173] [DEBUG   ] [MainThread  ] - waiting (0:19:00.927721) for threads: 3d:00.0
[2020-12-28 16:34:36,674] [DEBUG   ] [MainThread  ] - waiting (0:19:00.426487) for threads: 3d:00.0
(100%) [____________________] [38016/38016 bytes][Time:0:01:34.404933]
[2020-12-28 16:35:11,747] [INFO    ] applying update to 0000:3d:00.0
(100%) [____________________][Time:0:00:08.010363]
[2020-12-28 16:35:19,757] [INFO    ] update of 0000:3d:00.0 complete
[2020-12-28 16:35:19,758] [INFO    ] Secure update OK
[2020-12-28 16:35:19,758] [INFO    ] Total time: 0:01:42.536032
[2020-12-28 16:35:19,809] [DEBUG   ] [3d:00.0     ] - task completed in 0:01:42.707920
[2020-12-28 16:35:19,809] [DEBUG   ] [3d:00.0     ] - starting task: fpgasupdate /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4-26895-fw_Release.bin 0000:3d:00.0
[2020-12-28 16:35:19,932] [WARNING ] Update starting. Please do not interrupt.
[2020-12-28 16:35:19,934] [INFO    ] updating from file /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4-26895-fw_Release.bin with size 244864
[2020-12-28 16:35:20,039] [INFO    ] writing to staging area
(100%) [____________________] [244864/244864 bytes][Time:0:00:01.575939]
[2020-12-28 16:35:21,626] [INFO    ] applying update to 0000:3d:00.0
[2020-12-28 16:35:36,247] [DEBUG   ] [MainThread  ] - waiting (0:18:00.853465) for threads: 3d:00.0
[2020-12-28 16:35:36,748] [DEBUG   ] [MainThread  ] - waiting (0:18:00.352268) for threads: 3d:00.0
(100%) [____________________][Time:0:00:43.055355]
[2020-12-28 16:36:04,681] [INFO    ] update of 0000:3d:00.0 complete
[2020-12-28 16:36:04,682] [INFO    ] Secure update OK
[2020-12-28 16:36:04,682] [INFO    ] Total time: 0:00:44.749368
[2020-12-28 16:36:04,702] [DEBUG   ] [3d:00.0     ] - task completed in 0:00:44.892688
[2020-12-28 16:36:05,283] [INFO    ] [MainThread  ] - 1 board updated. A power-cycle is required.
[2020-12-28 16:36:05,284] [INFO    ] [MainThread  ] - super_rsu.pyc completed in: 0:02:28.187105
[2020-12-28 16:36:05,284] [INFO    ] [MainThread  ] - super-rsu exiting with code '0'

#Check the fme with fpgainfo to make sure it is updated
[]$ fpgainfo fme
Board Management Controller, microcontroller FW version 26895
Last Power Down Cause: POK_CORE
Last Reset Cause: None
//****** FME ******//
Object Id                     : 0xEB00000
PCIe s:b:d:f                  : 0000:3D:00:0
Device Id                     : 0x09C4
Socket Id                     : 0x00
Ports Num                     : 01
Bitstream Id                  : 0x124000200000367
Bitstream Version             : 1.2.4
Pr Interface Id               : 38d782e3-b612-5343-b934-2433e348ac4c
Boot Page                     : user

For the original issue with super-rsu my suggested solution is the following:

4) If this device exists, then the super-rsu command should complete successfully (or at least fail elsewhere).

Forum Discussion

Issues with updating Arria10 PAC for AFU

Recent Discussions

Cyclone 10 GX AS x4 Configuration Compatibility with Infineon S25FS512S (Custom Flash Definition)

Cold Temperature Issue

Thermal model for Max10 FPGA

Drop-in replacement for EPCS64SI16N

ModelSim-Intel FPGA Starter Edition 18.1 exits with code 211 when pressing Restart button