Issues with updating Arria10 PAC for AFU

Hi John,

I should clarify that we have 2x Arria10 PAC cards but they are in two separate servers both running the same version of Linux. I'm not sure that removing one card would change anything here since there is only one card in each server. Is there something else I could try, either just using the 1.2.0 firmware and an older stack or using a latest Github release, if available?

For more information - one card finishes the initial fpgaotsu update but fails at the super-rsu command (as in this post). The second card in a separate server won't even finish the initial fpgaotsu update process. I tried posting on the OPAE Github for this second issue (link here) but have not heard back yet.

[]$ sudo fpgaotsu /usr/share/opae/a10-gx-pac/fpgaotsu/base/otsu-09C4.json
[2020-09-02 21:12:14,385] [INFO    ] [MainThread] Intel PAC with Intel Arria 10 GX FPGA 0000:01:00.0 is not secure.
[2020-09-02 21:12:14,393] [WARNING ] [MainThread] Update starting. Please do not interrupt.
[2020-09-02 21:12:14,394] [INFO    ] [0000:01:00.0] Updating Intel PAC with Intel Arria 10 GX FPGA : 0000:01:00.0
[2020-09-02 21:12:14,396] [INFO    ] [0000:01:00.0] Erasing flash@0x7fd0000 for 196608 bytes
[2020-09-02 21:12:14,591] [INFO    ] [0000:01:00.0] Writing flash@0x7ff0000 for 104 bytes (RushCreek_Release_key_blk2)
(100%) [████████████████████] [104/104 bytes][Time:0:00:00.000068]              
[2020-09-02 21:12:14,593] [INFO    ] [0000:01:00.0] Reading flash@0x7ff0000 for 104 bytes for verification
(100%) [████████████████████] [104/104 bytes][Time:0:00:00.006021]              
[2020-09-02 21:12:14,600] [INFO    ] [0000:01:00.0] Verified flash@0x7ff0000 for 104 bytes (RushCreek_Release_key_blk2)
[2020-09-02 21:12:14,600] [INFO    ] [0000:01:00.0] Erasing flash@0x1800000 for 41943040 bytes
[2020-09-02 21:12:54,742] [INFO    ] [0000:01:00.0] Writing flash@0x1800000 for 41943040 bytes (dcp_1_2_1_rot_reversed.rpd)
(100%) [████████████████████] [41943040/41943040 bytes][Time:0:01:06.238681]    
[2020-09-02 21:14:01,045] [INFO    ] [0000:01:00.0] Reading flash@0x1800000 for 41943040 bytes for verification
(100%) [████████████████████] [41943040/41943040 bytes][Time:0:00:58.330970]    
[2020-09-02 21:14:59,404] [INFO    ] [MainThread] Total time: 0:02:45.010901
[2020-09-02 21:14:59,405] [ERROR   ] [MainThread] Intel PAC with Intel Arria 10 GX FPGA 0000:01:00.0: Verification of dcp_1_2_1_rot_reversed.rpd @0x25165824 failed
[2020-09-02 21:14:59,405] [ERROR   ] [MainThread] One-Time Secure Update failed

Hi @EricMunYew_C_Intel - we are using the AFU Getting Started Guide (UG 20166). I'm not sure if the process has changed with recent OPAE announcements in November, to be honest.

@JohnT_Intel- apologies for the long delay. The student working on this claimed 1 of the boards in "server3" was working ok for his designs. Server2 we've never been able to get working.

Server2 still can't be upgraded to a newer firmware version.

[server2]$sudo fpgainfo fme
Board Management Controller, microcontroller FW version 26889
Last Power Down Cause: POK_CORE
Last Reset Cause: External reset
Power-on-reset
//****** FME ******//
Object Id                     : 0xEC00000
PCIe s:b:d:f                  : 0000:01:00:0
Device Id                     : 0x09C4
Socket Id                     : 0x00
Ports Num                     : 01
Bitstream Id                  : 0x123000200000185
Bitstream Version             : 1.2.3
Pr Interface Id               : 69528db6-eb31-577a-8c36-68f9faa081f6
Boot Page                     : user

Server3 shows the correct bitstream version

[server3]$sudo fpgainfo fme
Board Management Controller, microcontroller FW version 26889
Last Power Down Cause: POK_CORE
Last Reset Cause: External reset
Power-on-reset
//****** FME ******//
Object Id                     : 0xEC00000
PCIe s:b:d:f                  : 0000:89:00:0
Device Id                     : 0x09C4
Socket Id                     : 0x00
Ports Num                     : 01
Bitstream Id                  : 0x124000200000367
Bitstream Version             : 1.2.4
Pr Interface Id               : 38d782e3-b612-5343-b934-2433e348ac4c
Boot Page                     : user

Hi @EricMunYew_C_Intel - answers inline below.

Have you tried on Ubuntu 16.04, kernel version 4.4 ?

• No, however the accleration stack user guide (UG 20166) indicates that the validated release is Ubuntu 18.04, kernel 4.15. It's unfortunately not feasible to reload the server just to test this since it is a production machine.

Do you have 48 GB of free memory ?

• Yes. This server has 384 GB of RAM.

Did you manually install OPAE ?

• No. We installed it from the 1.2.1 acceleration stack for development from this Intel site.

Which document you followed for upgrading to 1.2.1 ?

• We started directly with the 1.2.1 acceleration stack from the Intel site and worked from the acceleration stack user guide (UG 20166) in Appendix A to do the firmware upgrade.

Are you running virtual machine or non-virtual machine ?

• This is a standard Cascade Lake server with SuperMicro MB (server model SYS-4029GP-TRT with X11DPG-OT-CPU MB).

Can you check if your server is in the list below:

https://www.intel.com/content/www/us/en/programmable/products/boards_and_kits/dev-kits/altera/accele...

• It is not but we have specific vendors we have to purchase our servers from at our institution.

Did you get your bdf of your Intel PAC card correctly during upgrade,

lspci | grep 09c4

Output:

d8:00.0 Processing accelerators: Intel Corporation Device 09c4

./setup_fim_and_bmc -b <bus id> -d <device id> -f <function id> -p $OPAE_PLATFORM_ROOT

bus id d8

device id 00

function id 0

• I don't see this command (setup_fim_and_bmc) in the 1.2.1 acceleration stack release as it seems to be specific to the 1.2 acceleration stack release (it shows up in an older tarball as a10_gx_pac_ias_1_2_pv_rte_installer/components/setup_fim_and_bmc.sh). Section 5 of the user guide indicates we correctly have the 1.2 firmware, and Appendix A.1 seems to indicate we should not use this script to do a firmware upgrade since we already have the 1.2 firmware installed. Is there a benefit to downloading this additional script?

May I know where did you get fpgaotsu and super-rsu from ?

• These are from the standard 1.2.1 acceleration stack for development from this Intel site.

@EricMunYew_C_Intel I tried the fpgaotsu command, and this finished correctly this time. However, I'm back to the initial super-rsu error from my original post.

FPGAOTSU finishes correctly - the only change is the * as far as I can tell so not sure what changed. I am using the exact same AFU stack as previously.

fpgaotsu /usr/share/opae/a10*/fpgaotsu/base/otsu-09C4.json
[2020-12-28 14:40:06,356] [INFO    ] [MainThread] Intel PAC with Intel Arria 10 GX FPGA 0000:3d:00.0 is not secure.
[2020-12-28 14:40:06,368] [WARNING ] [MainThread] Update starting. Please do not interrupt.

...
[2020-12-28 14:49:05,510] [INFO    ] [0000:3d:00.0] Writing flash@0x3000000 for 71200768 bytes (dcp_1_2_1_rot_reversed.rpd)
(100%) [____________________] [71200768/71200768 bytes][Time:0:01:32.556300]
[2020-12-28 14:50:38,162] [INFO    ] [0000:3d:00.0] Reading flash@0x3000000 for 71200768 bytes for verification
(100%) [____________________] [71200768/71200768 bytes][Time:0:01:36.341767]
[2020-12-28 14:52:14,553] [INFO    ] [0000:3d:00.0] Verified flash@0x3000000 for 71200768 bytes (dcp_1_2_1_rot_reversed.rpd)
[2020-12-28 14:52:14,562] [INFO    ] [MainThread] Total time: 0:12:08.193506
[2020-12-28 14:52:14,563] [INFO    ] [MainThread] One-Time Secure Update OK

However, super-rsu does not finish correctly:

super-rsu --log-level trace /usr/share/opae/a10-gx-pac/super-rsu/base/rsu-09c4.json
[2020-12-28 16:14:13,362] [DEBUG   ] [MainThread  ] - found fpga objects: ['/sys/class/fpga/intel-fpga-dev.0']
[2020-12-28 16:14:13,363] [DEBUG   ] [MainThread  ] - found device at 0000:3d:00.0 -tree is
 [pci_address(0000:3a:00.0), pci_id(0x8086, 0x2030)]
    [pci_address(0000:3b:00.0), pci_id(0x10b5, 0x8747)]
        [pci_address(0000:3c:08.0), pci_id(0x10b5, 0x8747)]
            [pci_address(0000:3d:00.0), pci_id(0x8086, 0x09c4)]
        [pci_address(0000:3c:10.0), pci_id(0x10b5, 0x8747)]
            [pci_address(0000:3e:00.0), pci_id(0x198a, 0x385c)]

[2020-12-28 16:14:13,363] [DEBUG   ] [MainThread  ] - could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec_mgr/ifpga_sec*"
[2020-12-28 16:14:13,364] [WARNING ] [MainThread  ] - [0000:3d:00.0] does not support secure update
[2020-12-28 16:14:13,364] [ERROR   ] [MainThread  ] - missing one or more items required by rsu config
[2020-12-28 16:14:13,364] [INFO    ] [MainThread  ] - super-rsu exiting with code '78'

Looking at the working card/server it seems that the modules are the same - ifpga_sec_mgr module is loaded correctly on both systems, but the ifpga_sec_mgr device folder does not exist on the "non-working" server. I don't see any messages in dmesg that indicate any particular errors.

[]$ lsmod | grep ifpga
ifpga_sec_mgr 16384 1 intel_fpga_fme

#Check to see if the ifpga security manager device is loaded.
[]$ ls -all /sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec*
ls: cannot access '/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec*': No such file or directory

I don't see anything that sticks out in dmesg or /var/log/messages with respect to the security manager module.