OFS : Data Path of QSFP from HE_HSSI

Hi @JohnT_Intel ,
Is there any update on example design ?
In addition to my query I have modified few changes in N6001 AFU, but I am facing mbox_write timeout as below

sudo hssi hssi_100g

100G loopback test
port: 0
eth_loopback: on
num_packets: 1
gap: none
src_address: 11:22:33:44:55:66
(bits): 0x112233445566
dest_address: 77:88:99:aa:bb:cc
(bits): 0x778899aabbcc
pattern: random
start size: 40
end size: 2580
end select: pkt_num
eth:
continuous mode: off
monitor duration: 0 sec

No eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.
mbox_write timed out [a]

Hi @JohnT_Intel ,

Thanks for the update.
I tried with this FIM (https://github.com/otcshare/intel-ofs-fim/releases/download/ofs-3.2.1/FIM_x8_2x1x100G.zip ). Even the error is same.

[user@user ~]$ sudo fpgainfo phy
Intel Acceleration Development Platform N6001
Board Management Controller NIOS FW version: 3.15.1
Board Management Controller Build version: 3.15.1
//****** PHY ******//
Interface : DFL
Object Id : 0xF000000
PCIe s:b:d.f : 0000:01:00.0
Vendor Id : 0x8086
Device Id : 0xBCCE
SubVendor Id : 0x8086
SubDevice Id : 0x1770
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x501060255341BA8
Bitstream Version : 5.0.1
Pr Interface Id : 98c2df0a-7c39-5bda-9d5b-6a38550cd3d8
QSFP0 : Not Connected
QSFP1 : Not Connected
//****** HSSI information ******//
HSSI version : 1.0
Number of ports : 4
Port0 :100GCAUI-4 UP
Port4 :100GCAUI-4 UP
Port8 :100GCAUI-4 DOWN
Port12 :100GCAUI-4 DOWN

[user@user ~]$ sudo hssi hssi_100g
[2024-11-25 11:06:06.438] [hssi_100g] [error] no accelerator found with id: 43425ee6-92b2-4742-b03a-bd8d4a533812

Hi @JohnT_Intel ,

I have tried " sudo hssistats --clear" still the values used to be there.

We are using https://github.com/otcshare/intel-ofs-fim/releases/tag/ofs-4.1.1.
Yes, Ethernet configurations: 2x4x25GbE.

* Whether the FIM in this link is 10G ?

* Do you use other image for 2x4x25GbE?
* Can we know is there any FIM with 100G ?

Hi @JohnT_Intel ,

I have found some updation at Mellanox side, by enabling promiscuous mode, I could see the packets in Mellanox.

In Mellanox,
user@user:~$ sudo ifconfig enp1s0f0np0
enp1s0f0np0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
ether 0c:42:a1:8c:d5:cc txqueuelen 1000 (Ethernet)
RX packets 110 bytes 6600 (6.6 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

But in N6001, hssi stats I could see some random counts(4040438296505124148,....) in statistics parameters as in image. Can you please let me know what are those values?. Even after hssistats is cleared using "--clear", those values still exist under hssistats.
In N6001,

user@user:~$ sudo hssistats
Namespace(pcie_address=None, clear=False)
sbdf: 0000:01:00.0
FPGA dev: {'segment': 0, 'bus': 1, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:01:00.0'}
args.hssi_grps[('dfl_dev.4', ['/sys/bus/pci/devices/0000:01:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.4/uio/uio1'], '0000:01:00.0', 21)]
fpga uio dev:dfl_dev.4

--------HSSI INFO START-------
DFH :0x3000000010003015
HSSI ID :0x15
DFHv :0.5
guidl :0x99a078ad18418b9d
guidh :0x4118a7cbd9db4a9b
HSSI version :2.0
Firmware Version :16
HSSI num ports :4
Port0 :25GbE
Port1 :25GbE
Port2 :25GbE
Port3 :25GbE
--------HSSI INFO END-------

HSSI num ports: 4
HSSI Ports | port:0| port:1| port:2| port:3|

tx_packets |4040438296505124148|4040438296505124148|4040438296505124148| 110|
rx_packets |4040438296505124148|4040438296505124148|4040438296505124148| 6|
rx_crc_errors | 3735929054| 3735929054| 3735929054| 0|
rx_align_errors | 0| 0| 0| 0|
tx_payload_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 5060|
rx_payload_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 950|
tx_pause |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_pause |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_errors | 3735929054| 3735929054| 3735929054| 0|
tx_errors | 3735929054| 3735929054| 3735929054| 0|
rx_unicast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
rx_multicast |13644642147974742460|13644642147974742460|13644642147974742460| 4|
rx_broadcast |13644642147974742460|13644642147974742460|13644642147974742460| 2|
tx_discards | 0| 0| 0| 0|
tx_unicast |13644642147974742460|13644642147974742460|13644642147974742460| 110|
tx_multicast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
tx_broadcast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
ether_drops | 3735929054| 3735929054| 3735929054| 0|
rx_total_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 1058|
rx_total_packets |4040438318920698472|4040438318920698472|4040438318920698472| 6|
rx_undersize | 3735929054| 3735929054| 3735929054| 0|
rx_oversize | 3735929054| 3735929054| 3735929054| 0|
rx_64_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_65_127_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 4|
rx_128_255_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_256_511_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 2|
rx_512_1023_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_1024_1518_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_gte_1519_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_jabbers | 3735929054| 3735929054| 3735929054| 0|
rx_runts | 3735929054| 3735929054| 3735929054| 0|

Hi @JohnT_Intel

[user@user ~]$ sudo fpgainfo phy -B 0x8
Intel Acceleration Development Platform N6001
Board Management Controller NIOS FW version: 131.15.0
Board Management Controller Build version: 3.15.0
//****** PHY ******//
Interface : DFL
Object Id : 0xF100000
PCIe s:b:d.f : 0000:08:00.0
Vendor Id : 0x8086
Device Id : 0xBCCE
SubVendor Id : 0x8086
SubDevice Id : 0x1771
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x5010202FB22E12D
Bitstream Version : 5.0.1
Pr Interface Id : 8b153d79-5e54-5416-8096-25480eb9b02f
QSFP0 : Connected
QSFP1 : Connected
//****** HSSI information ******//
HSSI version : 1.0
Number of ports : 8
Port0 :25GbE UP
Port1 :25GbE UP
Port2 :25GbE UP
Port3 :25GbE UP
Port4 :25GbE UP
Port5 :25GbE UP
Port6 :25GbE UP
Port7 :25GbE UP
[user@user ~]$

[user@user ~]$ sudo hssiloopback --pcie-address 0000:08:00.0 --loopback disable
args Namespace(loopback='disable', ncsi_ch_sel=None, pcie_address='0000:08:00.0', port=0)
sbdf: 0000:08:00.0
FPGA dev: {'segment': 0, 'bus': 8, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:08:00.0'}
args.hssi_grps[('dfl_dev.3', ['/sys/bus/pci/devices/0000:08:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.3/uio/uio1'], '0000:08:00.0', 21)]
fpga uio dev:dfl_dev.3

--------HSSI INFO START-------
DFH :0x3000000010002015
HSSI ID :0x15
DFHv :0.5
guidl :0x99a078ad18418b9d
guidh :0x4118a7cbd9db4a9b
HSSI version :1.0
Firmware Version :1
HSSI num ports :8
Port0 :25GbE
Port1 :25GbE
Port2 :25GbE
Port3 :25GbE
Port4 :25GbE
Port5 :25GbE
Port6 :25GbE
Port7 :25GbE
--------HSSI INFO END-------

hssi loopback disabled to port0
[user@user ~]$

Hi @JohnT_Intel ,

Testcase1: In Loopback mode: Server1

Here, In server1 I am using HSSI application to send packets. I could see the packets received using hssistats in same Server1. I could see out the values of TX and RX packet counts in hssistats application. This testcase works.

Testcase2: In External Board to Board mode: Server1 -> Server2

Here, In server1 I am using HSSI application to send packets. In server2 I am using normal ifconfig. I could see no packets received in Server2. I could see out the values of TX counts in hssistats application in Server1. Even executed tcpdump in server2. This testcase doesnot work.

Let me know is there anyway I can show that HSSI sending packets to other cards. Since I have only one N6001 and a Mellanox card.

Hi @JohnT_Intel ,

I am connecting N6001 in Host1 and Mellanox NIC in Host2.

In Host1:
Application : HSSI, HSSISTATS

In Host2:
Application : ifconfig port0(interface), tcpdump on port0(interface). -- interface handled using Mellanox driver.

Its not normal ethernet (RJ45). I am using QSFP module on both the end devices connected via MTP cable. (Optical cable)

Hi @JohnT_Intel ,
Thanks for your response.

I had even tried adding destination address using --dst-addr in hssi applicaition. Still I was not able to detect the packet increment count in ifconfig.

Host1:

$ sudo hssi hssi_10g --eth-loopback off --num-packets 50 --src-addr 11:22:33:44:55:66 --dest-addr 04:3f:72:a4:11:bc --src-port 0
10G loopback test
Tx/Rx port: 99
Tx port: 0
Rx port: 0
eth_loopback: off
he_loopback: none
num_packets: 50
packet_length: 64
src_address: 11:22:33:44:55:66
(bits): 0x665544332211
dest_address: 04:3f:72:a4:11:bc
(bits): 0xbc11a4723f04
random_length: fixed
random_payload: incremental
rnd_seed0: 5eed0000
rnd_seed1: 5eed0001
rnd_seed2: 25eed
eth:

No eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.
0x40000 ETH_AFU_DFH: 0x1000010000001000
0x40008 ETH_AFU_ID_L: 0xbb370242ac130002
0x40010 ETH_AFU_ID_H: 0x823c334c98bf11ea
0x40030 TRAFFIC_CTRL_CMD: 0x0000000000000000
0x40038 TRAFFIC_CTRL_DATA: 0x0000000100000000
0x40040 TRAFFIC_CTRL_PORT_SEL: 0x0000000000000000
0x40048 AFU_SCRATCHPAD: 0x0000000045324511

0x3c00 number_packets: 0x00000032
0x3c01 random_length: 0x00000000
0x3c02 random_payload: 0x00000000
0x3c03 start: 0x00000000
0x3c04 stop: 0x00000000
0x3c05 source_addr0: 0x44332211
0x3c06 source_addr1: 0x00006655
0x3c07 dest_addr0: 0xa4723f04
0x3c08 dest_addr1: 0x0000bc11
0x3c09 packet_tx_count: 0x00000032
0x3c0a rnd_seed0: 0x5eed0000
0x3c0b rnd_seed1: 0x5eed0001
0x3c0c rnd_seed2: 0x00025eed
0x3c0d pkt_length: 0x00000040
0x3cf4 tx_sta_tstamp: 0x029238c8
0x3cf5 tx_end_tstamp: 0x02923a90
0x3d00 num_pkt: 0xffffffff
0x3d01 pkt_good: 0x00000000
0x3d02 pkt_bad: 0x00000000
0x3d07 avst_rx_err: 0x00000000
0x3d0b rx_sta_tstamp: 0x029427da
0x3d0c rx_end_tstamp: 0xaa91bc27
0x3e00 mac_loop: 0x00000000

HSSI performance:
Selected clock frequency : 402.832 MHz
Latency minimum : 529638 ns
Latency maximum : 6.99681e+09 ns
Achieved Tx throughput : 19.7882 GB/s

No eth interface, so not showing stats. Try using the hssistats command instead.

In Host2,

user@user-OptiPlex-7080:~$ sudo ifconfig enp1s0f0np0
enp1s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::694b:f9a2:b595:f22b prefixlen 64 scopeid 0x20<link>
ether 04:3f:72:a4:11:bc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Hi @JohnT_Intel ,
Yes I had disabled using --eth-loopback=off. Below logs contains interface stats after executing hssi commands.

In Host1 where N6001 is connected,
Log: "sudo hssi hssi_10g --eth-loopback off --num-packets 100"

log: sudo hssistats

In Host2 where Other NIC,
user@user-OptiPlex-7080:~$ sudo ifconfig enp1s0f0np0
enp1s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 04:3f:72:a4:11:bc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Host2 (NIC2) is validated with other tests and it receives data. But HE-HSSI applications the packet counts are 0

Hi @JohnT_Intel ,

Sorry for the inconvenience.
I was out of Office for many days.
I was testing these HSSI applications with --eth-loopback off parameter. But I could not see any packets transferred/received in another PC(where another NIC is connected). I was checking ifconfig <nic-interface> | grep 'RX\ packets' in another PC to know whether packets are received or not.
Is there any other applications/method I can validate that the packets moves out of QSFP1 of N6001 to QSFP2 of other NIC?

Thanks and regards,
Ayshathul Thuhara