Forum Discussion
OFS : Data Path of QSFP from HE_HSSI
Hi Team,
I am using N6001 with Intel OFS (OFS PCIe Attach) . I wanted to test data transfer from Host1 to Host2 as below.
Host1 -> N6001(Port0) -> OtherNIC(Port0) -> Host2.
Is there any sample application available in OPAE/OFS which performs this transaction? Can you please suggest.
In addition to above query, I wants to know how HE-HSSI sends data to QSFP?. The hssi application shows "No eth interface, so not honoring --eth-loopback". ( https://ofs.github.io/ofs-2024.1-1/hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/ Section 7. Send traffic through the 10G AFU). Does this mean the HE_HSSI will not communicate with E-Tile or QSFPs (FIM) ?
Do I need to enable any parameters to pass over QSFPs?
Thankyou.
44 Replies
- JohnT_Altera
Regular Contributor
HI,
I am working internally on this issue. Once I will keep you posted once I have progress on this.
- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
I have found some updation at Mellanox side, by enabling promiscuous mode, I could see the packets in Mellanox.In Mellanox,
user@user:~$ sudo ifconfig enp1s0f0np0
enp1s0f0np0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
ether 0c:42:a1:8c:d5:cc txqueuelen 1000 (Ethernet)
RX packets 110 bytes 6600 (6.6 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
But in N6001, hssi stats I could see some random counts(4040438296505124148,....) in statistics parameters as in image. Can you please let me know what are those values?. Even after hssistats is cleared using "--clear", those values still exist under hssistats.
In N6001,user@user:~$ sudo hssistats
Namespace(pcie_address=None, clear=False)
sbdf: 0000:01:00.0
FPGA dev: {'segment': 0, 'bus': 1, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:01:00.0'}
args.hssi_grps[('dfl_dev.4', ['/sys/bus/pci/devices/0000:01:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.4/uio/uio1'], '0000:01:00.0', 21)]
fpga uio dev:dfl_dev.4--------HSSI INFO START-------
DFH :0x3000000010003015
HSSI ID :0x15
DFHv :0.5
guidl :0x99a078ad18418b9d
guidh :0x4118a7cbd9db4a9b
HSSI version :2.0
Firmware Version :16
HSSI num ports :4
Port0 :25GbE
Port1 :25GbE
Port2 :25GbE
Port3 :25GbE
--------HSSI INFO END-------HSSI num ports: 4
HSSI Ports | port:0| port:1| port:2| port:3|tx_packets |4040438296505124148|4040438296505124148|4040438296505124148| 110|
rx_packets |4040438296505124148|4040438296505124148|4040438296505124148| 6|
rx_crc_errors | 3735929054| 3735929054| 3735929054| 0|
rx_align_errors | 0| 0| 0| 0|
tx_payload_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 5060|
rx_payload_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 950|
tx_pause |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_pause |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_errors | 3735929054| 3735929054| 3735929054| 0|
tx_errors | 3735929054| 3735929054| 3735929054| 0|
rx_unicast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
rx_multicast |13644642147974742460|13644642147974742460|13644642147974742460| 4|
rx_broadcast |13644642147974742460|13644642147974742460|13644642147974742460| 2|
tx_discards | 0| 0| 0| 0|
tx_unicast |13644642147974742460|13644642147974742460|13644642147974742460| 110|
tx_multicast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
tx_broadcast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
ether_drops | 3735929054| 3735929054| 3735929054| 0|
rx_total_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 1058|
rx_total_packets |4040438318920698472|4040438318920698472|4040438318920698472| 6|
rx_undersize | 3735929054| 3735929054| 3735929054| 0|
rx_oversize | 3735929054| 3735929054| 3735929054| 0|
rx_64_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_65_127_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 4|
rx_128_255_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_256_511_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 2|
rx_512_1023_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_1024_1518_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_gte_1519_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
rx_jabbers | 3735929054| 3735929054| 3735929054| 0|
rx_runts | 3735929054| 3735929054| 3735929054| 0|- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
In addition to above queries, the Transfer rate shows throughput nearly 25Gbps for 10G using 'sudo hssi hssi_10g --num-packets 1 '. But when I execute for 100G, I could see error message. Can I know what causes this issue? How to resolve this error?
$ sudo hssi hssi_100g --num-packets 10[hssi_100g] [error] no accelerator found with id: 43425XX6-9XX2-3XX2-bXXa-aXXX4b53XXX2
- JohnT_Altera
Regular Contributor
Hi,
Are you running " sudo hssistats --clear" to clear the stat? I am able to clear the stat from my side without any issue.
May I know which image are you using for your board? I suspect that you are using 10G FIM which is why it is not able to detect the AFU GUID for it to run the test. The provided image is 2x4x25GbE.
You will need to change the FIM design to 100G design
- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
I have tried " sudo hssistats --clear" still the values used to be there.
We are using https://github.com/otcshare/intel-ofs-fim/releases/tag/ofs-4.1.1.
Yes, Ethernet configurations: 2x4x25GbE.* Whether the FIM in this link is 10G ?
* Do you use other image for 2x4x25GbE?
* Can we know is there any FIM with 100G ?
- JohnT_Altera
Regular Contributor
Whether the FIM in this link is 10G ?
* Do you use other image for 2x4x25GbE?
Currently the official release image is 2x4x25GbE.
* Can we know is there any FIM with 100G ?
I am checking with engineering to see if we have this image.
- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
Thanks for the update.
I tried with this FIM (https://github.com/otcshare/intel-ofs-fim/releases/download/ofs-3.2.1/FIM_x8_2x1x100G.zip ). Even the error is same.[user@user ~]$ sudo fpgainfo phy
Intel Acceleration Development Platform N6001
Board Management Controller NIOS FW version: 3.15.1
Board Management Controller Build version: 3.15.1
//****** PHY ******//
Interface : DFL
Object Id : 0xF000000
PCIe s:b:d.f : 0000:01:00.0
Vendor Id : 0x8086
Device Id : 0xBCCE
SubVendor Id : 0x8086
SubDevice Id : 0x1770
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x501060255341BA8
Bitstream Version : 5.0.1
Pr Interface Id : 98c2df0a-7c39-5bda-9d5b-6a38550cd3d8
QSFP0 : Not Connected
QSFP1 : Not Connected
//****** HSSI information ******//
HSSI version : 1.0
Number of ports : 4
Port0 :100GCAUI-4 UP
Port4 :100GCAUI-4 UP
Port8 :100GCAUI-4 DOWN
Port12 :100GCAUI-4 DOWN
[user@user ~]$ sudo hssi hssi_100g
[2024-11-25 11:06:06.438] [hssi_100g] [error] no accelerator found with id: 43425ee6-92b2-4742-b03a-bd8d4a533812
- JohnT_Altera
Regular Contributor
Hi,
The reason is that particular FIM is having different Accelerator GUID. I am currently checking to see if we have example design for this testing.
- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
Is there any update on example design ?
In addition to my query I have modified few changes in N6001 AFU, but I am facing mbox_write timeout as belowsudo hssi hssi_100g
100G loopback test
port: 0
eth_loopback: on
num_packets: 1
gap: none
src_address: 11:22:33:44:55:66
(bits): 0x112233445566
dest_address: 77:88:99:aa:bb:cc
(bits): 0x778899aabbcc
pattern: random
start size: 40
end size: 2580
end select: pkt_num
eth:
continuous mode: off
monitor duration: 0 secNo eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.
mbox_write timed out [a]
- JohnT_Altera
Regular Contributor
Hi,
Unfortunately we do not have the design for the 100G. You will need to build the FIM with this command "./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_2x100.ofss n6001 work_n6001"
Thanks.
- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
Thanks for the update, I will compile and get back to you.
Meanwhile, with 10g_hssi tests, I could see throughput difference.TX Packet Count (N6001) TX Throughput(N6001) GB/s 1 25.7812 50 16.0559 500 15.5692 5000 15.5638 50000 15.5708 500000 15.5693 5000000 15.5692 9586980 15.5652 9586981 6.96E-07 10000000 0.643037 1000000000 0.177615 As packet count increases, the throughput decreases. In above table,
For Packet Count 1: The TX throughput = 25Gbps
For Packet Count 50: The TX throughput = 16.05GbpsCan I know why the "throughput" values are lesser for higher packet counts?
- JohnT_Altera
Regular Contributor
Hi,
It will be limited due to traffic generator. When you have too much packet to sent, it is impacting the traffic generator to generate the packet needed.
- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
With your previous reply "
Unfortunately we do not have the design for the 100G. You will need to build the FIM with this command "./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_2x100.ofss n6001 work_n6001"
"I had tried this command , and I face the below issue.
sudo hssi hssi_100g
100G loopback test
port: 0
eth_loopback: on
num_packets: 1
gap: none
src_address: 11:22:33:44:55:66
(bits): 0x112233445566
dest_address: 77:88:99:aa:bb:cc
(bits): 0x778899aabbcc
pattern: random
start size: 40
end size: 2580
end select: pkt_num
eth:
continuous mode: off
monitor duration: 0 secNo eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.
mbox_write timed out [a]
- JohnT_Altera
Regular Contributor
Hi,
Can you share with me your bitstream?
- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
I had attahced the bitstream generated using this command "./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_2x100.ofss n6001 work_n6001"https://drive.google.com/file/d/1Vz35lGX3VPandzAgsU8NGwGR-9Z6ItOw/view?usp=sharing
Please let us know any update on this
- JohnT_Altera
Regular Contributor
Hi,
I observed that you are sharing .pof file. Can you share .bin file that I can directly load into the board?
- attdoner
Occasional Contributor
- omprasad
New Contributor
Subscribing for the same
- JohnT_Altera
Regular Contributor
Hi,
I have try the bin file that you provided and I do not observed the mbox error that you mention.
- attdoner
Occasional Contributor
Hi @JohnT_Intel ,
Can you please share the output of 100g using below command
sudo hssi hssi_100g