Forum Discussion

attdoner's avatar
attdoner
Icon for New Contributor rankNew Contributor
2 years ago

OFS : Data Path of QSFP from HE_HSSI

Hi Team,

I am using N6001 with Intel OFS (OFS PCIe Attach) . I wanted to test data transfer from Host1 to Host2 as below.
Host1 -> N6001(Port0) -> OtherNIC(Port0) -> Host2.
Is there any sample application available in OPAE/OFS which performs this transaction? Can you please suggest.

In addition to above query, I wants to know how HE-HSSI sends data to QSFP?. The hssi application shows "No eth interface, so not honoring --eth-loopback". ( https://ofs.github.io/ofs-2024.1-1/hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/ Section 7. Send traffic through the 10G AFU). Does this mean the HE_HSSI will not communicate with E-Tile or QSFPs (FIM) ?

Do I need to enable any parameters to pass over QSFPs?

Thankyou.

Ofs

44 Replies

Show More
  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor
    2 years ago

    HI,


    I am working internally on this issue. Once I will keep you posted once I have progress on this.


    • attdoner's avatar
      attdoner
      Icon for New Contributor rankNew Contributor
      2 years ago

      Hi @JohnT_Intel ,

      I have found some updation at Mellanox side, by enabling promiscuous mode, I could see the packets in Mellanox.

      In Mellanox,
      user@user:~$ sudo ifconfig enp1s0f0np0
      enp1s0f0np0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
      ether 0c:42:a1:8c:d5:cc txqueuelen 1000 (Ethernet)
      RX packets 110 bytes 6600 (6.6 KB)
      RX errors 0 dropped 0 overruns 0 frame 0
      TX packets 0 bytes 0 (0 KB)
      TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

      But in N6001, hssi stats I could see some random counts(4040438296505124148,....) in statistics parameters as in image. Can you please let me know what are those values?. Even after hssistats is cleared using "--clear", those values still exist under hssistats.
      In N6001,

      user@user:~$ sudo hssistats
      Namespace(pcie_address=None, clear=False)
      sbdf: 0000:01:00.0
      FPGA dev: {'segment': 0, 'bus': 1, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:01:00.0'}
      args.hssi_grps[('dfl_dev.4', ['/sys/bus/pci/devices/0000:01:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.4/uio/uio1'], '0000:01:00.0', 21)]
      fpga uio dev:dfl_dev.4

      --------HSSI INFO START-------
      DFH :0x3000000010003015
      HSSI ID :0x15
      DFHv :0.5
      guidl :0x99a078ad18418b9d
      guidh :0x4118a7cbd9db4a9b
      HSSI version :2.0
      Firmware Version :16
      HSSI num ports :4
      Port0 :25GbE
      Port1 :25GbE
      Port2 :25GbE
      Port3 :25GbE
      --------HSSI INFO END-------

      HSSI num ports: 4
      HSSI Ports | port:0| port:1| port:2| port:3|

      tx_packets |4040438296505124148|4040438296505124148|4040438296505124148| 110|
      rx_packets |4040438296505124148|4040438296505124148|4040438296505124148| 6|
      rx_crc_errors | 3735929054| 3735929054| 3735929054| 0|
      rx_align_errors | 0| 0| 0| 0|
      tx_payload_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 5060|
      rx_payload_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 950|
      tx_pause |16045693110842147038|16045693110842147038|16045693110842147038| 0|
      rx_pause |16045693110842147038|16045693110842147038|16045693110842147038| 0|
      rx_errors | 3735929054| 3735929054| 3735929054| 0|
      tx_errors | 3735929054| 3735929054| 3735929054| 0|
      rx_unicast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
      rx_multicast |13644642147974742460|13644642147974742460|13644642147974742460| 4|
      rx_broadcast |13644642147974742460|13644642147974742460|13644642147974742460| 2|
      tx_discards | 0| 0| 0| 0|
      tx_unicast |13644642147974742460|13644642147974742460|13644642147974742460| 110|
      tx_multicast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
      tx_broadcast |13644642147974742460|13644642147974742460|13644642147974742460| 0|
      ether_drops | 3735929054| 3735929054| 3735929054| 0|
      rx_total_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 1058|
      rx_total_packets |4040438318920698472|4040438318920698472|4040438318920698472| 6|
      rx_undersize | 3735929054| 3735929054| 3735929054| 0|
      rx_oversize | 3735929054| 3735929054| 3735929054| 0|
      rx_64_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
      rx_65_127_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 4|
      rx_128_255_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
      rx_256_511_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 2|
      rx_512_1023_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
      rx_1024_1518_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
      rx_gte_1519_bytes |16045693110842147038|16045693110842147038|16045693110842147038| 0|
      rx_jabbers | 3735929054| 3735929054| 3735929054| 0|
      rx_runts | 3735929054| 3735929054| 3735929054| 0|

      • attdoner's avatar
        attdoner
        Icon for New Contributor rankNew Contributor
        2 years ago

        Hi @JohnT_Intel ,
        In addition to above queries, the Transfer rate shows throughput nearly 25Gbps for 10G using 'sudo hssi hssi_10g --num-packets 1 '. But when I execute for 100G, I could see error message. Can I know what causes this issue? How to resolve this error?


        $ sudo hssi hssi_100g --num-packets 10

        [hssi_100g] [error] no accelerator found with id: 43425XX6-9XX2-3XX2-bXXa-aXXX4b53XXX2
  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor
    2 years ago

    Hi,


    Are you running " sudo hssistats --clear" to clear the stat? I am able to clear the stat from my side without any issue.


    May I know which image are you using for your board? I suspect that you are using 10G FIM which is why it is not able to detect the AFU GUID for it to run the test. The provided image is 2x4x25GbE.


    You will need to change the FIM design to 100G design


  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor
    2 years ago

    Whether the FIM in this link is 10G ?

    * Do you use other image for 2x4x25GbE?

    Currently the official release image is 2x4x25GbE.

    * Can we know is there any FIM with 100G ?

    I am checking with engineering to see if we have this image.


    • attdoner's avatar
      attdoner
      Icon for New Contributor rankNew Contributor
      2 years ago

      Hi @JohnT_Intel ,

      Thanks for the update.
      I tried with this FIM (https://github.com/otcshare/intel-ofs-fim/releases/download/ofs-3.2.1/FIM_x8_2x1x100G.zip ). Even the error is same.

      [user@user ~]$ sudo fpgainfo phy
      Intel Acceleration Development Platform N6001
      Board Management Controller NIOS FW version: 3.15.1
      Board Management Controller Build version: 3.15.1
      //****** PHY ******//
      Interface : DFL
      Object Id : 0xF000000
      PCIe s:b:d.f : 0000:01:00.0
      Vendor Id : 0x8086
      Device Id : 0xBCCE
      SubVendor Id : 0x8086
      SubDevice Id : 0x1770
      Socket Id : 0x00
      Ports Num : 01
      Bitstream Id : 0x501060255341BA8
      Bitstream Version : 5.0.1
      Pr Interface Id : 98c2df0a-7c39-5bda-9d5b-6a38550cd3d8
      QSFP0 : Not Connected
      QSFP1 : Not Connected
      //****** HSSI information ******//
      HSSI version : 1.0
      Number of ports : 4
      Port0 :100GCAUI-4 UP
      Port4 :100GCAUI-4 UP
      Port8 :100GCAUI-4 DOWN
      Port12 :100GCAUI-4 DOWN


      [user@user ~]$ sudo hssi hssi_100g
      [2024-11-25 11:06:06.438] [hssi_100g] [error] no accelerator found with id: 43425ee6-92b2-4742-b03a-bd8d4a533812




  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor
    2 years ago

    Hi,


    The reason is that particular FIM is having different Accelerator GUID. I am currently checking to see if we have example design for this testing.


    • attdoner's avatar
      attdoner
      Icon for New Contributor rankNew Contributor
      12 months ago

      Hi @JohnT_Intel ,
      Is there any update on example design ?
      In addition to my query I have modified few changes in N6001 AFU, but I am facing mbox_write timeout as below

      sudo hssi hssi_100g

      100G loopback test
      port: 0
      eth_loopback: on
      num_packets: 1
      gap: none
      src_address: 11:22:33:44:55:66
      (bits): 0x112233445566
      dest_address: 77:88:99:aa:bb:cc
      (bits): 0x778899aabbcc
      pattern: random
      start size: 40
      end size: 2580
      end select: pkt_num
      eth:
      continuous mode: off
      monitor duration: 0 sec

      No eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.
      mbox_write timed out [a]

  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor
    12 months ago

    Hi,


    Unfortunately we do not have the design for the 100G. You will need to build the FIM with this command "./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_2x100.ofss n6001 work_n6001"


    Thanks.


    • attdoner's avatar
      attdoner
      Icon for New Contributor rankNew Contributor
      12 months ago

      Hi @JohnT_Intel ,
      Thanks for the update, I will compile and get back to you.

      Meanwhile, with 10g_hssi tests, I could see throughput difference.

      TX Packet Count (N6001)TX Throughput(N6001) GB/s
      125.7812
      5016.0559
      50015.5692
      500015.5638
      5000015.5708
      50000015.5693
      500000015.5692
      958698015.5652
      95869816.96E-07
      100000000.643037
      10000000000.177615

      As packet count increases, the throughput decreases. In above table,
      For Packet Count 1: The TX throughput = 25Gbps
      For Packet Count 50: The TX throughput = 16.05Gbps

      Can I know why the "throughput" values are lesser for higher packet counts?

  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor
    12 months ago

    Hi,


    It will be limited due to traffic generator. When you have too much packet to sent, it is impacting the traffic generator to generate the packet needed.


    • attdoner's avatar
      attdoner
      Icon for New Contributor rankNew Contributor
      12 months ago

      Hi @JohnT_Intel ,

      With your previous reply "
      Unfortunately we do not have the design for the 100G. You will need to build the FIM with this command "./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/hssi/hssi_2x100.ofss n6001 work_n6001"
      "

      I had tried this command , and I face the below issue.

      sudo hssi hssi_100g

      100G loopback test
      port: 0
      eth_loopback: on
      num_packets: 1
      gap: none
      src_address: 11:22:33:44:55:66
      (bits): 0x112233445566
      dest_address: 77:88:99:aa:bb:cc
      (bits): 0x778899aabbcc
      pattern: random
      start size: 40
      end size: 2580
      end select: pkt_num
      eth:
      continuous mode: off
      monitor duration: 0 sec

      No eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.
      mbox_write timed out [a]

  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor
    12 months ago

    Hi,


    I observed that you are sharing .pof file. Can you share .bin file that I can directly load into the board?


  • JohnT_Altera's avatar
    JohnT_Altera
    Icon for Regular Contributor rankRegular Contributor
    12 months ago

    Hi,


    I have try the bin file that you provided and I do not observed the mbox error that you mention.