Forum Discussion

AJ's avatar
AJ
Icon for New Contributor rankNew Contributor
6 years ago

Problem with HIP pcie gen3x16 two functions

Hi everyone,

Operating system: CentOS Linux release 7.4.1708

Base board: ASUS P8Z77-V LK

FPGA: 1SG280LN2F43E2VG

I have problem with pcie gen3x16 x2 (two functions). FPGA card inserted in slot PCIEX16_1 (blue), slot PCIEX16_2 is available (not used).

Base board provides maximum speed and width (gen3x16), all functions are displayed in the status (lspci):

01:00.0 Unassigned class [ff00]: Altera Corporation Device e10a (rev 01)

Subsystem: Device a106:2804

Flags: bus master, fast devsel, latency 0, IRQ 16

Memory at f0000000 (32-bit, non-prefetchable) [size=32M]

Memory at f2000000 (32-bit, non-prefetchable) [size=2M]

Memory at f22c0000 (32-bit, non-prefetchable) [size=64K]

Memory at f2280000 (32-bit, non-prefetchable) [size=256K]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/16 Maskable+ 64bit-

Capabilities: [70] Express Endpoint, MSI 00

Capabilities: [100] Advanced Error Reporting

Capabilities: [148] Virtual Channel

Capabilities: [178] Alternative Routing-ID Interpretation (ARI)

Capabilities: [188] #19

Capabilities: [b80] Vendor Specific Information: ID=1172 Rev=0 Len=05c <?>

Kernel driver in use: dd_board

01:00.1 Unassigned class [ff00]: Adobe Systems, Inc Device e10b (rev 01)

Subsystem: Device a106:2804

Flags: bus master, fast devsel, latency 0, IRQ 16

Memory at f2240000 (32-bit, non-prefetchable) [size=256K]

Memory at f2200000 (32-bit, non-prefetchable) [size=256K]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/16 Maskable+ 64bit-

Capabilities: [70] Express Endpoint, MSI 00

Capabilities: [100] Advanced Error Reporting

Capabilities: [178] Alternative Routing-ID Interpretation (ARI)

Kernel driver in use: dd_board2

No errors in drivers.

The error occurs when addressing the second function - function 1 is always ignored. The problem is in the decoding signal from HIP (rx_st_func_num_o), the signal value is always 0.

In the SignalTap, I can see that the packages for the first function are coming correct: header and data. The headers contain the bar addresses for the first function but rx_st_func_num_o = 0.

Because of this, all requests go to function 0. The project was built in two different software versions (18.0pro, 18.1pro).

The following pcie configurations work without errors (bar settings, drivers (and test soft) and slot (PCIEX16_1) are the same): gen3x16 with one function, gen3x8 with two functions or one function,

gen2x8 with two functions or one function.

Has anyone encountered such a problem?

Intel pcie HIP documentation:

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug_s10_pcie_avst.pdf

Base board documentation (page 33):

https://dlcdnets.asus.com/pub/ASUS/mb/LGA1155/P8Z77-V_LK/E8534_P8Z77-V_LK.pdf?_ga=2.83170837.2056190884.1548158451-500877768.1548158451

Best regards,

Alex

8 Replies

  • SengKok_L_Intel's avatar
    SengKok_L_Intel
    Icon for Regular Contributor rankRegular Contributor
    6 years ago
    Hi Sir, To better understand the problem here, are you facing the problem when using the SRIOV where the function 1 can't access? Difference virtual functions should have a difference BDF. Regards -SK
  • AJ's avatar
    AJ
    Icon for New Contributor rankNew Contributor
    6 years ago

    Hi SK Lim,

    It's true. I have two physical functions and tlp requests always come in zero function. There are two drivers for working with the device:

    dd_board and dd_board2. Driver dd_board only works with ph0 (01:00.0 - e10a), dd_board2 with ph1 (01:00.1 - e10b). In my tests, I only change

    the speed and width. BAR settings, drivers (and test soft) are the same. I wrote about this in detail.

    Best regards,

    Alex

  • AJ's avatar
    AJ
    Icon for New Contributor rankNew Contributor
    6 years ago

    I did new tests on the new PC (host1). First PC - host0.

    Host0:

    Operating system: CentOS Linux release 7.4.1708

    CPU: i7-3770K (host0_cpu file)

    Base board: ASUS P8Z77-V LK

    FPGA: 1SG280LN2F43E2VG

    Host1 (new):

    Operating system: CentOS Linux release 7.6.1810 (Core)

    CPU: i7 950 (host1_cpu file)

    Base board: ASUS P6X58D-E

    FPGA: 1SG280LN2F43E2VG

    The problems are the same. Test for host1 (slot PCIEX16_1(blue)): Gen3x16 (two functions) - failure. Gen2x16 (two functions), Gen3x8 (two functions) - OK.

    New test for host0: Card in PCIEX16_1: Gen2x16 (two functions) - OK, card in PCIEX16_2: Gen3x16 (two functions) - failure, Gen2x16 (two functions) - OK.

    I also attach the full output of pcie statuses for host0 and host1.

    Host1 Base board documentation (page 40):

    https://www.asus.com/Motherboards/P6X58DE/HelpDesk_Manual/

  • SengKok_L_Intel's avatar
    SengKok_L_Intel
    Icon for Regular Contributor rankRegular Contributor
    6 years ago

    ​HI Sir,

    What Quartus version you are using? Could you please try to use the latest version v18.1.1? Are the design can pass PCIe Gen3x8 with two Physical Function?

    Regards -SK

  • AJ's avatar
    AJ
    Icon for New Contributor rankNew Contributor
    6 years ago

    Hi SK Lim,

    The PCIE Gen3x8 with two functions passed the test (18.0). In the previous messages listed the tested settings.

    Explanations for my messages: 18.0 - 18.0.0 Build 219, 18.1 - 18.1.0 Build 222. These versions I use.

    Only one configuration does not pass tests - gen3x16 two functions (Avalon-ST 512). Project was compiled in two versions: 18.0, 18.1.

    Are you looked the hip settings? It's are correct settings?

    I will try to compile in the new version (v18.1.1).

    Best regards,

    Alex

  • AJ's avatar
    AJ
    Icon for New Contributor rankNew Contributor
    6 years ago

    Hi SK Lim,

    I found the cause of the error. Parameter enable_sriov_hwtcl must be equal to 1 (hip settings in hip_1811 file). I compiled the project with the corrected parameter in the latest version (v18.1.1).

    Gen3 x16 configuration passed tests.

    The reason is in the adapter.

    Path: .../hip/altera_pcie_s10_gen3x16_adapter_inst/u_rx_st_if

    Module: u_rx_st_if (altera_pcie_s10_gen3x16_rx_st_if)

    module altera_pcie_s10_gen3x16_rx_st_if

    #(

    parameter pld_rx_parity_ena = "enable",

    parameter enable_sriov_hwtcl = 0

    )(

    ...

    //avst rx interface

    ...

    input logic rx_st_func_num_i,

    ...

    output logic [1:0] rx_st_func_num_o,

    ...

    );

    localparam par_sriov = (pld_rx_parity_ena == "enable" && enable_sriov_hwtcl)? 1 : 0;

    localparam only_sriov = (pld_rx_parity_ena != "enable" && enable_sriov_hwtcl)? 1 : 0;

    localparam only_par = (pld_rx_parity_ena == "enable" && !enable_sriov_hwtcl)? 1 : 0;

    ...

    //async FIFO read interface

    always @ (posedge clk250)..

    begin

    ...

    if (rst_clk250)

    begin

    ...

    rx_st_func_num_o <= '0;

    ...

    end

    else

    begin

    ...

    rx_st_func_num_o <= (par_sriov) ? (rx_fifo_rdreq_q? {rx_fifo_rddata_hi[304], rx_fifo_rddata_lo[304]} : '0) : (only_sriov) ? (rx_fifo_rdreq_q? {rx_fifo_rddata_hi[272], rx_fifo_rddata_lo[272]} : '0) : '0 ;

    ...

    end

    end // always @ (posedge clk250)

    ...

    endmodule

    If enable_sriov_hwtcl = 0 then par_sriov = 0, only_sriov = 0 and rx_st_func_num_o is always zero.

    I have two questions.

    1. Is this a wrong description in the documentation?

    From documentation:

    4.2. Multifunction and SR-IOV System Settings

    Table 13. Multifunction and SR-IOV System Settings

    ...

    Enable SR-IOV Support On/Off When On, the variant supports multiple VFs. When Off, supports PFs only. SR-IOV is only available in H-Tile devices.

    ...

    My device is L-Tile. For non-Gen3x16 configurations, this parameter can be omitted (tested in 17.1.0, 18.0.0, 18.1.0, 18.1.1). For Gen3x16 physical function 1 is not available (enable_sriov_hwtcl = 0).

    My system is built using composition scripts. Adding an HIP instance with 2 functions:

    Before (non-Gen3x16 configurations):

    add_instance HIP altera_pcie_s10_hip_ast

    ...

    set_instance_parameter_value HIP enable_multi_func_hwtcl 1

    set_instance_parameter_value HIP total_pf_count_hwtcl 2

    ...

    After:

    add_instance HIP altera_pcie_s10_hip_ast

    ...

    set_instance_parameter_value HIP enable_multi_func_hwtcl 1

    set_instance_parameter_value HIP enable_sriov_hwtcl 1 <-- FIX Gen3x16

    set_instance_parameter_value HIP total_pf_count_hwtcl 2

    ...

    2. I use the parameter enable_multi_func_hwtcl, its was in 17.1:

    4 Parameters

    ...

    Table 10. System Settings

    ...

    Enable multiple physical functions On/Off When On, you can enable multiple physical functions (PFs) on the Multifunction and SR-IOV System Settings tab. You can also also enable TLP Processing

    Hints (TPH) and Address Translation Services (ATS) on the TPH/ATS Capabilities tab

    ...

    4.2 Multifunction and SR-IOV System Settings

    ...

    Enable SR-IOV Support On/Off When On, the variant supports multiple VFs. When Off, supports PFs only.

    ...

    This parameter is missing in the latest documentation, but I can assign it in the scripts. Do I need to use it?

    The description of SR-IOV is also different. In the latest edition there is a clarification about the tile. Is that the correct description?

    Best regards,

    Alex

  • SengKok_L_Intel's avatar
    SengKok_L_Intel
    Icon for Regular Contributor rankRegular Contributor
    6 years ago

    ​Hi Sir,

    Thank for this update. You are actually using L tile device, and the "multiple physical functions" is actually not available for the L tile device. If you open PCIe GUI for H-tile, you will see there is an option for "enable multiple physical functions". Since this feature is not officially available for L-tile, you probably can hack the lower level file to achieve the same feature, but this is not guarantee that the functional behavior is work as expected.

    Regards -SK