PCIe functions

In PCI, any device is addressed by a tuple of Bus–Device–Function. In original PCI there is a close relation between these three terms (Bus = the wires connecting your 4 PCI slots, Device = the chip on one of these extension boards, Function = one logical entity on this device) to the actual physical layout on the board. To some extent, this changed with PCIe as there are rarely multiple devices on a bus, so each PCIe link builds a separate bus. Today, only internal on-chip busses in the north/south bridge are enumerated as a bus with multiple devices.

Here is a »lspci -t« output from my computer:

--+-00.0
           +-02.0---+-00.0
           |            \-00.1
           +-09.0-----00.0
           +-0a.0-----00.0
           +-11.0
           +-12.0
           +-12.2
           +-13.0
           +-13.2
           +-14.0
           +-14.2
           +-14.3
           +-14.4---
           +-14.5
           +-15.0-----00.0
           +-15.1-----00.0
           +-15.2---
           +-15.3---
           +-16.0
           +-16.2
           +-18.0
           +-18.1
           +-18.2
           +-18.3
           +-18.4
           \-18.5

Here, the bus 0 is the internal bus inside my 890FX chipset, all devices on this bus are internal to it. Some of these devices (notably 02.0, 09.0, 0a.0, 14.4, 15.0, 15.1, 15.2, 15.3) are bridges to other PCI busses (see the bus numbers in brackets behind them). Some of these bridges have no devices connected to them (notably 14.4, 15.2 and 15.3), so these busses end in "--". All the other busses connect a single device, each, and only one of these devices, the one on bus 1, claims to integrate two functions, 01:00.0 and 01:00.1. Note from the following list that this is my nVidia Graphics card integrating both a graphics card and an audio adapter, used for HDMI audio output.

00:00.0 Host bridge: Advanced Micro Devices  nee ATI RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)
00:02.0 PCI bridge: Advanced Micro Devices  nee ATI RD890 PCI to PCI bridge (PCI express gpp port B)
00:09.0 PCI bridge: Advanced Micro Devices  nee ATI RD890 PCI to PCI bridge (PCI express gpp port H)
00:0a.0 PCI bridge: Advanced Micro Devices  nee ATI RD890 PCI to PCI bridge (external gfx1 port A)
00:11.0 SATA controller: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 SATA Controller  (rev 40)
00:12.0 USB controller: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices  nee ATI SBx00 SMBus Controller (rev 42)
00:14.2 Audio device: Advanced Micro Devices  nee ATI SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices  nee ATI SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices  nee ATI SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
00:15.1 PCI bridge: Advanced Micro Devices  nee ATI SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1)
00:15.2 PCI bridge: Advanced Micro Devices  nee ATI SB900 PCI to PCI bridge (PCIE port 2)
00:15.3 PCI bridge: Advanced Micro Devices  nee ATI SB900 PCI to PCI bridge (PCIE port 3)
00:16.0 USB controller: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices  nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices  Family 15h Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices  Family 15h Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices  Family 15h Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices  Family 15h Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices  Family 15h Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices  Family 15h Processor Function 5
01:00.0 VGA compatible controller: NVIDIA Corporation GF116  (rev a1)
01:00.1 Audio device: NVIDIA Corporation GF116 High Definition Audio Controller (rev a1)
02:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
03:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
06:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

The purpose of integrating multiple functions is that you can, as the name suggests, integrate two PCI function in one device. This is actually just one option of achieving this goal. If I were nvidia, I could build up a graphics card with integrated audio in the following different ways:

1. Route two PCIe links to my single graphics chip and integrate the graphics core on one link and audio functions on the other link. This is currently not possible on a PCIe connector and requires quite some added more resources in both the uplink switch and the device itself, additionally each such PCIe endpoint could only use its link’s lanes for data exchange. So even if the data rate for the audio function would be low, it would consume at least a full x1 PCIe link, wasting quite some bandwidth.

2. Integrate both cores as a hybrid PCI device. This would result in a bad architecture, as one would have to write a monolithic OS driver for both functions.

3. Declare the device as a multi-function PCI device. This is clearly the best of the solutions: Just one PCIe core is required, the bandwidth on the lanes is available to any function, only one PCIe link/port is consumed on the switch, separate drivers for each function on the OS level.

To implement multiple functions in your design (looks like a series V chip to me), consult the respective PCIe handbook. The interface typically transports a separate indication stating which function was addressed by an incoming TLP, on Arria V this signal is called rx_bar_dec_func_num. On TLP transmission, the function indication is given explicitly in the TLP and does not need additional signalling. A change is also required for interfacing to the configuration space like decoding the data from the Transaction Layer Configuration Space bus. Watch out for tl_cfg_add[6:4] representing the function number and the wide tl_cfg_sts array, possibly others.

– Matthias