Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
17 years ago

Ethernet question

I'm new to Ethernet interface and I have a few questions regarding my next project.

My requirements are to send data over ethernet from custom FPGA board to a PC over dedicated link. The throughput requirements are roughly 800Mbits/s. I need to receive commands other this link as well, but @ much lower rates.

I'm planning to use external 88E1111 RGMII module and TSE MAC on the FPGA side. From what I've researched so far, I need to implement a transport stack (either TCP or UDP). Also, from what I've read, SOPC approach with NIOS will yield ~110 Mbits/s at best. Is there any reference design on how to implement TCP/UDP in hardware? Will implementing these protocols in hardware provide me with desired throughput?

thanks in advance

12 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    You are really pushing the limit of Gigabit ethernet. By the time you take into account the overhead of the MAC and TCP and IP protocols and the time you're going to spend processing, you're not going to be able to get the full 800Mb/s. I'd really be surprised if your PC could keep up with this anyway. It has all the same issues. The stack is implemented in software on the PC.

    Jake
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    Are you open to other transport methods? Most people are going PCI Express these days. Can your data be easily compressed?

    Jake
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    Actually 800Mbits/s is our upper limit. We might be able to get away with 500Mbits/s, but from the sounds of it, this might also be a challenge.

    What would you say is a feasible upper limit on for this type of interface?

    PCIe is an option we're considering, but we'd like to run this board stand alone.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    To my opinion, you should be able to achieve almost Ethernet 1GBit hardware speed with a FPGA hardware network stack, reduced only by the protocol overhead. Comparable results are achieved e. g. by network switches. I also expect, that you can deliver the data stream at the FPGA side. But how about the PC side? What's the data sink? I suggest to perform some basic tests with PCs if the intended throughput is feasible.

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    Wouldn't the PC performance depend on NIC? Does the NIC handle network layer or is it the PC processor?

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    NIC handles MAC and PHY layers. All higher layers (IP->TCP/UDP->telnet/HTML/etc.) are normally done in software on the PC.

    Jake
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    We are currently doing a similar thing with 10GbE in a StratixII GX. Our data rate is 500MB/s (4Gbits/s) sustained. We are currently encapsulating our data in raw ethernet packets, but have plans to switch to UDP/IP. The computer on the receiving end has no trouble keeping up with this data rate as long as copies are minimized.

    Here are some issues to be aware of:

    1) The entire data path must be hardware. The NIOS cannot be in the data path at these rates. Hardware must therefore be able to split/merge the ethernet link between the high rate data packets and the control packets to/from the NIOS.

    2) Raw ethernet or UDP/IP may be implemented in hardware with relative ease. TCP is much more complex and requires more resources, like lots of RAM for the retransmit buffer. Also, TCP will put much higher load on the receiving computer. I would not recommend TCP for the data packet format. TCP can be used for the low rate control to the NIOS using a software stack.

    3) Overhead can be reduced by using jumbo frames (up to ~9kB), allowing throughput closer to the theoretical limit. The MAC, the receiving NIC and any switches on the network segment must explicitly support these. Also, your packet buffers will need to be much larger for jumbo frames. Jumbo frames may be required to hit 800Mbits/s on GbE.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    --- Quote Start ---

    TBut how about the PC side? What's the data sink? I suggest to perform some basic tests with PCs if the intended throughput is feasible.

    --- Quote End ---

    +1 on testing with PC's. Just write a simple transmit program and receive program that connect to each other over a raw ethernet or UDP socket. Have the receiver verify the data so it knows if packets start being dropped. Have the transmitter be able to throttle to a given data rate. You can also configure the NIC's to a specific MTU size.

    With this configuration, you can easily test the effects of different protocol and frame size combinations.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    Thank you for the interesting insights. Can you achieve the said speeds through UDP, when you connect at the socket driver level (winsock2 on windows OS), or do you need to connect the device driver at a lower level?

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    17 years ago

    --- Quote Start ---

    Thank you for the interesting insights. Can you achieve the said speeds through UDP, when you connect at the socket driver level (winsock2 on windows OS), or do you need to connect the device driver at a lower level?

    --- Quote End ---

    On Linux, we can achieve these throughputs with a socket connection using UDP, so it is through the OS's protocol stack and the standard sockets library. We have not benchmarked on Windows.

    We had to do a fair amount of tweaking driver settings to achieve 500MB/s. The settings that made the most difference were the ring buffer size and the IRQ affinity. The ring buffer needed to be huge. The NIC IRQ affinity had to be set to one CPU, and the affinity of all other IRQ's had to be assigned to the remaining CPU's, so that the NIC's IRQ essentially had a dedicated CPU on which it was the only interrupt that would be serviced.