Forum Discussion
Honored ContributorEthernet performance using TSE in uClinux
Hi all,
I am measuring ethernet performance on a NIOS II based board with an Altera TSE IP Core running uClinux, and it's been disappointing so far. I've done timing using TCP and UDP, and barely get up to 12Mbit/s. I did some timings using UDP and a 1024 byte message using a bind()ed and connect()ed socket: Total time to send message (when sendto() or write() is called until it returns): 650us Time spent in the driver (using atse.c driver, altera_tse.c driver gives similar performance): 140us Time in the driver waiting for the hardware to send data: 80us (seems it's doing 100Mbit instead of GbE, but that's a relatively small issue here considering overall performance) Kernel overhead before entering driver code: ~400us Kernel overhead after exiting driver code: ~100us Timing was done by writing to an output pin and using an osciloscope. It seems there's just a lot of overhead using the linux IP stack on the Nios II. Has anybody been able to get better performance or is it hopeless given the speed of the processor?55 Replies
Altera_ForumThis already has been discussed here several times. Out of the box, Linux is versatile, but not fast with TCP/IP. NIOS is a quite slow processor. So I don't think that you will achieve much more throughput on such a system without introducing special "router" stuff like the "Zero-Copy Stack" (which needs a special Ethernet driver).
-Michael- Altera_Forum
I've been doing some research on Zero-Copy stacks and haven't found any implementation for Linux that I could try to use. All I've found is sendfile() and a suggestion to use an mmapped file as a buffer as a hack to make use of sendfile() to remove the user-to-kernel copy normally done on send. I haven't been able to get this working because I get this error trying to use mmap for a size greater than a couple KB, even though malloc works fine to allocate the space:
Regardless, memcpying a 50KB buffer takes 32us, which is still only a small part of the overhead I'm seeing, so I don't think zero-copy will help all that much. Still, I'd be happy to hear any suggestions of how to implement it for uClinux on NIOS II. So far the only way I've found to get decent bandwidth is to send large UDP datagrams.Error mmapping the file: Cannot allocate memory - Altera_Forum
So you found that UDP is a lot faster than TCP, and you found that copying overhead is not the major problem with TCP ?
Of course UDP does impose less overhead than TCP but I did not think that this would be that important. Thanks for letting us know ! -Michael - Altera_Forum
I did mention I'm doing some tests with TCP as well, but all the above results are with udp. With UDP, there is over 500us of overhead per message, most of which by my understanding is not due to copying the buffer.
- Altera_Forum
You can improve this by increasing the size of cache or reduce memory interface latency.
- Hippo - Altera_Forum
Thanks for the suggestion hippo. Indeed the cache was set to the minimum. Setting it to 16+16k got the 1K time down to 420us (19Mbit/s) so that's definitely better but I'm still looking for ways to improve it. I'm also doing some receive tests and that's looking even worse so far.
- Altera_Forum
You probably thought about it already, but are you sure that you compile the kernel and code with full optimisations? The code generated with -O 0 can be quite slow...
- Altera_Forum
I was using -O2 most of the time but recently changed to -O3 with no noticeable change. The kernel doesn't seem to boot at all with -Os
- Altera_Forum
Sorry if this is too generic, but it might point you onto topics you might not have already addressed.
If I recall correctly, on the old niosforum, a guy had a whole slew of things he did to make pretty radical improvements. Like hippo suggests, reducing the memory latency is a good way to improve the performance. Putting the data buffers in SRAM or SSRAM rather than DDR, and several key pieces of network code, hels alot. I can't recall exactly how you specify certain files to be compiled into a specific portion of RAM, but there is a way to do it. Also if the CRC checksum algorithm is done in software, a hardware implementation helps. I recall one post where he was getting around 60 Mbits/sec on the opencores 10/100 core with udp. Another with 10/100/1000 got over 100 Mbits/sec. If it can be believed, this should be your expectations. Edit: I am not sure this was under uClinux though. - Altera_Forum
I take it that thread got lost with the year or so of lost posts?
Those are interesting suggestions. Any idea/link how to implement loading network code in SSRAM or offloading CRC checksums? My searching hasn't turned up anything useful. Thanks.
