Hi all, I am measuring ethernet performance on a NIOS II based board with an Altera TSE IP Core running uClinux, and it's been disappointing so far. I've done timing using TCP and UDP, and barely get up to 12Mbit/s. I did some timings using UDP and a 1024 byte message using a bind()ed and connect()ed socket: Total time to send message (when sendto() or write() is called until it returns): 650us Time spent in the driver (using atse.c driver, altera_tse.c driver gives similar performance): 140us Time in the driver waiting for the hardware to send data: 80us (seems it's doing 100Mbit instead of GbE, but that's a relatively small issue here considering overall performance) Kernel overhead before entering driver code: ~400us Kernel overhead after exiting driver code: ~100us Timing was done by writing to an output pin and using an osciloscope. It seems there's just a lot of overhead using the linux IP stack on the Nios II. Has anybody been able to get better performance or is it hopeless given the speed of the processor?

This already has been discussed here several times. Out of the box, Linux is versatile, but not fast with TCP/IP. NIOS is a quite slow processor. So I don't think that you will achieve much more throughput on such a system without introducing special "router" stuff like the "Zero-Copy Stack" (which needs a special Ethernet driver). -Michael

I've been doing some research on Zero-Copy stacks and haven't found any implementation for Linux that I could try to use. All I've found is sendfile() and a suggestion to use an mmapped file as a buffer as a hack to make use of sendfile() to remove the user-to-kernel copy normally done on send. I haven't been able to get this working because I get this error trying to use mmap for a size greater than a couple KB, even though malloc works fine to allocate the space: Error mmapping the file: Cannot allocate memory Regardless, memcpying a 50KB buffer takes 32us, which is still only a small part of the overhead I'm seeing, so I don't think zero-copy will help all that much. Still, I'd be happy to hear any suggestions of how to implement it for uClinux on NIOS II. So far the only way I've found to get decent bandwidth is to send large UDP datagrams.

So you found that UDP is a lot faster than TCP, and you found that copying overhead is not the major problem with TCP ? Of course UDP does impose less overhead than TCP but I did not think that this would be that important. Thanks for letting us know ! -Michael

I did mention I'm doing some tests with TCP as well, but all the above results are with udp. With UDP, there is over 500us of overhead per message, most of which by my understanding is not due to copying the buffer.

You can improve this by increasing the size of cache or reduce memory interface latency. - Hippo

Ethernet performance using TSE in uClinux | Altera Community

55 Replies

Altera_Forum
Honored Contributor
16 years ago
Hi,

--- Quote Start ---

That is, of course, if the TSE does not already implement it in hardware.

--- Quote End ---

Altera's TSE has hardware CRC generators. Please check the datasheet.
http://www.altera.co.jp/support/ip/interface-protocols/ips-inp-tse.html

Almost all MAC have hardware CRC generators, because they need real-time CRC generation. Ethernet has CSMA/CD functions, so when the collision occurs, the MAC must add 'JAM' data which does'nt match the CRC. In ancient period, this task was too hard for those CPUs, so MAC has its CRC generator by historical reasons.

About IP or UDP, they also have their checksum, but those are 16bit one's complement sum, not CRC. This is done by software, and in uClinux, the code is
'/home/***/nios2-linux/linux-2.6/include/asm-nios2/checksum.h'. For IP packets, these checksum are done only for their headers. For UDP packets, the checksum is applied for whole data include their headers, and this eats CPU power, but it is optional.

For the question of ' Ethernet performance using TSE in uClinux', please see the directory of '/home/***/nios2-linux/linux-2.6/net/ipv4', and feel how many things Nios CPU must do to send only 'ONE' UDP packet.

Kazu
Altera_Forum
Honored Contributor
16 years ago
OK. Then it was either an exercise in futility, or and educational exercise on my part. Whatever it was, I was just trying to help. :)
Altera_Forum
Honored Contributor
16 years ago
--- Quote Start ---
The Kernel Building configuration process ("make menuconfig" and friends) sets a lot of values in a configuration file that results in# define values (to be used by the Compiler) and other things (e.g. to be used in the linker script, but which I don't have much clue of yet).
--- Quote End ---

I know about the header with memory addresses and all, but I still can't find any auto-generated linker script. There is only the static one that comes with elf2flt that defines only one flat memory area, and another static one at linux-2.6/arch/nios2/kernel/vmlinux.lds.S that doesn't list a MEMORY. So how does the uClinux linker know where to put everything? Does it just grow from 0 and if it reaches out of main RAM you're screwed?
Altera_Forum
Honored Contributor
16 years ago
I feel that this is an important issue.
You might want to ask this in the uClinux and/or uClinux-NIOS mailing lists, where all the experts will listen

-Michael
Altera_Forum
Honored Contributor
16 years ago
The other problem I'm having trying to implement this is that the SSRAM seems to be used for something already. There is data there, and if I try to write to it, whether it's the buffer or running a program that uses code with its text section there (I tried modifying vmlinux.lds.S), the system freezes. As far as I can tell the hardware doesn't use the SSRAM for anything and I haven't previously modified uClinux to use it in any way.
Altera_Forum
Honored Contributor
16 years ago
I suppose you need to define another TCM in your hardware design for this use.

-Michael
Altera_Forum
Honored Contributor
16 years ago
I'm not sure what you mean. There is already a memory block for SSRAM in sopc builder and writing to some random addresses there works, but there is something there and I don't know where it's coming from and overwriting any significant amount of data there freezes the system.
#define na_ssram 0x05000000 # define na_ssram_size 0x00400000 # define na_ssram_end 0x05400000 # define na_ssram_clock_freq 100000000
Altera_Forum
Honored Contributor
16 years ago
I suppose the system uses DRAM for normal purposes. So I, too, have no idea what the SSRAM might be used for.

I suggested TCM (internal RAM in the FPGA that does not use the Cache and the Avalon bus), as same would give the best possible performance.

-Michael
Altera_Forum
Honored Contributor
16 years ago
Hi,

--- Quote Start ---
I know about the header with memory addresses and all, but I still can't find any auto-generated linker script. There is only the static one that comes with elf2flt that defines only one flat memory area, and another static one at linux-2.6/arch/nios2/kernel/vmlinux.lds.S that doesn't list a MEMORY. So how does the uClinux linker know where to put everything? Does it just grow from 0 and if it reaches out of main RAM you're screwed?
--- Quote End ---

The linker uses the script 'home/***/nios2-linux/uClinux-dist/linux-2.6.x/arch/nios2/kernel/vmlinux.lds' and this script is generated from '/home/***/nios2-linux/linux-2.6/arch/nios2/kernel/vmlinux.lds.S', etc. The generator is '/home/***/nios2-linux/uClinux-dist/linux-2.6.x/arch/nios2/kernel/.vmlinux.lds.cmd' (hidden file), I think, and this generator is generated from... , I don't know !
But anyway, you can control the linker from 'vmlinux.lds.S' and 'vmlinux.lds.inc'. Also you can check the symbol's locations by a map file '/home/***/nios2-linux/uClinux-dist/linux-2.6.x/System.map'.

Kazu

Altera_Forum

Honored Contributor

16 years ago

Hi,

--- Quote Start ---

The other problem I'm having trying to implement this is that the SSRAM seems to be used for something already. There is data there, and if I try to write to it, whether it's the buffer or running a program that uses code with its text section there (I tried modifying vmlinux.lds.S), the system freezes. As far as I can tell the hardware doesn't use the SSRAM for anything and I haven't previously modified uClinux to use it in any way.

--- Quote End ---

I forgot to mention this. Where did you locate the exception hook codes? In the file '/home/***/nios2-linux/linux-2.6/arch/nios2/kernel/head.S',


    /* This is the default location for the exception 
     * handler. Code in jump to our handler
     */
exc_hook:    
    movia    r24,inthandler
    jmp    r24
1:        
    /*
     * After flushing the instruction cache, we must flush the data
     * cache.
     */
    movia    r1,NIOS2_DCACHE_SIZE
    movi    r2,NIOS2_DCACHE_LINE_SIZE
data_flush:
    flushd    0(r1)
    sub    r1,r1,r2
    bgt    r1,r0,data_flush
# ifdef CONFIG_BREAK_ON_START
    break# endif //CONFIG_BREAK_ON_START
# if defined(CPU_EXCEPT_ADDRESS_ASM) && (CPU_EXCEPT_ADDRESS_ASM != (LINUX_SDRAM_START + 0x20))
/*    Copy an instruction sequence to put at the exception address */    
        movia    r2,exc_hook
        movia    r3,CPU_EXCEPT_ADDRESS_ASM
        ldw     r1,0(r2)
        stw     r1,0(r3)
        ldw     r1,4(r2)
        stw     r1,4(r3)
        ldw     r1,8(r2)
        stw     r1,8(r3)
        flushd  0(r3)
        flushd  4(r3)
        flushd  8(r3)
        flushi  r3
        addi    r3,r3,4
        flushi  r3
        addi    r3,r3,4
        flushi  r3# endif

the code 'exc_hook' is copied to the new location.

Kazu

Forum Discussion

Ethernet performance using TSE in uClinux

55 Replies

Recent Discussions

Ashling RISC Free IDE fails to download ELF file

Dual Port Nios V BSP error

set_output_delay explained for dummies

[Test only]How to verify Agilex 3 in radiation tolerant environment

gdb server problem when debugging