Ethernet performance using TSE in uClinux

Thank you Kazuyasu and mschnell!

I've made some progress. There was a hardware issue with the SSRAM on our board as well as a Linux issue. With the hardware issue fixed, I was getting an EFAULT ("bad address") trying to send a buffer from SSRAM. It turns out the kernel checks that user pointers are within the program memory in _access_ok in include/asm-nios2/uaccess.h. Adding the SSRAM area to that check fixed the EFAULT problem, but I just went ahead and took that check out. Since I can access all of memory from userspace anyways, it's not really protecting me from anything and the error was a pain to track down.

-#define access_ok(type,addr,size)      _access_ok((unsigned long)(addr),(size))
+#define access_ok(type,addr,size) 1

Putting the buffer in external SSRAM instead of DDR SDRAM got me from 14Mbps to 16Mbps. I then changed the linker script to add an .ssram section and make sure everything else goes in main memory (DDR). diff is attached in case anyone is interested. I then went through the trace of sending a message (17 functions from the system call to the driver for UDP!) and added

__attribute__ ((section (".ssram")))

to all the functions. For anyone else trying it who hasn't used attributes before either: that goes right before the function definition like so:

__attribute__ ((section (".ssram")))
static int atse_hard_start_xmit(struct sk_buff *skb, struct net_device *ndev)
{

linux-2.6.x/System.map verifies that they get linked at the right addresses (started on TCP and rx stuff here):

05000000 T irq_exit
05000088 T do_sync_write
050001ac T vfs_write
050002a8 T sys_write
05000340 t atse_hard_start_xmit
05000684 T sock_sendmsg
0500076c T sys_connect
0500080c T sys_sendto
05000918 T sys_send
05000938 T sys_socketcall
05000b20 T release_sock
05000c3c T dev_hard_start_xmit
05000f00 T dev_queue_xmit
050012ec T netif_receive_skb
0500163c t process_backlog
05001748 t net_rx_action
05001908 T neigh_resolve_output
05001c7c t neigh_timer_handler
05002160 T __qdisc_run
0500249c t dst_output
050024bc t ip_finish_output2
050027ac T ip_queue_xmit
05002be0 T ip_output
05002cb4 T ip_push_pending_frames
05003120 t __tcp_ack_snd_check
050031c0 t tcp_transmit_skb
050039a0 T tcp_connect
05003db8 T tcp_send_ack
05003ea8 T tcp_v4_connect
05004330 T tcp_v4_rcv
05004ae8 T udp_flush_pending_frames
05004b1c t udp_push_pending_frames
05004f90 T udp_sendmsg
050056b0 T arp_xmit
050056c8 T arp_send
05005724 t arp_solicit
050059e4 T inet_stream_connect
05005d44 T inet_sendmsg
05005dc0 t packet_sendmsg_spkt
05005ff8 t packet_sendmsg

(What do those t's and T's mean?)

This along with 8k caches got me up to 19Mbps sending 1KB UDP messages. Still dismal for a gigabit connection, but it's an improvement. Unfortunately tightly coupled/internal memory is likely not an option because FPGA resources are limited, but now that I've got this figured out I'll try that once I know if that memory is available.

There are still problems though. This configuration only works with kernel debugging enabled. There are three options there that are enabled by default (CONFIG_DETECT_SOFTLOCKUP, CONFIG_DETECT_HUNG_TASK, and CONFIG_SCHED_DEBUG) and if I disable any of them the system doesn't boot with the modified linkage.

Also I've double and triple checked the MAC and PHY registers and it seems everything is configured for gigabit, but as my timings in the original post showed, I still seem to only be getting 100Mbps out of the PHY.

I'd like to submit/bring up some of these changes to the -devel list but I'm not sure how to make it more general.

--- Quote Start ---

Hi,

I forgot to mention this. Where did you locate the exception hook codes? In the file '/home/***/nios2-linux/linux-2.6/arch/nios2/kernel/head.S',

[snip]

Kazu

--- Quote End ---

I haven't modified anything to do with that. I'm not sure what you are suggesting -- putting the exception address somewhere other than 0x20? (which on this system is in DDR SDRAM)

Hi,

--- Quote Start ---

I haven't modified anything to do with that. I'm not sure what you are suggesting -- putting the exception address somewhere other than 0x20? (which on this system is in DDR SDRAM)

--- Quote End ---

Of course, this depends on the MACRO

# if defined(CPU_EXCEPT_ADDRESS_ASM) && (CPU_EXCEPT_ADDRESS_ASM != (LINUX_SDRAM_START + 0x20))

.

If you select the location of exception address in SSRAM with SOPC builder, those scripts automatically set the new address of SSRAM to CPU_EXCEPT_ADDRESS_ASM. So the exception handler is copied to SSRAM. And this is not controlled by linker scripts. If you select the location in DDR SDRAM, it's OK, no problem.

--- Quote Start ---

(What do those t's and T's mean?)

--- Quote End ---

Those are 'Symbol Types'. 'T' means that this symbol belongs to 'text' segment and is global.

Kazu

I am currently trying to put some functions in SSRAM on the MMU distribution, but having trouble. I added the section to the linker script as before, but offset for the kernel region:

        NOTES
+       . = KERNEL_REGION_BASE + SSRAM_BASE;
+       .ssram :
+       {
+               . = ALIGN(4);
+               *(.ssram)
+       }
+
        DISCARDS

I then tried adding a couple kernel functions to the section.

The results from System.map:

c04b59ac A _end
c5000000 T do_sync_write
c5000128 T vfs_write

When I add just do_sync_write, the system hangs after "RPC: Registered tcp NFSv4.1 backchannel transport module." When I add both, I get this at the same place:

r1:  c1c16000 r2:  00000000 r3:  00000000 r4:  c1c2b620
r5:  c1c681e0 r6:  00007e20 r7:  c1c16e1c r8:  6f6f7468
r9:  6f6f0000 r10: c1c43450 r11: c1c681e0 r12: 00000000
r13: c1c681e0 r14: c1c43452 r15: 006c6f6f              
ra:  c0072514 fp:  c01f8000 sp:  c1c16e18 gp:  00000000
ea:  c5000128 estatus:  00000001                       
Unaligned access from kernel mode, this might be a hardware
problem, dump registers and restart the instruction        
  BADADDR 0xc4fff891                                       
  cause   7                                                
  op-code 0x36bdd976

The address given is not a symbol, which makes me think some data address is getting shifted when I move the functions out of DDR, which makes me think this is a latent problem somewhere, but that's just guesswork.

Also, if I use IO_REGION_BASE instead of KERNEL_REGION_BASE, which by my understanding should have the same effect but non-cacheable, I get this link error:

fs/built-in.o: In function `vfs_write':
(.text+0x2ec8): relocation truncated to fit: R_NIOS2_CALL26 against `do_sync_write'
fs/built-in.o: In function `do_sync_write':
(.ssram+0xb4): relocation truncated to fit: R_NIOS2_CALL26 against `wait_on_retry_sync_kiocb'

--- Quote Start ---

When I add just do_sync_write, the system hangs after "RPC: Registered tcp NFSv4.1 backchannel transport module." When I add both, I get this at the same place:

--- Quote End ---

I figured it out and realized I had the same problem with the non-MMU version: when changing the sections, I have to load in the linux.initramfs.gz image instead of the zImage. The code that loads the image from the link offset is missing code to copy extra sections to the right places.

Hi,

Sorry I'm new to all of this so I have a few questions from all of your threads.

1. How do I compile the kernel with full optimizations?

2. How do I increase cache size?

3. How do I decrease memory latency?

4. How do I enable jumbo frames?

I'm using a Cyclone III Starter board with the HSMC-Net daugher card. I have GigE UDP working at the moment, but it's quite slow. I need to increase the speed significantly. I'm connecting the HSMC-Net card to my Macbook directly.

Thanks!

Benjamin

What performance do you expect ? As said in this thread, a NIOS LINUX system will by far not be able to keep up with a line speed of 1 GBit (100 MByte/sec netto) . I doubt even 100 MBit (some 10 MByte/sec netto) performance can be achieved. This system is not viable to do a router kind of application.

-Michael

To be honest, it needs to be very fast (> 60 MByte / sec data). We need to send 200 MBytes of data from our board to a PC, and based on the specs we figured GigE was the most versatile and configurable (and also the cheapest).

Will I never achieve that throughput using UDP, Nios II and the TSE core?

My understanding was that the TCP/IP component of the Nios Processor was really slow, but that using Jumbo frames and UDP, we could get very high data rates.

Not with this CPU alone... Reaching 60MBytes/s with a CPU running at ~100MHz can be a challenge, with any architecture.

You will probably need a hardware acceleration, such as the one in this design example (http://www.nioswiki.com/exampledesigns/nios2udpoffloadexample). Then you can reach speeds close to the theoretical limit, if the target is able to handle it.

Pity that there is no demo example for my dev board. I'll try deciphering what they've done....

Thanks for the help and advice.