Ethernet performance using TSE in uClinux

Yes, it was sadly lost. It was great information.

If found this. In general, you can use a compiler directive like this:

__attribute__ ((section (".ssram")));

after your function name. You use a similar __attribute__ directive after data variable declarations to put them in an alternate data section.

However, I believe the guy used a custom linker script, and explicitly placed the names of the object files that needed to be located into a fast memory section there.

I am by no means remotely familiar with linker scripts, so you are on your own for this subject.

As far as the hardware CRC checksum, I would be surprised if Altera's TSE does not already do this and tack it on at the end of the packet being sent automatically. Likewise, calculated on the fly when receiving a packet for packet integrity validation. It would have been relatively simple to have incorporated it into the IP. You'd have to check the software driver implementation to verify if it calculates the CRC with software or not.

If not, you'd generally have to make an avalon memory mapped slave with addresses for writing which would a) initialize the CRC, and b) calculate a new CRC with data written, and one read address to read the final calculated CRC value. The new 32 bit CRC value can be calculated with a new 32 bit data input in 1 clock cycle using combinational xor logic, so the IP can execute with zero waitstates. Sorry, I can't go into details of writing such a module--too much explaning. If I recall correctly (this may be wrong), the CRC value is complemented when sending it out for ethernet. You need to run some experiments to verify the output of your hardware CRC IP matches the output of the software calculated one. And of course, you have to modify the TSE driver to use your CRC IP to calculate it.

However, here is a great link to generate the combinational logic code in Verilog for any CRC data width, polynomial expression, and arbitrary input data width. http://outputlogic.com/ Click on CRC generator.

Wikipedia has the polynomial expression for many standards including ethernet. Search for CRC.

Wow. I found the link I was thinking about. It actually was in this forum.

http://www.alteraforum.com/forum/showthread.php?t=675 (http://www.alteraforum.com/forum/archive/index.php?t-675.html)

Yes. that was me :)

I experimented with the ucOS, but I guess the same method could be used with Linux.

You would need to run a profiler to find what functions are called most.

If you have a big FPGA I would suggest to store the packets on chip instead of the SSRAM, as suggested in this design example (http://www.altera.com/support/examples/nios2/exm-ethernet-acceleration.html?gsa_pos=1&wt.oss_r=1&wt.oss=ethernet%20accelerate). The Ethernet driver will need some rewrite to use the packet memory, I don't know how easy it is to do for Linux.

If you can also put some code in tightly coupled memory, it will help too. The thread linked by Nails shows how to move functions in different memories using a custom linker script. A bigger CPU instruction cache could be enough though, and is a lot easier to do.

From the other thread:

--- Quote Start ---

You must compile the system library with the -ffunction-sections option. That way each function is compiled in a separate section (.text.function_name).

Then I adapted the linker script (tci is the name of the memory attached to the tightly coupled instruction memory:

SECTIONS
{
    .tci :
    {
        PROVIDE (_alt_partition_tci_start = ABSOLUTE(.));
        . = ALIGN(32 / 8);
                
        * (.text.cksum)
        * (.text.asm_cksum)
        * (.text.irq_Unmask)
        * (.text.memcpy)
	* (.text.sosend)
	* (.text.udp_send)
	* (.text.ip_write_internal)
	* (.text.pk_alloc)
	* (.text.udp4_socksend)
	* (.text.ip2mac)
	* (.text.ip_write)
	* (.text.t_sendto)
	* (.text.m_freem)
	* (.text.m_getnbuf)
	* (.text.udp_maxalloc)
	* (.text.udp_usrreq)
	* (.text.tcp_wakeup)
	* (.text.udp_send)
	* (.text.udp_alloc)
	* (.text.m_free)
	* (.text.udp_maxalloc)
	* (.text.ip_mymach)
	* (.text.send_via_arp)
	* (.text.pk_free)
	* (.text.tse_mac_raw_send)
	* (.text.memmove) 		
	* (.text.putq)
	* (.text.pk_validate)
	* (.text.irq_Mask)
	* (.text.alt_remap_uncached) 		
	* (.text.alt_avalon_sgdma_construct_mem_to_stream_desc)
	* (.text.tse_mac_sTxWrite)
	* (.text.alt_remap_cached)
	* (.text.alt_dcache_flush) 		
	* (.text.alt_avalon_sgdma_do_sync_transfer)
	* (.text.alt_remap_cached)
	* (.text.getq)
	* (.text.qdel) 				
        
        PROVIDE (_alt_partition_tci_end = ABSOLUTE(.));
    } > tci
    .text :
    {
        *(.text .stub .text.* .gnu.linkonce.t.*)
}

The tci soction must be listed before the text section. You need to change one line in the .text section, as indicated, to put all the .text.* functions that are not defined in the tci section back in the text section.

I also noticed that each packet sent generates lots of function calls. I wonder if it would be possible to put the Interniche task stack in a tightly coupled data memory and gain a bit more.

Of course if you still need more speed you can do some UDP communication with hardware, as in the nios2 udp offload example (http://www.nioswiki.com/exampledesigns/nios2udpoffloadexample).

--- Quote End ---

This is really useful information but I am not sure how to apply it. I think I found the current linker script used by uClinux (elf2flt/elf2flt.ld) but it uses a flat memory map for all the memory so I don't know how to apply the change to add another section.

I also tried the attribute solution [adding __attribute__ ((section (".ssram")))] to a method. I get this error:

/opt/build/nios2/bin//nios2-linux-uclibc-ld.real: section .ssram  overlaps section .text 

I don't think there is an sram or ssram section defined anywhere, so looks like this is trying to create one in the middle of the text section, which isn't at all what I want.

Here's a guess: Your linker script probably doesn't have a MEMORY definition for SSRAM, therefore a section referring to SSRAM memory doesn't exist. You will need to specify the actual address where the SSRAM resides, and it's length. This can be found by looking in SOPC builder.

Nothing's ever easy is it?

I think I'm at the end of my ability to help without actually digging into it myself. You're gonna have to start reading about linker scripts, as I have no real experience in the matter, only a general "feel" for what needs to be done.

It appears that Daixiwen has spelled out 95% of how to do this, so I hope you can get it with a little more effort.

Good luck, and please let us know if you have any succuss and what was the stumbling block for you.

As in the NIOS uCLinux formus, there have been requests for faster TCP/IP performance already multiple times in the past, I feel it would be viable to include a configuration setting in the Kernel building options for appropriate use of internal memory for the critical code and/or data sections.

Of course this should be done for the MMU enabled distribution, as same will be used in the future supposedly nearly exclusively.

(And of course only if the performance gain is decently proven.)

Should we at least add a paragraph in the "ToDo" section of the NiosWiki ?

-Michael

I don't know how the linux building process works, but I guess it is similar to the ones I know (uC OS and eCOS).

The SOPC system is described in a ptf file, and some gtf scripts are used on it to generate all the files required for the software build process. This includes the system.h include file that contains the base addresses for all the hardware, and the linking script.

The first thing that you should check is if the SSRAM controller is included in your SOPC project. If yes it should contains entries in the system.h file and the linker script.

The Kernel Building configuration process ("make menuconfig" and friends) sets a lot of values in a configuration file that results in# define values (to be used by the Compiler) and other things (e.g. to be used in the linker script, but which I don't have much clue of yet).

When creating the (Kernel etc) binaries (with "make"), the SOPC-builder output is read by some scripts that move the appropriate values (e.g addresses) into the header files included by the C sources and and Linker scripts etc.

Thus in "make menueconfig" it should be possible to set e.g. the name of an SOPC-Builder introduced internal memory block and due to that have the make process use same for the critical network memory (if the Linux Kernel really can be configured in that way at all.).

-Michael

I suspect the linker script is generated in the the "make vendor_hw_select=..." process. I haven't turned my linux box on in months, so I can't verify.

As a side note, if anyone is interested in performing a test with a harware CRC calculation IP, implemented as an avalon slave, I am open to writing the verilog code. It would only take me an hour or so. That is, of course, if the TSE does not already implement it in hardware.

Here's some files if anyone is interested in testing this for me.

I'll admit it took much more than an hour. :-)

It simulates as expected in modelsim with CRC results identical to an app. note I found elsewhere.

It takes 5 clock cycles when computing a new CRC value with 4 data bytes.

I have NOT tested it in a project, however, but it integrates into SOPC builder just fine.

There should be enough comments in the crc32_ethernet.v file to describe it's usage/for making a small driver for it.

Edit: There was a slight error in crc32_ethernet.v file which would have caused a compiler warning, but the results would still be valid. It is now fixed. However, I understand from the next post that this isn't practically useful for anything.