Forum Discussion
Altera_Forum
Honored Contributor
Honored ContributorAltera TSE driver and example program for lwIP (1.3.2)
After many many requests and complaints about lack of support and/or documentation for support of lwIP for the Altera TSE, I have developed a drop-in TSE driver and example program and made this available to the NIOS II community. This was done for NIOS II 8.1 SP0.01. I don't expect difficulty with version 9.x.
This is for the latest version of lwIP (the latest is as of this post) for a minimal program and HTTP server based on the http server in the lwIP contrib folder. The lwIP TSE driver uses the altera_avalon_tse driver and SGDMA as-is. There is a complete (as in 41-step) set of instructions on creating the project and example program. More information and the link to the driver is available here: http://lwip.wikia.com/wiki/available_device_drivers#lwip_1.3.2 Please direct any questions, changes for NIOS II 9.1, or comments to this thread. 12-16-2010 update: This example works with NIOS Version 10.0 with some tweaks to the procedure to create the project. Also, a lwIP 1.4 release candidate has been out for a while and it drops into this example (in place of 1.3) without changes. Bill257 Replies
Altera_ForumI'm happy to run something if you can post some code I can add to the lwIP example and produce the problem. It would determine if it's lwIP options or hardware. Or if it's a real problem with the driver and I'm willing to spend time to try to make it right if I can.
Note my programs are *all* RAW API based. I wrote my own streaming "socket" on top of it. I've been meaning to publish my "socket" because it is very efficient - it uses pbuf chains to hold incoming data and allows the application to read one or more bytes from the front of this pbuf list. Actually this isn't that hard to implement just going off me stating how I did it. Bill- Altera_Forum
Bill,
I’ve attached a code. This is basically tcpecho_raw from lwip contrib, all the tests are inside echo_accept(). I run this under FreeRTOS port by DipSwitch with lwip 1.4.0 (don’t have runable non-os project on the moment), my lwipopts.h is also attached. It looks like if the first tcp_write is NOCOPY, then lwip doesn’t use oversized pbuf and forwards all the chunks straight to the driver. I fully support Dsl hypothesis about byte enables. According to Avalon-ST spec tse_mac must ignore empty[] in the middle of packet (see Chapter 5.3, empty signal description: “If endofpacket is not asserted, this signal is not interpreted”). Also, there is an anomaly in sgdma, which potentially may affect short transfers. Here is a quotation from Nios II EDS 11.1 errata: --- Quote Start --- Unaligned Transfers of Small Payloads Fail on SG-DMA Description The Scatter Gather DMA SOPC Builder peripheral does not correctly handle unaligned transfers with small payloads. A payload length smaller than the data width causes erroneous data transfers. Workaround Avoid using DMA devices to transfer small payloads. If absolutely necessary, for a 32-bit SG-DMA, a minimum length of 4 bytes guarantees that data is transferred correctly. --- Quote End --- As for the “On-Chip FIFO Memory Core” – this core in MM->ST configuration may perform similarly to sgdma for transmitting (we use sgdma synchronously, isn’t it?), but it does not suffer from the “short writes” anomaly. That is why I thought it can be used as a workaround. Unfortunately this doesn’t help with short writes in the middle of the packet. That is why I withdraw this “proposal”. Igor - Altera_Forum
Igor,
Building your program (changing it to NO_SYS=1 and no RTOS) doesn't download - some debugger complaint about running 2 programs? Anyway, adding echo.c to my version of the example does run and I can confirm your finding. This is pretty bad - I don't know why I don't find this problem in real applications - even those using telnet. Anyway, you cannot send more bytes than required to meet the minimum 4 quantity. The only option is to copy the pbuf chains to a single pbuf. I think this sucks! But so it is. This works:
You could optimize it by first checking if any chains are less than 4 and doing this addition only when one or more is, but I kept it simple - just do it if there are chains. Maybe it's risky but I intend to not change my lwIP-based products. They don't show the problem which I don't know why. Maybe I could add this code and it would never be called??? [Update: Probably should check for q != NULL and exit if so. Of course this will result in errors as well!] Bill/* @Function Description - TSE transmit API to send data to the MAC * * * @API TYPE - Public * @param net - NET structure associated with the TSE MAC instance * @param data - pointer to the data payload * @param data_bytes - number of bytes of the data payload to be sent to the MAC * @return SUCCESS if success, else a negative value */ err_t tse_mac_raw_send(struct netif *netif, struct pbuf *pkt) { int tx_length; unsigned len; struct pbuf *p, *q = NULL; alt_u32 *data; tse_mac_trans_info *mi; lwip_tse_info *tse_ptr; struct ethernetif *ethernetif; unsigned int *ActualData; /* Intermediate buffers used for temporary copy of frames that cannot be directrly DMA'ed*/ char buf2; ethernetif = netif->state; tse_ptr = ethernetif->tse_info; mi = &tse_ptr->mi; if(pkt->next != NULL) // Unwind pbuf chains { q = pbuf_alloc(PBUF_RAW, pkt->tot_len, pkt->type); for(len = 0, p = pkt; p != NULL; p = p->next) { memcpy(q->payload + len, p->payload, p->len); len += p->len; } pkt = q; } for(p = pkt; p != NULL; p = p->next) { data = p->payload; len = p->len; if(((unsigned long)data & 0x03) != 0) { /* * Copy data to temporary buffer <buf2>. This is done because of allignment * issues. The SGDMA cannot copy the data directly from (data + ETH_PAD_SIZE) * because it needs a 32-bit alligned address space. */ memcpy(buf2,data,len); data = (alt_u32 *)buf2; } ActualData = (void *)alt_remap_uncached (data, len<4 ? 4 : len); printf("<%d @ 0x%08X/0x%08X>", len, (unsigned int)p->payload, (unsigned int)ActualData); if(len<4) len=4; /* Write data to Tx FIFO using the DMA */ alt_avalon_sgdma_construct_mem_to_stream_desc( (alt_sgdma_descriptor *) &tse_ptr->desc, // descriptor I want to work with (alt_sgdma_descriptor *) &tse_ptr->desc,// pointer to "next" (alt_u32*)ActualData, // starting read address (len), //# bytes 0, // don't read from constant address p == pkt, // generate sop p->next == NULL, // generate endofpacket signal 0); // atlantic channel (don't know/don't care: set to 0) tx_length = tse_mac_sTxWrite(mi,&tse_ptr->desc); ethernetif->bytes_sent += tx_length; } if(q != NULL) pbuf_free(q); LINK_STATS_INC(link.xmit); return ERR_OK; } - Altera_Forum
Bill,
Thank you for the confirmation and for the patch. With modified tse_mac_raw_send() both raw_api- and netconn-based tests work fine in my configuration. I think that your simple solution should be OK for TCP, but may severely impact performance of udp applications if someone tries to achieve zero-copy by chaining headers-only pbuf with ROM- or REF-type payload-only pbuf. The both pbufs could be properly aligned and unnecessary and expensive unwinding may become a bottleneck. I don’t use this approach in my code currently, but advertised it earlier in this thread. Below is my implementation of chain scanning as you suggested. I changed also to do unwinding in buf2[]. Is it ok, or you had good reasons to allocate separate pbuf for this purpose?err_t tse_mac_raw_send(struct netif *netif, struct pbuf *pkt) { int tx_length; unsigned len; struct pbuf *p; alt_u32 *data; tse_mac_trans_info *mi; lwip_tse_info *tse_ptr; struct ethernetif *ethernetif; unsigned int *ActualData; int unwind; /* Intermediate buffers used for temporary copy of frames that cannot be directrly DMA'ed*/ struct pbuf unwind_pbuf; char buf2; ethernetif = netif->state; tse_ptr = ethernetif->tse_info; mi = &tse_ptr->mi; unwind = 0; for(p = pkt; p != NULL; p = p->next) { if (((unsigned long)p->payload & 0x03) != 0 || p->len < 4 || ((p->len & 3) != 0 && p->next)) { unwind = 1; break; } } if (unwind) { // Unwind pbuf chains if (pkt->tot_len > sizeof(buf2)) { // no space for unwinding; drop the packet return ERR_OK; } for(len = 0, p = pkt; p != NULL; p = p->next) { /* * Copy data to temporary buffer <buf2>. This is done because of allignment * issues. The SGDMA cannot copy the data directly from (data + ETH_PAD_SIZE) * because it needs a 32-bit alligned address space. */ memcpy(buf2 + len, p->payload, p->len); len += p->len; } unwind_pbuf.len = unwind_pbuf.tot_len = len; unwind_pbuf.payload = buf2; unwind_pbuf.next = NULL; pkt = &unwind_pbuf; } for(p = pkt; p != NULL; p = p->next) { data = p->payload; len = p->len; // No need for re-alignment and length-checking here ActualData = (void *)alt_remap_uncached (data, len); //printf("<%d @ 0x%08X/0x%08X>", len, (unsigned int)p->payload, (unsigned int)ActualData); /* Write data to Tx FIFO using the DMA */ alt_avalon_sgdma_construct_mem_to_stream_desc( (alt_sgdma_descriptor *) &tse_ptr->desc, // descriptor I want to work with (alt_sgdma_descriptor *) &tse_ptr->desc,// pointer to "next" (alt_u32*)ActualData, // starting read address (len), //# bytes 0, // don't read from constant address p == pkt, // generate sop p->next == NULL, // generate endofpacket signal 0); // atlantic channel (don't know/don't care: set to 0) tx_length = tse_mac_sTxWrite(mi,&tse_ptr->desc); ethernetif->bytes_sent += tx_length; } LINK_STATS_INC(link.xmit); return ERR_OK; } - Altera_Forum
Maybe you could accumulate the data sent by the user in an aligned buffer until the tcp stack actually sends the data?
For TCP it might be ok (if slightly unexpected) to resend the previous 1-3 bytes in order to always do aligned sends. If you are willing to do that, then the application send() could write directly into a pre-allocated buffer used for retransmissions and indexed by the tcp byte sequence number. Might be worth doing that anyway - but with a single buffer for 'realignment' transmits. - Altera_Forum
Dsl, I think that the easiest application level workaround is simply to not use zero-copy versions of tcp_write and netconn_write. This is appropriate for me because I need TCP for primitive debug console only. Other workarounds are also possible of course.
- Altera_Forum
Guys,
Comments: First, pbuf payload is guaranteed aligned. I use custom version of this driver (more optimized and improves on SGDMA and PHY handling plus some bug fixes in the Altera code) and took out the alignment check and added an assertion and it never asserts. If user code sets payload (a bad practice) than it's not guaranteed, but pbuf_alloc aligns payload. An application that uses UDP for high data rates is probably using a custom client in which case the payload size could be enforced. I use a zero-copy UDP high speed reliable protocol (the hardware writes in the UDP payload and checksum) and in this case the payload is large and always aligned. You only need to unwind if pkt->next isn't NULL. An unchained pbuf can never have a payload size less then 4 because of the IP header. I'd put back the pkt->next != NULL test before the for loop. Minor:
Can't occur. lwIP will never chain more than the MTU. It can't because 802.11 cannot support it. This bug in the SGDMA (it's a bug if you can't sent 1-x bytes) maybe why InterNiche is so slow and full of copies. It's too bad it cripples lwIP which otherwise is much more efficient. I wonder if we can write the small payload bytes right to the MAC. I.e. don't use SGDMA but use a for loop to write 1 to 3 bytes to the MAC buffer (same place the SGDMA writes to)? Does either of you or anyone know if this could be done? I think I'll put in a service request for this as a SGDMA bug. Great discussion - thanks! Bill// Unwind pbuf chains if (pkt->tot_len > sizeof(buf2)) { // no space for unwinding; drop the packet return ERR_OK; }- Msg06484
New Contributor
Hi, I downloaded the latest LWIP. It somewhat works in that I have to start two cmd shells and send ping requests from both so that the second will "unbuffer" the first. I tried replacing the err_t tse_mac_raw_send(struct netif *netif, struct pbuf *pkt) with yours above, but the compiler complains that
ALT_LINK_ERROR("alt_remap_uncached() is not available because Nios II Gen2 cores with data caches don't support mixing cacheable and uncacheable data on the same line.");
From alt_remap_uncached.c
I tried putting it back to use
ActualData = (void*)(((alt_u32)data)); which does compile but does not actually reply to the ping.
below is the tse_mac_raw_send() found in the latest lwip lwip_tse_mac.c
I am not sure if this is causing the issue, it seems like the packet is coming in and getting buffered until the next packet arrives that is why having 2 ping running makes everything appear to work.
Does this make sense? I may be completely wrong, any help is appreciated.
err_t tse_mac_raw_send_orig (struct netif *netif, struct pbuf *pkt){int tx_length;unsigned len;struct pbuf *p;alt_u32 *data;tse_mac_trans_info *mi;lwip_tse_info *tse_ptr;struct ethernetif *ethernetif;alt_u32 *ActualData;/* Intermediate buffers used for temporary copy of frames that cannot be directrly DMA'ed*/char buf2[1560];ethernetif = netif->state;tse_ptr = ethernetif->tse_info;mi = &tse_ptr->mi;for(p = pkt; p != NULL; p = p->next){data = p->payload;len = p->len;// just in case we have an unaligned buffer, this should never occurif(((unsigned long)data & 0x03) != 0){/** Copy data to temporary buffer <buf2>. This is done because of alignment* issues. The SGDMA cannot copy the data directly from (data + ETH_PAD_SIZE)* because it needs a 32-bit aligned address space.*/memcpy(buf2,data,len);data = (alt_u32 *)buf2;}// uncache the ethernet frameActualData = (void*)(((alt_u32)data));/* Write data to Tx FIFO using the DMA */alt_avalon_sgdma_construct_mem_to_stream_desc((alt_sgdma_descriptor *) &tse_ptr->desc[ALTERA_TSE_FIRST_TX_SGDMA_DESC_OFST], // descriptor I want to work with(alt_sgdma_descriptor *) &tse_ptr->desc[ALTERA_TSE_SECOND_TX_SGDMA_DESC_OFST],// pointer to "next"(alt_u32*)ActualData, // starting read address(len), // # bytes0, // don't read from constant addressp == pkt, // generate sopp->next == NULL, // generate endofpacket signal0); // atlantic channel (don't know/don't care: set to 0)tx_length = tse_mac_sTxWrite(mi,&tse_ptr->desc[ALTERA_TSE_FIRST_TX_SGDMA_DESC_OFST]);if (tx_length != p->len)dprintf(("failed to send all bytes, send %d out of %d\r\n", tx_length, p->len));ethernetif->bytes_sent += tx_length;}LINK_STATS_INC(link.xmit);return ERR_OK;}
- Altera_Forum
Bill,
--- Quote Start --- First, pbuf payload is guaranteed aligned. I use custom version of this driver (more optimized and improves on SGDMA and PHY handling plus some bug fixes in the Altera code) and took out the alignment check and added an assertion and it never asserts. If user code sets payload (a bad practice) than it's not guaranteed, but pbuf_alloc aligns payload. --- Quote End --- As we discovered all together under some circumstances innocent tcp_write(pcb, long_help_message, len, NOCOPY) can have an effect of “user code setting payload”. How could I expect this in advance? --- Quote Start --- An application that uses UDP for high data rates is probably using a custom client in which case the payload size could be enforced. I use a zero-copy UDP high speed reliable protocol (the hardware writes in the UDP payload and checksum) and in this case the payload is large and always aligned. --- Quote End --- Yes, for UDP alignment is under developer’s control. I also had to reinvent some kind of “zero-copy reliable UDP”. I use LWIP for retransmitting only – this makes life easier. Actually, if you cook UDP packets in hardware, you can retransmit a packet without LWIP intervention at all – just instruct one SGDMA to ship out exactly what had been delivered into memory by another SGDMA. --- Quote Start --- You only need to unwind if pkt->next isn't NULL. An unchained pbuf can never have a payload size less then 4 because of the IP header. I'd put back the pkt->next != NULL test before the for loop. --- Quote End --- Agree. --- Quote Start --- Minor:
Can't occur. lwIP will never chain more than the MTU. It can't because 802.11 cannot support it. --- Quote End --- Would be better to turn into ASSERT. (For those who like building pbufs by hand, like me :) ). --- Quote Start --- This bug in the SGDMA (it's a bug if you can't sent 1-x bytes) maybe why InterNiche is so slow and full of copies. It's too bad it cripples lwIP which otherwise is much more efficient. I wonder if we can write the small payload bytes right to the MAC. I.e. don't use SGDMA but use a for loop to write 1 to 3 bytes to the MAC buffer (same place the SGDMA writes to)? Does either of you or anyone know if this could be done? I think I'll put in a service request for this as a SGDMA bug. --- Quote End --- If you only need to insert 1-3 bytes at the _end_ of packet, you may try standard “On-Chip FIFO Memory Core” (or very simple custom component) and mux its output with the SGDMA (though I never implemented this in hardware). I don’t know readily available solution for general case (when incomplete words are inside the packet). Should not be very difficult to code. The main challenge seems to be repacking of the stream to become Avalon-ST complaint: e.g. <sop4><4><1><4><2eop> into <sop4><4><4><3eop>. --- Quote Start --- Great discussion - thanks! --- Quote End --- Thank you also!// Unwind pbuf chains if (pkt->tot_len > sizeof(buf2)) { // no space for unwinding; drop the packet return ERR_OK; } - Altera_Forum
Igor,
--- Quote Start --- How could I expect this in advance? --- Quote End --- I didn't mean to be condescending. I guess we have to remember that zero-copy means "use the given pointer to the payload". Even so, the unaligned pointer was not your issue here. That was handled by the driver (and now I see why it's good to not take it out even with there being a copy). What got you is the "me called" SGDMA bug. Aligned or not the 1-byte transfer was not going to work. --- Quote Start --- Yes, for UDP alignment is under developer’s control. I also had to reinvent some kind of “zero-copy reliable UDP”. I use LWIP for retransmitting only – this makes life easier. Actually, if you cook UDP packets in hardware, you can retransmit a packet without LWIP intervention at all – just instruct one SGDMA to ship out exactly what had been delivered into memory by another SGDMA. --- Quote End --- If you NAK each packet, yes. And sending is faster. I have a multi-nak protocol which might have to go back in the buffer and resend one or more packets. I actually build and checksum the IP and UDP headers in a static position and only patch the checksum before sending the packet header and payload. This is because most of the header doesn't change. lwIP could incorporate this too because once connected to a PCB, much of the header and that checksum part is static. --- Quote Start --- Would be better to turn into ASSERT. (For those who like building pbufs by hand, like me :) ). --- Quote End --- Sure, good idea. --- Quote Start --- If you only need to insert 1-3 bytes at the _end_ of packet, you may try standard “On-Chip FIFO Memory Core” (or very simple custom component) and mux its output with the SGDMA (though I never implemented this in hardware). I don’t know readily available solution for general case (when incomplete words are inside the packet). Should not be very difficult to code. The main challenge seems to be repacking of the stream to become Avalon-ST complaint: e.g. <sop4><4><1><4><2eop> into <sop4><4><4><3eop>. --- Quote End --- Too bad there's no MAC register to "queue a byte into the frame buffer". In reality if it were designed well you should be able to use the MAC in either SGDMA mode or direct CPU write mode. It's just receiving data. You would need START, DATA, and STOP registers to start, send data, and terminate a packet. Bill - Altera_Forum
There may be a much faster way to handle these 1-3 byte SGDMA transfers.
I believe it to be true that there would only be one 1-byte pbuf in a chain, is this true? If it *is* true, you could ensure that all pbufs are allocated with an extra 4 bytes of payload. This would be done for PBUF_POOL (not sure about PBUF_RAM). Take the 1 to 3 bytes of the next pbuf and add it to the end of the payload of the preceding one. Increase that len by 1 to 3 and set the second one to 0. Leave the chain in place but skip pbufs with len==0. This would be very efficient for the common case of a trailing short pbuf chain. For PBUF_REF you may have to unwind to a new pbuf. Or ensure 0-copy doesn't use small payloads (probably not hard to manage). Bill
