lwip MicroC/OS-II throughput

This is an update on the reply that I posted a couple of days ago. I've found out why I had such different time measurements between the 1.1 toolkit and the 5.0 toolkit. There was one other modification that I had done to the 1.1 environment that I hadn't bothered to move forward. I had no idea that it had speed implications, but in fact it did.

The 1.1 toolkit's lwIP version had problems in the protection of pbuf's in a multitasking environment. The only way to get effective protection was to specify SYS_LIGHTWEIGHT_PROT. That forced the lwIP library to used interrupt disables rather than a semaphore to protect pbuf's and it's various memory pools. (A couple of other declarations were also required to make this work.)

The 5.0 toolkit's lwIP version resolved the problems in the protection of pbuf's, and the default distribution used semaphores rather than the "lightweight" mechanism. Because of this, I didn't bother to pull forward the changes I had made in lwipopts.h to specify and facilitate SYS_LIGHTWEIGHT_PROT. Yesterday, for reasons unrelated to speed tuning, I had to pull this change into the 5.0 toolkit. Much to my surprise, I got back most of the time that had been lost. It amounted to almost 400us in the handling of two outgoing packets and one incoming ACK. I'm now back in the vicinity of 1.5ms for these operations. This is just under 11Mbps for the payloads in question.

It seems unlikely that semaphores would be so costly, but lwIP does load them down with a mechanism of its own for executing specified functions at specified times. Even with this, though, there must be a lot of semaphore calls in the processing of normal packets.

If you are tuning for performance, I strongly recommend changing to the lightweight protection mechanism. This may well be more important than any of the loop unrolling that I did.