TSE MAC catatonic

Yes, I get that window with the cancel button. The system behavior is identical whether I click Cancel or not, both in 9.1sp2 and 11.0, which seems very strange.

I tried instantiating the OpenCore+ evaluation timeout indicator core; the timeout indicator works correctly (asserts only after the Cancel button is pressed). I wonder if the TSE core is timing out immediately after programming, since core timeout matches the problem description.

No in that case the problem is somewhere else... If the opencore timeout indicator says everything is okay, then the TSE should work too.

Can you add the statistics counter option when generating the TSE core, and have your software read them? You should be able to see if the received packets counter and/or error counter are increased.

I supposed you did it already, but check that the TSE core isn't in loopback mode.

--- Quote Start ---

Can you add the statistics counter option when generating the TSE core, and have your software read them? You should be able to see if the received packets counter and/or error counter are increased.

--- Quote End ---

I tried this. etherStatsOctets increases in a plausible way (sees network broadcast traffic). ifInErrors goes to 1 after the first received packet but never increases after that. The MAC should be configured to pass along RX error frames (RX_ERR_DISC cleared) but it does not appear to be doing so, nothing ever happens on the RX ST interface. None of the counters relevant to the TX side seem to be incrementing.

Here is a partial dump of some of the counters after a few seconds, during which several packets should have been successfully transferred in both directions:

 aFramesTransmittedOK=00000000    
 aFramesReceivedOK=00000000   
 aFramesCheckSequenceErrors=00000000
 aAlignmentErrors=00000000          
 aOctetsTransmittedOK=00000000
 aOctetsReceivedOK=00000000   
 ifInErrors=00000001       
 ifOutErrors=00000000
 etherStatsDropEvent=00000000
 etherStatsOctets=000009a2   
 etherStatsPkts=00000001  
 etherStatsUndersizePkts=00000000
 etherStatsOversizePkts=00000000 
 etherStatsJabbers=00000000     
 etherStatsFragments=00000000

--- Quote Start ---

I supposed you did it already, but check that the TSE core isn't in loopback mode.

--- Quote End ---

It is not, if I turn loopback mode on then the octet counters take off counting at high speed. No actual packets are transferred though. I bet this is an implementation artifact (maybe they assume there is valid MII data every cycle).

That's odd, it seems that something gets stuck inside the TSE core. But at least the fact that the octets counter increases shows that the opencore evaluation is still running.

I don't remember if this is possible with the small TSE core, but can you disable CRC checking?

Are the clocks correctly connected between the TSE core and the PHY chip?

Does the hardware design complies with all the timing requirements?

If you could replace the DMAs with Altera's SGDMAs and try to run a standard software example with uc/OS and Niche stack (such as the sockets server example) you could check if this as a hardware of software issue.

--- Quote Start ---

I don't remember if this is possible with the small TSE core, but can you disable CRC checking?

--- Quote End ---

I already have it off. I do need the full core anyway; the small TSE can't do TX store-and-forward mode, which my design will need to eliminate a potential source of TX packet loss.

--- Quote Start ---

Are the clocks correctly connected between the TSE core and the PHY chip?

--- Quote End ---

As far as I can tell, yes. For plain MII mode I am getting both RX and TX clocks directly from the PHY. Both clocks are definitely toggling, the TSE will get stuck in its software reset if they don't (I had it happen). Further, the RX MII data looks perfect in SignalTap.

--- Quote Start ---

Does the hardware design complies with all the timing requirements?

--- Quote End ---

Aside from a couple of harmless issues, yes. (There is an asynch reset that feeds all the reset-CDCs which has overly tight constraints on it). All clock domains related to the TSE should be good. The entire rest of the system (3 clock domains) works flawlessly.

--- Quote Start ---

If you could replace the DMAs with Altera's SGDMAs and try to run a standard software example with uc/OS and Niche stack (such as the sockets server example) you could check if this as a hardware of software issue.

--- Quote End ---

Hm, now we come to the tough one. This isn't a Nios system, it is a 4-CPU superscalar MIPS with full cache coherence (yes, cache-coherent DMA). The CPUs and system design are well-proven, and the fact that the MAC and ST interfaces are what's stuck should hopefully rule out all the really tough problems. But unfortunately I can't just drop in Altera's software stack and test it out.

One particularly notable issue related to the different CPU type is the I/O access width when poking the TSE registers. The driver I wrote is very careful to do only aligned full-word reads/writes, so that byte-enables are unnecessary for the TSE. The TSE registers do get written with the intended values so I know this is done right. The Avalon-MM transactions look correct on SignalTap as well. If it would help to post and annotate the exact sequence of read/write transactions to the TSE let me know.

Yes a capture of the Avalon slave interface could be useful. I have a (working!) board with some MII PHY chips, I can do it on my side too.

There are two more things that would come to mind, since you are using a different CPU... one would be to check you are using the correct endianness (I think you do, but you never know, it can be worth mentioning) and that the register addresses described in the TSE datasheet are double word addresses, not byte addresses.

While trying to get you a clean capture of the MAC initialization, I found the bug... it was quite embarassing. I missed a ~ operator when masking out the half-duplex enable bit on link-up, so when the link came up the Command_Config register got mistakenly cleared. Oops.

It looks like things are working more now, I have some DMA alignment issues to take care of but the MAC seems to be OK. I did have to switch to Quartus 11.0 after encountering an RX FIFO bug in 9.1sp2 (underflow, most likely).

Thanks for the help though!

--- Quote Start ---

There is an option in the MAC to add/remove two extra null bytes at the beginning of the packet.

--- Quote End ---

I'm aware of this option, under Linux there is a constant NET_IP_ALIGN set to either 0 or (more typically) 2 which determines whether the network stack expects the extra 2 bytes or not. On mipsel it is 2, so my driver sets RX_SHIFT16 and TX_SHIFT16 and adjusts the DMA pointers as it should. Everything here works as expected. (The existing altera_tse.c and atse.c drivers seem to completely misunderstand it)

The main alignment issue is because my system busses and DMA are 64-bit wide, as are the CPUs' cache interfaces, but the CPUs are otherwise 32-bit. This means my buffers are only aligned on 4-byte boundaries, not 8. The unaligned transfer support in the mSGDMA controllers does work as advertised, which is an easy fix.

With those problems sorted I can boot my system on a uniprocessor kernel over the network for about 2 minutes, then the transmit DMA deadlocks due to a missed TX completion. With all 4 CPUs active the DMA lockup happens within a second.

The issue is racing between the RX/TX IRQ handlers and the DMA hardware. It is possible for the mSGDMA to complete a descriptor and push it to the response buffer at the same time the CPU drains the response buffer and clears the interrupt flag. Altera's SGDMA has similar problems, the descriptor update can race with the descriptor chain walk and the CPU can wind up not seeing freshly completed packets.

Altera's TSE/SGDMA driver appears to avoid this problem by simply polling for TX completion... which is an utter travesty. The entire point of having scatter-gather DMA is to queue multiple packets and then move on to processing something else, not to sit there busy-looping.

It looks like I am going to have to write my own DMA controller to avoid this bug. I don't want to change the mSGDMA code, it is already far more complicated than it really should be. The DMA hardware needs to have cleaner IRQ behavior to close the race windows without polling hacks. It will just take me a few days to sort out properly.