Linux with MMU on NEEK

Hi,

--- Quote Start ---

I do hope that this gets into the official distribution, so that we are able to use it just by setting the appropriate config option.

--- Quote End ---

I think that Hippo is the main maintainer of this distri, so he will decide it. (We need a kind of dictator for these things;).)

--- Quote Start ---

Moreover it then would be great to have a Wiki page describing the ways possibilities and how to do this using Eclipse as a gdb frontend.

(Do you know the book "The Art of Debugging with GDB, DDD, and Eclipse" ? )

--- Quote End ---

Of course, I know DDD and Eclipse well. And I think to connect these front-ends are not so hard task. But I have less interest for those things, because I'm not a software programmer. Or rather, I have much interest for 'futex' or 'SMP', though we have many problems for atomic codes and cache coherency. But is it possible?

Please let me know your opinion.

Kazu

Thank you for the wonderful works, Kazu.

Surely, I will commit them to the server. Though I am extremely busy on a hardware project this month.

About Eclipse and DDD, it is almost the same as what is in nommu wiki. Just change the compiler prefix and give the solib path. Threads (NPTL) work.

Cheers,

Hippo

--- Quote Start ---

Or rather, I have much interest for 'futex' or 'SMP', though we have many problems for atomic codes and cache coherency. But is it possible?Kazu

--- Quote End ---

I'm very interested in threaded applications and thus in Futex and thus in atomic user land code.

I do know that Hippo is working on this for the MMU based Linux.

As NIOS does not have atomic operations, we need support by the OS for user-land atomic operations. Hippo is planing to do this by defining an "atomic area" to hold the appropriate supported atomic functions within the RAM page that is used for the MMU-cache and that is located within the FPGA as a TCM (and thus fast and not cacheable). You may want to take a look at the appropriate thread in the NIOS uClinux mailing list, where Hippo and I discussed that issue.

I do intend to help on that area, but right now I'm too busy with other (no-NIOS) stuff. At least I already did create a testing program that can be used to verify and to do some timing tests with atomic code and the various possible variants of doing a MUTETX / FUTEX.

I'm sure Hippo (and I) will be very happy to have you in the boat to make FUTEX work for NIOS / MMU.

SMP is a quite different beast. Hippo decided (IMHO wisely) do delay this issue and first provide a single CPU FUTEX.

I already did discuss (with Hippo and with an Altera FEA) doing SMP-save user-land atomic operations by introducing a "Hardware-FUTEX" by doing a multi-CPU-atomic "compare and xchange" custom instruction. One problem is that a custom instruction can't access memory through the cache and thus it only can decently handle a limited dedicated internal memory area for the "FUTEX"es and thus it only can be used a means to protect the user-code sequences handling the normal memory based Linux-FUTEXes. This two stage method of course asks for some "interesting" code ;).

Supposedly something like that is needed for SMP-Kernel-code, as well, but I did not take a look into that, yet.

At best, Altera would add some decent atomic support for NIOS (e.g. "load locked / store conditional" as introduced with the modern ARM CPUs). But I doubt that this will happen some day soon. Maybe Hippo has or can get some information on Altera's plans on this.

Maybe we should discuss all this on the (IMHO more appropriate) mailng list. Where I'm sure Hippo and others will jump in.

-Michael

Hi,

Thank you, Michael. I've read your post in the mailing list and learned a lot. The next is my humble opinion.

For the 'futex',

If we use no-MMU and non-SMP Nios uClinux system, I think the 'futex' is an easy problem, because we can control the Nios's interrupt enable bit from the userland. In this case, only interrupts and task switches are the factors which destroy the atomicity.

With MMU and non-SMP case. your plan to make custom instructions 'lock1' and 'lock2' are dangerous for the OS's security, because it means that we can control the interrupts from our userland. Moreover, we can't perceive the codes flows in the Nios's pipeline and get the exact timing of accepting interruptions.

It seems to me that the idea of 'Common Area for atomic functions' deprives the flexibility of 'futex'. From the viewpoint of userlands, 'futex' is only an int-type counter variable that is located in a shared memory. We can access it from anywhere (of course from our userland) and the kernel doesn't know it till the contention will occur.

Fundamentally, we should not revise the kernel's machine-independent codes or add new codes to there, and should keep the 'look & feel' of userland interfaces (not only for the 'futex').

So I think that the idea of emulations by 'hardware mutex' like Mr. Ben Twijnstra

http://sopc.et.ntust.edu.tw/pipermail/nios2-dev/2009-february/002439.html

is the best method, though we must change the futex's integer valuable to a pointer that points a 'hardware mutex'. I'm not sure how this will influence the kernel codes, but maybe, we can hide everything under the machine-dependent codes and their macros.

For the possibility of SMP by Nios CPU, I will write it later.

Kazu

Kazu,

With No-MMU, Futex is of very limited interest, as the no-MMU toolchain uses gcc3 while the MMU toolchain uses gcc4. Gcc4 can't compile for no-MMU hardware and gcc3 can't do TLS (Thread loacl storage: "_thread" variables in C) and thus is quite unusable for a decent threaded application, and does not support NPTL (Native Posix Thread Library).

As Linux allows the user land code to create as many Futexes as it desires, the Futex variables need to be located in normal RAM, and thus the standard Futex user-library and Kernel code needs to be used. The only thing that needs to be done architecture-specific is the userland atomic instructions.

Here Hippo suggested to follow the way that the BlackFin people, featuring the same problem, go: defining an "atomic region" that holds the appropriate functions and gets special handling by the Kernel's interrupt code.

This method is well proven and works without any hardware support.

Some hardware support (disabling the interrupt for a limited time that is automatically cut by the hardware) might be constructed at a later time if desired and optionally be enabled when configuring the system. But this should be not discussed right now.

-Michael

Hi,

Thank you for your quick reply.

--- Quote Start ---

As Linux allows the user land code to create as many Futexes as it desires, the Futex variables need to be located in normal RAM, and thus the standard Futex user-library and Kernel code needs to be used. The only thing that needs to be done architecture-specific is the userland atomic instructions.

--- Quote End ---

Of course, I'm thinking not only one 'hardware mutex' but special memory regions that emulate atomic codes. Each futex 'pointer' for their counters points one special 'hardware mutex' respectively. 'Hardware mutex' will limits the amount of futex counters, but do you need those more than one thousand? Of course, if there is a law that prohibits to change the type of futex counters, this method is impossible.

--- Quote Start ---

Here Hippo suggested to follow the way that the BlackFin people, featuring the same problem, go: defining an "atomic region" that holds the appropriate functions and gets special handling by the Kernel's interrupt code.

--- Quote End ---

If the main maintainer supports this plan, I will obey it.

--- Quote Start ---

Some hardware support (disabling the interrupt for a limited time that is automatically cut by the hardware) might be constructed at a later time if desired and optionally be enabled when configuring the system.

--- Quote End ---

I think that 'disabling the interrupt for a limited time' is quite difficult because we can't detect the instruction despatch and know what pipeline stage deals with the interruptions.

Kazu

--- Quote Start ---

I think that 'disabling the interrupt for a limited time' is quite difficult because we can't detect the instruction despatch and know what pipeline stage deals with the interruptions.

--- Quote End ---

With embedded software we can assume the user processes to be fair, and thus we just should try to avoid a hanging system due to blocked interrupts and so I think a simple timeout of - say - 1000 clock cycles should do here. Of course the user land process would need to re-enable the interrupt as soon as possible.

But I do see another major issue: The interrupt disabling by a custom instruction needs to be done outside of the CPU and thus the timing will be quite sloppy regarding the instructions. So I suppose 100 % atomicness can't be granted and I think the hardware support should be implemented on top of Hippo's suggestion, just in 99,9 % of the cases preventing the necessity for the ISR to bother with the atomic user code, as the ISR is normally not called any more if the user code is in the atomic region.

The mainline macro for an atomic operations would be

___ custom disable interrupt

___ call appropriate_function_in_atomic_page

___ custom enable interrupt

instead of just

___ call appropriate_function_in_atomic_page

if no custom disable interrupt hardware is available

The Kernel code would stay the same.

This problem might make the SMP implementation even more interesting...

-Michael

--- Quote Start ---

Hi, all.

I'm testing Linux MMU version, on my NEEK.

http://www.nioswiki.com/linux

It works fine and I can use "bash" shell. This is the evident proof that we are using the true 'fork' instead of 'vfork'.

May be this will depends on the version, but TSE driver claims an error and doesn't work on this design. The error is

ERROR: altera_tse.c:1666: request_mem_region() failed

I think that this error is caused by misunderstanding of the usage for the function request_mem_region(). Inside of the request_mem_region(), the function __request_region() is called. If the resource has been already registered, this function returns a non-NULL value, that is the pointer for its resource. But the resource 'sgdma_rx_base' is already registered in the initialization process, so this function returns the 'conflict' and

    if (!request_mem_region(sgdma_rx_base, sgdma_rx_size, "altera_tse")) {

is always true. So I made a dirty patch,

    if (!request_mem_region(sgdma_rx_base, sgdma_rx_size, "altera_tse")) {
        reg_resource = __request_region(&iomem_resource, sgdma_rx_base, sgdma_rx_size, "altera_tse", 0);
        if (reg_resource != NULL && reg_resource->flags & IORESOURCE_BUSY) {
            printk(KERN_ERR "ERROR: %s:%d: request_mem_region() failed\n", __FILE__, __LINE__);
            ret = -EBUSY;
            goto out_sgdma_rx;
        }
    }

Moreover, the author is forgetting that the DMA is working in the physical address world,

so we need to set the pointers of descripters like

//    desc->source = read_addr;
    desc->source = virt_to_phys(read_addr);
//    desc->destination = write_addr;
    desc->destination = virt_to_phys(write_addr);
//    desc->next = (unsigned int *)next;
    desc->next = (unsigned int *)((unsigned long)next & 0x1fffffffUL);

and so on.

--- Quote End ---

Did you get any of these patches into the distribution? I am starting to work with the MMU and both of the Altera TSE drivers are broken with this version (more so than before ;)). I have fixed the atse.c driver to use the new macros for addresses, but need some more work on it to put ioremap()s in the right places because it uses phycal addresses for MAC control registers and descriptor memory. I am fairly new to driver programming so I am also a bit confused between ioremap, request_mem_region, phys_to_virt and all. Did you have to do anything else to get the altera_tse.c driver working?

Hi,

--- Quote Start ---

Did you have to do anything else to get the altera_tse.c driver working?

--- Quote End ---

Please see and try this.

Kazu

P.S. Why can't we attach *.c and *.h files directly?

If this code does work why not upload it as a patch ?

-Michael