NIOS Software appears to restart

If you aren't using a multitask operating system, then the stack and the heap memory will share the same memory space. The heap will just begin to allocate the memory from the beginning of the area, whereas the stack will start from it's end and go down as you use it. I'm not sure a malloc() call will in fact report insufficient memory before it collides with the stack.

Do you allocate a lot of memory in your code? Are you using a mechanism that puts a lot on the stack, such as a recursive algorithm? If you use the debugger, you should be able to see the addresses of the allocated buffers returned by malloc() and monitor the stack pointer, to check for any collisions.