Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
16 years ago

Debug user space application with MMU

I am trying to use the gdbserver as described here:

http://nioswiki.com/linux/debug_user_space

At some point, this was working, but sometimes it wouldn't work and now I haven't had it working for a while. The application and gdb dies on a crash and doesn't let me debug. This isn't just this particular crash, I've had this happen with almost every crash while developing.

On the server:

--- Quote Start ---

root:/> gdbserver localhost:9999 myprogram

Process myprogram created; pid = 50

Listening on port 9999

Remote debugging from host 192.168.100.28

<program output...>

Child terminated with signal = b

Child terminated with signal = 0xb (SIGSEGV)

GDBserver exiting

root:/>

--- Quote End ---

On the client:

--- Quote Start ---

Program terminated with signal SIGSEGV, Segmentation fault.

The program no longer exists.

(gdb) backtrace

No stack.

--- Quote End ---

Any advice?

Also, the wiki says "we will add the gdbserver source to uclinux-dist later" -- is there any progress on this? Could the gdbserver currently in uclinux-dist work?

16 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor

    This may be related to troubles debugging. I am trying to print a stack trace from within the program, but that doesn't work either. I tried with a very simple non-threaded C program:

    #include <stdio.h>
    # include <execinfo.h>
    # include <stdlib.h>
    void print_trace() {
      void *array;
      size_t size;
      // get void*'s for all entries on the stack
      size = backtrace(array, 10);
      // print out all the frames to stderr
      fprintf(stderr, "trace size %zu:\n", size);
      backtrace_symbols_fd(array, size, 2);
      exit(1);
    }
    void baz() {
      print_trace();
    }
    void bar() { baz(); }
    void foo() { bar(); }
    int main(int argc, char **argv) {
      foo();
    }
    

    When I build this on my PC (Ubuntu 10.10 x86_64) I get a nice stacktrace:

    gcc -g -rdynamic -o test_bt test_bt.c

    trace size 7:
    ./test_bt(print_trace+0x19)
    ./test_bt(baz+0xe)
    ./test_bt(bar+0xe)
    ./test_bt(foo+0xe)
    ./test_bt(main+0x19)
    /lib/libc.so.6(__libc_start_main+0xfe)
    ./test_bt

    On Nios II Linux I just get this:

    nios2-linux-gnu-gcc -g -rdynamic -o test_bt test_bt.c

    trace size 1:
    

    I've been trying to mess with compiler flags, but haven't found any magic so far. Any ideas?
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor

    --- Quote Start ---

    I am trying to use a core file now. I am running `nios2-linux-gnu-gdb mynios2program`, with set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/ in .gdbinit. Then core-file mycorefile, but all I can get is:

    warning: Couldn't find general-purpose registers in core file.
    # 0  0x00000000 in ?? ()

    --- Quote End ---

    I hit the same problem here too and traced it down to the kernel writing a faulty corefile, missing the .reg and .reg/<N> section. The cause for this was ELF_NGREG in arch/nios2/include/asm/elf.h being too small ((sizeof (struct pt_regs) / sizeof(elf_greg_t)), while ELF_CORE_COPY_REGS also writes all registers from struct switch_stack to the buffer.

    If ELF_NGREG is defined as in the following patch, the corefile is generated properly and can be used with nios2-linux-gnu-gdb:

    
    -#define ELF_NGREG (sizeof (struct pt_regs) / sizeof(elf_greg_t))
    +#define ELF_NGREG ((sizeof(struct pt_regs) + sizeof(struct switch_stack)) / sizeof(elf_greg_t))
    

    I'll also apply this fix to the nios2 branch of the nios2-linux kernel git repository.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor

    Tobias, I just saw this fix in the commit log and was surprised to see a link back to my report here because I missed your response in the thread. I will try this fix!

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor

    --- Quote Start ---

    This may be related to troubles debugging. I am trying to print a stack trace from within the program, but that doesn't work either.

    --- Quote End ---

    Have you get it to work, ykozlov? I couldn't print out a stacktrace in a user-level program with backtrace() / backtrace_symbols()...
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor

    --- Quote Start ---

    Have you get it to work, ykozlov? I couldn't print out a stacktrace in a user-level program with backtrace() / backtrace_symbols()...

    --- Quote End ---

    No, I never got that working, but tklauser's kernel fix did fix debugging using a core dump.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor

    --- Quote Start ---

    No, I never got that working, but tklauser's kernel fix did fix debugging using a core dump.

    --- Quote End ---

    And I fixed the backtrace() function... or bypass it and wrote my own!

    stackframe.c
    # define _GNU_SOURCE
    # include <stdio.h># include <inttypes.h># include <stdint.h># include <ucontext.h># include <dlfcn.h># include <string.h>
    struct frame {
        void *fp; // frame pointer
        void *ra; // return address
    };
    void print_symbol(void *address) {
        Dl_info info;
        char    library;
        char    function;
        void   *symbol_address;
        if (dladdr(address, &info) == 0) {
            strncpy(function, "???", sizeof(function));
            strncpy(library, "???", sizeof(library));
            symbol_address = 0;
        } else {
            strncpy(function, info.dli_sname, sizeof(function));
            strncpy(library, info.dli_fname, sizeof(library));
            symbol_address = info.dli_saddr;
        }
        printf("0x%08X %s: %s\n", (uint32_t) symbol_address, library, function);
    }
    void backtrace() {
        struct frame *frame;
        // frame = CURRENT_FRAME_POINTER
        __asm("mov %0,r28" : "=r"(frame));
        // Last frames
        while (frame && frame->fp) {
            printf("fp = 0x%08X ra = 0x%08X\n", (uint32_t) frame->fp, (uint32_t) frame->ra);
            print_symbol((void *) frame->ra);
            frame = (struct frame *) frame->fp;
        }
    }
    void test3() {
        backtrace();
    }
    void test2() {
        test3();
    }
    void test1() {
        test2();
    }
    int main(int argc, char *argv) {
        test1();
        return 0;
    }
    Makefile
    
    CC             = nios2-linux-gnu-gcc
    CFLAGS         = -Wall -fno-omit-frame-pointer
    LDFLAGS        = -rdynamic -ldl
    SOURCES = stackframe.c
    OBJECTS = $(SOURCES:%.c=%.o)
    PROGRAM = stackframe
    .PHONY: all clean
    all: $(PROGRAM)
    clean:
        rm -rf $(OBJECTS) $(PROGRAM)
    build: clean all
    $(PROGRAM): $(OBJECTS)
        $(CC) -o $@ $(OBJECTS) $(LDFLAGS)
    %.o: %.c
        $(CC) $(CFLAGS) -c $<
    
    Build:
    $ make
    nios2-linux-gnu-gcc -Wall -fno-omit-frame-pointer -c stackframe.c
    nios2-linux-gnu-gcc -o stackframe stackframe.o -rdynamic -ldl
    
    Execute:
    # stackframe
    fp = 0x7FE81D24 ra = 0x00001838
    0x00001824 stackframe: test3
    fp = 0x7FE81D2C ra = 0x00001860
    0x0000184C stackframe: test2
    fp = 0x7FE81D34 ra = 0x00001888
    0x00001874 stackframe: test1
    fp = 0x7FE81D44 ra = 0x000018B8
    0x0000189C stackframe: main