RiscFree debugging fails after first debug

I have a simple Nios V system that prints "Hello World" on a DE-10. The design is generated using Quartus Std Lite 23.1.1. The first time I try to debug, everything is fine. The second time I try to debug, Ashling returns and error when a try and add a break point:

"Operation Failed: Command aborted"

Looking at the GDB output I get:

Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0x6a8

The only way I can recover is to program the .sof again, but it only works for one debug.

Is there anything that I can do because its annoying to have to program the SOF every time I want to debug?

Notes:

There is something strange in the at the ash-riscv-gdb-server output. On the first (successful) run it looks like:

Ashling GDB Server for RISC-V (ash-riscv-gdb-server).
v23.4.1, 03-Nov-2023, (c)Ashling Microsystems Ltd 2023.

Initializing connection ...
Checking for an active debug connection using the selected debug probe (SN: 1):
Connected to target device with IDCODE 0x2d020dd using USB-Blaster-2 via JTAG at 16.00MHz.
Info : Active Harts Detected : 1
Info : [0] System architecture : RV32
Info : [0] Number of hardware breakpoints available : 1 
Info : [0] Number of program buffers: 8
Info : [0] Number of data registers: 2
Info : [0] Memory access -> Program buffer
Info : [0] Memory access -> Abstract access memory
Info : [0] CSR & FP Register access -> Abstract commands

Waiting for debugger connection on port 3333 for core 0.
Press 'Q' to Quit.

On the second (failed) an subsequent runs it looks like:

Ashling GDB Server for RISC-V (ash-riscv-gdb-server).
v23.4.1, 03-Nov-2023, (c)Ashling Microsystems Ltd 2023.

Initializing connection ...
Checking for an active debug connection using the selected debug probe (SN: 1):
Connected to target device with IDCODE 0x2d020dd using USB-Blaster-2 via JTAG at 16.00MHz.
Info : Active Harts Detected : 1
Info : [0] System architecture : RV32
Info : [0] Number of hardware breakpoints available : 1 
Info : [0] Number of program buffers: 8
Info : [0] Number of data registers: 2
Info : [0] Memory access -> System bus (Data width : 128 Address size : 57)
Info : [0] Memory access -> Program buffer
Info : [0] Memory access -> Abstract access memory
Info : [0] CSR & FP Register access -> Abstract commands

Waiting for debugger connection on port 3333 for core 0.
Press 'Q' to Quit.

It looks like it has detected Memory access -> System bus (Data width : 128 Address size : 57) as a way to access memory. This seems strange because it didn't exist on the first run. Also the values for the width + address size are just wrong (and can change between runs).

This also happens if I use ash-riscv-gdb-server from the command line outside the RiscFree IDE. I have also tried using OpenOCD. It never fails. However, it causes ash-riscv-gdb-server to fail in the same way when its executed after OpenOCD.

Externally I was also able to debug multiple times provided I don't shutdown the ash-riscv-gdb-server. Everything starts failing when ash-riscv-gdb-server is re-run.

Nios Cpu

tehjingy_Altera
2 years ago
Hi

Sorry in that cause you will need to wait for the release of Quartus Standard 24.1 in Q1 next year.

Regards
Jingyang, Teh

20 Replies

tehjingy_Altera
Regular Contributor
2 years ago
Hi

After each debug did you close and remove the previous debug connection in the IDE ?

Regards
Jingyang, Teh
BrendenB
New Contributor
2 years ago
I hit the "stop" button which exits the debugger and ash-riscv-gdb-server.
I'm not 100% sure what you mean by removed, but I never deleted the run configuration. I just hit "stop", waited 15 seconds, then hit "debug" again.
I was also able to reproduce the error using the cli directly. The steps:
Program the .sof to start in clean state using: quartus_pgm --no_banner --mode=jtag -o "p;$PWD/../output_files/top.sof@2".
Run the ash-riscv-gdb-server: ~/intelFPGA/23.1.1/riscfree/debugger/gdbserver-riscv/ash-riscv-gdb-server --probe-type usb-blaster-2 --transport-type jtag --device 0x02D020DD --auto-detect true --jtag-frequency 24 --gdb-port 3333. This server starts without Memory access -> System bus.
Connect to the server using the debugger: ~/intelFPGA/23.1.1/riscfree/toolchain/riscv32-unknown-elf/bin/riscv32-unknown-elf-gdb -ex "target remote :3333" -ex "load" -ex "break main" -ex "c" app/build/Debug/hello.elf
You can connect to the server as many times as you want and it will work provided you don't shut it down.
Killed the server with 'Q'.
Run the ash-riscv-gdb-server with same command line above. The server starts with the Memory access -> System bus.
Connect to the server using the debugger with the command above. End up getting the message "Cannot insert breakpoint 1....Cannot access memory at address 0x6b0".
I added a bad_niosv.txt of all the output above.
I also did some digging. The debugging spec seems to give three different ways of letting the debugger access memory.
System bus
Program buffer
Abstract access memory
On the first run only Program buffer and Abstract access memory are available. The System bus method does not appear to be implemented by the Nios V (just guessing here). The System Bus Access Control and Status (sbcs, at 0x38) from the debugging spec reports if the System Bus method is supported. It would seem that this register is changed between the two invocations of the ash-riscv-gdb-server. I'm assuming the debugger is trying to use the system bus to write an instruction to implement the break point but fails because the system bus method is not supported and as such fails.
I also wanted to note that if you run openocd in step 2 instead of ash-riscv-gdb-server it also can cause Memory access -> System bus to show up in step 6. It doesn't happen always but its pretty common. Also note that openocd will use Program buffer to write memory over system bus and abstract access memory which is why is might be working while ash-riscv-gdb-server is failing (I'm assuming that it prioritizes System bus for memory access). Remember, if I use openocd in step 2 and step 6 then it all works.
Some other things I have tried to fix. Nothing worked:
Slowed the clock on my design down.
Slowed the JTAG clock down to 1MHZ.
Tried on both Nios V/g and Nios V/m with the same results. I am using the Nios V/m in my design.
bad_niosv.txt7 KB
XleraSolutions
New Contributor
2 years ago
Hi all,
I see this same issue, for about 6mo now. I had thought it was because I was running the Risc-V from external DDR3. However, I eventually built a reduced case that had nothing but the minimum required for Risc-V, running from internal blockram. Still the issue persists. Only way to get it to work is reload the SOF.
A similar issue it that the processor traps on a bad event when loading the elf and running. Often just hitting restart several times will get it to make it to main().
Nios-II never exhibited this behavior. A working theory is that the Risc-V core is executing garbage or still executing your elf from the last run. I've noticed that stopping or disconnecting seems to release the processor to run from there (I don't think the Nios-II did that). In any event, something seems to not get reset correctly in the Risc-V core when reloading.
BrendenB
New Contributor
2 years ago
I just wanted to mention a work around I have been using for debugging. I have been using OpenOCD running externally from the RiscFree IDE.
Terminal: openocd-cfg-gen debugger.cfg
Terminal: openocd -f debugger.cfg
IDE: Create "GDB OpenOCD Debugging Launcher"
IDE: On Main Configure executable.
IDE: Setup to compile the program before launch.
IDE: On Debugger tab Uncheck "Start OpenOCD locally". I'm running it externally to make things easier. I guess you can launch it from here if you wanted.
IDE: Executable name ${eclipse_home}/../toolchain/riscv32-unknown-elf/bin/riscv32-unknown-elf-gdb. This was automatically filled.
IDE: Commands: set mem inaccessible-by-default off. Again automatically filled.
IDE: Remote Target Host: localhost Port: 3333. Again automatically filled.
IDE: I left the other settings at their defaults.
This seems to be pretty reliable to me (but slower).
tehjingy_Altera
Regular Contributor
2 years ago

Hi
I mean removing the connection in the debug tab.
You could right click on the connection and removing the connection.
(Please ignore the connection name in the screenshot below)
Have you tried using the setting from the how-to in the below link?
https://www.intel.com/content/www/us/en/docs/programmable/730783/23-4/setting-debug-configurations-and-downloading-59901.html
Regards
Jingyang, Teh
tehjingy_Altera
Regular Contributor
2 years ago
Hi

Any update on this case?

Regards
Jingyang, Teh
BrendenB
New Contributor
2 years ago
I deleted the connection and tried again. Same result. Works the first time. Wont work again until I re-program the SOF.
I also tried deleting the connection after the error occurs and creating a new one. The connection can never be re-established until I re-program the SOF.
As I mentioned above, I am able to reproduce this outside of the IDE just using the command line tools manually (running them the same way the IDE does in the background). I'm not sure the IDE is playing a part here. Seems to be something wrong with the ash-riscv-gdb-server and/or the IP itself.
XleraSolutions
New Contributor
2 years ago
I agree, I have been dealing with the same behavior to the point that reloading the SOF is a habit. Posing a problem now that development is further along (plus QSPI booting now), since reloading the SOF is a full reset to everything.
Thought the issue was related to using off chip DDR, but reduced the case to minimal nios-v running from block ram with only the essentials.

Nios-II didn’t have this issue.
tehjingy_Altera
Regular Contributor
2 years ago
Hi
Could you try enabling the "Issue TAP_Reset" option in the debugger configuration?
Regards
Jingyang, Teh
KianHinT_altera
Frequent Contributor
2 years ago
Hi,
May I know the status of this case, whether anyone tried on the enabling the "Issue TAP_Reset" option in the debugger configuration?
Thanks
Regards
Kian