Proper flow for debugging code loaded from ROM (QSPI)

I have a board where I loaded my code into QSPI for loaded by the HPS at boot time. This works. But when I connect with the debugger, and load my symbols for that code, I cannot set/hit breakpoints, ...