Multicycle custom instruction repeated calls, hangs, signals

Try finding your custom instruction on the RTL viewer.

For combinatorial ones it shows quite cleary how they connect to the main cpu.

Signaltap might also be informative.

I'm not vhdl expert, but I'd have thought that you'd need to set DONE to zero in the reset clause - setting things to 'U' is probably only relevenant for simulation.