Multicycle custom instruction repeated calls, hangs, signals

I think you should be setting DONE to '0' not 'U'.

Under simulation 'U' is probably different from both '0' and '1', on real silicon it will either be '0' or '1'.

So quite likely the value doesn't change - and you need it to be 0.

If it has a fixed value of '1' then the instruction would terminate immediately - which is one thing you are seeing.