Trouble making timing with large number of memory banks

I have a design that has 14 memory banks. Each bank is 64xM20K blocks.

There is block (MUX) that reads/writes one bank, while another block (FFT) that reads/writes another bank.

There is a 1 to 14 registered router between the blocks and memories.

The clock speed is 180Mhz.

I am getting setup errors on read and write inputs to this memory from the registered router.

Depending on the build, -.5ns to -1.5ns.

My initial idea was to give the tool more registers between the router and memories. I double registered each of the 14 memories' input and outputs.

(The latency is not important in my design)

I did that, and when I look at final placement, all my extra registers got placed next to the router, and not somewhere half way between router and memories.

The router and memories are in the same vhdl file. But MUX and FFT are outside blocks.

Initially, the M20K usage was up to 99%. I brought it down to 82%, and I still see the same errors.

What do you recommend that I do to have my extra router registers be placed ½ way between router and memories ?