Why does this design not fit and how to reason about congestion

This report shows that failure is occurring during fitting due to lack of DSPs (or maybe DSPs that match placement requirements). Does the compiler also generate the file named top.fit.summary in this case? That report should make it clear whether you are really running out of DSPs or not (even though I regularly see negative or overflown numbers in that file).

With block size 16 for your first code, I get this with Quartus 16.1.2 and Arria 10:

+--------------------------------------------------------------------+
; Estimated Resource Usage Summary                                   ;
+----------------------------------------+---------------------------+
; Resource                               + Usage                     ;
+----------------------------------------+---------------------------+
; Logic utilization                      ;   65%                     ;
; ALUTs                                  ;   31%                     ;
; Dedicated logic registers              ;   35%                     ;
; Memory blocks                          ;   40%                     ;
; DSP blocks                             ;   90%                     ;
+----------------------------------------+---------------------------;

I would expect this to be very hard to fit on the FPGA, especially since the logic utilization is probably underestimated.

Your second code takes a very long time to do the first stage of compilation with a block size of 16, so I gave up on it, but the estimation seems to report lower number of DSPs for a block size of 8, compared to the first code.

Both codes seem to compile very nicely with all loops being fully-pipelined with an II of one, and based on the system overview from Quartus v16.1.2, the system has 4 fully-coalesced memory ports and a low-latency peipeline (100-200 clocks). I guess this case is simply a problem of high area utilization. You could reduce your area overhead by converting some of your less important fully-unrolled loops to partially-unrolled ones.