Optimizing my first VHDL project, big complexion of algorithm

1) The compiler should handle that just fine, makes no difference in logic complexity.

2) Signals and variables don't have a direct difference in complexity. They do have different behaviors and you should use whichever best describes the behavior you need.

3) Functions can be OK or not, depending on the logic they implement.

4) Loops can be a big problem...

5) I'm slightly amazed Quartus even compiled that...

I think you need to go back to basics and find a tutorial for digital logic and synthesisable VHDL. In terms of synthesis, your project is so complex I can't even begin to tell you where to fix it.

You do need to be able to think what your code will become in terms of hardware.

For example, the following translates into a combinatory multiplier block (a*b), whose result is fed to rising edge flip-flop bank with asynchronous reset.

process(reset, clk) begin
if reset = '1' then
y <= (others => '0');
elsif rising_edge(clk) then
y <= a*b;
end if;
end process;

The following translates to actually N copies of the above.

process(reset, clk) begin
if reset = '1' then
	for i in 0 to N-1 loop
		y(i) <= (others => '0');
	end loop;
elsif rising_edge(clk) then
	for i in 0 to N-1 loop 
		y(i) = a(i) * b(i);
	end loop;
end if;
end process;

And this translates into a chain of N-1 multiplier blocks feeding into a register bank.

The chain will also have N-1 times the delay, so this will not be able to work at a high frequency.

process(reset, clk) 
variable t : some_type;
begin
if reset = '1' then
	for i in 0 to n-1 loop
		y(i) <= (others => '0');
	end loop;
elsif rising_edge(clk) then
	t := a(0);
	for i in 1 to n-1 loop 
		t := t * a(i);
	end loop;
 	y <= t;
end if;
end process;