I try your example and the result it's wrong...
using language vhdl y can't be an output .. because the compiler send me the error that an output can't be read..
so i use always signal and i use output only for the debug...: so i write this to test your example
signal y1,y2,y3 : integer :=0;
shared variable a1,a2,b0,b1,b2;
process
wait until clk='1'
a1:=2;
a2:=1;
b0:=1;
b1:=2;
b2:=3;
y1<=x_in*b2+y*a2;
y1debug<=y1;
y2<=x_in*b1+y*a1+y1;
y2debug<=y2;
end process;
y<=b0*x_in+y2;
yout<=y;
end architecture;
using the coefficient a1=2 a2=1 b0=1 b1=2 b2=3 and x(0)=1 x(1)=0 x(2)=0
y must be:
y(0)=1
y(1)=4
y(2)=12
y(3)=28
y(4)=68 ecc...
but using your code (readapted) the output is show in figure example.pdf simulate with qsim.
and the number are different.. there is something that not function well...