On further investigation I have found the following:
You need to maintain the sign after the shift, and you need to exit to state STOP earlier otherwise you exit to STOP at the end of the 9th cycle.
Here are the mod's:
process (Resetn, Clock)
begin
if Resetn = '0' then
state <= START;
z <= (others => '0');
elsif (Clock'event and Clock = '1') then
case state is
when START =>
Count <= "00001";
A <= x & "000000000";
S <= (not x + 1) & "000000000";
P <= "00000000" & y & '0';
state <= RUN;
when RUN =>
P(0) <= Ptmp(1);
P(1) <= Ptmp(2);
P(2) <= Ptmp(3);
P(3) <= Ptmp(4);
P(4) <= Ptmp(5);
P(5) <= Ptmp(6);
P(6) <= Ptmp(7);
P(7) <= Ptmp(8);
P(8) <= Ptmp(9);
P(9) <= Ptmp(10);
P(10) <= Ptmp(11);
P(11) <= Ptmp(12);
P(12) <= Ptmp(13);
P(13) <= Ptmp(14);
P(14) <= Ptmp(15);
P(15) <= Ptmp(16);
P(16) <= Ptmp(16); -- Keep the sign
Count <= Count + 1;
if Count > 7 then -- Exit at the end of the 8th cycle
state <= STOP;
end if;
when others =>
z <= P(16 downto 1);
state <= STOP;
end case;
end if;
end process;
-- Took this out of the process and made it combinatorial.
-- Output of the adder is registers by the shift register.
with P(1 downto 0) select
adderIp <=
"00000000000000000" when "00",
"00000000000000000" when "11",
A when "01",
S when others;
Ptmp <= P + adderIp;
I have simulated this and it works for a few test cases, but you will need to check it as I am not that familiar with Booth's algorithm.
As an asside the 17 lines of code for the shift register could be replaced with one line:
P <= Ptmp(16) & Ptmp(16 downto 1);
Steve