5 bit arithmetic with negative value

I am currently doing a project where I need to display the result of an arithmetic operation. Everything seems to go perfectly, however when I do an operation such as (+A) + (-B), my result is incorrect. For example, doing (+5) + (-3) is giving me a result of -14. Below is my code where I have the magnitude and sign (+/-) for 'a', 'b' and 'r' (result), as well as 'i' which I'm using to select the operation for each combination. The problem obviously stems from somewhere in the 'i' select that my condition is not being properly met, however it all makes sense to me. I would appreciate if someone would point me in the right direction. Thank you!

library ieee;
use ieee.std_logic_1164.all;
use ieee.std_logic_unsigned.all;
entity sm_arith is 
port ( 
  a, b: in std_logic_vector(4 downto 0); 
  add: in std_logic; 
  r: out std_logic_vector(4 downto 0) 
); 
end sm_arith; 
architecture arch of sm_arith is
    signal as, bs, rs: std_logic;
    signal am, bm, rm: std_logic_vector(4 downto 0);
    signal i: std_logic_vector(2 downto 0);
begin
    as <= a(4);
    bs <= b(4);
    am <= a(4 downto 0);
    bm <= b(4 downto 0);
    i(2) <= not(add);
    i(1) <= as xor bs;
    i(0) <= '0' when am >= bm else '1';
with i select
    rm <= am + bm when "000",
            am - bm when "010",
            bm - am when "011",
            am - bm when "100",
            bm - am when "101",
            am + bm when "110",
            bm + am when others;
rs <= as when am > bm else (bs xor i(2));
r(4) <= rs;
r(3 downto 0) <= rm(3 downto 0);
end arch; 

**Update #1:**

After adding the numeric library and changing my i select to:

   with i select
 
         rm <= std_logic_vector(signed(am) + signed(bm)) when "000",
 
               std_logic_vector(signed(am) - signed(bm)) when "010", etc.

I am still running into the same problem when B is negative and A is positive. This was fixed when I changed my unsigned library to signed, however I would run into a similar problem when my A was negative and B was positive. Eg, (-5) + (+3) = +14.