simulate a vhdl code of a function with ISE

In this line

sum <= sum + x(k)* h(i-k);

as sum is a signal, it will only get its value from the simulator after the process was executed. There is always asmall delay (called delta delay) before a signal gets it's value. Therefore sum will always stay at 0. If you want to make an accumulator that is updated without waiting between each operation, you need to make sum a variable.

But this still isn't synthesizable, as you are using reals. Even if it where, as you are doing all the operations simultaneously it will generate a very complicated hardware as you increase the values of n and m. I would suggest to use fixed point types instead of reals, and add a clock to do only one multiplication/accumulation per clock cycle.