quartus 2: can't determine definition of operator "*"

Remember that 1/4 = 0 because it is integer division, and the VHDL rules mean that division is completed before multiply unless you put expressions in brackets. This is just standard programming when it comes to integer division

So 1/4 * 100 = 0

100 / 4 = 25.

Just put the /4 after the multiply:

(imagemanterior(x,y+1) - imagemanterior(x,y) + imagemanterior (x+1,y+1) - imagemanterior(x+1,y) + imagem(x,y+1) - imagem(x,y)+imagem(x+1,y+1)-imagem(x,y+1)) /4

/4 is safe in this case as it is just a power of 2, which is really just a bit shift and free in terms of logic. If you need any non-power of 2 division, then you need to use the division IP core.

The problem is because you are mixing signed and unsigned. Make everything signed and it should work.