Altera_Forum
Honored Contributor
14 years agoALTFP_INV_SQRT problem
I have instanced ALTFP_INV_SQRT (y = 1/sqrt(y)) mega function in Quartus 10. I’m using single precision. When I calculate X”3F800001" (almost 1) the result is X”3F000000” (0.5). Calculating X” 3F800000" and X” 3F800002" is OK as shown below.
Simulations: X"3F800000" => X"3F800000" or 1,0000000000000000 => 1,0000000000000000 X"3F800001" => X"3F000000" or 1,0000001192092896 => 0,5000000000000000 -ERROR X"3F800002" => X"3F7FFFFE" or 1,0000002384185791 => 0,9999998807907104