Moving object detection using D5M/DE1-SoC

The D5M camera has an image sensor with rolling shutter. This may explain why your algorithm doesn't work for fast moving objects. At 640 x 480 you should be able to get more frames per second, by shortening the exposure time and at the same time increasing the illumination of the scene.

Also detecting black pixels comparing them to (only) 0 is a bit optimistic. When moving (fast) a pixel sees a mixture of black and white, especially when you have a relatively long exposure time. Setting a threshold of 1/10th (or so) would probably work better.