Moving object detection using D5M/DE1-SoC

To debug the design, I removed whole algorithm from the design. For now, its just transferring pixel data as given by camera, to VGA. And it doesn't buffer the frame. It transfer pixel by pixel, as it gets from camera using Avalon Streaming. The data bus in Avalon Streaming is 24bit [8:8:8][RGB]. The only algo i have is a small comparator which checks for the presence of black pixels[0x000000]. If i move my black object slowly the algo asserts a flag but as soon I increase its speed, it de-asserts flag. It means it couldn't even detect the object in frame because it was moving. What can be done to achieve the moving object detection to a more better extent?