Restricted Fmax due to minimum period

I have created a simple project with a 500 MHz clock fed into a clock pin set for LVDS. This clock goes only to a divide-by-4 block (two flip-flops) and the resulting 125 MHz is fed to an output pin....