Unless your oscilloscope and probe bandwidth is 2 GHz or better above, you won't see anything but a sine. Of course you need a suitable differential I/O standard with correct termination. If you don't have a multi GHz active probe, a direct connected 50 ohm cable can reproduce the waveform.
Weird behaviour of your "verilog hardware" is rather a problem of timing violations or other design errors. To make it understandable, you should briefly tell what you are trying to achieve and how you detect "weird behaviour".
P.S.: I see, that Stratix III can also achieve up to 1000 MHz toggle rate with some single ended IO standards. With the DE3 board, it's probably best to use the SMA clock output for this tests. The external output's waveform hasn't anything to do with waveforms of the internal clock tree, but frequency and jitter performance can be monitored.