Drive pins to ground instead of using real ground

No. When an FPGA drives to "ground" its not really, its driving an output low.

Think about what you are asking a little bit more; your adapter card sends a single-ended signal over to the FPGA, and the signal return current returns via the ground pins.

If you had no ground pins at all on your connector, and only had FPGA pins. Driving some of the pins to logic low, does not create a "ground" connection.

You need to connect the "reference plane" on both boards together, i.e., the grounds.

The return current from your signal over the cable will not route through the FPGA and then out the pins that are driving logic low.

You can also think of this in terms of current; the FPGA output is low, due to the fact its sinking current on the I/O pin through the low CMOS gate to ground. If your signal return path current needs to take this path, then the current needs to go through the FPGA I/O pin in the wrong direction.

Cheers,

Dave