Hi Terry,
There are several approaches for you.
The "standard" way is to write a driver in kernel (the way Microtronix did for PIO buttons), implementing your PIO device as some kind of device Linux knows (the button is implemented as a character device). Then you can add the device node under /dev/ directory and read/write it as a file.
You can also take advantage of uClinux (no virtual memory and no memory protection) and read/write your device directly, but remember to bypass the cache. This is a quick (but dirty) approach. (It looks like you have tried this approach, but as long as you turned on the 31 bit of the USER_PIO_BASE, I don't think it caused the system crash).
Good luck,
wentao