can I lock system memory through pci express

No, you cannot. You have to maintain ownership of the memory areas by means of having the driver interact with your OS.

The typical process is:

1. Driver allocates kernel memory, i.e. memory that can be accessed by hardware.

2. Driver maps the memory to hardware, and the OS states the hardware accessible address of that memory space.

3. Driver remembers that the memory is allocated for the device, i.e. it is not allowed to manipulate or free the memory space at that time.

4. Driver informs hardware of the memory address. This can be done easily by a PIO write or, more efficiently/more complicated by entry in a DMA descriptor table.

5. Hardware writes data. Just normal write accesses (probably with relaxed ordering bit set for efficiency).

6. Hardware informs driver of the transfer success. This is done either by a INTx, a MSI/MSI-X, or in case of the DMA descriptor table, by an update of the descriptor entry. From that moment on, hardware is not allowed to write to that memory anymore.

7. Driver gets informed of data update by interrupt or polling (e.g. Linux NAPI). It forwards data straight to the application or to the kernel. Once the memory is out of reach for the driver, it will restart at (1).

So, what you need is a kind of handshake protocol between the driver and the hardware so that there is no ambiguity about who “owns” a memory area and successively is allowed to write to or read from it.