ACP port under Linux

Hi,

I am trying to use the functions to sync the memory buffer so the FPGA can handle the data, but I am getting a segmentation fault when calling the dma_sync_single_for_device function, so I wonder if I am using the wrong parameters,

what is the dma_addr_t dma_handle parameter? is it the physical address? or the virtual address after the mem allocation? I read that the dma_addr_t is not the same as the physical address, or am I missing something before calling the functions, something like doing a dma map before?

switch(cmd){
    case MY_SYNC_DEVICE:
        //device now owns buffer to read
        dma_sync_single_for_device(&d->pdev->dev,
            d->my_buf.buf_phys_addr, d->myfx_buf.buf_len,
            DMA_TO_DEVICE);
        break;
    case MY_SYNC_CPU:
        //cpu now owns buffer to read
        dma_sync_single_for_cpu(&d->pdev->dev, d->my_buf.buf_phys_addr, d->my_buf.buf_len,
            DMA_FROM_DEVICE);
        break;
    default:
      dev_err(&d->pdev->dev, "Improper ioctl command %d.\n", cmd);
      return -1;
  }

--- Quote Start ---

Hmm... I've always allocated all the memory available to the linux system so the kernel has access to all of it. That way, I can allocate as much or as little buffer space in the driver as needed. I'm not sure how to keep memory off-limits to the kernel but still accessible to drivers, is it possible? Sorry, not sure.

--- Quote End ---

What I did was to boot the kernel with the argument mem=896M, so the kernel can only see that memory, then in the kernel module I request access to missing memory by using the request_mem_region function, then from the user app I call the mmap to actual remap the physical region from the kernel to the user app space by using the remap_pfn_range function, so if I call the dma_sync_single_for_device function with the physical start address of that region I get the segmentation fault.

Any advice on how to handle the reserved memory from the booting process, or the reason on getting the segmentation fault?

Thanks.

Hello again,

I will try to allocate some memory in the same way the example you provided using the link https://github.com/jpendlum/user-peripheral-kmod/blob/master/user_peripheral.c,

d->xx_buf = alloc_pages(GFP_ATOMIC, PAGE_ORDER);

But from you code hint,

d->my_buf.buf_phys_addr, d->myfx_buf.buf_len,

I don't know what is my_buf, it seem for me an array of structs, so how were you getting the allocated memory? what is the relation between the virtual address you got from this and the variable buf_phys_addr?

is the buf_phys_addr the physical address of a given start virtual address of a buffer or is it the dma address?

If I would like to use the ACP port, I need to send to the FPGA module the physical address | 0x80000000, right? so how did you get the buffer's physical address you allocated? Using the same function page_to_phys?

Just I would like to give a try on using the ACP because I was requested to use it and see if my implementation of the driver is working.

Another thing would be, what is the address I need to use as a parameter for the function call dma_sync_single_for_device?

Thanks!

--- Quote Start ---

I am trying to use the functions to sync the memory buffer so the FPGA can handle the data, but I am getting a segmentation fault when calling the dma_sync_single_for_device function, so I wonder if I am using the wrong parameters,

what is the dma_addr_t dma_handle parameter? is it the physical address? or the virtual address after the mem allocation? I read that the dma_addr_t is not the same as the physical address, or am I missing something before calling the functions, something like doing a dma map before?

--- Quote End ---

Here's how my driver gets buf_phys_addr for dma_handle parameter:

typedef struct _yx_buffer {
  struct page           *buf;
  unsigned long         buf_phys_addr;
  size_t                buf_len;
} Buffer;
static int
_allocBuffer(struct platform_device *pdev, Buffer *b, int page){
  dma_addr_t    dma;
  b->buf = alloc_pages(GFP_ATOMIC, page);
  if (!b->buf) {
    dev_err(&pdev->dev, "Error allocating buffer 2^%d pages of %lu\n",
      page, PAGE_SIZE);
    return -ENOMEM;
  }
  dma = dma_map_page(&pdev->dev, b->buf, 0, page, DMA_BIDIRECTIONAL);
  if(dma != page_to_phys(b->buf)){
    dev_err(&pdev->dev, "BAD DMA ASSUMPTION: page address is %u, and dma is %u\n",
      page_to_phys(b->buf), dma);
    return -ENOMEM;
  }
  b->buf_phys_addr = page_to_phys(b->buf);
  b->buf_len = (1 << page) * PAGE_SIZE;
  dev_info(&pdev->dev, "allocBuffer, %lx at size %u bytes\n",
      b->buf_phys_addr, b->buf_len);
  return 0;
}

Also, if you're using ioctl control between app and driver, make sure you define it with the _IOR macro. I first made the mistake of using a simple number which collides with existing ioctl.

# define SYNC_DEVICE      _IOR(0xAF, 1, unsigned long)# define SYNC_CPU          _IOR(0xAF, 2, unsigned long)

On the memory mapping, I guess that should work but I don't have working experience with that arrangement.

--- Quote Start ---

If I would like to use the ACP port, I need to send to the FPGA module the physical address | 0x80000000, right? so how did you get the buffer's physical address you allocated? Using the same function page_to_phys?

--- Quote End ---

Yes, page_to_phys in the driver is where it is first assigned. The GnuRadio FPGA module approach I linked to earlier takes that address and creates a device attribute out of it which is then read by the user-space application using the udev library given the device tree node of the accelerator (which looks something like "ff203000.accel" on my system).

Yes, definitely test using the ACP because that's simpler than the DMA scheme.

Hello yxi95,

I successfully got the driver and the user application working using the dma approach you told me.

I realized that the dma_addr I got by using the dma mapping was the same value as the physical address of the buffer I allocated using the alloc_pages, I think it is only applicable for my current architecture and its Linux.

Yes, I ended up using the ioctl to sync the data between the CPU and the FPGA, I will apply your advice.

I am not sure if taking some time to develop the ACP way will provide me something better than the current DMA approach you said you got something not so good when using the ACP port.

How did you measure the performance comparison between using the ACP port and FPGA-SDRAM port?

If I would use the ACP port, do I need to cast the buffer to an unsigned long long to manage the data in 64 bits format? is it required? My FPGA module has a 32 bits bus data.

Also from the kernel driver I allocated 32 buffers (4MB each one by calling the alloc_pages function several times) to get a total of 128MB of contiguous physical memory, I am planning on making a custom malloc function (or memory pools) to provide dynamic memory allocation to a user application. Is there a different way to get +100MB of contiguous physical memory without calling the alloc_pages multiple times? Maybe another function?

Thanks for your support!

Hello again,

I am trying to setup the ACP port, so I have some questions:

1. Do I need to use the FPGA to HPS port instead the FPGA to SDRAM ports?

2. Do I need to cast the buffer I allocated in the driver to a 'unsigned long long' to read/write data? is it required?

3. Is it required to set the f2h_AR[AW]CACHE attributes as you said in the second post?

4. After sending the command to perform the DMA task using the ACP port (phy_addr | 0x80000000) I got a "Kernel panic - not syncing", do you have a repository where I can take a look at the code of your driver when using the ACP?

And the FPGA qsys system.

Also this is the Linux I am using: Linux cyclone5 3.10.31-ltsi-altera# 1 SMP Thu Feb 11 22:06:58 UTC 2016 armv7l GNU/Linux and the DE1-SoC board.

Thanks.

--- Quote Start ---

How did you measure the performance comparison between using the ACP port and FPGA-SDRAM port?

--- Quote End ---

Running the clock at 150Mhz and seeing how much streaming data we could get over the bus from an mSGDMA. From my notes: ACP was 280MB/s, f2h_axi was 380MB/s, f2h_sdram was 564MB/s.

--- Quote Start ---

If I would use the ACP port, do I need to cast the buffer to an unsigned long long to manage the data in 64 bits format? is it required? My FPGA module has a 32 bits bus data.

--- Quote End ---

I think you'll get best performance with a 64-bit bus, but I don't think the datatype of the structure matters since all software accesses to memory will go through the optimized cache/memory-interfaces first. But if you do any kind of random access in the FPGA module (as opposed to streaming), though, that should probably be in 64-bit chunks and might turn out to be easier with the ACP port, but will of course be much slower than streaming.

--- Quote Start ---

I am trying to setup the ACP port, so I have some questions:

1. Do I need to use the FPGA to HPS port instead the FPGA to SDRAM ports?

--- Quote End ---

Yes, only the FPGA to HPS works for ACP. f2h_axi on my qsys.

--- Quote Start ---

2. Do I need to cast the buffer I allocated in the driver to a 'unsigned long long' to read/write data? is it required?

--- Quote End ---

I don't see any need to per above.

--- Quote Start ---

3. Is it required to set the f2h_AR[AW]CACHE attributes as you said in the second post?

--- Quote End ---

Check if latest Qsys does this automatically now. But it might still be needed.

--- Quote Start ---

After sending the command to perform the DMA task using the ACP port (phy_addr | 0x80000000) I got a "Kernel panic - not syncing", do you have a repository where I can take a look at the code of your driver when using the ACP?

--- Quote End ---

Sorry, no, but I'm using configuration identical to GnuRadio. I do recall spending a lot of time getting the right boot-loader/device-tree/kernel/, any chance it is that? If so, try to get any kind of working system at all first so you can rule out these issues. Then, just add/subtract your HDL to this working system. When I started my project late 2014, I did everything from the SocKit Linaro Desktop at rocketboards, that was my golden model. I could add/delete software and hardware incrementally from that version and know immediately if I'd done something wrong.

Thanks for your reply.

I have a working system, using yocto I can build the bootloader, kernel image, device tree and the file system. Is the GnuRadio ACP port configuration similar for the Altera devices?

I have configures the AXI port attributes as you said and also sending to the HDL module the physical address | 0x80000000, but I am getting a kernel panic after starting writing to that memory location, did you configure something for the ACP port before using it?

--- Quote Start ---

I have configures the AXI port attributes as you said and also sending to the HDL module the physical address | 0x80000000, but I am getting a kernel panic after starting writing to that memory location, did you configure something for the ACP port before using it?

--- Quote End ---

Does it work without "| 0x80000000"? That bypasses the cache so needs non-cached memory (pgprot_noncached) but should work correctly. If you get kernel panic on both, it has to be a driver typo/issue. If it only happens on ACP, I don't know. The AXI port attributes were the only thing I needed to change for ACP, no software changes were needed other than the flag.

--- Quote Start ---

The FPGA module I have developed has an Altera Avalon MM master which is connected to the AXI port FPGA2HPS, the port that your FPGA design has is an AXI or Avalon port?

--- Quote End ---

It's Avalon MM master too. At least one test case just used Altera's modular SGDMA connected to f2h_axi_slave.