Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
15 years ago

Extract pixel data from video stream

Hi,

I am very new to FPGA. I am trying to configure a test system using a video input via Bitec QV card and to output pixel data to an external SPI-controlled display device. The interface to the external device requries manipulation of pixel data (16-bit RGB565) with delays between bytes sent and special handling for each new frame.

I am reasonably able to create a video flow to receive input video, clip, scale, mix etc and to output this stream on the DVI out on the Bitec QV card. I have successfully modified the VIP Ref Design using Quartus 9.1SP2 on a CycloneIII Dev Kit board (3C120).

However, I cannot figure out how to access the pixel data for output to my external device.

I know that I need a framebuffer to hold the incoming data and to manage the different frame rates between input and output. Using the VIP Ref Design the framebuffer uses DDR2 mem. I hoped that I could use the reader component of this framebuffer then to access the direct pixel data and to control the data flow to the external device. But I cannot determine if this is even possible let alone how to do it.

It would be great if someone experienced can tell me if this is possible or what modules I need to do this.

I also considered using DMA to access the framebuffer memory but this does not solve the problem of how to manage the output flow rate or even where the pixel data is in memory for the current 'valid' frame.

Any help would be greatly appreciated, thanks!

9 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Hi,

    I fail to see why the Frame Buffer included in the VIP reference design is not appropriate in your case. The output of the Frame Buffer is Avalon-ST with a ready signal so you can still do your processing after the frame buffer and apply back-pressure as needed to let the frame buffer drop input frames and handle frame rate conversion from you. Are you trying to access pixels in non-raster scan order?

    Otherwise, there is a Frame Reader in the VIP Suite that would do what you are asking for but there is no corresponding Frame Writer. Trying to figure out where the writer component of the Frame Buffer has put frames in memory to read them back with the Frame Reader will probably prove to be a bit tricky.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Hi,

    Thank you very much for replying. I must not fully understand how to use the framebuffer.

    I looked again at the VIP user guide and now I see that if I write 0 to the control reg this will stop the output component from updating while the input side drops frames as appropriate. I presume this is what you are referring to how it should be used.

    I do want to read the pixels in raster order. If possible, can you give me some hints of how to access this data from the Avalon-ST? Is there a module to use for this or do I need to provide some interpretation logic to access them?

    Thanks again for your help.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Hi,

    there is no need to try to control or interact with the frame buffer. Everything should be automatic. You just need to tick the check boxes in the GUI to enable dropping and repeating of frames so that the frame buffer becomes a triple buffer.

    The triple buffer behaviour is determined by what happens on the Avalon-ST wires. If the flow of valid pixels at the input is small and the output is always ready for data then the output side of the triple buffer will start repeating frame to meet the demand. On the contrary, when the output is slow, as in your case, you can apply back-pressure whenever is appropriate between pixels by setting ready low, the input side of the triple buffer will drop incoming frames to meet your output frame rate.

    > can you give me some hints of how to access this data from the Avalon-ST

    I believe you will have to build a bridge between the Avalon-ST Video output of the frame buffer and your device. The Avalon-ST Video protocol used by the Video IP is described in the user guide. This will describe how you can access image data, moderate the flow, ...
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Hi,

    Thanks again for your explanation. It is starting to make more sense. I'll try to figure out and understand the bridge which will enact the ready signal to control my slower uptake of pixel data.

    Many thanks for your help!
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Maybe this will help: http://www.nioswiki.com/exampledesigns/modular_sgdma_video_frame_buffer

    If you look at the C code that'll give you a good idea how difficult (or easy...) it is to manage the frame buffers with that hardware. That design has a bursting bug so if you want to use it and need bursting let me know and I can send you the fix for the read master (I'm working on a final bug fix and want to post them at the same time).
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Thanks for that. I'll check it out.

    From vgs's helpful posts, I was thinking of implementing a new component in SOPC with a Avalon-ST sink to feed into a deep FIFO. The framebuffer can feed into this custom component and the FIFO-full output can control the READY signal. I can then retrieve the data at my own pace from the FIFO into my external device.

    Of course I don't know how difficult this is but it seems to provide the necessary logic.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Hi mmcd98,

    I have same problem with you.How Can I extract the data pixels from avalon video protocol. I am freezing the video and I want to take this image from ddr which is controlled by the frame buffer.However it is said in the vip user guide that frame buffer stores ddr also non-image data packets.Can you succeed obtaining the data pixels.If so, can you help me?

    Thanks,
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    hi

    i knoe u want to get the pixels data so u need to make a frame grabber ip with sgdma to store the pixels data more important u need to knoe the avalon video protocol . u cannot get frame from frame buffer.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Hi daisyhome,

    It's been many months since I did this, so difficult to remember details. My eventual solution was all done in hardware. I created a custom sink module in SOPC builder into which I fed the video stream. I then added my own hardware design to take the video pixel data and output this via SPI to my custom device.

    The video stream is actually pretty straightforward and is explained in the video protocol spec document. The data (in my design at least) is transmitted in parallel RGB, scanned from top left pixel of a frame to bottom right. There is a sync marker fransmitted before every new frame (see video spec for details). My SPI hardware detected the start of frame and then loaded the parallel pixel data into a deep FIFO. I used the FIFO full signal to handshake with the SOPC design to prevent overflow. I then used a separate circuit to extract the data from the FIFO and forward to my device via SPI.

    I did not access the pixel data via SOPC CPU or with software as the hardware solution was much better for my implementation.

    I hope this is of some help.