Forum Discussion

EvanHamiltonJAVS's avatar
EvanHamiltonJAVS
Icon for New Contributor rankNew Contributor
2 years ago

Problem W/ Altera VIP Frame Buffer II

I have configured an FPGA design for my linux device that is displaying SDI and HMDI inputs. It was originally configured to receive YCbCr 4:2:2 video format. Since I am experiencing green spill on images when trying to chroma key, I am attempting to change the input data to RGBA.

For a test, I have implemented a conversion into the FPGA that handles the RGB input from the frame buffer. I believe this conversion is correct because the color values come out as expected.

FPGA Design Here:

FB -> CSC -> CRS -> Output

This is the same for all four mixers.

I am using gstreamer to input the color bars pattern into the pipeline.

(gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb3")

There are 4 frame buffers located at "/dev/fb0", "/dev/fb1", and so on.

Basically the issue is these frame buffer's graphics are overlapping and changing the memory regions in the DTS file does not fix the issue:

"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb0"

"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb1"

"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb3"

I can even see the bottom portion of the color bars pattern when changing the height to 720.

"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=720 ! videoconvert ! filesink location="/dev/fb3"

The drivers being used for the frame buffer is altvipfb.c and altvipfb2-plat.c

Here is the DTS snippet of these frame buffer components and the boot logs where the frame buffers are initialized:

DTS snippetBoot Logs

The virtual address and length seen in these boot logs do not change at all when DTS changes are made.

I also get this error when inputting a pipeline:

If anyone could help me approach this issue it would be greatly appreciated.

Thanks,

Evan

Video

22 Replies

Show More
  • KianHinT_altera's avatar
    KianHinT_altera
    Icon for Frequent Contributor rankFrequent Contributor
    2 years ago

    Hi Evan,

    Thanks for the the info, I will wait for your results.

    Thanks

    Regards

    Kian

    • EvanHamiltonJAVS's avatar
      EvanHamiltonJAVS
      Icon for New Contributor rankNew Contributor
      2 years ago

      Hey Kian,

      I have implemented this new approach into my design. I am inputting YCbCr 4:2:2 through a gstreamer pipeline and converting it to RGB before the chroma keying. I have also changed the chroma keying logic to handle RGB data.

      Input YCbCr 4:2:2 -> Convert and resample to RGB 4:4:4 -> Chroma key into RGBA 4:4:4 -> Separate RGB and A -> Convert and resample RGB 4:4:4 to YCbCr 4:2:2 -> Combine YCbCr with Alpha -> Output

      I used intel's Color Plane Sequencer to separate RGB and A. I was not able to use this same component to combine YCbCr with Alpha since YCbCr is in 4:2:2 and has different bit sizes for each color space. I made the custom combiner component to handle this.

      Y(8 bits) Cb(4 bits) Cr(4 bits)

      I have not been getting adequate results so I learnt how to use the Signal Tap Logic Analyzer to debug. The custom combiner I made does not seem to be outputting any data. I will provide my new chroma keyer and custom combiner component. The chroma keyer signals and ports are still the same names as before because I haven't manually changed them yet but the logic has been changed to handle RGB.

      The goal is to chroma key RGB 4:4:4 video data to remove the green residuals leftover from chroma keying YCbCr 4:2:2, but the output still needs to receive YCbCr w/ A 4:2:2

      Thanks,

      Evan

  • KianHinT_altera's avatar
    KianHinT_altera
    Icon for Frequent Contributor rankFrequent Contributor
    2 years ago

    Hi Evan,

    Thanks for the information. Let me check with the team and get back to you.

    Thanks

    Regards

    Kian

  • KianHinT_altera's avatar
    KianHinT_altera
    Icon for Frequent Contributor rankFrequent Contributor
    2 years ago

    Hi Evan,

    Our video team provided some links and commented that

    "If the customer has different bit sizes for each color space they will run into trouble. They need to pad or crop color planes so that the IPs see the same number of planes and then use the IPs. If they want to consider adding protocol convertors so they can use the VVP IPs, the bits per color sample does this. "

    See :

    https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/avalon-streaming-video-protocol-versus.html

    https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/protocol-converter-ip-parameters.html

    https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/bits-per-color-sample-adapter-ip-parameters.html

    https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/color-plane-manager.html

    https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/about-the-chroma-resampler-ip.html

    Thanks

    Regards

    Kian

  • KianHinT_altera's avatar
    KianHinT_altera
    Icon for Frequent Contributor rankFrequent Contributor
    2 years ago

    Hi Evan,

    Just checking up on this case status whether you have any updates?

    Thanks

    Regards

    Kian

  • KianHinT_altera's avatar
    KianHinT_altera
    Icon for Frequent Contributor rankFrequent Contributor
    2 years ago

    Hi Evan,

    I would like to inquire if there have been any updates.

    Looking forward to hearing back from you soon.

    Regards,
    Kian

    • EvanHamiltonJAVS's avatar
      EvanHamiltonJAVS
      Icon for New Contributor rankNew Contributor
      2 years ago
      Hey Kian,

      I’m sorry for the late response, we are in the later stages of development for a different project and have been prioritizing it as of late. I haven’t been able to work much more on this yet since my last update. This project is still important to us, and I will let you know as soon as we start back up. I think the VVP IP components could help a lot and save me from making a custom converter from scratch.

      Thanks,
      Evan
      • KianHinT_altera's avatar
        KianHinT_altera
        Icon for Frequent Contributor rankFrequent Contributor
        2 years ago

        Hi Evan,

        Thanks for the reply and no worries on the response. Since you're currently tied up with another project, may I know whether possible to set this case to close first, and then once you're free to work on this again, do re-open a new forum thread (if still have any issues) and tag this forum thread link in the new thread so we can trace back the history of the case and support from there on.

        Thanks

        Regards

        Kian

  • KianHinT_altera's avatar
    KianHinT_altera
    Icon for Frequent Contributor rankFrequent Contributor
    2 years ago

    Hi Evan,

    As I could not suspend the thread case tracking in system and being flagged out for idling, I will set this thread to close-pending first and revert to community support. Once you're free up to continue work on this issue, do create a new thread and tag this thread link and we will resume support from there on.

    Thanks for using Intel-Altera forums and sorry for the inconvenience cause. Please do let us know once you're able to resume work on this area and still facing issues on the video conversion.

    Thanks

    Regards

    Kian