Forum Discussion
Problem W/ Altera VIP Frame Buffer II
I have configured an FPGA design for my linux device that is displaying SDI and HMDI inputs. It was originally configured to receive YCbCr 4:2:2 video format. Since I am experiencing green spill on images when trying to chroma key, I am attempting to change the input data to RGBA.
For a test, I have implemented a conversion into the FPGA that handles the RGB input from the frame buffer. I believe this conversion is correct because the color values come out as expected.
FPGA Design Here:
FB -> CSC -> CRS -> Output
This is the same for all four mixers.
I am using gstreamer to input the color bars pattern into the pipeline.
(gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb3")
There are 4 frame buffers located at "/dev/fb0", "/dev/fb1", and so on.
Basically the issue is these frame buffer's graphics are overlapping and changing the memory regions in the DTS file does not fix the issue:
"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb0"
"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb1"
"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb3"
I can even see the bottom portion of the color bars pattern when changing the height to 720.
"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=720 ! videoconvert ! filesink location="/dev/fb3"
The drivers being used for the frame buffer is altvipfb.c and altvipfb2-plat.c
Here is the DTS snippet of these frame buffer components and the boot logs where the frame buffers are initialized:
DTS snippetBoot Logs
The virtual address and length seen in these boot logs do not change at all when DTS changes are made.
I also get this error when inputting a pipeline:
If anyone could help me approach this issue it would be greatly appreciated.
Thanks,
Evan
22 Replies
- ZH_Intel
Frequent Contributor
Hi Evan,
Thank you for reaching out.
Apologize for the delayed response as we encounter some technical difficulty.
Just to let you know that Intel has received your support request and currently we are confirming the details with our internal team.
I shall come back to you with findings.
Thank you for your patience.
Best Regards,
ZH_Intel
- KianHinT_altera
Frequent Contributor
Hi Evan,
From the team feedback, referring to the VIP user guide (https://www.intel.com/content/dam/support/us/en/programmable/support-resources/bulk-container/pdfs/literature/ug/archives/ug-vip-18-1.pdf) section 15.7 Motion Adaptive Mode Bandwidth Requirements
Screenshot (just in case you dont have access to the pdf link):
This is the consideration on the bandwidth requirements for video frame data based on deinterlacer, but it is the same process for the frame buffer as well.
Contrast this with the DDR configuration you have configured – width of data bus, DDR speed – use 90% of this figure to account for refresh and inefficiency through the EMIF. If your requirements exceed this 90% value, then this will be the problem. Certainly by changing from YCbCr to RGB this will consume 50% more bandwidth so it sounds highly likely this is the issue.
Thanks
Regards
Kian
- KianHinT_altera
Frequent Contributor
Hi Evan,
Just got the confirmation from the team on the static frames. So basically the video is static is irrelevant to the frame buffer, each pixel still consumes the same amount of bandwidth. As per your reply previously, calculating for one buffer and multiplying by the number of buffers is correct, assuming that all 4 buffers are active at the same time, which is likely in this case since you have all 4 displays up displaying the same thing. Another thing suggested by the team is to try is to increase the amount of available bandwidth by increasing the DDR frequency.
Thanks
Regards
Kian
- ZH_Intel
Frequent Contributor
Hi All,
Just to note for internal team and customer:
- DTS and boot are related to HPS
- The drivers being used for the frame buffer is altvipfb.c and altvipfb2-plat.c, this is also driver for HPS
Screenshot
Moving to the right SME for this case.
- KianHinT_altera
Frequent Contributor
Hi Evan
Apologies for the delay, I'm Kian and got assigned to this case. Regarding this issue, I've already contacted our VIP team to get their insight into this issue, will get back to you later once we have findings on our end.
Thanks
Regards
Kian
- KianHinT_altera
Frequent Contributor
Hi Evan,
The team here is asking for more information on what are the base addresses are and the associated settings in the 4 frame buffers? In addition, did you do any maximum bandwidth calculations to check that the EMIF can service all 4 frame buffers at the same time if required?
Thanks
Regards
Kian
- EvanHamiltonJAVS
New Contributor
Hey Khtan,
The frame buffer's base addresses and associated settings can be seen in the DTS snippet I provided:
In this DTS file image you can see the virtual base addresses, max width and height, bits per symbol, and port width.
Here you can see the frame buffers initializing to their physical addresses:
The EMIF has had no issues before handling all 4 frame buffers, but since I have changed the video format from YCbCr to RGB I guess that could be surpassing the maximum bandwidth. How would I conduct those calculations to check if the EMIF can handle all 4 frame buffers with RGB?
Thanks,
Evan
- EvanHamiltonJAVS
New Contributor
Hi Kian,
Thank you for your response, I did not see it till now. I am attempting to do these calculations, but I am a little confused on how to calculate for all 4 frame buffers. I'm assuming I would calculate for 1 of them and multiply it by 4, but while doing this I realized I forgot to mention we are sending static images as of right now. The input is a RGB gstreamer pipeline with the default smpte color bars pattern that is in a paused state so the bandwidth shouldn't be exceeded. I could understand this becoming an issue when I try to input a RGB live feed in playing state into the frame buffers, but this should not be the problem while the pipeline is in a paused state. Also, while in paused state the frame rate does not apply, so I am not sure how I would calculate for the paused frame rate.
Thanks
Evan
- KianHinT_altera
Frequent Contributor
Hi Evan,
Sorry for the delay in response, I'm trying to confirm some of the information with our expert over here before able to address your question, will get back to you early this week once he is back to office.
Thanks
Regards
Kian
- EvanHamiltonJAVS
New Contributor
Hi Kian,
I believe you are correct. Our DDR4 is already maxed out at 1200 MHz and from my calculations I believe we are using nearly all of our bandwidth, like around 98%. Do you have any recommendations on what video format I should use? The reason we are navigating away from YCbCr is because our chroma keyer feature cannot handle small detailed graphics. There is always a residual green spill leftover from detailed graphics. The only thing we could think to do to accommodate for this was switching to RGBA, which is why I started conducting those RGB tests with the frame buffers. I will attach a image of the green spill I am talking about. Thank you again for your time and help!
Thank you,Evan
- KianHinT_altera
Frequent Contributor
Hi Evan,
Sorry for the delay in getting back to you as I was out of office till next week. The team suggested that if the DDR is maxed out then you have to use YCbCr but this will suffer from the residual green spill kind of thing. You might be able to mitigate against it by using the color space convertors to go back up to RGBA prior to doing the chroma keying.
Thanks
Regards
Kian
- KianHinT_altera
Frequent Contributor
Hi Evan,
Just checking on this issue , whether does the suggestion on using color space convertor able to mitigate the residual green spill?
Thanks
Regards
Kian
- EvanHamiltonJAVS
New Contributor
Hey Kian,
I believe your suggestion has a good chance of working. I am still working on this new approach. I believe I have the conversion down, but I am not sure if my Chroma Keying logic is good for handling RGB. I am also having to make a custom color plane combiner because I have to separate RGBA and A in order to convert back to YCbCr. The custom component I made combines YCbCr w/ Alpha. I will let you know if I get better results once I get it working.
Thanks,
Evan