Tone Mapping Operator IP Malfunctioning

Hi,

May I know what device that you are targeting on this project ?

There is some idea out of my mind where we can try to narrow down the issue further bit

1. before TMO insert a gain on R only (e.g., scale R up). If ROI stays black but the rest of the frame changes, the input is RGB; if nothing changes, the source into TMO may be packed/zeroes.
2. set ROI to almost full frame and enable it. If black goes away → ROI alignment/width issue.
3. Do you try other lower/higher resolution rather than 2k ? is this issue happen in other resolution ?

Regards,

Wincent_Altera

Hello Wincent,

Thank you for the quick response. The target device is 10AX115H2F34E2SG.

1. I adjusted the red gain via a color space converter and the ROI region remained black while the color bars changed.
2. When I attempt to use larger ROI values typically the function call to intel_vvp_tmo_set_roi_horizontal_x0x1 or intel_vvp_tmo_set_roi_vertical_y0y1 will hang the nios2 processor. On one attempt I was able to set vertical 0->1060 and horizontal to 0->1900. The black area still appeared with a small bit of the test pattern visible.
3. I also happened to try a 800x800 camera feed and the results were the same.

Color Bar with Red Gain Set to ~8x:

800x800 Video:

Hi Wincent,

I reviewed the example code you sent me. The setup of the core is here:

 intel_vvp_tmo_instance tmo;
 intel_vvp_tmo_base_t TMO_PIPE_0_TMO_BASE = reinterpret_cast<intel_vvp_tmo_base_t>(VIDEO_PROC_PIPE_0_TMO_BASE);
 intel_vvp_tmo_init_instance(&tmo, TMO_PIPE_0_TMO_BASE);
 intel_vvp_tmo_set_bypass(&tmo, 1);
 intel_vvp_tmo_set_volume(&tmo, 45);
 intel_vvp_tmo_set_threshold(&tmo, 4500);
 intel_vvp_tmo_set_resolution(&tmo, PIPE_DIM.x, PIPE_DIM.y);

I changed my setup to exactly match this order:

intel_vvp_tmo_init_instance(&cam_tmo, (intel_vvp_core_base)VVP_MAIN_PATH_VVP_MAIN_INTEL_TONEMAP_BASE);
intel_vvp_tmo_set_bypass(&cam_tmo, 1);
intel_vvp_tmo_set_volume(&cam_tmo, 45);
intel_vvp_tmo_set_threshold(&cam_tmo, 4500);
intel_vvp_tmo_set_resolution(&cam_tmo, 1920, 1080);

I verified that frames were flowing through the IP by checking the field count registers of the VVP IPs (IMG_INFO_FIELD_COUNT) before the TMO IP and after. The resolution registers (IMG_INFO_WIDTH/IMG_INFO_HEIGHT) for these IPs also match the set resolution.

I then ran a terminal command that calls intel_vvp_tmo_set_bypass(&cam_tmo, 0);

Sometimes the processor will hang on this write. Other times it does not freeze but the other IPs report a non-changing field count.

As you can see, there are issues even if not using ROI.

Best,

David

Hi Wincent,

I have discovered that part of the issue is that this core requires the Lite variant of the video protocol and I was passing the Full variant to the core. I have since fixed this by adding protocol converters on both ends of the IP.

However, I am now facing new issues related to the resolution of the incoming video.

I have four sources of 60FPS video with varying resolutions:
800x800
1944x1092
3268x1840
1920x1080

When the video source changes, I call intel_vvp_tmo_set_resolution and set the lite to full protocol converter after the IP to the same resolution.

For 800x800 and 1920x1080 resolutions the tone mapping operation appears to apply correctly.

However for 1944x1092 and 3268x1840 the IP appears to be incorrectly applying the operation.

Processing Mode 3268x1840

Bypass Mode 3268x1840

Does this IP have undocumented restrictions on input resolution? The only restriction in the documentation I see is "Video resolutions up to 8192x4320 at 60 fps or 8192x8192 at 30 fps". Does the width or height need to be a multiple of the pixels in parallel? Does the ratio of width and height matter?

Thanks,

David

Hi Wincent,

Can you confirm with the IP designers if there are other restrictions? Why does the IP malfunction if 1944x1092
3268x1840 are used since both widths are divisible by 4? Reminder that 800x800 and 1920x1080 function correctly in 4 pixels-in-parallel mode.

Thank you,

David

Hi Wincent,

Thank you for this information. I used a upstream clipper to experimentally arrive at a similar ruleset. My findings are slightly different as the rule appears to be:

Resolution must be (N*8*PIP) x (M*8), where N and M are integers

For PIP=4: N*32 x M*8

For example:

Testing height rule:
1920x1080 (div by 4,8): Working
1920x1076 (div by 4): Looks ok at first, but image flickers on certain scenes
1920x1072 (div by 4,8): Working

Testing width rule:

1920x1080 (div by 16,32): Working
1904x1080 (div by 16): Creates 4 vertical white lines spaced across the screen
1888x1080 (div by 16,32): Working

I am not sure why I need double your recommendations but with this rule I can proceed evaluating the TMO IP.

I highly recommend that this rule be stated in the next revision of the "Video and Vision Processing Suite IP User Guide" so that others do not run into this issue. I have a feeling that there is a similar rule needed for the Warp IP as I had issues using it while warping at 798x798 image until I switched to 800x800.

Thank you for your support,
David