[PointCloud] Set texel intrinsic & extrinsic based on selected stream

[Arun-Prasad-V]

Tracked on LRS-514.

The PointCloud filter supports generating rs2::points for below scenarios:

1. Input: Depth frame and Color frame
2. Input: Depth frame and Colorized depth frame

Overview of the issue:

• Scenario 2 works fine if only Depth stream is enabled.
• If the user enables both Color and Depth streams and want to generate pointcloud for scenario 2, then the output doesn't look good.

Rootcause:

• Whenever Color stream is enabled, Color sensor's texel intrinsic and extrinsic is getting used instead of Depth sensor's.

[Nir-Az]

Why do we want the prev filter to be color?

[maloel]

The way I read it:

• Whenever Color stream is enabled, Color sensor's texel intrinsic and extrinsic is getting used instead of Depth sensor's.

That means that we want to take the color intrinsics only when we have scenario 1: we actually want to use Color.

What I don't quite understand is why that's reflected in _prev_stream_filter, but I guess we just have to stare at the code some more :)

[Arun-Prasad-V]

The way I read it:

• Whenever Color stream is enabled, Color sensor's texel intrinsic and extrinsic is getting used instead of Depth sensor's.

That means that we want to take the color intrinsics only when we have scenario 1: we actually want to use Color.

Yes, @maloel. That's right.

What I don't quite understand is why that's reflected in _prev_stream_filter, but I guess we just have to stare at the code some more :)

If we look at this function inspect_other_frame(...) , user will pass the required frame:

• Color frame for scenario 1
• Colorized depth frame for scenario 2
• or even Infrared frame also possible I think

Some default intrinsic and extrinsic values are set for the first time at line 191 and 192 in this same function.
Then, later in this callback, if the Color stream is enabled, they are overwritten by Color sensor's intrinsic and extrinsic.
So, to prevent this overwrite, added a condition to check whether the user want's Color stream or not.

[maloel]

This code surrounding _prev_stream_filter is very fishy:

• Not initialized that I can see...
• I'm still unclear about its usage -- is it simply a way of saying "we want to do this only once and never again"??

[Arun-Prasad-V]

If we see the flow of the API calls:

Top level API is map_to(frame mapped). refer rs_processing.h

• Here _stream_filter will be set as per user's need
  • Color for scenario 1
  • Depth for scenario 2
• Then, process(...) function is called
  • which in turn calls inspect_other_frame(...) and it sets _prev_stream_filter

[Arun-Prasad-V]

at line 115 in this same function, it is initialized.

[maloel]

But it's used in line 113 😄

[Arun-Prasad-V]

Looks like, it is getting initialized with default values (RS2_STREAM_ANY, RS2_FORMAT_ANY, ...) in the constructor stream_filter class in synthetic-stream.h

[maloel]

Hmm right you are.
Then why compare the previous stream filter and not simply _stream_filter? I think that would be preferable?

[Arun-Prasad-V]

yeah, that makes sense. will update.