Skip to content

remove image-msg::frame-id; use timestamp - #11751

Merged
maloel merged 9 commits into
realsenseai:ddsfrom
maloel:timestamp
May 1, 2023
Merged

remove image-msg::frame-id; use timestamp#11751
maloel merged 9 commits into
realsenseai:ddsfrom
maloel:timestamp

Conversation

@maloel

@maloel maloel commented Apr 30, 2023

Copy link
Copy Markdown
Contributor

Follow-up to #11665, this introduces timestamp-based synchronization of metadata to images:

  • Changed frame-id in the MD to frame-number to be consistent, since that's what it's called in librs
  • It's now a number, not a string
  • Frames will now have a timestamp, even without metadata
  • Timestamp is communicated as nanoseconds - using doubles is asking for trouble
  • Removed frame_id field in image
  • Frame numbers are now still communicated, but are optional on the client since MD is the only way to communicate them
    • When not available, librs will number them sequentially - meaning we lose the ability to use the numbers for drop detection
  • The frame_id in the ROS2 image format is a string denoting the device & sensor - this needs more work

Along the way:

  • Fix behavior of dds_time with negatives, and added a dds-time unit-test that clarifies a lot of its (mis)behavior
  • Removed old image format
  • Refactoring of the message formats to all be similar: image_msg, flexible_msg, located under topics/
  • Improve rspy.test.remote.Error output
  • Separated publish_image from publish_motion

As before, I suggest looking at the commits one by one; they're fairly self-contained.

Next is adding the ros2imu format and getting that to work.

@maloel
maloel requested a review from OhadMeir April 30, 2023 09:31

bool is_streaming() const override { return _streaming; }
void start_streaming( const image_header & header );
void start_streaming( const image_header & );

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you remove the variable name?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because saying image_header header is redundant and unhelpful. If the type conveys the meaning, then everything else is extra. But that's in hindsight -- maybe I changed something and then undid it...?

raw_image.encoding() = _image_header.format.to_string();
raw_image.height() = _image_header.height;
raw_image.width() = _image_header.width;
raw_image.step() = uint32_t( image.raw_data.size() / _image_header.height );

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check _image_header.height != 0 (can be in start_streaming)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to address this when I do the IMU work, since an image_header is the wrong thing to use when the data isn't an image.

raw_image.height() = _image_header.height;
raw_image.width() = _image_header.width;
raw_image.step() = uint32_t( image.raw_data.size() / _image_header.height );
raw_image.is_bigendian() = false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is true for Jetson?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's reflective of our images (whoever created the data) rather than the system. For now it should do.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I'm not entirely sure what it's indicative of. It might refer to the endian-ness of the message header, since the image format is pretty much set in stone...
I think we'll address this issue when we get there.

if( ! is_streaming() )
DDS_THROW( runtime_error, "stream '" + name() + "' cannot publish before start_streaming()" );

if( image.height != _image_header.height )

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need this? motion does not have height and width.
Same question for step

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a copy of publish_image until I get to the IMU.

DDS_API_CALL( _writer->get()->write( &raw_image ) );
}

} // namespace realdds No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check new line at EOF

server->publish( static_cast< const uint8_t * >( f.get_data() ), f.get_data_size(), f.get_frame_number() );
realdds::topics::image_msg image;
auto data = static_cast< const uint8_t * >( f.get_data() );
image.raw_data.assign( data, data + f.get_data_size() );

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move the data?
Maybe construct a new vector on this range and move to raw_data

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constructing a new vector (meaning a copy) and moving it is the same as assigning the data. We assigned the data inside publish before, too, so this is not a change.

if( res.rem < 0 )
{
--res.quot;
res.rem += 1000000000ull;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using this number a lot. Good opportunity to use constexpr uint64_t nanos_in_second = 1'000'000'000ull

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might be a good idea, but I don't want to introduce this in a header at this point...

@OhadMeir OhadMeir left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maloel
maloel merged commit 4074adb into realsenseai:dds May 1, 2023
@maloel
maloel deleted the timestamp branch May 1, 2023 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants