Skip to content

Add hardware timestamp reporting - #86

Open
C-Achard wants to merge 16 commits into
cy/improve-encoding-speedfrom
cy/hardware-timestamps
Open

Add hardware timestamp reporting#86
C-Achard wants to merge 16 commits into
cy/improve-encoding-speedfrom
cy/hardware-timestamps

Conversation

@C-Achard

@C-AchardC-Achard commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds frame capture metadata and recording timestamp support, with best-effort Basler hardware timestamp integration.

This PR refactors camera reads to allow returning extra objects alongside the frame, such as the existing host-side software timestamp, as well as optional backend-provided timestamp metadata.
The software timestamp remains the primary source, while hardware timestamps are added only when the backend can report them.

Main changes

  • Introduces a CapturedFrame wrapper for camera backends
  • Adds generic timestamp metadata support
    • Supports hardware camera-clock timestamps, raw device ticks, optional wall-clock timestamps, and backend-specific extras
  • Adds best-effort Basler hardware timestamp support
    • Reads device ticks count reported by Basler grab results when available
    • Falls back to an assumed one gigahertz tick frequency when the camera does not report one, and records that assumption in metadata. See Basler docs, generally devices which cannot report tick frequency operate at 1GHz, hence the fallback.
    • Treats zero-valued Basler timestamps as unavailable, which handles cases such as pylon emulation or cameras that expose the API but do not provide usable timestamps, as per Basler docs.
  • Updates non-Basler backends to return the new captured-frame object.
    • Aravis, GenTL, and OpenCV currently preserve software timestamps and do not provide hardware timestamp metadata for now.
  • Threads timestamp metadata through the recording path.
    • RecordingManager forwards timestamp metadata into VideoRecorder.
  • Updates recorder timestamp sidecar output.
    • Timestamp sidecars now use schema version 2.
    • Per-recording timestamp source metadata is stored once.
    • Hardware timestamp fields are omitted when unavailable.
  • Improves Basler exposure compatibility
    • Sets both ExposureTime and ExposureTimeAbs when writable as they both control the same property on different devices

What this does not do yet

  • Hardware timestamps are best effort and backend-dependent.
  • This does not change recording throughput or move recorder enqueueing off the GUI thread.

Testing

Added and updated tests accordingly; tested and confirmed to work as intended with 1x a2A1920-160umPRO.

@C-Achard
C-Achard requested a review from CopilotJune 29, 2026 14:10
@C-AchardC-Achard self-assigned this Jun 29, 2026
@C-AchardC-Achard added enhancement New feature or request camera Related to cameras and camera backends labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds optional per-frame timestamp metadata (including best-effort Basler hardware timestamps) and threads that metadata through capture, controller signals, recording, and the timestamp sidecar output format.

Changes:

  • Introduces CapturedFrame as the standard return type for camera backend read() calls, carrying software timestamp plus optional timestamp metadata.
  • Adds FrameTimestampMetadata and integrates Basler tick-based timestamps (with tick-frequency discovery/fallback) into capture and recording.
  • Updates VideoRecorder timestamp sidecar JSON to schema v2 with per-recording timestamp source metadata and per-frame timestamp records.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Show a summary per file
FileDescription
tests/utils/test_timestamps.pyAdds unit tests for FrameTimestampMetadata dict splitting and default selection behavior.
tests/services/test_video_recorder.pyUpdates/extends tests for schema v2 timestamp sidecar output and hardware metadata inclusion.
tests/services/test_multicam_controller.pyAdjusts signal handler signatures and adds coverage for forwarding timestamp metadata.
tests/gui/test_rec_manager.pyAdds coverage ensuring RecordingManager.write_frame() forwards timestamp metadata to recorder.
tests/conftest.pyUpdates test backend and recorder fakes to use CapturedFrame and pass through timestamp metadata.
tests/cameras/backends/test_opencv_backend.pyUpdates OpenCV backend tests for CapturedFrame return type.
tests/cameras/backends/test_gentl_trigger.pyUpdates GenTL trigger test to read .frame from CapturedFrame.
tests/cameras/backends/test_gentl_backend.pyUpdates GenTL backend tests for CapturedFrame return type.
tests/cameras/backends/test_basler_backend.pyUpdates Basler backend tests and adds coverage for Basler timestamp metadata.
tests/cameras/backends/test_aravis_backend.pyUpdates Aravis backend tests for CapturedFrame return type and minor unused var cleanup.
tests/cameras/backends/conftest.pyExtends fake Basler SDK objects to provide timestamp and tick-frequency nodes.
dlclivegui/utils/timestamps.pyIntroduces FrameTimestampMetadata dataclass and serialization helpers.
dlclivegui/services/video_recorder.pyThreads timestamp metadata through queue/write loop and writes schema v2 timestamp sidecar JSON.
dlclivegui/services/multi_camera_controller.pyExtends capture/recording signals and internal plumbing to propagate timestamp metadata.
dlclivegui/gui/recording_manager.pyForwards timestamp metadata into VideoRecorder.write().
dlclivegui/gui/main_window.pyPasses timestamp metadata from controller signal into recording manager.
dlclivegui/cameras/base.pyAdds CapturedFrame, updates backend interface to return it, and extends capabilities.
dlclivegui/cameras/backends/opencv_backend.pyUpdates OpenCV backend to return CapturedFrame on success/failure paths.
dlclivegui/cameras/backends/gentl_backend.pyUpdates GenTL backend to return CapturedFrame.
dlclivegui/cameras/backends/basler_backend.pyAdds Basler timestamp tick-frequency handling and per-frame timestamp metadata generation.
dlclivegui/cameras/backends/aravis_backend.pyUpdates Aravis backend to return CapturedFrame.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threaddlclivegui/services/video_recorder.py Outdated
@C-AchardC-Achard added this to the Performance improvements milestone Jul 7, 2026
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from 3e34e7b to 9ab0670CompareJuly 14, 2026 09:01
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from 9ab0670 to 620ee46CompareJuly 14, 2026 09:44
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from 620ee46 to ea2389bCompareJuly 16, 2026 09:47
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from ea2389b to 19d98b3CompareJuly 16, 2026 11:15
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from 19d98b3 to 5969b0cCompareAugust 10, 2026 12:11
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from 5969b0c to 97ad476CompareAugust 10, 2026 14:27

@deruyter92deruyter92 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just minor comments. Looks good!

Comment threaddlclivegui/cameras/base.py Outdated
Comment threaddlclivegui/cameras/backends/basler_backend.py
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch 2 times, most recently from 1ad45ec to e576e6fCompareAugust 11, 2026 09:24
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from e576e6f to 176818dCompareAugust 11, 2026 11:56
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from 176818d to 3a7c1faCompareAugust 11, 2026 13:23
@C-Achard
C-Achard marked this pull request as ready for review August 12, 2026 08:26
Introduce `FrameTimestampMetadata` in `dlclivegui/utils/timestamps.py` to standardize optional backend/hardware timestamp data for captured frames. The dataclass captures source/backend fields, converted and raw timestamp values, conversion metadata, and backend extras, and adds helper methods to serialize source-level data, per-frame values, full dictionaries, and the configured default reported timestamp.
C-Achardand others added 15 commits August 12, 2026 11:26
Adds end-to-end support for optional per-frame hardware timestamp metadata. Camera backends now return a `CapturedFrame` object (while preserving tuple unpacking), multi-camera signals and recording paths carry timestamp metadata, and `VideoRecorder` persists richer timestamp records. The timestamp JSON output is upgraded to schema v2 with backward-compatible software `timestamps` plus source metadata and per-frame hardware timestamp fields.
Refactors backend tests to use the new `read()` return payload (`CapturedFrame`) instead of tuple unpacking, including frame/timestamp access updates and minor unused-variable cleanup. Test fixtures were also aligned with timestamp metadata support by returning `CapturedFrame` in the fake backend and extending fake recorder/frame callback signatures to accept `timestamp_metadata`.
Updates the recording metadata schema in `video_recorder.py` by renaming `hardware_frame_timestamps` to `frame_timestamps`. This aligns timestamp data with a more general key name while preserving the same underlying values.
Expands test coverage for frame timestamp metadata end-to-end: Basler backend reads now validate hardware timestamp extraction, controller/recording manager tests assert metadata forwarding, and video recorder tests verify schema_version 2 sidecar output for both software-only and hardware-backed timestamps. Also adds focused unit tests for `FrameTimestampMetadata` source/frame field splitting and default-reported value behavior.
Improve Basler hardware timestamp robustness by treating support as best-effort, recording the tick-frequency source, falling back to an assumed 1 GHz clock when frequency is unavailable, and ignoring zero-value camera timestamps as missing data. The frame timestamp metadata now includes the frequency source in `extra`, and unused last-frame timestamp state was removed. Video recorder metadata also drops the legacy top-level `timestamps` field in favor of the structured `timestamp_sources` schema.
Updates `test_video_recorder.py` to stop asserting the top-level `timestamps` list in sidecar JSON fixtures. The tests now focus on schema v2 fields that remain authoritative (`frame_timestamps`, `start_time`, `end_time`, `duration_seconds`, and timestamp source metadata), aligning expectations with current sidecar output.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Rename `patch_basler_sdk` to `patched_basler_sdk` and have it return the patched Basler backend module directly. Tests now consume that module from the fixture instead of re-importing it, and the backend patcher mapping was updated accordingly. The fixture also patches `genicam` with `raising=False` for more robust monkeypatching.
Extend Basler backend tests to cover `_make_timestamp_metadata` behavior when the SDK timestamp is zero and when tick frequency is missing or invalid. The new cases assert that zero ticks return `None` and invalid frequencies fall back to reporting raw tick values.
@C-Achard
C-Achardforce-pushed the cy/hardware-timestamps branch from 9078a4e to 29c8c11CompareAugust 12, 2026 09:31
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cameraRelated to cameras and camera backendsenhancementNew feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@C-Achard@deruyter92