feat(datasets): browse local episodes with interrupted-recording handling - #105
Merged
Merged
Conversation
A dataset on disk currently can't be viewed at all. The viewer is a deep-link to the hosted lerobot/visualize_dataset Space, which renders from a Hub repo path, so Landing sends local datasets to /upload instead — you have to push a dataset to the Hub before you can look at what you just recorded. /edit-dataset has been a routed placeholder. Fill it in: pick a dataset, page its episodes, watch every camera frame by frame, with a film strip and an aggregate joint-motion trace. Backend — episode_media.py reads the v3.0 layout (meta/episodes/*.parquet + videos/) directly and decodes with PyAV. It deliberately avoids LeRobotDataset so browsing never builds a robot config or imports torch. No new dependencies: PyAV, numpy, pyarrow and Pillow all arrive with lerobot[core_scripts], which is already required. Notes on two things that are easy to get wrong: - Episodes share an mp4. Locating one resolves the file *and* its from_/to_timestamp window inside it. /dataset-video is therefore addressed by chunk/file, not by episode, so the URL stays stable as you page between episodes: the player seeks instead of reloading, and playback survives the switch. An episode-addressed URL hands the browser a new identifier for the same bytes and silently drops playback on every click. - duration_s is length/fps, not to_timestamp - from_timestamp. An episode accepted early stops its parquet at accept while the camera ran to the configured limit, so the window over-reports the demonstration. The window is still exposed for the transport. Frontend — the Landing picker now opens local/both datasets in the browser rather than diverting to /upload; Hub-only datasets still go to the Space, since there's nothing here to decode. Upload stays reachable from the browse page. Tests build a real (tiny) v3.0 dataset on disk — a genuine PyAV-encoded mp4 plus parquet metadata — rather than mocking the readers: the point of the module is that it agrees with the on-disk format, which a mock can't check.
extract_thumbnails dropped an undecodable frame with `continue` and returned a plain list of pngs, which handle_episode_thumbnails paired back to the requested indices positionally. A single mid-strip decode failure then shifted every later tile up one slot: a tile labelled "frame 8" showed a later frame's image, clicking it seeked the player somewhere the picture never showed, and the strip ended short of the last frame — all with success:true and valid PNGs. It surfaces on a corrupt/truncated recording, which is exactly when the strip is being scrutinised. Return (frame_index, png) pairs so the index travels with its image; a dropped frame is simply absent rather than a positional gap. Regression tests at the unit and route level inject a mid-strip decode failure and assert the survivors keep their true indices. Reported by @nicolas-rabault in review. Also joins the episode_media.py copyright with HuggingFace's, per review.
…owser # Conflicts: # CLAUDE.md
…owser # Conflicts: # lelab/server.py
…frame Out-of-range frame indices used to decode cleanly into the NEXT episode's footage on a packed mp4 and come back as a 200. Now bounded twice: the route checks the episode's length, and extract_frame_png itself refuses targets outside the episode's video window — which also covers rows with no usable length, where the route check never fires. Negative indices are rejected unconditionally. Pinned by test_frame_route_404s_past_episode_end, test_frame_route_404s_past_end_without_a_usable_length and test_extract_frame_png_refuses_frames_outside_the_window. frameUrl gets its call site: the viewer's video elements use the episode's first frame as their poster, so the box shows the scene while the mp4 downloads instead of sitting black. Two adjacent 500s found while fuzzing the route are fixed too: a non-positive max_width is a no-op instead of a PIL ValueError, and a NUL byte in repo_id maps to 404 instead of an unhandled ValueError out of Path.resolve(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
seekToFrame seeked to the frame's boundary timestamp; the timeupdate recompute (floor((t - fromTs) * fps)) then sat epsilon below the boundary and floored back a frame — ArrowRight moved +1 and snapped back, ArrowLeft jumped 2. Seeking to (frame + 0.5) / fps makes the floor immune to float error in either direction; the scrubber and film-strip clicks ride the same path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Path safety, legacy-video fallback, stale episode state, keyboard accessibility, and request bounding need correction.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds local LeRobot episode browsing with video playback, frame navigation, thumbnails, and motion traces.
Changes:
- Adds backend media APIs and dataset parsing.
- Replaces the placeholder editor with an episode browser.
- Adds Python/frontend tests and refreshes committed assets.
File summaries
| File | Description |
|---|---|
lelab/episode_media.py |
Reads local episode media. |
lelab/datasets.py |
Adds browsing handlers. |
lelab/server.py |
Exposes browsing routes. |
tests/test_episode_media.py |
Tests media parsing/decoding. |
tests/test_datasets.py |
Tests browsing endpoints. |
frontend/src/pages/Landing.tsx |
Routes local datasets to browser. |
frontend/src/pages/EditDataset.tsx |
Implements browser page. |
frontend/src/lib/datasetApi.ts |
Adds browser API client. |
frontend/src/hooks/useEpisodes.ts |
Adds episode data hooks. |
frontend/src/hooks/useEpisodes.test.tsx |
Tests request races. |
frontend/src/components/dataset/EpisodeList.tsx |
Renders episode navigation. |
frontend/src/components/dataset/EpisodeViewer.tsx |
Implements media playback. |
frontend/src/components/dataset/EpisodeViewer.test.tsx |
Tests playback looping. |
frontend/src/components/dataset/FilmStrip.tsx |
Renders thumbnails. |
frontend/src/components/dataset/MotionTrace.tsx |
Renders joint motion. |
frontend/src/contexts/ApiContext.tsx |
Restricts API URLs to HTTP(S). |
frontend/src/components/replay/DatasetCombobox.tsx |
Simplifies repository validation regex. |
frontend/package.json |
Adds Vitest and jsdom. |
frontend/package-lock.json |
Locks frontend test dependencies. |
.github/workflows/quality.yml |
Runs frontend tests in CI. |
README.md |
Documents browsing. |
CLAUDE.md |
Documents backend architecture. |
frontend/dist/index.html |
References rebuilt assets. |
frontend/dist/assets/index-DJuBJMLM.css |
Adds rebuilt styles. |
frontend/dist/assets/index-1vZ-UTJX.css |
Removes stale styles. |
frontend/dist/assets/index-BX9JKcmw.js |
Adds rebuilt application bundle. |
frontend/dist/assets/index-Cl3yeJ80.js |
Removes stale application bundle. |
Review details
Files not reviewed (1)
- frontend/package-lock.json: Generated file
- Files reviewed: 21/27 changed files
- Comments generated: 6
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if length <= 0: | ||
| return {"success": False, "message": f"Episode {episode_index} has no frames"} | ||
|
|
||
| count = max(1, min(count, length)) |
Comment on lines
+107
to
+112
| features = read_info(dataset_dir).get("features") or {} | ||
| return sorted( | ||
| key.removeprefix(VIDEO_KEY_PREFIX) | ||
| for key, feat in features.items() | ||
| if key.startswith(VIDEO_KEY_PREFIX) and isinstance(feat, dict) and feat.get("dtype") == "video" | ||
| ) |
Comment on lines
+171
to
+174
| const onKey = (e: KeyboardEvent) => { | ||
| const el = e.target as HTMLElement | null; | ||
| if (el && ["INPUT", "TEXTAREA", "SELECT"].includes(el.tagName)) return; | ||
| if (e.code === "Space") { |
Comment on lines
+61
to
+68
| useEffect(() => { | ||
| if (!repoId || episodeIndex === null) { | ||
| setDetail(null); | ||
| return; | ||
| } | ||
| const controller = new AbortController(); | ||
| setLoading(true); | ||
| setError(null); |
Comment on lines
+187
to
+189
| const m = Math.floor(seconds / 60); | ||
| const s = Math.round(seconds % 60); | ||
| return `${m}m ${String(s).padStart(2, "0")}s`; |
Comment on lines
+413
to
+417
| location = _locate_via_meta(dataset_dir, episode_idx, cam, fps) or _locate_fallback( | ||
| dataset_dir, episode_idx, cam, fps | ||
| ) | ||
| if location is None or not location.path.exists(): | ||
| raise EpisodeNotFoundError(f"Episode {episode_idx} for camera {cam!r} not found on disk") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continue #68 so local recordings can be browsed in-app without uploading to the Hub, including camera frames and episode navigation. Includes @ravediamond’s interrupted-recording fix (16e4644), merges current main, and refreshes the committed frontend bundle. Original commits and authors are preserved; #68 remains open while this replacement is validated.
Also keeps the episode pagination clear of the floating tour button and pins two compatible browser-data lockfile entries to versions accepted by the registry’s release-age policy.
Validation: 268 Python tests pass; all pre-commit checks pass; TypeScript and targeted episode-browser ESLint pass; production build reproduces the committed bundle. Tested the production UI against a real 50-episode dataset, including frame and episode navigation. Full frontend lint still reports six existing errors in unchanged main files.
API URL overrides and saved API URLs now accept only HTTP(S), addressing the CodeQL findings on video sources. Four focused regression cases cover rejecting JavaScript/data URLs and preserving HTTP(S) endpoints.