Uh oh!
There was an error while loading. Please reload this page.
Add interactive annotation ability via .pl.annotate() - #684
Merged
Conversation
Add plans/interactive-selection.md documenting the v0 design for sdata.pl.interactive(...): in-notebook selector widget that draws a region on a spatialdata-plot canvas and persists it back into the SpatialData object as a ShapesModel. Includes resolved Q1-Q4, coordinate- system rules, downsampling strategy, persistence policy, and a 12-task implementation queue. Add a pixi `interactive` dep-group (ipympl, ipywidgets, squidpy) and a new `dev-interactive-py313` environment for prototyping. Register a dedicated `sdata-plot-interactive` kernel-install task to avoid the existing `pixi-dev` kernel name collision. Rewrite the broken [tool.pixi] inline-dotted block to explicit table headers ([tool.pixi.workspace], etc.) so pixi 0.54.2 actually loads the manifest. This commit records the ipympl-based prototype iteration. The notebook prototype (Sandbox.ipynb in lustre, not tracked here) revealed that websocket-streamed PNG frames are too laggy over SSH for full-slide interactive drawing; the next iteration switches to Plotly's client-side draw tools while keeping the same spec and task queue. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add anywidget and plotly>=5.20,<6 to the pixi interactive dep-group so the prototype notebook can render a custom HTML5/SVG drawing widget. anywidget is the canonical path for traitlet-based widget sync in VSCode-Remote; plotly is pinned to 5.x because its 6.0 anywidget-backed FigureWidget does not relay client-side relayout events back to Python (so layout.shapes never syncs there). The Sandbox.ipynb prototype itself lives outside this repo (/home/.../lustre/projects/spatialdata-plot/), but its current state implements a working anywidget-based draw canvas: pure client-side SVG drawing (rectangle drag, polygon click-then-Close-polygon, lasso freehand drag), shapes pushed back via the `shapes` traitlet, pixel→CS coordinate mapping that respects matplotlib's origin='upper' image axis, multi-shape commit per Save, and an explicit "Write last to disk" button for persistence. Sandbox.anywidget-v0.ipynb is preserved alongside as a reference snapshot before optimization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Productionises the Sandbox.ipynb prototype as a user-facing method on PlotAccessor. Public surface is a single function: sdata.pl.annotate(coordinate_system, element, *, persist=True) -> None Both args are required positional. The function validates that the image element is registered in the given coordinate system, renders it to a PNG, constructs an internal _InteractiveSession with anywidget-driven drawing tools (rectangle / polygon / lasso), and displays the widget. Drawn shapes are written into sdata.shapes[name] on click of the Save button; the optional "Write to disk" button persists via sdata.write_element. Module layout (src/spatialdata_plot/pl/interactive/): - _canvas.py DrawCanvas anywidget class - static/draw_canvas.js ESM module read from disk by anywidget (HMR-friendly) - _render.py render_to_png: sdata.pl → PNG + ax extent - _commit.py pixel-coord shape → CS-coord shapely Polygon → ShapesModel - _persist.py commit_to_memory + persist_to_disk (collision policy) - _session.py _InteractiveSession orchestrating the widget The new optional extra `interactive` (anywidget, ipykernel, ipywidgets) gates this feature behind a clear ImportError when missing: pip install 'spatialdata-plot[interactive]' The prototype iteration explored ipympl (rejected: PNG-over-websocket latency unusable over SSH) and plotly's FigureWidget (rejected: client- side relayout events don't sync back to Python in VSCode-Remote, plus plotly 6's anywidget-backed FigureWidget broke the comm path entirely). The custom anywidget approach was the only architecture that worked reliably over SSH while staying responsive. Drawing UX: - Tools: rect (drag), polygon (click + snap-close), lasso (drag freehand) - Wheel zoom, shift-drag pan, alt-click shape to delete - Ctrl+Z undo, R/P/L tool shortcuts, F fit view, Enter close polygon - Multi-shape bundling: each Save commits all canvas shapes as one ShapesModel with multiple rows under a single name Tests cover the unit surface (pixel→CS conversion, ShapesModel transform registration, render-to-PNG correctness, commit/persist policy, widget smoke). Spec at plans/interactive-selection.md updated to document the architectural pivot from the original ipympl approach. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Tests: import `DrawCanvas` from `._canvas` (internal class is not re-exported), and gate `test_canvas` with `pytest.importorskip` so CI envs without the `interactive` extra skip rather than fail. - `_persist.py`: replace deprecated `datetime.utcnow()` with timezone-aware `datetime.now(timezone.utc)`. Document on-disk overwrite behaviour (asymmetric with in-memory rename-on-collision). - `_render.py`: wrap render in `try/finally` so figures don't leak if `render_images().show()` or `savefig` raises. - `draw_canvas.js`: Delete/Backspace now removes the most recent shape (matches Ctrl+Z) instead of wiping the whole canvas — the toolbar Clear button covers the wipe case. - `basic.py` docstring: note that the canvas clears on every Save and that the Write-to-disk button overwrites same-named on-disk elements. - Add `tests/test_interactive/test_annotate.py` covering the three validation paths (`unknown CS`, `unknown element`, `element not in CS`) by stubbing `_InteractiveSession.show`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@## main #684 +/- ##
==========================================
- Coverage 77.79% 76.01% -1.79%
==========================================
Files 11 14 +3 Lines 3693 3869 +176 Branches 877 896 +19 ==========================================
+ Hits 2873 2941 +68 - Misses 490 597 +107 - Partials 330 331 +1
🚀 New features to boost your workflow:
|
Reuse / convention fixes: - Delete `_persist.persist_to_disk` — `SpatialData.write_element` already raises `ValueError` when `path is None`. Inline `write_element(name, overwrite=True)` in `_on_persist` and fix the docstring claim about overwrite semantics. - Match project import convention: `from spatialdata.transformations. operations import get_transformation` and `...transformations import Identity`, replacing `sd.transformations.*` references in `_session`, `_commit`, and tests. - `_validate` uses `sdata[element]` indexing + `get_transformation` to match the established pattern in `pl/utils.py` / `pl/render.py`. Quality: - Introduce frozen `RenderExtent` dataclass returned by `render_to_png`; collapses the 5-tuple return + 4 cached attrs on `_InteractiveSession` into one object. `pixel_shape_to_polygon(shape, extent)` drops 4 args. - `traitlets.Enum(TOOLS, ...)` for `DrawCanvas.tool` so a typo raises. - `BannerKind = Literal["info","success","error","hint"]`; drop the silent `.get(..., default)` fallback so a banner-kind typo raises. - Factor `_trigger_btn(description, icon, trait_name, after=...)` — replaces 4 near-identical `_on_close_polygon` / `_on_undo` / `_on_clear` / `_on_fit` methods. - Split `_on_save` into `_collect_polygons` / `_commit_polygons` / `_reset_canvas_state`; orchestrator stays ~10 lines. - Drop redundant `spine.set_visible(False)` loop after `set_axis_off()`. - Guard `persist_btn` construction behind `persist=True`; `_on_persist` early-returns if disabled. - Strip restate-the-code comments (`_canvas` module docstring, `_render` v0/v1 narration, `_commit` lasso-restating comment). - Underscore truly-private attrs (`_sdata`, `_commits`); keep `canvas` un-underscored since `_validate` callers in tests still need a way in. Efficiency (JS): - Incremental in-progress shape update during rect/lasso drag — keep a stable reference to the in-progress SVG node; mutate its attributes in `onMouseMove` instead of full `redraw()` (60 Hz × O(N) DOM ops → O(1)). - Lasso vert-push gated on viewbox-px ≥ 1 from the last vert; cuts vertex count ~5-10× for typical drags and the kernel-side traitlet payload on commit. - `setShapes` early-returns on `next === shapes` or both-empty; the `clear_trigger` handler routes through `setShapes([])` and skips when there is nothing to clear or cancel. - `zoomAt` / `panBy` / `fitView` snapshot the vbox pre-clamp and skip `applyViewbox` + `redraw` if the clamped vbox is unchanged. - `change:tool` only redraws if there was an in-progress shape to clear. - Dedup: `popLastShape` helper used by Ctrl+Z, Delete/Backspace, and `change:undo_trigger`. `shapeNode` extracts the common stroke/fill attrs and uses a single `pointsAttr` formatter for polygon/polyline. Tests: - Pytest `no_display` fixture replaces three duplicate `monkeypatch. setattr(...)` calls in `test_annotate.py`. - `pixel_shape_to_polygon` tests updated to the `RenderExtent` signature via a small `_extent(...)` helper. - `test_render` reads `extent.image_w` / `extent.xlim` from the dataclass instead of unpacking a 5-tuple. - Drop the two `test_persist` tests for the deleted `persist_to_disk` wrapper; `commit_to_memory` policy tests remain. All 18 interactive tests pass in dev-interactive-py313. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
UX: - Responsive canvas via `width: 100%` + `max-width: Npx` + CSS `aspect-ratio` on the wrap/container divs. Removes the fixed `DISP_MAX = 760` pixel box. Below `max_width` the canvas scales with the surrounding column; above it the canvas caps and centers. - New `max_width: int = 880` kwarg on `sdata.pl.annotate(...)` plumbed through `_InteractiveSession.__init__` and a new `max_display_width` `Int` traitlet on `DrawCanvas`. Pure display hint; underlying PNG render is unchanged (840 × 840). - Toolbar reflow: replace `HBox` with `Box(layout=Layout(flex_flow= "row wrap", ...))` so the tool toggle + icon buttons wrap onto a second row under narrow notebook widths instead of overflowing. - Icon-only auxiliary buttons (Close polygon / Undo / Clear / Fit / Write to disk) — `description=""`, 36px square, tooltip carries the affordance. Save button keeps its text label. - Drop the standalone "0 shape(s) on canvas" row and the `description= "Tool:"` / `description="Name:"` widget-side labels — none of them added information, all cost vertical space. Behaviour: - Remove the UTC-timestamp collision rename in `commit_to_memory`. Same name now overwrites in-memory (and on-disk via `write_element`, which we already pass `overwrite=True`). Drops the `datetime` import and the rename-on-collision banner branch. Reviewer flagged the prior rename as off-convention vs upstream spatialdata. - Update `test_commit_to_memory_renames_on_collision` → `test_commit_to_memory_overwrites_on_collision`. Other tests unchanged. Lint: - Pre-commit pass: `_dt.timezone.utc` → `_dt.UTC` (UP017) before the whole datetime block was dropped; ruff PT018 split assertion into two lines in `_collect_polygons`; biome + ruff-format normalised the changed Python and JS. All 18 interactive tests pass in dev-interactive-py313. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LucaMarconato
commented
May 22, 2026
Member
Hi, really useful feature! What's the relationship to this scverse/2026_04_hackathon_padua#22? |
Replace the custom anywidget + JS drawing canvas with the external `anybioimage` library's `BioImageViewer`. Removes ~1100 lines of bespoke client-side drawing code in favour of a well-maintained widget that already implements rectangle / polygon / point tools, wheel zoom + pan, multi-channel display, and a stable kernel↔JS sync. API: - `sdata.pl.annotate()` now behaves as a terminal step on a render chain: it rasterises the accumulated `plotting_tree` and hands the resulting RGB to the viewer, so overlays composed upstream (`render_images` / `render_shapes` / `render_points` / `render_labels`) appear on the annotation canvas. Args are keyword-only: `coordinate_systems`, `point_radius_frac`, `figsize`, `dpi`. - The chain pattern `sdata.pl.render_images(...).pl.annotate()` writes drawn shapes back into the user's original `sdata`. `_copy()` now propagates a `_source_sdata` reference forward through chained `render_*` calls so `annotate()` can resolve the real writeback target via `getattr(self._sdata, "_source_sdata", self._sdata)`. Without this, saves landed in the anonymous copy that the chain returns and the user's `sdata.shapes` silently stayed empty. - Points are stored as buffered circle polygons so the resulting `ShapesModel` stays uniform-type (no `radius` column). Module: - `pl/interactive/_session.py` — `_InteractiveSession` wraps `BioImageViewer` with an ipywidgets card (Name field, Save, Clear, banner). RGB channels are pre-swapped to `[G, R, B]` so anybioimage's default `CHANNEL_COLORS=[green, red, blue]` lands true RGB without post-hoc `_channel_settings` mutation (which fights the text field's focus). - `pl/interactive/_commit.py` — pure functions that read the viewer's `_rois_data` / `_polygons_data` / `_points_data` traits and map pixel coords back to the chosen CS. - Deleted: `_canvas.py`, `_render.py`, `_persist.py`, `static/draw_canvas.js` and their test files. Deps: - Add `anybioimage>=0.3,<0.4` to the published `interactive` extra. - Drop the `interactive-extras` dep-group (ipympl + plotly<6 + squidpy): it existed only for the abandoned anywidget/plotly sandbox prototype and is no longer referenced. Drop it from the `dev-interactive-py313` pixi env too. Tests: 21 pass against `dev-interactive-py313`. Full suite (644 tests) still passes — the `_copy()` change is additive (attaches one private attribute to chained copies) and does not affect render-only paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
.pl.annotate()timtreis added a commit
to scverse/spatialdata-plot-tutorials
that referenced
this pull request
May 27, 2026
The widget pivoted from the custom anywidget+JS canvas to the external `anybioimage` library's `BioImageViewer` (see scverse/spatialdata-plot#684). Update the tutorial to match: - Intro: replace the anywidget/SVG architecture blurb with a description of the `BioImageViewer` backend. - Launch section: rewrite the API signature from the old positional `annotate(coordinate_system, element, persist=True)` to the new keyword-only chain-step form `.pl.render_images(...).pl.annotate( coordinate_systems=...)`. Drop the lasso tool (no longer present) and the JS-canvas-specific shortcut list. Replace the "Write to disk" button with a `sdata.write_element(<name>)` follow-up note. - Polygon query cell: pass the actual shapely geometry rather than the whole `GeoDataFrame`. spatialdata's `polygon_query` overload for `DataArray`/`DataTree` calls `GeoDataFrame(geometry=[polygon])` internally, which fails with `GeometryTypeError: Unknown geometry type: 'featurecollection'` if passed a nested gdf. - Watermark: drop `anywidget` (transitive dep that isn't imported by any executable cell) to keep the watermark report honest. - Add a `warnings.filterwarnings("ignore")` housekeeping cell ahead of the squidpy dataset load so the notebook's recorded outputs don't leak Zarr v3 deprecation chatter into the docs build. - Add the recorded GIF (`_static/img/interactive_annotate.gif`) and a placeholder PNG thumbnail (regenerate before merging). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The notebook drives the packaged `sdata.pl.annotate` chain on the `squidpy.datasets.visium_hne_sdata` H&E section: render → annotate (BioImageViewer widget) → name → Save → inspect `sdata.shapes`. Useful as a manual smoke test alongside the automated test suite, and as a kernel of the (re-executed) Sandbox runs we keep referring to in PR discussion. Local prototyping artefacts that don't belong in the repo — the two prior architecture variants (`Sandbox.anywidget-v0.ipynb`, `Sandbox.ipympl-v0.ipynb`), the SSH-rendering probe (`verify_ssh_annotate.ipynb`), and the persisted zarr under `sandbox_data/` — go into `.gitignore` so they stay locally visible without sneaking into future commits. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Both were scratch / planning artefacts that don't belong in the published package. Sandbox.ipynb is added to .gitignore so it can stay locally as a manual smoke notebook without re-entering the repo. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
timtreis
marked this pull request as ready for review
May 27, 2026 14:20
timtreis
commented
May 27, 2026
MemberAuthor
I've now refactored the feature to use |
`set_aspect("equal")` inside `show()` shrinks the axes box so the figure
has blank padding around the data when the figure aspect doesn't match
the data extent. The previous `savefig` wrote the whole figure (padding
included), but `_make_px_to_cs` assumed PNG dims == axes dims — so drawn
shapes got squeezed toward the center along the padded axis (zero at
center, max offset at edges).
Crop the saved PNG to the axes bbox via `bbox_inches=` so PNG pixels map
1:1 to (xlim, ylim) and the existing px→cs transform stays correct.
Add a regression test that drives the full annotate → save loop on a
non-square (32 wide × 64 tall) image with a square figsize=(5, 5) and
asserts the round-tripped corner polygon matches the CS extent.
Also annotate the previously untyped `_commit.py` helpers with a shared
`PxToCs` callable alias so pre-commit mypy passes on the touched files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>Uh oh!
There was an error while loading. Please reload this page.
timtreis added a commit
to scverse/spatialdata-plot-tutorials
that referenced
this pull request
May 27, 2026
0.4.0 ships `sdata.pl.annotate()` (scverse/spatialdata-plot#684), so the tutorial can require the published wheel instead of an editable path. Re-executed the notebook to refresh the watermark output to 0.4.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
sdata.pl.annotate(), a terminal step on a render chain that hands the rasterised RGB toanybioimage'sBioImageViewer, then commits drawn shapes back tosdata.shapesas aShapesModelin the chosen CS withIdentity().pip install 'spatialdata-plot[interactive]'._copy()carries a_source_sdataref so writeback lands on the user'ssdata, not the chain copy.