Skip to content

feat(zarr): warn on stale hand-edited encoding['chunks'] via metadata baseline - #3

Open
d-v-b wants to merge 1 commit into
claude/zarr-metadata-fragmentfrom
claude/zarr-chunks-drift-warning
Open

feat(zarr): warn on stale hand-edited encoding['chunks'] via metadata baseline#3
d-v-b wants to merge 1 commit into
claude/zarr-metadata-fragmentfrom
claude/zarr-chunks-drift-warning

Conversation

@d-v-b

@d-v-bd-v-b commented Jul 4, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Summary

Uses the read-only encoding["zarr_array_metadata"] document (added in the parent PR) as a provenance baseline to warn about stale, hand-edited encoding["chunks"].

On write, if a variable still carries the stored metadata document but its encoding["chunks"] matches neither the source array's chunk shape nor the variable's own current (dask) chunking, it was almost certainly edited by hand and is now stale → DeprecationWarning.

What it deliberately does NOT warn on

  • A plain round-trip (read → write unchanged) — encoding["chunks"] still equals the document.
  • .chunk(...) — changes the dask chunking but leaves encoding["chunks"] untouched (chunks.py:116-117).
  • overwrite_encoded_chunks=True — sets encoding["chunks"] to the variable's current chunking, i.e. consistent with the data being written.

Honest scope note

Fully deprecating any use of the chunks key is not feasible: overwrite_encoded_chunks=True is a legitimate first-party writer of encoding["chunks"], and at write time it's indistinguishable from a hand-edit by value alone. So this scopes the warning to the genuinely stale/inconsistent case (differs from both the source and the data), which is the actual footgun, rather than "any use of the key." encoding is a plain mutable dict, so this drift-vs-baseline check at the consumption boundary is the most reliable signal available without wrapping the dict.

This also generalizes: the same baseline diff could flag user overrides of other derived keys (compressors, fill_value, …).

Testing

  • test_encoding_chunks_drift_warns: hand-editing encoding["chunks"] on a round-tripped variable warns.
  • test_encoding_chunks_no_drift_no_warn: .chunk(...) then write does not warn.
  • Existing test_manual_chunk (which exercises overwrite_encoded_chunks=True) stays green.

Stacked on claude/zarr-metadata-fragment (PR #2).

[This is Claude Code on behalf of Davis Vann Bennett]

🤖 Generated with Claude Code

@d-v-b
d-v-bforce-pushed the claude/zarr-metadata-fragment branch from be29ee6 to 776bd2fCompareJuly 30, 2026 09:08
@d-v-b
d-v-bforce-pushed the claude/zarr-chunks-drift-warning branch from 21939fa to 243f404CompareJuly 30, 2026 09:10
@d-v-b
d-v-bforce-pushed the claude/zarr-metadata-fragment branch from 776bd2f to 7668a40CompareJuly 30, 2026 09:21
@d-v-b
d-v-bforce-pushed the claude/zarr-chunks-drift-warning branch from 243f404 to 03ac970CompareJuly 30, 2026 09:21
@d-v-b
d-v-bforce-pushed the claude/zarr-metadata-fragment branch from 7668a40 to 34e4e81CompareJuly 30, 2026 09:26
@d-v-b
d-v-bforce-pushed the claude/zarr-chunks-drift-warning branch from 03ac970 to 81694c5CompareJuly 30, 2026 09:26
@d-v-b
d-v-bforce-pushed the claude/zarr-metadata-fragment branch from 34e4e81 to 943f9b4CompareJuly 30, 2026 10:03
… baseline
Use the stored zarr_array_metadata document as a provenance baseline on write:
if a variable's encoding['chunks'] matches neither the source array's chunk
shape nor the variable's current chunking, it was almost certainly hand-edited
and is stale, so emit a DeprecationWarning. Plain round-trips, .chunk(...), and
overwrite_encoded_chunks=True do not warn.
Assisted-by: ClaudeCode:claude-fable-5
Co-authored-by: Claude <noreply@anthropic.com>
@d-v-b
d-v-bforce-pushed the claude/zarr-chunks-drift-warning branch from 81694c5 to 95359efCompareJuly 30, 2026 10:04
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@d-v-b