Reading and F-Order Files in v2 - #95

Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom
konstibob:fix/v2-order-f
Open

Reading and F-Order Files in v2#95
konstibob wants to merge 1 commit into
zarr-developers:mainfrom
konstibob:fix/v2-order-f

Conversation

@konstibob

@konstibobkonstibob commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

.zarray's order was parsed into ArrayMetadata.order and then never read, so "order": "F" was silently treated as row-major in both directions:

  • Reading an F-order store written by zarr-python returned the elements of each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7], [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].

  • Writing was worse: withOrder(Order.F) put "order": "F" into .zarray while laying the chunk out row-major, so zarr-java emitted stores that every other implementation misreads. zarr-python read such a store as [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the round trip through zarr-java itself returned the correct values and no Java-only test could see the corruption.

@konstibob@claude
Honour the Zarr v2 order key instead of ignoring it
50c7aca
`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:
* Reading an F-order store written by zarr-python returned the elements of
each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
[8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
* Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
while laying the chunk out row-major, so zarr-java emitted stores that
every other implementation misreads. zarr-python read such a store as
[[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
round trip through zarr-java itself returned the correct values and no
Java-only test could see the corruption.
Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.
`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.
Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:
* `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
total, no compressor, so the chunk files are raw element bytes), covering a
C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
`src/test/python-scripts/generate_v2_order_golden.py` regenerates them.
* `ZarrV2OrderTest` checks both directions against those fixtures offline,
with no Python at test time: reads must yield `arange`, and writing
`arange` must reproduce zarr-python's chunk bytes exactly. It also pins
down that C and F differ on disk (uncompressed and compressed), that the
two coincide for rank 1, that a single-full-chunk read is physically
contiguous, and that `order` survives `resize` and attribute updates.
7 of its 15 tests fail without this change.
* `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
the installed zarr-python in both directions so that a divergence from the
committed fixtures is noticed after a zarr-python upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@konstibobkonstibob changed the title Reading F-Order Files in v3 and v2Reading and F-Order Files in v2Aug 27, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@konstibob
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Reading and F-Order Files in v2 - #95
Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Open
Reading and F-Order Files in v2#95
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Conversation
@konstibob
@konstibobkonstibob
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
.zarray's order was parsed into ArrayMetadata.order and then never read, so "order": "F" was silently treated as row-major in both directions:
  • Reading an F-order store written by zarr-python returned the elements of each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7], [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
  • Writing was worse: withOrder(Order.F) put "order": "F" into .zarray while laying the chunk out row-major, so zarr-java emitted stores that every other implementation misreads. zarr-python read such a store as [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the round trip through zarr-java itself returned the correct values and no Java-only test could see the corruption.
@konstibob@claude
Honour the Zarr v2 order key instead of ignoring it
50c7aca
`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:
* Reading an F-order store written by zarr-python returned the elements of
each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
[8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
* Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
while laying the chunk out row-major, so zarr-java emitted stores that
every other implementation misreads. zarr-python read such a store as
[[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
round trip through zarr-java itself returned the correct values and no
Java-only test could see the corruption.
Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.
`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.
Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:
* `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
total, no compressor, so the chunk files are raw element bytes), covering a
C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
`src/test/python-scripts/generate_v2_order_golden.py` regenerates them.
* `ZarrV2OrderTest` checks both directions against those fixtures offline,
with no Python at test time: reads must yield `arange`, and writing
`arange` must reproduce zarr-python's chunk bytes exactly. It also pins
down that C and F differ on disk (uncompressed and compressed), that the
two coincide for rank 1, that a single-full-chunk read is physically
contiguous, and that `order` survives `resize` and attribute updates.
7 of its 15 tests fail without this change.
* `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
the installed zarr-python in both directions so that a divergence from the
committed fixtures is noticed after a zarr-python upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@konstibobkonstibob
changed the title
Reading F-Order Files in v3 and v2Reading and F-Order Files in v2Aug 27, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@konstibob

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Reading and F-Order Files in v2 - #95
Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Open
Reading and F-Order Files in v2#95
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Conversation
@konstibob
@konstibobkonstibob
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
.zarray's order was parsed into ArrayMetadata.order and then never read, so "order": "F" was silently treated as row-major in both directions:
  • Reading an F-order store written by zarr-python returned the elements of each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7], [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
  • Writing was worse: withOrder(Order.F) put "order": "F" into .zarray while laying the chunk out row-major, so zarr-java emitted stores that every other implementation misreads. zarr-python read such a store as [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the round trip through zarr-java itself returned the correct values and no Java-only test could see the corruption.
@konstibob@claude
Honour the Zarr v2 order key instead of ignoring it
50c7aca
`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:
* Reading an F-order store written by zarr-python returned the elements of
each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
[8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
* Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
while laying the chunk out row-major, so zarr-java emitted stores that
every other implementation misreads. zarr-python read such a store as
[[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
round trip through zarr-java itself returned the correct values and no
Java-only test could see the corruption.
Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.
`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.
Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:
* `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
total, no compressor, so the chunk files are raw element bytes), covering a
C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
`src/test/python-scripts/generate_v2_order_golden.py` regenerates them.
* `ZarrV2OrderTest` checks both directions against those fixtures offline,
with no Python at test time: reads must yield `arange`, and writing
`arange` must reproduce zarr-python's chunk bytes exactly. It also pins
down that C and F differ on disk (uncompressed and compressed), that the
two coincide for rank 1, that a single-full-chunk read is physically
contiguous, and that `order` survives `resize` and attribute updates.
7 of its 15 tests fail without this change.
* `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
the installed zarr-python in both directions so that a divergence from the
committed fixtures is noticed after a zarr-python upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@konstibobkonstibob
changed the title
Reading F-Order Files in v3 and v2Reading and F-Order Files in v2Aug 27, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@konstibob

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Reading and F-Order Files in v2 - #95
Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Open
Reading and F-Order Files in v2#95
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Conversation
@konstibob
@konstibobkonstibob
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
.zarray's order was parsed into ArrayMetadata.order and then never read, so "order": "F" was silently treated as row-major in both directions:
  • Reading an F-order store written by zarr-python returned the elements of each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7], [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
  • Writing was worse: withOrder(Order.F) put "order": "F" into .zarray while laying the chunk out row-major, so zarr-java emitted stores that every other implementation misreads. zarr-python read such a store as [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the round trip through zarr-java itself returned the correct values and no Java-only test could see the corruption.
@konstibob@claude
Honour the Zarr v2 order key instead of ignoring it
50c7aca
`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:
* Reading an F-order store written by zarr-python returned the elements of
each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
[8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
* Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
while laying the chunk out row-major, so zarr-java emitted stores that
every other implementation misreads. zarr-python read such a store as
[[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
round trip through zarr-java itself returned the correct values and no
Java-only test could see the corruption.
Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.
`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.
Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:
* `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
total, no compressor, so the chunk files are raw element bytes), covering a
C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
`src/test/python-scripts/generate_v2_order_golden.py` regenerates them.
* `ZarrV2OrderTest` checks both directions against those fixtures offline,
with no Python at test time: reads must yield `arange`, and writing
`arange` must reproduce zarr-python's chunk bytes exactly. It also pins
down that C and F differ on disk (uncompressed and compressed), that the
two coincide for rank 1, that a single-full-chunk read is physically
contiguous, and that `order` survives `resize` and attribute updates.
7 of its 15 tests fail without this change.
* `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
the installed zarr-python in both directions so that a divergence from the
committed fixtures is noticed after a zarr-python upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@konstibobkonstibob
changed the title
Reading F-Order Files in v3 and v2Reading and F-Order Files in v2Aug 27, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@konstibob

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Reading and F-Order Files in v2 - #95
Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Open
Reading and F-Order Files in v2#95
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Conversation
@konstibob
@konstibobkonstibob
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
.zarray's order was parsed into ArrayMetadata.order and then never read, so "order": "F" was silently treated as row-major in both directions:
  • Reading an F-order store written by zarr-python returned the elements of each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7], [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
  • Writing was worse: withOrder(Order.F) put "order": "F" into .zarray while laying the chunk out row-major, so zarr-java emitted stores that every other implementation misreads. zarr-python read such a store as [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the round trip through zarr-java itself returned the correct values and no Java-only test could see the corruption.
@konstibob@claude
Honour the Zarr v2 order key instead of ignoring it
50c7aca
`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:
* Reading an F-order store written by zarr-python returned the elements of
each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
[8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
* Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
while laying the chunk out row-major, so zarr-java emitted stores that
every other implementation misreads. zarr-python read such a store as
[[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
round trip through zarr-java itself returned the correct values and no
Java-only test could see the corruption.
Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.
`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.
Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:
* `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
total, no compressor, so the chunk files are raw element bytes), covering a
C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
`src/test/python-scripts/generate_v2_order_golden.py` regenerates them.
* `ZarrV2OrderTest` checks both directions against those fixtures offline,
with no Python at test time: reads must yield `arange`, and writing
`arange` must reproduce zarr-python's chunk bytes exactly. It also pins
down that C and F differ on disk (uncompressed and compressed), that the
two coincide for rank 1, that a single-full-chunk read is physically
contiguous, and that `order` survives `resize` and attribute updates.
7 of its 15 tests fail without this change.
* `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
the installed zarr-python in both directions so that a divergence from the
committed fixtures is noticed after a zarr-python upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@konstibobkonstibob
changed the title
Reading F-Order Files in v3 and v2Reading and F-Order Files in v2Aug 27, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@konstibob

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Reading and F-Order Files in v2 - #95
Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Open
Reading and F-Order Files in v2#95
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Conversation
@konstibob
@konstibobkonstibob
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
.zarray's order was parsed into ArrayMetadata.order and then never read, so "order": "F" was silently treated as row-major in both directions:
  • Reading an F-order store written by zarr-python returned the elements of each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7], [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
  • Writing was worse: withOrder(Order.F) put "order": "F" into .zarray while laying the chunk out row-major, so zarr-java emitted stores that every other implementation misreads. zarr-python read such a store as [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the round trip through zarr-java itself returned the correct values and no Java-only test could see the corruption.
@konstibob@claude
Honour the Zarr v2 order key instead of ignoring it
50c7aca
`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:
* Reading an F-order store written by zarr-python returned the elements of
each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
[8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
* Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
while laying the chunk out row-major, so zarr-java emitted stores that
every other implementation misreads. zarr-python read such a store as
[[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
round trip through zarr-java itself returned the correct values and no
Java-only test could see the corruption.
Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.
`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.
Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:
* `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
total, no compressor, so the chunk files are raw element bytes), covering a
C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
`src/test/python-scripts/generate_v2_order_golden.py` regenerates them.
* `ZarrV2OrderTest` checks both directions against those fixtures offline,
with no Python at test time: reads must yield `arange`, and writing
`arange` must reproduce zarr-python's chunk bytes exactly. It also pins
down that C and F differ on disk (uncompressed and compressed), that the
two coincide for rank 1, that a single-full-chunk read is physically
contiguous, and that `order` survives `resize` and attribute updates.
7 of its 15 tests fail without this change.
* `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
the installed zarr-python in both directions so that a divergence from the
committed fixtures is noticed after a zarr-python upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@konstibobkonstibob
changed the title
Reading F-Order Files in v3 and v2Reading and F-Order Files in v2Aug 27, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@konstibob

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Reading and F-Order Files in v2 - #95
Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Open
Reading and F-Order Files in v2#95
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Conversation
@konstibob
@konstibobkonstibob
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
.zarray's order was parsed into ArrayMetadata.order and then never read, so "order": "F" was silently treated as row-major in both directions:
  • Reading an F-order store written by zarr-python returned the elements of each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7], [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
  • Writing was worse: withOrder(Order.F) put "order": "F" into .zarray while laying the chunk out row-major, so zarr-java emitted stores that every other implementation misreads. zarr-python read such a store as [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the round trip through zarr-java itself returned the correct values and no Java-only test could see the corruption.
@konstibob@claude
Honour the Zarr v2 order key instead of ignoring it
50c7aca
`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:
* Reading an F-order store written by zarr-python returned the elements of
each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
[8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
* Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
while laying the chunk out row-major, so zarr-java emitted stores that
every other implementation misreads. zarr-python read such a store as
[[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
round trip through zarr-java itself returned the correct values and no
Java-only test could see the corruption.
Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.
`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.
Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:
* `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
total, no compressor, so the chunk files are raw element bytes), covering a
C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
`src/test/python-scripts/generate_v2_order_golden.py` regenerates them.
* `ZarrV2OrderTest` checks both directions against those fixtures offline,
with no Python at test time: reads must yield `arange`, and writing
`arange` must reproduce zarr-python's chunk bytes exactly. It also pins
down that C and F differ on disk (uncompressed and compressed), that the
two coincide for rank 1, that a single-full-chunk read is physically
contiguous, and that `order` survives `resize` and attribute updates.
7 of its 15 tests fail without this change.
* `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
the installed zarr-python in both directions so that a divergence from the
committed fixtures is noticed after a zarr-python upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@konstibobkonstibob
changed the title
Reading F-Order Files in v3 and v2Reading and F-Order Files in v2Aug 27, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@konstibob

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Reading and F-Order Files in v2 - #95
Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Open
Reading and F-Order Files in v2#95
konstibob wants to merge 1 commit into
zarr-developers:mainfrom 
konstibob:fix/v2-order-f
Conversation
@konstibob
@konstibobkonstibob
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
.zarray's order was parsed into ArrayMetadata.order and then never read, so "order": "F" was silently treated as row-major in both directions:
  • Reading an F-order store written by zarr-python returned the elements of each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7], [8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
  • Writing was worse: withOrder(Order.F) put "order": "F" into .zarray while laying the chunk out row-major, so zarr-java emitted stores that every other implementation misreads. zarr-python read such a store as [[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the round trip through zarr-java itself returned the correct values and no Java-only test could see the corruption.
@konstibob@claude
Honour the Zarr v2 order key instead of ignoring it
50c7aca
`.zarray`'s `order` was parsed into `ArrayMetadata.order` and then never read,
so `"order": "F"` was silently treated as row-major in both directions:
* Reading an F-order store written by zarr-python returned the elements of
each chunk transposed. A 3x4 int32 array holding [[0,1,2,3],[4,5,6,7],
[8,9,10,11]] read back as [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11].
* Writing was worse: `withOrder(Order.F)` put `"order": "F"` into `.zarray`
while laying the chunk out row-major, so zarr-java emitted stores that
every other implementation misreads. zarr-python read such a store as
[[0,3,6,9],[1,4,7,10],[2,5,8,11]]. Because the reader shared the bug, the
round trip through zarr-java itself returned the correct values and no
Java-only test could see the corruption.
Zarr v3 dropped `order` and expresses the same layout with the `transpose`
codec, so `order` is modelled the same way here: a new `FortranOrderCodec`
array->array codec reverses all axes ahead of the `BytesCodec`, whose row-major
serialization of the reversed view is exactly the column-major serialization of
the logical chunk. Reversing axes is its own inverse, so one operation serves
both directions. The codec is never serialized; `order` stays the on-disk
representation, and it is only inserted for rank > 1, where the two orders
actually differ.
`decode` materializes the result with `Array.copy()`. A permuted view answers
iterators and `get(int[])` correctly but keeps a column-major backing store, so
linear accessors such as `getInt(int)` would walk it in the wrong order — and
the single-full-chunk fast path in `core.Array.read` hands a decoded chunk
straight to the caller.
Testing uses zarr-python as the oracle, since a Java-only round trip cannot
detect a symmetric bug:
* `testdata/golden/` gains four fixtures written by zarr-python (1.3 kB in
total, no compressor, so the chunk files are raw element bytes), covering a
C/F pair, rank 3, and a chunk grid that leaves partial edge chunks.
`src/test/python-scripts/generate_v2_order_golden.py` regenerates them.
* `ZarrV2OrderTest` checks both directions against those fixtures offline,
with no Python at test time: reads must yield `arange`, and writing
`arange` must reproduce zarr-python's chunk bytes exactly. It also pins
down that C and F differ on disk (uncompressed and compressed), that the
two coincide for rank 1, that a single-full-chunk read is physically
contiguous, and that `order` survives `resize` and attribute updates.
7 of its 15 tests fail without this change.
* `ZarrPythonTests` gains `testReadOrderV2`/`testWriteOrderV2`, which drive
the installed zarr-python in both directions so that a divergence from the
committed fixtures is noticed after a zarr-python upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@konstibobkonstibob
changed the title
Reading F-Order Files in v3 and v2Reading and F-Order Files in v2Aug 27, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@konstibob