Skip to content

feat: Implement ZEP 8 URL syntax support for zarr-python - #3567

Open
ianhi wants to merge 26 commits into
zarr-developers:mainfrom
ianhi:ian/jhamman/zep8
Open

feat: Implement ZEP 8 URL syntax support for zarr-python #3567
ianhi wants to merge 26 commits into
zarr-developers:mainfrom
ianhi:ian/jhamman/zep8

Conversation

@ianhi

@ianhiianhi commented Nov 4, 2025

Copy link
Copy Markdown
Contributor

Continuation of @jhamman's #3369 with addressing my review comments
Claude summary of the changes on top of Joe's PR:

  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations

Test Coverage: 167 tests passing (1 skipped), up from ~150 tests

Missing ZEP 8 Features Analysis

Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:

✅ Currently Implemented (9 adapters)

  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)

❌ Missing from Spec (11 adapters)

Storage/Database Adapters:

  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)

Compression Adapters:
3. gzip: - Transparent gzip decompression
4. zstd: - Transparent zstd decompression

Data Format Adapters:
5. n5: - N5 format support
6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters
7. neuroglancer-precomputed: - Neuroglancer format
8. json: - JSON pointer access

Utility Adapters:
9. byte-range:start-end - Byte range extraction
10. ..: - Parent directory traversal (for relative URLs)

Other Missing Features:

  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)

jhammanand others added 26 commits August 11, 2025 13:13
@jhamman
feat: Implement ZEP 8 URL syntax support for zarr-python
0e179d3
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@jhamman
fixup
c5aefd3
@jhamman
rework remote adapter and add back zip adapter
26192cb
@jhamman
add logging adapter
b67edcf
@jhamman
fix doc build
dc5acd3
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
227c214
…into feature/zep8-url-support
@jhamman
more test coverage
ebe9660
@jhamman
remove gcs scheme support
3b55281
@jhamman
add s3 compatible object store support with s3+https://...
c8941ba
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
2126a3b
…into feature/zep8-url-support
@jhamman
fix partial writes check in chained store
0cbeb72
@jhamman
skip for old fsspec versions
14c2817
@jhamman
windows fix
20a35f0
@jhamman
windows fix
7533d95
@jhamman
fixup + tests
542c3ec
@jhamman
fixup
4e8a925
@jhamman
fixup
0315e0b
@jhamman
fixup for win
5666478
@jhamman
cleanup zep8 tests more
35526a5
@ianhi
don't unpack storage_options through multiple adapters
523b9e2
@ianhi
refactor url logic - now works with s3+https scheme
5af145f
@ianhi
cleanup execeptions and store_options validation
c580fe0
@ianhi
revert typing change
4f3fa66
@ianhi
modularize
410e22a
@ianhi
bubble zarr_format up
f57f430
@ianhi
nested zip adapter support
56cf8f1
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Nov 4, 2025
@ianhiianhi mentioned this pull request Nov 24, 2025
Open
@jbms

jbms commented Nov 25, 2025

Copy link
Copy Markdown

Note: I have moved the URL pipeline proposal over from the ZEP repo to a separate repository:

https://github.com/jbms/url-pipeline

@maxrjones

maxrjones commented Jun 16, 2026

Copy link
Copy Markdown
Member

Hey @ianhi, I hope you're doing well! I just wanted to check in to see if you're interested in continuing this work?

@jhammanjhamman mentioned this pull request Jul 26, 2026
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ianhi@jbms@maxrjones@jhamman
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
 feat: Implement ZEP 8 URL syntax support for zarr-python by ianhi · Pull Request #3567 · zarr-developers/zarr-python · GitHub
Skip to content
 feat: Implement ZEP 8 URL syntax support for zarr-python  - #3567
Open
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Open
 feat: Implement ZEP 8 URL syntax support for zarr-python #3567
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Conversation
@ianhi
@ianhiianhi
commented
Nov 4, 2025

Copy link


Copy Markdown

Contributor
Continuation of @jhamman's #3369 with addressing my review comments

Claude summary of the changes on top of Joe's PR:
  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations
Test Coverage: 167 tests passing (1 skipped), up from ~150 tests
Missing ZEP 8 Features Analysis
Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:
✅ Currently Implemented (9 adapters)
  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)
❌ Missing from Spec (11 adapters)
Storage/Database Adapters:
  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)
Compression Adapters:

3. gzip: - Transparent gzip decompression

4. zstd: - Transparent zstd decompression
Data Format Adapters:

5. n5: - N5 format support

6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters

7. neuroglancer-precomputed: - Neuroglancer format

8. json: - JSON pointer access
Utility Adapters:

9. byte-range:start-end - Byte range extraction

10. ..: - Parent directory traversal (for relative URLs)
Other Missing Features:
  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    
implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)
jhammanand others
added 26 commits
August 11, 2025 13:13
@jhamman
feat: Implement ZEP 8 URL syntax support for zarr-python
0e179d3
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@jhamman
fixup
c5aefd3
@jhamman
rework remote adapter and add back zip adapter
26192cb
@jhamman
add logging adapter
b67edcf
@jhamman
fix doc build
dc5acd3
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
227c214
…into feature/zep8-url-support
@jhamman
more test coverage
ebe9660
@jhamman
remove gcs scheme support
3b55281
@jhamman
add s3 compatible object store support with s3+https://...
c8941ba
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
2126a3b
…into feature/zep8-url-support
@jhamman
fix partial writes check in chained store
0cbeb72
@jhamman
skip for old fsspec versions
14c2817
@jhamman
windows fix
20a35f0
@jhamman
windows fix
7533d95
@jhamman
fixup + tests
542c3ec
@jhamman
fixup
4e8a925
@jhamman
fixup
0315e0b
@jhamman
fixup for win
5666478
@jhamman
cleanup zep8 tests more
35526a5
@ianhi
don't unpack storage_options through multiple adapters
523b9e2
@ianhi
refactor url logic - now works with s3+https scheme
5af145f
@ianhi
cleanup execeptions and store_options validation
c580fe0
@ianhi
revert typing change
4f3fa66
@ianhi
modularize
410e22a
@ianhi
bubble zarr_format up
f57f430
@ianhi
nested zip adapter support
56cf8f1
@github-actionsgithub-actionsBot
added
the 
needs release notes
Automatically applied to PRs which haven't added release notes
label
Nov 4, 2025
@ianhiianhi
mentioned this pull request
Nov 24, 2025

Open

@jbms
jbms
commented
Nov 25, 2025

Copy link


Copy Markdown

Note: I have moved the URL pipeline proposal over from the ZEP repo to a separate repository:
https://github.com/jbms/url-pipeline
@maxrjones
maxrjones
commented
Jun 16, 2026

Copy link


Copy Markdown

Member
Hey @ianhi, I hope you're doing well! I just wanted to check in to see if you're interested in continuing this work?
@jhammanjhamman
mentioned this pull request
Jul 26, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@ianhi@jbms@maxrjones@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
 feat: Implement ZEP 8 URL syntax support for zarr-python by ianhi · Pull Request #3567 · zarr-developers/zarr-python · GitHub
Skip to content
 feat: Implement ZEP 8 URL syntax support for zarr-python  - #3567
Open
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Open
 feat: Implement ZEP 8 URL syntax support for zarr-python #3567
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Conversation
@ianhi
@ianhiianhi
commented
Nov 4, 2025

Copy link


Copy Markdown

Contributor
Continuation of @jhamman's #3369 with addressing my review comments

Claude summary of the changes on top of Joe's PR:
  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations
Test Coverage: 167 tests passing (1 skipped), up from ~150 tests
Missing ZEP 8 Features Analysis
Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:
✅ Currently Implemented (9 adapters)
  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)
❌ Missing from Spec (11 adapters)
Storage/Database Adapters:
  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)
Compression Adapters:

3. gzip: - Transparent gzip decompression

4. zstd: - Transparent zstd decompression
Data Format Adapters:

5. n5: - N5 format support

6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters

7. neuroglancer-precomputed: - Neuroglancer format

8. json: - JSON pointer access
Utility Adapters:

9. byte-range:start-end - Byte range extraction

10. ..: - Parent directory traversal (for relative URLs)
Other Missing Features:
  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    
implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)
jhammanand others
added 26 commits
August 11, 2025 13:13
@jhamman
feat: Implement ZEP 8 URL syntax support for zarr-python
0e179d3
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@jhamman
fixup
c5aefd3
@jhamman
rework remote adapter and add back zip adapter
26192cb
@jhamman
add logging adapter
b67edcf
@jhamman
fix doc build
dc5acd3
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
227c214
…into feature/zep8-url-support
@jhamman
more test coverage
ebe9660
@jhamman
remove gcs scheme support
3b55281
@jhamman
add s3 compatible object store support with s3+https://...
c8941ba
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
2126a3b
…into feature/zep8-url-support
@jhamman
fix partial writes check in chained store
0cbeb72
@jhamman
skip for old fsspec versions
14c2817
@jhamman
windows fix
20a35f0
@jhamman
windows fix
7533d95
@jhamman
fixup + tests
542c3ec
@jhamman
fixup
4e8a925
@jhamman
fixup
0315e0b
@jhamman
fixup for win
5666478
@jhamman
cleanup zep8 tests more
35526a5
@ianhi
don't unpack storage_options through multiple adapters
523b9e2
@ianhi
refactor url logic - now works with s3+https scheme
5af145f
@ianhi
cleanup execeptions and store_options validation
c580fe0
@ianhi
revert typing change
4f3fa66
@ianhi
modularize
410e22a
@ianhi
bubble zarr_format up
f57f430
@ianhi
nested zip adapter support
56cf8f1
@github-actionsgithub-actionsBot
added
the 
needs release notes
Automatically applied to PRs which haven't added release notes
label
Nov 4, 2025
@ianhiianhi
mentioned this pull request
Nov 24, 2025

Open

@jbms
jbms
commented
Nov 25, 2025

Copy link


Copy Markdown

Note: I have moved the URL pipeline proposal over from the ZEP repo to a separate repository:
https://github.com/jbms/url-pipeline
@maxrjones
maxrjones
commented
Jun 16, 2026

Copy link


Copy Markdown

Member
Hey @ianhi, I hope you're doing well! I just wanted to check in to see if you're interested in continuing this work?
@jhammanjhamman
mentioned this pull request
Jul 26, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@ianhi@jbms@maxrjones@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
 feat: Implement ZEP 8 URL syntax support for zarr-python by ianhi · Pull Request #3567 · zarr-developers/zarr-python · GitHub
Skip to content
 feat: Implement ZEP 8 URL syntax support for zarr-python  - #3567
Open
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Open
 feat: Implement ZEP 8 URL syntax support for zarr-python #3567
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Conversation
@ianhi
@ianhiianhi
commented
Nov 4, 2025

Copy link


Copy Markdown

Contributor
Continuation of @jhamman's #3369 with addressing my review comments

Claude summary of the changes on top of Joe's PR:
  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations
Test Coverage: 167 tests passing (1 skipped), up from ~150 tests
Missing ZEP 8 Features Analysis
Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:
✅ Currently Implemented (9 adapters)
  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)
❌ Missing from Spec (11 adapters)
Storage/Database Adapters:
  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)
Compression Adapters:

3. gzip: - Transparent gzip decompression

4. zstd: - Transparent zstd decompression
Data Format Adapters:

5. n5: - N5 format support

6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters

7. neuroglancer-precomputed: - Neuroglancer format

8. json: - JSON pointer access
Utility Adapters:

9. byte-range:start-end - Byte range extraction

10. ..: - Parent directory traversal (for relative URLs)
Other Missing Features:
  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    
implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)
jhammanand others
added 26 commits
August 11, 2025 13:13
@jhamman
feat: Implement ZEP 8 URL syntax support for zarr-python
0e179d3
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@jhamman
fixup
c5aefd3
@jhamman
rework remote adapter and add back zip adapter
26192cb
@jhamman
add logging adapter
b67edcf
@jhamman
fix doc build
dc5acd3
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
227c214
…into feature/zep8-url-support
@jhamman
more test coverage
ebe9660
@jhamman
remove gcs scheme support
3b55281
@jhamman
add s3 compatible object store support with s3+https://...
c8941ba
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
2126a3b
…into feature/zep8-url-support
@jhamman
fix partial writes check in chained store
0cbeb72
@jhamman
skip for old fsspec versions
14c2817
@jhamman
windows fix
20a35f0
@jhamman
windows fix
7533d95
@jhamman
fixup + tests
542c3ec
@jhamman
fixup
4e8a925
@jhamman
fixup
0315e0b
@jhamman
fixup for win
5666478
@jhamman
cleanup zep8 tests more
35526a5
@ianhi
don't unpack storage_options through multiple adapters
523b9e2
@ianhi
refactor url logic - now works with s3+https scheme
5af145f
@ianhi
cleanup execeptions and store_options validation
c580fe0
@ianhi
revert typing change
4f3fa66
@ianhi
modularize
410e22a
@ianhi
bubble zarr_format up
f57f430
@ianhi
nested zip adapter support
56cf8f1
@github-actionsgithub-actionsBot
added
the 
needs release notes
Automatically applied to PRs which haven't added release notes
label
Nov 4, 2025
@ianhiianhi
mentioned this pull request
Nov 24, 2025

Open

@jbms
jbms
commented
Nov 25, 2025

Copy link


Copy Markdown

Note: I have moved the URL pipeline proposal over from the ZEP repo to a separate repository:
https://github.com/jbms/url-pipeline
@maxrjones
maxrjones
commented
Jun 16, 2026

Copy link


Copy Markdown

Member
Hey @ianhi, I hope you're doing well! I just wanted to check in to see if you're interested in continuing this work?
@jhammanjhamman
mentioned this pull request
Jul 26, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@ianhi@jbms@maxrjones@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
 feat: Implement ZEP 8 URL syntax support for zarr-python by ianhi · Pull Request #3567 · zarr-developers/zarr-python · GitHub
Skip to content
 feat: Implement ZEP 8 URL syntax support for zarr-python  - #3567
Open
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Open
 feat: Implement ZEP 8 URL syntax support for zarr-python #3567
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Conversation
@ianhi
@ianhiianhi
commented
Nov 4, 2025

Copy link


Copy Markdown

Contributor
Continuation of @jhamman's #3369 with addressing my review comments

Claude summary of the changes on top of Joe's PR:
  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations
Test Coverage: 167 tests passing (1 skipped), up from ~150 tests
Missing ZEP 8 Features Analysis
Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:
✅ Currently Implemented (9 adapters)
  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)
❌ Missing from Spec (11 adapters)
Storage/Database Adapters:
  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)
Compression Adapters:

3. gzip: - Transparent gzip decompression

4. zstd: - Transparent zstd decompression
Data Format Adapters:

5. n5: - N5 format support

6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters

7. neuroglancer-precomputed: - Neuroglancer format

8. json: - JSON pointer access
Utility Adapters:

9. byte-range:start-end - Byte range extraction

10. ..: - Parent directory traversal (for relative URLs)
Other Missing Features:
  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    
implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)
jhammanand others
added 26 commits
August 11, 2025 13:13
@jhamman
feat: Implement ZEP 8 URL syntax support for zarr-python
0e179d3
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@jhamman
fixup
c5aefd3
@jhamman
rework remote adapter and add back zip adapter
26192cb
@jhamman
add logging adapter
b67edcf
@jhamman
fix doc build
dc5acd3
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
227c214
…into feature/zep8-url-support
@jhamman
more test coverage
ebe9660
@jhamman
remove gcs scheme support
3b55281
@jhamman
add s3 compatible object store support with s3+https://...
c8941ba
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
2126a3b
…into feature/zep8-url-support
@jhamman
fix partial writes check in chained store
0cbeb72
@jhamman
skip for old fsspec versions
14c2817
@jhamman
windows fix
20a35f0
@jhamman
windows fix
7533d95
@jhamman
fixup + tests
542c3ec
@jhamman
fixup
4e8a925
@jhamman
fixup
0315e0b
@jhamman
fixup for win
5666478
@jhamman
cleanup zep8 tests more
35526a5
@ianhi
don't unpack storage_options through multiple adapters
523b9e2
@ianhi
refactor url logic - now works with s3+https scheme
5af145f
@ianhi
cleanup execeptions and store_options validation
c580fe0
@ianhi
revert typing change
4f3fa66
@ianhi
modularize
410e22a
@ianhi
bubble zarr_format up
f57f430
@ianhi
nested zip adapter support
56cf8f1
@github-actionsgithub-actionsBot
added
the 
needs release notes
Automatically applied to PRs which haven't added release notes
label
Nov 4, 2025
@ianhiianhi
mentioned this pull request
Nov 24, 2025

Open

@jbms
jbms
commented
Nov 25, 2025

Copy link


Copy Markdown

Note: I have moved the URL pipeline proposal over from the ZEP repo to a separate repository:
https://github.com/jbms/url-pipeline
@maxrjones
maxrjones
commented
Jun 16, 2026

Copy link


Copy Markdown

Member
Hey @ianhi, I hope you're doing well! I just wanted to check in to see if you're interested in continuing this work?
@jhammanjhamman
mentioned this pull request
Jul 26, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@ianhi@jbms@maxrjones@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
 feat: Implement ZEP 8 URL syntax support for zarr-python by ianhi · Pull Request #3567 · zarr-developers/zarr-python · GitHub
Skip to content
 feat: Implement ZEP 8 URL syntax support for zarr-python  - #3567
Open
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Open
 feat: Implement ZEP 8 URL syntax support for zarr-python #3567
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Conversation
@ianhi
@ianhiianhi
commented
Nov 4, 2025

Copy link


Copy Markdown

Contributor
Continuation of @jhamman's #3369 with addressing my review comments

Claude summary of the changes on top of Joe's PR:
  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations
Test Coverage: 167 tests passing (1 skipped), up from ~150 tests
Missing ZEP 8 Features Analysis
Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:
✅ Currently Implemented (9 adapters)
  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)
❌ Missing from Spec (11 adapters)
Storage/Database Adapters:
  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)
Compression Adapters:

3. gzip: - Transparent gzip decompression

4. zstd: - Transparent zstd decompression
Data Format Adapters:

5. n5: - N5 format support

6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters

7. neuroglancer-precomputed: - Neuroglancer format

8. json: - JSON pointer access
Utility Adapters:

9. byte-range:start-end - Byte range extraction

10. ..: - Parent directory traversal (for relative URLs)
Other Missing Features:
  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    
implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)
jhammanand others
added 26 commits
August 11, 2025 13:13
@jhamman
feat: Implement ZEP 8 URL syntax support for zarr-python
0e179d3
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@jhamman
fixup
c5aefd3
@jhamman
rework remote adapter and add back zip adapter
26192cb
@jhamman
add logging adapter
b67edcf
@jhamman
fix doc build
dc5acd3
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
227c214
…into feature/zep8-url-support
@jhamman
more test coverage
ebe9660
@jhamman
remove gcs scheme support
3b55281
@jhamman
add s3 compatible object store support with s3+https://...
c8941ba
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
2126a3b
…into feature/zep8-url-support
@jhamman
fix partial writes check in chained store
0cbeb72
@jhamman
skip for old fsspec versions
14c2817
@jhamman
windows fix
20a35f0
@jhamman
windows fix
7533d95
@jhamman
fixup + tests
542c3ec
@jhamman
fixup
4e8a925
@jhamman
fixup
0315e0b
@jhamman
fixup for win
5666478
@jhamman
cleanup zep8 tests more
35526a5
@ianhi
don't unpack storage_options through multiple adapters
523b9e2
@ianhi
refactor url logic - now works with s3+https scheme
5af145f
@ianhi
cleanup execeptions and store_options validation
c580fe0
@ianhi
revert typing change
4f3fa66
@ianhi
modularize
410e22a
@ianhi
bubble zarr_format up
f57f430
@ianhi
nested zip adapter support
56cf8f1
@github-actionsgithub-actionsBot
added
the 
needs release notes
Automatically applied to PRs which haven't added release notes
label
Nov 4, 2025
@ianhiianhi
mentioned this pull request
Nov 24, 2025

Open

@jbms
jbms
commented
Nov 25, 2025

Copy link


Copy Markdown

Note: I have moved the URL pipeline proposal over from the ZEP repo to a separate repository:
https://github.com/jbms/url-pipeline
@maxrjones
maxrjones
commented
Jun 16, 2026

Copy link


Copy Markdown

Member
Hey @ianhi, I hope you're doing well! I just wanted to check in to see if you're interested in continuing this work?
@jhammanjhamman
mentioned this pull request
Jul 26, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@ianhi@jbms@maxrjones@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
 feat: Implement ZEP 8 URL syntax support for zarr-python by ianhi · Pull Request #3567 · zarr-developers/zarr-python · GitHub
Skip to content
 feat: Implement ZEP 8 URL syntax support for zarr-python  - #3567
Open
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Open
 feat: Implement ZEP 8 URL syntax support for zarr-python #3567
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Conversation
@ianhi
@ianhiianhi
commented
Nov 4, 2025

Copy link


Copy Markdown

Contributor
Continuation of @jhamman's #3369 with addressing my review comments

Claude summary of the changes on top of Joe's PR:
  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations
Test Coverage: 167 tests passing (1 skipped), up from ~150 tests
Missing ZEP 8 Features Analysis
Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:
✅ Currently Implemented (9 adapters)
  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)
❌ Missing from Spec (11 adapters)
Storage/Database Adapters:
  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)
Compression Adapters:

3. gzip: - Transparent gzip decompression

4. zstd: - Transparent zstd decompression
Data Format Adapters:

5. n5: - N5 format support

6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters

7. neuroglancer-precomputed: - Neuroglancer format

8. json: - JSON pointer access
Utility Adapters:

9. byte-range:start-end - Byte range extraction

10. ..: - Parent directory traversal (for relative URLs)
Other Missing Features:
  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    
implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)
jhammanand others
added 26 commits
August 11, 2025 13:13
@jhamman
feat: Implement ZEP 8 URL syntax support for zarr-python
0e179d3
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@jhamman
fixup
c5aefd3
@jhamman
rework remote adapter and add back zip adapter
26192cb
@jhamman
add logging adapter
b67edcf
@jhamman
fix doc build
dc5acd3
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
227c214
…into feature/zep8-url-support
@jhamman
more test coverage
ebe9660
@jhamman
remove gcs scheme support
3b55281
@jhamman
add s3 compatible object store support with s3+https://...
c8941ba
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
2126a3b
…into feature/zep8-url-support
@jhamman
fix partial writes check in chained store
0cbeb72
@jhamman
skip for old fsspec versions
14c2817
@jhamman
windows fix
20a35f0
@jhamman
windows fix
7533d95
@jhamman
fixup + tests
542c3ec
@jhamman
fixup
4e8a925
@jhamman
fixup
0315e0b
@jhamman
fixup for win
5666478
@jhamman
cleanup zep8 tests more
35526a5
@ianhi
don't unpack storage_options through multiple adapters
523b9e2
@ianhi
refactor url logic - now works with s3+https scheme
5af145f
@ianhi
cleanup execeptions and store_options validation
c580fe0
@ianhi
revert typing change
4f3fa66
@ianhi
modularize
410e22a
@ianhi
bubble zarr_format up
f57f430
@ianhi
nested zip adapter support
56cf8f1
@github-actionsgithub-actionsBot
added
the 
needs release notes
Automatically applied to PRs which haven't added release notes
label
Nov 4, 2025
@ianhiianhi
mentioned this pull request
Nov 24, 2025

Open

@jbms
jbms
commented
Nov 25, 2025

Copy link


Copy Markdown

Note: I have moved the URL pipeline proposal over from the ZEP repo to a separate repository:
https://github.com/jbms/url-pipeline
@maxrjones
maxrjones
commented
Jun 16, 2026

Copy link


Copy Markdown

Member
Hey @ianhi, I hope you're doing well! I just wanted to check in to see if you're interested in continuing this work?
@jhammanjhamman
mentioned this pull request
Jul 26, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@ianhi@jbms@maxrjones@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
 feat: Implement ZEP 8 URL syntax support for zarr-python by ianhi · Pull Request #3567 · zarr-developers/zarr-python · GitHub
Skip to content
 feat: Implement ZEP 8 URL syntax support for zarr-python  - #3567
Open
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Open
 feat: Implement ZEP 8 URL syntax support for zarr-python #3567
ianhi wants to merge 26 commits into
zarr-developers:mainfrom 
ianhi:ian/jhamman/zep8
Conversation
@ianhi
@ianhiianhi
commented
Nov 4, 2025

Copy link


Copy Markdown

Contributor
Continuation of @jhamman's #3369 with addressing my review comments

Claude summary of the changes on top of Joe's PR:
  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations
Test Coverage: 167 tests passing (1 skipped), up from ~150 tests
Missing ZEP 8 Features Analysis
Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:
✅ Currently Implemented (9 adapters)
  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)
❌ Missing from Spec (11 adapters)
Storage/Database Adapters:
  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)
Compression Adapters:

3. gzip: - Transparent gzip decompression

4. zstd: - Transparent zstd decompression
Data Format Adapters:

5. n5: - N5 format support

6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters

7. neuroglancer-precomputed: - Neuroglancer format

8. json: - JSON pointer access
Utility Adapters:

9. byte-range:start-end - Byte range extraction

10. ..: - Parent directory traversal (for relative URLs)
Other Missing Features:
  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    
implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)
jhammanand others
added 26 commits
August 11, 2025 13:13
@jhamman
feat: Implement ZEP 8 URL syntax support for zarr-python
0e179d3
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@jhamman
fixup
c5aefd3
@jhamman
rework remote adapter and add back zip adapter
26192cb
@jhamman
add logging adapter
b67edcf
@jhamman
fix doc build
dc5acd3
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
227c214
…into feature/zep8-url-support
@jhamman
more test coverage
ebe9660
@jhamman
remove gcs scheme support
3b55281
@jhamman
add s3 compatible object store support with s3+https://...
c8941ba
@jhamman
Merge branch 'main' ofhttps://github.com/zarr-developers/zarr-python…
2126a3b
…into feature/zep8-url-support
@jhamman
fix partial writes check in chained store
0cbeb72
@jhamman
skip for old fsspec versions
14c2817
@jhamman
windows fix
20a35f0
@jhamman
windows fix
7533d95
@jhamman
fixup + tests
542c3ec
@jhamman
fixup
4e8a925
@jhamman
fixup
0315e0b
@jhamman
fixup for win
5666478
@jhamman
cleanup zep8 tests more
35526a5
@ianhi
don't unpack storage_options through multiple adapters
523b9e2
@ianhi
refactor url logic - now works with s3+https scheme
5af145f
@ianhi
cleanup execeptions and store_options validation
c580fe0
@ianhi
revert typing change
4f3fa66
@ianhi
modularize
410e22a
@ianhi
bubble zarr_format up
f57f430
@ianhi
nested zip adapter support
56cf8f1
@github-actionsgithub-actionsBot
added
the 
needs release notes
Automatically applied to PRs which haven't added release notes
label
Nov 4, 2025
@ianhiianhi
mentioned this pull request
Nov 24, 2025

Open

@jbms
jbms
commented
Nov 25, 2025

Copy link


Copy Markdown

Note: I have moved the URL pipeline proposal over from the ZEP repo to a separate repository:
https://github.com/jbms/url-pipeline
@maxrjones
maxrjones
commented
Jun 16, 2026

Copy link


Copy Markdown

Member
Hey @ianhi, I hope you're doing well! I just wanted to check in to see if you're interested in continuing this work?
@jhammanjhamman
mentioned this pull request
Jul 26, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@ianhi@jbms@maxrjones@jhamman