Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes - #2430

Merged
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix
Nov 5, 2024
Merged

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes#2430
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix

Conversation

@jhamman

@jhammanjhamman commented Oct 22, 2024

Copy link
Copy Markdown
Member

This PR implements delete_dir teaches the Group and Array classes to use it when overwriting nodes in a hierarchy.

Also included here:
- delete_prefix - not used but perhaps we should just use it instead of delete_dir

  • changes list_prefix to return absolute paths instead of stripping prefix. Given that list_prefix was unused before, this wasn't very disruptive but calling it out so others can help decide if this is what we want.

Still needs:

  • new store tests for delete_dir
  • new group tests for edge cases around delete_dir

closes#2191
first steps toward #2108 and #2359

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
jhamman marked this pull request as ready for review October 22, 2024 13:44
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/api/asynchronous.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/storage/local.py Outdated
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py

async def delete_dir(self, prefix: str, recursive: bool = True) -> None:
"""
Remove all keys and prefixes in the store that begin with a given prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make life much easier for Store implementers if we declare that prefix will always be the path of a group or a array. Otherwise, all implementations need to deal with unnecessary edge cases like delete_dir("path/to/array/c/0") or worse.

Icechunk is an example of a store that could very cleanly and efficiently implement the case for a group/array prefix, but it would struggle to handle other prefixes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for recursive = False ? If zarr itself doesn't have one, I'd remove the argument.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see a use case for delete_dir("path/to/array/c/") (remove all chunks from an array) so I think we should allow that -- even if it makes life a little harder in icechunk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's totally fine. Let's document the three possible prefix cases, and say that stores are allowed to do undefined behavior for other prefixes

Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py
if not self.supports_listing:
raise NotImplementedError
self._check_writable()
async for key in self.list_prefix(prefix):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document what's the expected return value of list_prefix? Absolute vs. relative, final "/" or not, etc.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done in the list_prefix docstring

@TomAugspurger

Copy link
Copy Markdown
Contributor

Looks good overall. I've been struggling a bit with whether Stores should be a bit more aware about the context the operation is occurring in (e.g. deleting a group vs a specific chunk of an array; I think this relates to the conversation at #2430 (comment)). But I think this is a good improvement on its own.

@jhammanjhamman mentioned this pull request Oct 28, 2024
6 tasks
@jhamman

Copy link
Copy Markdown
MemberAuthor

@TomAugspurger / @d-v-b / @paraseba - any final comments here. I'd like to get this in ahead of the larger store mode refactor in #2442

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores, since RemoteStore will return keys relative to its path attribute, and similarly LocalStore with its root attribute.

@jhamman

Copy link
Copy Markdown
MemberAuthor

regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores

Would defining these as "relative to the root of the store" be clearer?

Comment threadsrc/zarr/abc/store.py Outdated
@jhamman

Copy link
Copy Markdown
MemberAuthor

pre-commit.ci autofix

@jhammanjhamman closed this Nov 5, 2024
@jhammanjhamman reopened this Nov 5, 2024
@jhamman
jhamman merged commit a31046c into zarr-developers:mainNov 5, 2024
@jhamman
jhamman deleted the feature/store_erase_prefix branch November 5, 2024 04:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3] deleting a group does not delete subgroups

4 participants

@jhamman@TomAugspurger@paraseba@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes - #2430

Merged
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix
Nov 5, 2024
Merged

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes#2430
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix

Conversation

@jhamman

@jhammanjhamman commented Oct 22, 2024

Copy link
Copy Markdown
Member

This PR implements delete_dir teaches the Group and Array classes to use it when overwriting nodes in a hierarchy.

Also included here:
- delete_prefix - not used but perhaps we should just use it instead of delete_dir

  • changes list_prefix to return absolute paths instead of stripping prefix. Given that list_prefix was unused before, this wasn't very disruptive but calling it out so others can help decide if this is what we want.

Still needs:

  • new store tests for delete_dir
  • new group tests for edge cases around delete_dir

closes#2191
first steps toward #2108 and #2359

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
jhamman marked this pull request as ready for review October 22, 2024 13:44
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/api/asynchronous.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/storage/local.py Outdated
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py

async def delete_dir(self, prefix: str, recursive: bool = True) -> None:
"""
Remove all keys and prefixes in the store that begin with a given prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make life much easier for Store implementers if we declare that prefix will always be the path of a group or a array. Otherwise, all implementations need to deal with unnecessary edge cases like delete_dir("path/to/array/c/0") or worse.

Icechunk is an example of a store that could very cleanly and efficiently implement the case for a group/array prefix, but it would struggle to handle other prefixes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for recursive = False ? If zarr itself doesn't have one, I'd remove the argument.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see a use case for delete_dir("path/to/array/c/") (remove all chunks from an array) so I think we should allow that -- even if it makes life a little harder in icechunk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's totally fine. Let's document the three possible prefix cases, and say that stores are allowed to do undefined behavior for other prefixes

Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py
if not self.supports_listing:
raise NotImplementedError
self._check_writable()
async for key in self.list_prefix(prefix):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document what's the expected return value of list_prefix? Absolute vs. relative, final "/" or not, etc.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done in the list_prefix docstring

@TomAugspurger

Copy link
Copy Markdown
Contributor

Looks good overall. I've been struggling a bit with whether Stores should be a bit more aware about the context the operation is occurring in (e.g. deleting a group vs a specific chunk of an array; I think this relates to the conversation at #2430 (comment)). But I think this is a good improvement on its own.

@jhammanjhamman mentioned this pull request Oct 28, 2024
6 tasks
@jhamman

Copy link
Copy Markdown
MemberAuthor

@TomAugspurger / @d-v-b / @paraseba - any final comments here. I'd like to get this in ahead of the larger store mode refactor in #2442

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores, since RemoteStore will return keys relative to its path attribute, and similarly LocalStore with its root attribute.

@jhamman

Copy link
Copy Markdown
MemberAuthor

regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores

Would defining these as "relative to the root of the store" be clearer?

Comment threadsrc/zarr/abc/store.py Outdated
@jhamman

Copy link
Copy Markdown
MemberAuthor

pre-commit.ci autofix

@jhammanjhamman closed this Nov 5, 2024
@jhammanjhamman reopened this Nov 5, 2024
@jhamman
jhamman merged commit a31046c into zarr-developers:mainNov 5, 2024
@jhamman
jhamman deleted the feature/store_erase_prefix branch November 5, 2024 04:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3] deleting a group does not delete subgroups

4 participants

@jhamman@TomAugspurger@paraseba@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes - #2430

Merged
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix
Nov 5, 2024
Merged

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes#2430
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix

Conversation

@jhamman

@jhammanjhamman commented Oct 22, 2024

Copy link
Copy Markdown
Member

This PR implements delete_dir teaches the Group and Array classes to use it when overwriting nodes in a hierarchy.

Also included here:
- delete_prefix - not used but perhaps we should just use it instead of delete_dir

  • changes list_prefix to return absolute paths instead of stripping prefix. Given that list_prefix was unused before, this wasn't very disruptive but calling it out so others can help decide if this is what we want.

Still needs:

  • new store tests for delete_dir
  • new group tests for edge cases around delete_dir

closes#2191
first steps toward #2108 and #2359

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
jhamman marked this pull request as ready for review October 22, 2024 13:44
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/api/asynchronous.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/storage/local.py Outdated
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py

async def delete_dir(self, prefix: str, recursive: bool = True) -> None:
"""
Remove all keys and prefixes in the store that begin with a given prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make life much easier for Store implementers if we declare that prefix will always be the path of a group or a array. Otherwise, all implementations need to deal with unnecessary edge cases like delete_dir("path/to/array/c/0") or worse.

Icechunk is an example of a store that could very cleanly and efficiently implement the case for a group/array prefix, but it would struggle to handle other prefixes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for recursive = False ? If zarr itself doesn't have one, I'd remove the argument.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see a use case for delete_dir("path/to/array/c/") (remove all chunks from an array) so I think we should allow that -- even if it makes life a little harder in icechunk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's totally fine. Let's document the three possible prefix cases, and say that stores are allowed to do undefined behavior for other prefixes

Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py
if not self.supports_listing:
raise NotImplementedError
self._check_writable()
async for key in self.list_prefix(prefix):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document what's the expected return value of list_prefix? Absolute vs. relative, final "/" or not, etc.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done in the list_prefix docstring

@TomAugspurger

Copy link
Copy Markdown
Contributor

Looks good overall. I've been struggling a bit with whether Stores should be a bit more aware about the context the operation is occurring in (e.g. deleting a group vs a specific chunk of an array; I think this relates to the conversation at #2430 (comment)). But I think this is a good improvement on its own.

@jhammanjhamman mentioned this pull request Oct 28, 2024
6 tasks
@jhamman

Copy link
Copy Markdown
MemberAuthor

@TomAugspurger / @d-v-b / @paraseba - any final comments here. I'd like to get this in ahead of the larger store mode refactor in #2442

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores, since RemoteStore will return keys relative to its path attribute, and similarly LocalStore with its root attribute.

@jhamman

Copy link
Copy Markdown
MemberAuthor

regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores

Would defining these as "relative to the root of the store" be clearer?

Comment threadsrc/zarr/abc/store.py Outdated
@jhamman

Copy link
Copy Markdown
MemberAuthor

pre-commit.ci autofix

@jhammanjhamman closed this Nov 5, 2024
@jhammanjhamman reopened this Nov 5, 2024
@jhamman
jhamman merged commit a31046c into zarr-developers:mainNov 5, 2024
@jhamman
jhamman deleted the feature/store_erase_prefix branch November 5, 2024 04:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3] deleting a group does not delete subgroups

4 participants

@jhamman@TomAugspurger@paraseba@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes - #2430

Merged
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix
Nov 5, 2024
Merged

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes#2430
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix

Conversation

@jhamman

@jhammanjhamman commented Oct 22, 2024

Copy link
Copy Markdown
Member

This PR implements delete_dir teaches the Group and Array classes to use it when overwriting nodes in a hierarchy.

Also included here:
- delete_prefix - not used but perhaps we should just use it instead of delete_dir

  • changes list_prefix to return absolute paths instead of stripping prefix. Given that list_prefix was unused before, this wasn't very disruptive but calling it out so others can help decide if this is what we want.

Still needs:

  • new store tests for delete_dir
  • new group tests for edge cases around delete_dir

closes#2191
first steps toward #2108 and #2359

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
jhamman marked this pull request as ready for review October 22, 2024 13:44
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/api/asynchronous.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/storage/local.py Outdated
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py

async def delete_dir(self, prefix: str, recursive: bool = True) -> None:
"""
Remove all keys and prefixes in the store that begin with a given prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make life much easier for Store implementers if we declare that prefix will always be the path of a group or a array. Otherwise, all implementations need to deal with unnecessary edge cases like delete_dir("path/to/array/c/0") or worse.

Icechunk is an example of a store that could very cleanly and efficiently implement the case for a group/array prefix, but it would struggle to handle other prefixes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for recursive = False ? If zarr itself doesn't have one, I'd remove the argument.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see a use case for delete_dir("path/to/array/c/") (remove all chunks from an array) so I think we should allow that -- even if it makes life a little harder in icechunk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's totally fine. Let's document the three possible prefix cases, and say that stores are allowed to do undefined behavior for other prefixes

Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py
if not self.supports_listing:
raise NotImplementedError
self._check_writable()
async for key in self.list_prefix(prefix):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document what's the expected return value of list_prefix? Absolute vs. relative, final "/" or not, etc.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done in the list_prefix docstring

@TomAugspurger

Copy link
Copy Markdown
Contributor

Looks good overall. I've been struggling a bit with whether Stores should be a bit more aware about the context the operation is occurring in (e.g. deleting a group vs a specific chunk of an array; I think this relates to the conversation at #2430 (comment)). But I think this is a good improvement on its own.

@jhammanjhamman mentioned this pull request Oct 28, 2024
6 tasks
@jhamman

Copy link
Copy Markdown
MemberAuthor

@TomAugspurger / @d-v-b / @paraseba - any final comments here. I'd like to get this in ahead of the larger store mode refactor in #2442

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores, since RemoteStore will return keys relative to its path attribute, and similarly LocalStore with its root attribute.

@jhamman

Copy link
Copy Markdown
MemberAuthor

regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores

Would defining these as "relative to the root of the store" be clearer?

Comment threadsrc/zarr/abc/store.py Outdated
@jhamman

Copy link
Copy Markdown
MemberAuthor

pre-commit.ci autofix

@jhammanjhamman closed this Nov 5, 2024
@jhammanjhamman reopened this Nov 5, 2024
@jhamman
jhamman merged commit a31046c into zarr-developers:mainNov 5, 2024
@jhamman
jhamman deleted the feature/store_erase_prefix branch November 5, 2024 04:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3] deleting a group does not delete subgroups

4 participants

@jhamman@TomAugspurger@paraseba@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes - #2430

Merged
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix
Nov 5, 2024
Merged

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes#2430
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix

Conversation

@jhamman

@jhammanjhamman commented Oct 22, 2024

Copy link
Copy Markdown
Member

This PR implements delete_dir teaches the Group and Array classes to use it when overwriting nodes in a hierarchy.

Also included here:
- delete_prefix - not used but perhaps we should just use it instead of delete_dir

  • changes list_prefix to return absolute paths instead of stripping prefix. Given that list_prefix was unused before, this wasn't very disruptive but calling it out so others can help decide if this is what we want.

Still needs:

  • new store tests for delete_dir
  • new group tests for edge cases around delete_dir

closes#2191
first steps toward #2108 and #2359

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
jhamman marked this pull request as ready for review October 22, 2024 13:44
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/api/asynchronous.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/storage/local.py Outdated
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py

async def delete_dir(self, prefix: str, recursive: bool = True) -> None:
"""
Remove all keys and prefixes in the store that begin with a given prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make life much easier for Store implementers if we declare that prefix will always be the path of a group or a array. Otherwise, all implementations need to deal with unnecessary edge cases like delete_dir("path/to/array/c/0") or worse.

Icechunk is an example of a store that could very cleanly and efficiently implement the case for a group/array prefix, but it would struggle to handle other prefixes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for recursive = False ? If zarr itself doesn't have one, I'd remove the argument.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see a use case for delete_dir("path/to/array/c/") (remove all chunks from an array) so I think we should allow that -- even if it makes life a little harder in icechunk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's totally fine. Let's document the three possible prefix cases, and say that stores are allowed to do undefined behavior for other prefixes

Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py
if not self.supports_listing:
raise NotImplementedError
self._check_writable()
async for key in self.list_prefix(prefix):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document what's the expected return value of list_prefix? Absolute vs. relative, final "/" or not, etc.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done in the list_prefix docstring

@TomAugspurger

Copy link
Copy Markdown
Contributor

Looks good overall. I've been struggling a bit with whether Stores should be a bit more aware about the context the operation is occurring in (e.g. deleting a group vs a specific chunk of an array; I think this relates to the conversation at #2430 (comment)). But I think this is a good improvement on its own.

@jhammanjhamman mentioned this pull request Oct 28, 2024
6 tasks
@jhamman

Copy link
Copy Markdown
MemberAuthor

@TomAugspurger / @d-v-b / @paraseba - any final comments here. I'd like to get this in ahead of the larger store mode refactor in #2442

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores, since RemoteStore will return keys relative to its path attribute, and similarly LocalStore with its root attribute.

@jhamman

Copy link
Copy Markdown
MemberAuthor

regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores

Would defining these as "relative to the root of the store" be clearer?

Comment threadsrc/zarr/abc/store.py Outdated
@jhamman

Copy link
Copy Markdown
MemberAuthor

pre-commit.ci autofix

@jhammanjhamman closed this Nov 5, 2024
@jhammanjhamman reopened this Nov 5, 2024
@jhamman
jhamman merged commit a31046c into zarr-developers:mainNov 5, 2024
@jhamman
jhamman deleted the feature/store_erase_prefix branch November 5, 2024 04:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3] deleting a group does not delete subgroups

4 participants

@jhamman@TomAugspurger@paraseba@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes - #2430

Merged
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix
Nov 5, 2024
Merged

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes#2430
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix

Conversation

@jhamman

@jhammanjhamman commented Oct 22, 2024

Copy link
Copy Markdown
Member

This PR implements delete_dir teaches the Group and Array classes to use it when overwriting nodes in a hierarchy.

Also included here:
- delete_prefix - not used but perhaps we should just use it instead of delete_dir

  • changes list_prefix to return absolute paths instead of stripping prefix. Given that list_prefix was unused before, this wasn't very disruptive but calling it out so others can help decide if this is what we want.

Still needs:

  • new store tests for delete_dir
  • new group tests for edge cases around delete_dir

closes#2191
first steps toward #2108 and #2359

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
jhamman marked this pull request as ready for review October 22, 2024 13:44
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/api/asynchronous.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/storage/local.py Outdated
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py

async def delete_dir(self, prefix: str, recursive: bool = True) -> None:
"""
Remove all keys and prefixes in the store that begin with a given prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make life much easier for Store implementers if we declare that prefix will always be the path of a group or a array. Otherwise, all implementations need to deal with unnecessary edge cases like delete_dir("path/to/array/c/0") or worse.

Icechunk is an example of a store that could very cleanly and efficiently implement the case for a group/array prefix, but it would struggle to handle other prefixes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for recursive = False ? If zarr itself doesn't have one, I'd remove the argument.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see a use case for delete_dir("path/to/array/c/") (remove all chunks from an array) so I think we should allow that -- even if it makes life a little harder in icechunk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's totally fine. Let's document the three possible prefix cases, and say that stores are allowed to do undefined behavior for other prefixes

Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py
if not self.supports_listing:
raise NotImplementedError
self._check_writable()
async for key in self.list_prefix(prefix):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document what's the expected return value of list_prefix? Absolute vs. relative, final "/" or not, etc.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done in the list_prefix docstring

@TomAugspurger

Copy link
Copy Markdown
Contributor

Looks good overall. I've been struggling a bit with whether Stores should be a bit more aware about the context the operation is occurring in (e.g. deleting a group vs a specific chunk of an array; I think this relates to the conversation at #2430 (comment)). But I think this is a good improvement on its own.

@jhammanjhamman mentioned this pull request Oct 28, 2024
6 tasks
@jhamman

Copy link
Copy Markdown
MemberAuthor

@TomAugspurger / @d-v-b / @paraseba - any final comments here. I'd like to get this in ahead of the larger store mode refactor in #2442

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores, since RemoteStore will return keys relative to its path attribute, and similarly LocalStore with its root attribute.

@jhamman

Copy link
Copy Markdown
MemberAuthor

regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores

Would defining these as "relative to the root of the store" be clearer?

Comment threadsrc/zarr/abc/store.py Outdated
@jhamman

Copy link
Copy Markdown
MemberAuthor

pre-commit.ci autofix

@jhammanjhamman closed this Nov 5, 2024
@jhammanjhamman reopened this Nov 5, 2024
@jhamman
jhamman merged commit a31046c into zarr-developers:mainNov 5, 2024
@jhamman
jhamman deleted the feature/store_erase_prefix branch November 5, 2024 04:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3] deleting a group does not delete subgroups

4 participants

@jhamman@TomAugspurger@paraseba@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes - #2430

Merged
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix
Nov 5, 2024
Merged

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes#2430
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix

Conversation

@jhamman

@jhammanjhamman commented Oct 22, 2024

Copy link
Copy Markdown
Member

This PR implements delete_dir teaches the Group and Array classes to use it when overwriting nodes in a hierarchy.

Also included here:
- delete_prefix - not used but perhaps we should just use it instead of delete_dir

  • changes list_prefix to return absolute paths instead of stripping prefix. Given that list_prefix was unused before, this wasn't very disruptive but calling it out so others can help decide if this is what we want.

Still needs:

  • new store tests for delete_dir
  • new group tests for edge cases around delete_dir

closes#2191
first steps toward #2108 and #2359

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
jhamman marked this pull request as ready for review October 22, 2024 13:44
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/api/asynchronous.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/storage/local.py Outdated
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py

async def delete_dir(self, prefix: str, recursive: bool = True) -> None:
"""
Remove all keys and prefixes in the store that begin with a given prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make life much easier for Store implementers if we declare that prefix will always be the path of a group or a array. Otherwise, all implementations need to deal with unnecessary edge cases like delete_dir("path/to/array/c/0") or worse.

Icechunk is an example of a store that could very cleanly and efficiently implement the case for a group/array prefix, but it would struggle to handle other prefixes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for recursive = False ? If zarr itself doesn't have one, I'd remove the argument.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see a use case for delete_dir("path/to/array/c/") (remove all chunks from an array) so I think we should allow that -- even if it makes life a little harder in icechunk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's totally fine. Let's document the three possible prefix cases, and say that stores are allowed to do undefined behavior for other prefixes

Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py
if not self.supports_listing:
raise NotImplementedError
self._check_writable()
async for key in self.list_prefix(prefix):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document what's the expected return value of list_prefix? Absolute vs. relative, final "/" or not, etc.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done in the list_prefix docstring

@TomAugspurger

Copy link
Copy Markdown
Contributor

Looks good overall. I've been struggling a bit with whether Stores should be a bit more aware about the context the operation is occurring in (e.g. deleting a group vs a specific chunk of an array; I think this relates to the conversation at #2430 (comment)). But I think this is a good improvement on its own.

@jhammanjhamman mentioned this pull request Oct 28, 2024
6 tasks
@jhamman

Copy link
Copy Markdown
MemberAuthor

@TomAugspurger / @d-v-b / @paraseba - any final comments here. I'd like to get this in ahead of the larger store mode refactor in #2442

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores, since RemoteStore will return keys relative to its path attribute, and similarly LocalStore with its root attribute.

@jhamman

Copy link
Copy Markdown
MemberAuthor

regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores

Would defining these as "relative to the root of the store" be clearer?

Comment threadsrc/zarr/abc/store.py Outdated
@jhamman

Copy link
Copy Markdown
MemberAuthor

pre-commit.ci autofix

@jhammanjhamman closed this Nov 5, 2024
@jhammanjhamman reopened this Nov 5, 2024
@jhamman
jhamman merged commit a31046c into zarr-developers:mainNov 5, 2024
@jhamman
jhamman deleted the feature/store_erase_prefix branch November 5, 2024 04:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3] deleting a group does not delete subgroups

4 participants

@jhamman@TomAugspurger@paraseba@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes - #2430

Merged
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix
Nov 5, 2024
Merged

Feature: store learns to delete prefixes when overwriting/creating hierarchy nodes#2430
jhamman merged 19 commits into
zarr-developers:mainfrom
jhamman:feature/store_erase_prefix

Conversation

@jhamman

@jhammanjhamman commented Oct 22, 2024

Copy link
Copy Markdown
Member

This PR implements delete_dir teaches the Group and Array classes to use it when overwriting nodes in a hierarchy.

Also included here:
- delete_prefix - not used but perhaps we should just use it instead of delete_dir

  • changes list_prefix to return absolute paths instead of stripping prefix. Given that list_prefix was unused before, this wasn't very disruptive but calling it out so others can help decide if this is what we want.

Still needs:

  • new store tests for delete_dir
  • new group tests for edge cases around delete_dir

closes#2191
first steps toward #2108 and #2359

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
jhamman marked this pull request as ready for review October 22, 2024 13:44
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/api/asynchronous.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/storage/local.py Outdated
Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py

async def delete_dir(self, prefix: str, recursive: bool = True) -> None:
"""
Remove all keys and prefixes in the store that begin with a given prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make life much easier for Store implementers if we declare that prefix will always be the path of a group or a array. Otherwise, all implementations need to deal with unnecessary edge cases like delete_dir("path/to/array/c/0") or worse.

Icechunk is an example of a store that could very cleanly and efficiently implement the case for a group/array prefix, but it would struggle to handle other prefixes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for recursive = False ? If zarr itself doesn't have one, I'd remove the argument.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see a use case for delete_dir("path/to/array/c/") (remove all chunks from an array) so I think we should allow that -- even if it makes life a little harder in icechunk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's totally fine. Let's document the three possible prefix cases, and say that stores are allowed to do undefined behavior for other prefixes

Comment threadsrc/zarr/abc/store.py Outdated
Comment threadsrc/zarr/abc/store.py
if not self.supports_listing:
raise NotImplementedError
self._check_writable()
async for key in self.list_prefix(prefix):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document what's the expected return value of list_prefix? Absolute vs. relative, final "/" or not, etc.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is done in the list_prefix docstring

@TomAugspurger

Copy link
Copy Markdown
Contributor

Looks good overall. I've been struggling a bit with whether Stores should be a bit more aware about the context the operation is occurring in (e.g. deleting a group vs a specific chunk of an array; I think this relates to the conversation at #2430 (comment)). But I think this is a good improvement on its own.

@jhammanjhamman mentioned this pull request Oct 28, 2024
6 tasks
@jhamman

Copy link
Copy Markdown
MemberAuthor

@TomAugspurger / @d-v-b / @paraseba - any final comments here. I'd like to get this in ahead of the larger store mode refactor in #2442

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores, since RemoteStore will return keys relative to its path attribute, and similarly LocalStore with its root attribute.

@jhamman

Copy link
Copy Markdown
MemberAuthor

regarding the change to list_prefix, I will note that the definition of "absolute paths" is a bit shaky across different stores

Would defining these as "relative to the root of the store" be clearer?

Comment threadsrc/zarr/abc/store.py Outdated
@jhamman

Copy link
Copy Markdown
MemberAuthor

pre-commit.ci autofix

@jhammanjhamman closed this Nov 5, 2024
@jhammanjhamman reopened this Nov 5, 2024
@jhamman
jhamman merged commit a31046c into zarr-developers:mainNov 5, 2024
@jhamman
jhamman deleted the feature/store_erase_prefix branch November 5, 2024 04:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v3] deleting a group does not delete subgroups

4 participants

@jhamman@TomAugspurger@paraseba@d-v-b