Skip to content

Use implicit fill values for zarr v2 - #2274

Merged
TomAugspurger merged 6 commits into
zarr-developers:v3from
TomAugspurger:user/tom/fix/v2-no-fill-value
Oct 10, 2024
Merged

Use implicit fill values for zarr v2#2274
TomAugspurger merged 6 commits into
zarr-developers:v3from
TomAugspurger:user/tom/fix/v2-no-fill-value

Conversation

@TomAugspurger

@TomAugspurgerTomAugspurger commented Sep 30, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Closes#2271

I'm not sure if this is a good idea, but this, consolidated metadata, and #2270 lets xarray round trip a dataset. The basic idea is to let ArrayV2Metadata represent the v2 metadata a bit more faithfully. Currently, we "eagerly" interpret the lack of fill_value in a V2 document as the default fill value for some dtype (e.g. 0 for int64). This PR changes that, to allow ArrayV2Metadata.fill_value to be None if it's unspecified in the metadata. But we do need some fill value when reading data. So that's where we finally substitute in the default.

This doesn't really affect Zarr directly, but it is useful for downstream libraries that want to inspect the array metadata before reading and make some decision based on whether or not fill_value is specified.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@TomAugspurgerTomAugspurger mentioned this pull request Sep 30, 2024
Closed
jhamman
jhamman reviewed Oct 7, 2024
View reviewed changes

@jhammanjhamman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Seems reasonable though.

Comment threadsrc/zarr/codecs/pipeline.py Outdated
Comment threadsrc/zarr/core/metadata/v2.py Outdated
@TomAugspurger
TomAugspurger marked this pull request as ready for review October 7, 2024 15:26
rabernat
rabernat reviewed Oct 8, 2024
View reviewed changes

@rabernatrabernat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. My one suggestion would be to consider a special case for object / string dtypes.

Comment threadsrc/zarr/core/metadata/v2.py Outdated
Comment threadsrc/zarr/core/metadata/v2.py Outdated
Comment threadtests/v3/test_v2.py Outdated
rabernat
rabernat approved these changes Oct 8, 2024
View reviewed changes

@rabernatrabernat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, nice work Tom!

@TomAugspurgerTomAugspurger added the downstream Downstream libraries using zarr label Oct 9, 2024
jhamman
jhamman reviewed Oct 9, 2024
View reviewed changes
Comment threadsrc/zarr/codecs/pipeline.py
out[out_selection] = chunk_spec.fill_value
fill_value = chunk_spec.fill_value

if fill_value is None:

@jhammanjhammanOct 9, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we may consider in the future is to parameterize the CodecPipeline class with the zarr_format of the calling Array. This would allow us feel more confident that workarounds like this one only apply to v2 data.

jhamman
jhamman approved these changes Oct 9, 2024
View reviewed changes

@jhammanjhamman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @TomAugspurger!

@jhammanjhamman added this to the 3.0.0.beta milestone Oct 9, 2024
@jhammanjhamman added the V3 label Oct 9, 2024
@TomAugspurger
TomAugspurger merged commit 81a87d6 into zarr-developers:v3Oct 10, 2024
@TomAugspurger
TomAugspurger deleted the user/tom/fix/v2-no-fill-value branch October 10, 2024 01:23
@LDeakinLDeakin mentioned this pull request Feb 5, 2025
Merged
6 tasks
@moradologymoradology mentioned this pull request Feb 6, 2025
Open
@LDeakinLDeakin mentioned this pull request Feb 28, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rabernatrabernatrabernat approved these changes
@jhammanjhammanjhamman approved these changes

Assignees

No one assigned

Labels

downstreamDownstream libraries using zarr

Projects

Zarr-Python - 3.0
Status: Done

Milestone

3.0.0.beta

Development

Successfully merging this pull request may close these issues.

Don't infer a default fill_value for zarr_format=2?

3 participants

@TomAugspurger@rabernat@jhamman
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Use implicit fill values for zarr v2 by TomAugspurger · Pull Request #2274 · zarr-developers/zarr-python · GitHub
Skip to content
Use implicit fill values for zarr v2 - #2274
Merged
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Oct 10, 2024
Merged
Use implicit fill values for zarr v2#2274
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Conversation
@TomAugspurger
@TomAugspurgerTomAugspurger
commented
Sep 30, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Closes#2271
I'm not sure if this is a good idea, but this, consolidated metadata, and #2270 lets xarray round trip a dataset. The basic idea is to let ArrayV2Metadata represent the v2 metadata a bit more faithfully. Currently, we "eagerly" interpret the lack of fill_value in a V2 document as the default fill value for some dtype (e.g. 0 for int64). This PR changes that, to allow ArrayV2Metadata.fill_value to be None if it's unspecified in the metadata. But we do need some fill value when reading data. So that's where we finally substitute in the default.
This doesn't really affect Zarr directly, but it is useful for downstream libraries that want to inspect the array metadata before reading and make some decision based on whether or not fill_value is specified.
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 30, 2024

Closed

jhamman
jhamman
reviewed
Oct 7, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Seems reasonable though.
Comment threadsrc/zarr/codecs/pipeline.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

@TomAugspurger
TomAugspurger
marked this pull request as ready for review
October 7, 2024 15:26
rabernat
rabernat
reviewed
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. My one suggestion would be to consider a special case for object / string dtypes.
Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadtests/v3/test_v2.py
Outdated

rabernat
rabernat
approved these changes
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, nice work Tom!
@TomAugspurgerTomAugspurger
added
the 
downstream
Downstream libraries using zarr
label
Oct 9, 2024
jhamman
jhamman
reviewed
Oct 9, 2024
View reviewed changes
Comment threadsrc/zarr/codecs/pipeline.py
 out[out_selection] = chunk_spec.fill_value
 fill_value = chunk_spec.fill_value

 if fill_value is None:
@jhammanjhammanOct 9, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we may consider in the future is to parameterize the CodecPipeline class with the zarr_format of the calling Array. This would allow us feel more confident that workarounds like this one only apply to v2 data.
jhamman
jhamman
approved these changes
Oct 9, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @TomAugspurger!
@jhammanjhamman
added this to the 3.0.0.beta milestone
Oct 9, 2024
@jhammanjhamman
added
the 
V3

label
Oct 9, 2024
@TomAugspurger
TomAugspurger
merged commit 81a87d6
into
zarr-developers:v3Oct 10, 2024
@TomAugspurger
TomAugspurger
deleted the

user/tom/fix/v2-no-fill-value

branch
October 10, 2024 01:23
@LDeakinLDeakin
mentioned this pull request
Feb 5, 2025
 Merged

6 tasks
@moradologymoradology
mentioned this pull request
Feb 6, 2025

Open

@LDeakinLDeakin
mentioned this pull request
Feb 28, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@rabernatrabernatrabernat approved these changes
@jhammanjhammanjhamman approved these changes

Assignees


No one assigned


Labels

downstreamDownstream libraries using zarr

Projects

Zarr-Python - 3.0

Status: Done


Milestone

3.0.0.beta

Development

Successfully merging this pull request may close these issues.

Don't infer a default fill_value for zarr_format=2?


3 participants

@TomAugspurger@rabernat@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Use implicit fill values for zarr v2 by TomAugspurger · Pull Request #2274 · zarr-developers/zarr-python · GitHub
Skip to content
Use implicit fill values for zarr v2 - #2274
Merged
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Oct 10, 2024
Merged
Use implicit fill values for zarr v2#2274
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Conversation
@TomAugspurger
@TomAugspurgerTomAugspurger
commented
Sep 30, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Closes#2271
I'm not sure if this is a good idea, but this, consolidated metadata, and #2270 lets xarray round trip a dataset. The basic idea is to let ArrayV2Metadata represent the v2 metadata a bit more faithfully. Currently, we "eagerly" interpret the lack of fill_value in a V2 document as the default fill value for some dtype (e.g. 0 for int64). This PR changes that, to allow ArrayV2Metadata.fill_value to be None if it's unspecified in the metadata. But we do need some fill value when reading data. So that's where we finally substitute in the default.
This doesn't really affect Zarr directly, but it is useful for downstream libraries that want to inspect the array metadata before reading and make some decision based on whether or not fill_value is specified.
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 30, 2024

Closed

jhamman
jhamman
reviewed
Oct 7, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Seems reasonable though.
Comment threadsrc/zarr/codecs/pipeline.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

@TomAugspurger
TomAugspurger
marked this pull request as ready for review
October 7, 2024 15:26
rabernat
rabernat
reviewed
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. My one suggestion would be to consider a special case for object / string dtypes.
Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadtests/v3/test_v2.py
Outdated

rabernat
rabernat
approved these changes
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, nice work Tom!
@TomAugspurgerTomAugspurger
added
the 
downstream
Downstream libraries using zarr
label
Oct 9, 2024
jhamman
jhamman
reviewed
Oct 9, 2024
View reviewed changes
Comment threadsrc/zarr/codecs/pipeline.py
 out[out_selection] = chunk_spec.fill_value
 fill_value = chunk_spec.fill_value

 if fill_value is None:
@jhammanjhammanOct 9, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we may consider in the future is to parameterize the CodecPipeline class with the zarr_format of the calling Array. This would allow us feel more confident that workarounds like this one only apply to v2 data.
jhamman
jhamman
approved these changes
Oct 9, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @TomAugspurger!
@jhammanjhamman
added this to the 3.0.0.beta milestone
Oct 9, 2024
@jhammanjhamman
added
the 
V3

label
Oct 9, 2024
@TomAugspurger
TomAugspurger
merged commit 81a87d6
into
zarr-developers:v3Oct 10, 2024
@TomAugspurger
TomAugspurger
deleted the

user/tom/fix/v2-no-fill-value

branch
October 10, 2024 01:23
@LDeakinLDeakin
mentioned this pull request
Feb 5, 2025
 Merged

6 tasks
@moradologymoradology
mentioned this pull request
Feb 6, 2025

Open

@LDeakinLDeakin
mentioned this pull request
Feb 28, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@rabernatrabernatrabernat approved these changes
@jhammanjhammanjhamman approved these changes

Assignees


No one assigned


Labels

downstreamDownstream libraries using zarr

Projects

Zarr-Python - 3.0

Status: Done


Milestone

3.0.0.beta

Development

Successfully merging this pull request may close these issues.

Don't infer a default fill_value for zarr_format=2?


3 participants

@TomAugspurger@rabernat@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Use implicit fill values for zarr v2 by TomAugspurger · Pull Request #2274 · zarr-developers/zarr-python · GitHub
Skip to content
Use implicit fill values for zarr v2 - #2274
Merged
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Oct 10, 2024
Merged
Use implicit fill values for zarr v2#2274
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Conversation
@TomAugspurger
@TomAugspurgerTomAugspurger
commented
Sep 30, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Closes#2271
I'm not sure if this is a good idea, but this, consolidated metadata, and #2270 lets xarray round trip a dataset. The basic idea is to let ArrayV2Metadata represent the v2 metadata a bit more faithfully. Currently, we "eagerly" interpret the lack of fill_value in a V2 document as the default fill value for some dtype (e.g. 0 for int64). This PR changes that, to allow ArrayV2Metadata.fill_value to be None if it's unspecified in the metadata. But we do need some fill value when reading data. So that's where we finally substitute in the default.
This doesn't really affect Zarr directly, but it is useful for downstream libraries that want to inspect the array metadata before reading and make some decision based on whether or not fill_value is specified.
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 30, 2024

Closed

jhamman
jhamman
reviewed
Oct 7, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Seems reasonable though.
Comment threadsrc/zarr/codecs/pipeline.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

@TomAugspurger
TomAugspurger
marked this pull request as ready for review
October 7, 2024 15:26
rabernat
rabernat
reviewed
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. My one suggestion would be to consider a special case for object / string dtypes.
Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadtests/v3/test_v2.py
Outdated

rabernat
rabernat
approved these changes
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, nice work Tom!
@TomAugspurgerTomAugspurger
added
the 
downstream
Downstream libraries using zarr
label
Oct 9, 2024
jhamman
jhamman
reviewed
Oct 9, 2024
View reviewed changes
Comment threadsrc/zarr/codecs/pipeline.py
 out[out_selection] = chunk_spec.fill_value
 fill_value = chunk_spec.fill_value

 if fill_value is None:
@jhammanjhammanOct 9, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we may consider in the future is to parameterize the CodecPipeline class with the zarr_format of the calling Array. This would allow us feel more confident that workarounds like this one only apply to v2 data.
jhamman
jhamman
approved these changes
Oct 9, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @TomAugspurger!
@jhammanjhamman
added this to the 3.0.0.beta milestone
Oct 9, 2024
@jhammanjhamman
added
the 
V3

label
Oct 9, 2024
@TomAugspurger
TomAugspurger
merged commit 81a87d6
into
zarr-developers:v3Oct 10, 2024
@TomAugspurger
TomAugspurger
deleted the

user/tom/fix/v2-no-fill-value

branch
October 10, 2024 01:23
@LDeakinLDeakin
mentioned this pull request
Feb 5, 2025
 Merged

6 tasks
@moradologymoradology
mentioned this pull request
Feb 6, 2025

Open

@LDeakinLDeakin
mentioned this pull request
Feb 28, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@rabernatrabernatrabernat approved these changes
@jhammanjhammanjhamman approved these changes

Assignees


No one assigned


Labels

downstreamDownstream libraries using zarr

Projects

Zarr-Python - 3.0

Status: Done


Milestone

3.0.0.beta

Development

Successfully merging this pull request may close these issues.

Don't infer a default fill_value for zarr_format=2?


3 participants

@TomAugspurger@rabernat@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Use implicit fill values for zarr v2 by TomAugspurger · Pull Request #2274 · zarr-developers/zarr-python · GitHub
Skip to content
Use implicit fill values for zarr v2 - #2274
Merged
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Oct 10, 2024
Merged
Use implicit fill values for zarr v2#2274
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Conversation
@TomAugspurger
@TomAugspurgerTomAugspurger
commented
Sep 30, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Closes#2271
I'm not sure if this is a good idea, but this, consolidated metadata, and #2270 lets xarray round trip a dataset. The basic idea is to let ArrayV2Metadata represent the v2 metadata a bit more faithfully. Currently, we "eagerly" interpret the lack of fill_value in a V2 document as the default fill value for some dtype (e.g. 0 for int64). This PR changes that, to allow ArrayV2Metadata.fill_value to be None if it's unspecified in the metadata. But we do need some fill value when reading data. So that's where we finally substitute in the default.
This doesn't really affect Zarr directly, but it is useful for downstream libraries that want to inspect the array metadata before reading and make some decision based on whether or not fill_value is specified.
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 30, 2024

Closed

jhamman
jhamman
reviewed
Oct 7, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Seems reasonable though.
Comment threadsrc/zarr/codecs/pipeline.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

@TomAugspurger
TomAugspurger
marked this pull request as ready for review
October 7, 2024 15:26
rabernat
rabernat
reviewed
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. My one suggestion would be to consider a special case for object / string dtypes.
Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadtests/v3/test_v2.py
Outdated

rabernat
rabernat
approved these changes
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, nice work Tom!
@TomAugspurgerTomAugspurger
added
the 
downstream
Downstream libraries using zarr
label
Oct 9, 2024
jhamman
jhamman
reviewed
Oct 9, 2024
View reviewed changes
Comment threadsrc/zarr/codecs/pipeline.py
 out[out_selection] = chunk_spec.fill_value
 fill_value = chunk_spec.fill_value

 if fill_value is None:
@jhammanjhammanOct 9, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we may consider in the future is to parameterize the CodecPipeline class with the zarr_format of the calling Array. This would allow us feel more confident that workarounds like this one only apply to v2 data.
jhamman
jhamman
approved these changes
Oct 9, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @TomAugspurger!
@jhammanjhamman
added this to the 3.0.0.beta milestone
Oct 9, 2024
@jhammanjhamman
added
the 
V3

label
Oct 9, 2024
@TomAugspurger
TomAugspurger
merged commit 81a87d6
into
zarr-developers:v3Oct 10, 2024
@TomAugspurger
TomAugspurger
deleted the

user/tom/fix/v2-no-fill-value

branch
October 10, 2024 01:23
@LDeakinLDeakin
mentioned this pull request
Feb 5, 2025
 Merged

6 tasks
@moradologymoradology
mentioned this pull request
Feb 6, 2025

Open

@LDeakinLDeakin
mentioned this pull request
Feb 28, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@rabernatrabernatrabernat approved these changes
@jhammanjhammanjhamman approved these changes

Assignees


No one assigned


Labels

downstreamDownstream libraries using zarr

Projects

Zarr-Python - 3.0

Status: Done


Milestone

3.0.0.beta

Development

Successfully merging this pull request may close these issues.

Don't infer a default fill_value for zarr_format=2?


3 participants

@TomAugspurger@rabernat@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Use implicit fill values for zarr v2 by TomAugspurger · Pull Request #2274 · zarr-developers/zarr-python · GitHub
Skip to content
Use implicit fill values for zarr v2 - #2274
Merged
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Oct 10, 2024
Merged
Use implicit fill values for zarr v2#2274
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Conversation
@TomAugspurger
@TomAugspurgerTomAugspurger
commented
Sep 30, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Closes#2271
I'm not sure if this is a good idea, but this, consolidated metadata, and #2270 lets xarray round trip a dataset. The basic idea is to let ArrayV2Metadata represent the v2 metadata a bit more faithfully. Currently, we "eagerly" interpret the lack of fill_value in a V2 document as the default fill value for some dtype (e.g. 0 for int64). This PR changes that, to allow ArrayV2Metadata.fill_value to be None if it's unspecified in the metadata. But we do need some fill value when reading data. So that's where we finally substitute in the default.
This doesn't really affect Zarr directly, but it is useful for downstream libraries that want to inspect the array metadata before reading and make some decision based on whether or not fill_value is specified.
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 30, 2024

Closed

jhamman
jhamman
reviewed
Oct 7, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Seems reasonable though.
Comment threadsrc/zarr/codecs/pipeline.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

@TomAugspurger
TomAugspurger
marked this pull request as ready for review
October 7, 2024 15:26
rabernat
rabernat
reviewed
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. My one suggestion would be to consider a special case for object / string dtypes.
Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadtests/v3/test_v2.py
Outdated

rabernat
rabernat
approved these changes
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, nice work Tom!
@TomAugspurgerTomAugspurger
added
the 
downstream
Downstream libraries using zarr
label
Oct 9, 2024
jhamman
jhamman
reviewed
Oct 9, 2024
View reviewed changes
Comment threadsrc/zarr/codecs/pipeline.py
 out[out_selection] = chunk_spec.fill_value
 fill_value = chunk_spec.fill_value

 if fill_value is None:
@jhammanjhammanOct 9, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we may consider in the future is to parameterize the CodecPipeline class with the zarr_format of the calling Array. This would allow us feel more confident that workarounds like this one only apply to v2 data.
jhamman
jhamman
approved these changes
Oct 9, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @TomAugspurger!
@jhammanjhamman
added this to the 3.0.0.beta milestone
Oct 9, 2024
@jhammanjhamman
added
the 
V3

label
Oct 9, 2024
@TomAugspurger
TomAugspurger
merged commit 81a87d6
into
zarr-developers:v3Oct 10, 2024
@TomAugspurger
TomAugspurger
deleted the

user/tom/fix/v2-no-fill-value

branch
October 10, 2024 01:23
@LDeakinLDeakin
mentioned this pull request
Feb 5, 2025
 Merged

6 tasks
@moradologymoradology
mentioned this pull request
Feb 6, 2025

Open

@LDeakinLDeakin
mentioned this pull request
Feb 28, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@rabernatrabernatrabernat approved these changes
@jhammanjhammanjhamman approved these changes

Assignees


No one assigned


Labels

downstreamDownstream libraries using zarr

Projects

Zarr-Python - 3.0

Status: Done


Milestone

3.0.0.beta

Development

Successfully merging this pull request may close these issues.

Don't infer a default fill_value for zarr_format=2?


3 participants

@TomAugspurger@rabernat@jhamman

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
})();
Use implicit fill values for zarr v2 by TomAugspurger · Pull Request #2274 · zarr-developers/zarr-python · GitHub
Skip to content
Use implicit fill values for zarr v2 - #2274
Merged
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Oct 10, 2024
Merged
Use implicit fill values for zarr v2#2274
TomAugspurger merged 6 commits into
zarr-developers:v3from 
TomAugspurger:user/tom/fix/v2-no-fill-value
Conversation
@TomAugspurger
@TomAugspurgerTomAugspurger
commented
Sep 30, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Closes#2271
I'm not sure if this is a good idea, but this, consolidated metadata, and #2270 lets xarray round trip a dataset. The basic idea is to let ArrayV2Metadata represent the v2 metadata a bit more faithfully. Currently, we "eagerly" interpret the lack of fill_value in a V2 document as the default fill value for some dtype (e.g. 0 for int64). This PR changes that, to allow ArrayV2Metadata.fill_value to be None if it's unspecified in the metadata. But we do need some fill value when reading data. So that's where we finally substitute in the default.
This doesn't really affect Zarr directly, but it is useful for downstream libraries that want to inspect the array metadata before reading and make some decision based on whether or not fill_value is specified.
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 30, 2024

Closed

jhamman
jhamman
reviewed
Oct 7, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Seems reasonable though.
Comment threadsrc/zarr/codecs/pipeline.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

@TomAugspurger
TomAugspurger
marked this pull request as ready for review
October 7, 2024 15:26
rabernat
rabernat
reviewed
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. My one suggestion would be to consider a special case for object / string dtypes.
Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadsrc/zarr/core/metadata/v2.py
Outdated

Comment threadtests/v3/test_v2.py
Outdated

rabernat
rabernat
approved these changes
Oct 8, 2024
View reviewed changes
@rabernatrabernat
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, nice work Tom!
@TomAugspurgerTomAugspurger
added
the 
downstream
Downstream libraries using zarr
label
Oct 9, 2024
jhamman
jhamman
reviewed
Oct 9, 2024
View reviewed changes
Comment threadsrc/zarr/codecs/pipeline.py
 out[out_selection] = chunk_spec.fill_value
 fill_value = chunk_spec.fill_value

 if fill_value is None:
@jhammanjhammanOct 9, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we may consider in the future is to parameterize the CodecPipeline class with the zarr_format of the calling Array. This would allow us feel more confident that workarounds like this one only apply to v2 data.
jhamman
jhamman
approved these changes
Oct 9, 2024
View reviewed changes
@jhammanjhamman
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @TomAugspurger!
@jhammanjhamman
added this to the 3.0.0.beta milestone
Oct 9, 2024
@jhammanjhamman
added
the 
V3

label
Oct 9, 2024
@TomAugspurger
TomAugspurger
merged commit 81a87d6
into
zarr-developers:v3Oct 10, 2024
@TomAugspurger
TomAugspurger
deleted the

user/tom/fix/v2-no-fill-value

branch
October 10, 2024 01:23
@LDeakinLDeakin
mentioned this pull request
Feb 5, 2025
 Merged

6 tasks
@moradologymoradology
mentioned this pull request
Feb 6, 2025

Open

@LDeakinLDeakin
mentioned this pull request
Feb 28, 2025
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@rabernatrabernatrabernat approved these changes
@jhammanjhammanjhamman approved these changes

Assignees


No one assigned


Labels

downstreamDownstream libraries using zarr

Projects

Zarr-Python - 3.0

Status: Done


Milestone

3.0.0.beta

Development

Successfully merging this pull request may close these issues.

Don't infer a default fill_value for zarr_format=2?


3 participants

@TomAugspurger@rabernat@jhamman