rolling keep_attrs & default True by mathause · Pull Request #4510 · pydata/xarray

mathause · 2020-10-14T23:12:16Z

Closesds.rolling() drops attributes and name #4497
Tests added
Passes isort . && black . && mypy . && flake8
User visible changes (including notable bug fixes) are documented in whats-new.rst

Only global attributes were retained for rolling operations. DataArrays would loose their attrs.

As per #3891 (comment) I also changed the default to True.

Note #4497 also mentions propagation of the name. This is not yet implemented.

mathause · 2020-10-14T23:15:22Z

+ If True (default), the object's attributes (`attrs`) will be copied
+from the original object to the new one. If False, the new


Suggested change
IfTrue (default), theobject'sattributes (`attrs`) willbecopied
fromtheoriginalobjecttothenewone.IfFalse, thenew
IfNone (default)orTrue, theobject'sattributes (`attrs`) willbe
copiedfromtheoriginalobjecttothenewone. IfFalse, thenew

I'd replace the (default) and the , optional in the type spec with , default: None

max-sixty · 2020-10-15T01:31:16Z

Great, thanks!

Are the global configs still being honored in rolling? No strong view on whether they should be, though it affects whether we have defaults of None.

mathause · 2020-10-15T12:15:29Z

As mentioned in #4513 I should probably switch from ds.rolling(..., keep_attrs=True).mean() to ds.rolling(...).mean( keep_attrs=True). Let's see if there is feedback/ a consensus on #4513

mathause

This is ready for a review. It was a bit more complicated than anticipated - I had to thread keep_attrs through a lot of functions. So may not be trivial to check.

To summarize:

Per #4513 this switches from ds.rolling(..., keep_attrs=False).mean() to ds.rolling(...).mean(keep_attrs=False) & prints a deprecation warning.
Per # #3891 this now keeps attrs per default (it respects the global keep_attrs setting)
Fixes #4497

mathause · 2020-10-26T09:53:02Z

+ f"Reductions are applied along the rolling dimension(s) "
+ f"'{self.dim}'. Passing the 'dim' kwarg to reduction "
+ f"operations has no effect.",


The warning used to be:
"and will raise an error in xarray 0.16.0"
I decided to continue warning and not raise an error - objections?

max-sixty · 2020-10-26T17:22:00Z

Wow, this was quite the herculean effort @mathause . Thanks. That solves any questions I had about whether the global configs are being honored...

To the extent we have to do more of these, we could consider whether we need to add keep_attrs kwargs, or we should recommend users removing attrs separately where needed. We'd still need to through the global defaults, though.

max-sixty · 2020-10-30T00:40:56Z

Any other thoughts before we merge?

mathause · 2020-10-30T13:01:38Z

I added a test to ensure da.name is conserved & the tests passed. So this is finished from my side. I guess this is not so easy to review as I change one or two lines per function in a lot of functions. So maybe someone could just review the tests? Or maybe you already did?

While working on this I thought rolling could profit from some refactoring but I kept it out of this PR. I might follow up with do this if I find time.

max-sixty

I think this looks great. I read through the tests carefully and browed the code. I would merge unless others have thoughts

mathause

Testing a global default and kwarg revealed some more issues. Should be fixed now.

I realize this does still not preserve the encoding. Not sure what our general take is on that.

mathause · 2020-10-30T15:50:40Z

 else:
- dataset[key] = da
- return Dataset(dataset, coords=self.obj.coords).isel(
+ dataset[key] = da.copy()


I now do a deep copy here. I think this is safer. Else non-rolled DataArrays were connected to the original obj.

max-sixty · 2020-11-09T19:10:23Z

Thanks again @mathause !

BREAKING CHANGE: Change keep_attrs default from False to True This changes the default behavior of xarray operations to preserve attributes by default, which better aligns with user expectations and scientific workflows where metadata preservation is critical. Migration guide: - To restore previous behavior globally: xr.set_options(keep_attrs=False) - To restore for specific operations: use keep_attrs=False parameter - Alternative: use .drop_attrs() method after operations Closespydata#3891, pydata#4510, pydata#9920

* feat: Preserve attributes by default in all operations BREAKING CHANGE: Change keep_attrs default from False to True This changes the default behavior of xarray operations to preserve attributes by default, which better aligns with user expectations and scientific workflows where metadata preservation is critical. Migration guide: - To restore previous behavior globally: xr.set_options(keep_attrs=False) - To restore for specific operations: use keep_attrs=False parameter - Alternative: use .drop_attrs() method after operations Closes#3891, #4510, #9920 * Fix Dataset.map to properly handle coordinate attrs when keep_attrs=False The merge incorrectly preserved coordinate attributes even when keep_attrs=False. Now coordinates have their attrs cleared when keep_attrs=False, consistent with data variables. * Optimize Dataset.map coordinate attribute handling - When keep_attrs=True: restore attrs from original coords (func may have dropped them) - When keep_attrs=False: clear all attrs - More efficient than previous implementation * Simplify Dataset.map attribute handling code Group attribute operations by keep_attrs value for cleaner, more readable code with identical functionality. * Remove temporal 'now' references from comments Per Stefan's review, remove 'now' from comments that describe behavior changes, as these become stale over time. Replace with timeless descriptions that simply state the current behavior. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address remaining review comments from Stefan - Remove PR-specific comment from variable.py that only makes sense in context - Remove redundant comment from test_computation.py - Clarify comment in test_dataarray.py about argmin preserving attrs * Use drop_conflicts for binary operations attribute handling Instead of only preserving the left operand's attributes, binary operations now combine attributes from both operands using the drop_conflicts strategy: - Matching attributes (same key, same value) are kept - Conflicting attributes (same key, different values) are dropped - Non-conflicting attributes from both operands are preserved This provides more intuitive behavior when combining data with partially overlapping metadata. * Fix binary ops attrs: only merge when both operands have attrs For backward compatibility, when one operand has no attributes (None), keep the left operand's attributes instead of merging. This maintains the existing test expectations while still providing drop_conflicts behavior when both operands have attributes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix binary ops attrs handling when operands have no attrs When one operand has no attributes (either None or empty dict), the result should have no attributes. This maintains backward compatibility and fixes test_1d_math failures. The issue was compounded by a side effect in the attrs property getter that mutates _attrs from None to {} when accessed. * Simplify binary ops attrs handling Use attrs property directly instead of checking _attrs, since the property normalizes None to {}. This simplifies the logic while maintaining the same behavior. * Clarify comments about attrs handling differences Variable uses None for no attrs, Dataset uses {} for no attrs. Updated comments to make this distinction clear. * Implement true drop_conflicts behavior for binary operations Previously we were dropping all attrs if either operand had no attrs. Now we properly merge attrs and only drop conflicting ones, which is what drop_conflicts should do: - If one has {"a": 1} and other has {}, result is {"a": 1} - If one has {"a": 1} and other has {"a": 2}, result is {} - If one has {"a": 1} and other has {"b": 2}, result is {"a": 1, "b": 2} Updated tests to reflect this correct behavior. * Remove unnecessary conversion of {} to None Variable constructor already normalizes empty dict to None internally, so the explicit conversion is redundant. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

rolling keep_attrs & default True
b929492

mathause mentioned this pull request Oct 14, 2020
ds.rolling() drops attributes and name #4497
Closed

mathause commented Oct 14, 2020
View reviewed changes

mathause commented Oct 15, 2020
View reviewed changes

Comment threadxarray/core/rolling.py Outdated

mathause mentioned this pull request Oct 15, 2020
where should keep_attrs be set in groupby, resample, weighted etc.? #4513
Closed

keewis reviewed Oct 15, 2020
View reviewed changes

Comment threaddoc/whats-new.rst Outdated

mathause added 12 commits October 19, 2020 23:01

WIP
90e202f

Merge branch 'master' into rolling_keep_attrs
8477384

remove docstr on keep_attrs
e478843

adapt tests
df22347

rolling WIP
cb30ce1

small update
8fcb2d9

update docs
dbc56d5

update tests
dddc4ba

undo refactoring
52698b0

Merge branch 'master' into rolling_keep_attrs
82bc596

some more fixes
896513f

more doc fixes
e9de7d5

mathause commented Oct 26, 2020
View reviewed changes

max-sixty reviewed Oct 26, 2020
View reviewed changes

Comment threadxarray/core/common.py

keewis mentioned this pull request Oct 26, 2020
don't remove the trailing newline for blocks keewis/blackdoc#52
Closed

mathause added 2 commits October 30, 2020 11:41

Merge branch 'master' into rolling_keep_attrs
b56b9ee

test the name is conserved
3596208

max-sixty approved these changes Oct 30, 2020
View reviewed changes

Comment threadxarray/tests/test_dataarray.py

test global default and kwarg
8ce9044

mathause added 2 commits October 30, 2020 16:29

more fixes...
54371df

do a deep copy
1ee45f5

mathause commented Oct 30, 2020
View reviewed changes

mathause added the topic-metadata Relating to the handling of metadata (i.e. attrs and encoding) label Oct 30, 2020

mathause commented Nov 9, 2020
View reviewed changes

Comment threadxarray/core/common.py
Comment threadxarray/core/common.py
Comment threadxarray/core/common.py

mathauseand others added 2 commits November 9, 2020 15:02

Apply suggestions from code review
502ca77

Merge branch 'master' into rolling_keep_attrs
a44af88

mathause merged commit 1735892 into pydata:masterNov 9, 2020

mathause deleted the rolling_keep_attrs branch November 9, 2020 15:35

keewis mentioned this pull request Feb 17, 2024
Use ruff for formatting #8761
Draft
1 task

		If True (default), the object's attributes (`attrs`) will be copied
		from the original object to the new one. If False, the new

-IfTrue (default), theobject'sattributes (`attrs`) willbecopied
-fromtheoriginalobjecttothenewone.IfFalse, thenew
+IfNone (default)orTrue, theobject'sattributes (`attrs`) willbe
+copiedfromtheoriginalobjecttothenewone. IfFalse, thenew

Uh oh!

Uh oh!

Conversation

mathause commented Oct 14, 2020

Uh oh!

mathauseOct 14, 2020

Choose a reason for hiding this comment

Uh oh!

keewisOct 15, 2020

Choose a reason for hiding this comment

Uh oh!

max-sixty commented Oct 15, 2020

Uh oh!

Uh oh!

mathause commented Oct 15, 2020

Uh oh!

Uh oh!

mathause left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mathauseOct 26, 2020

Choose a reason for hiding this comment

Uh oh!

max-sixty commented Oct 26, 2020

Uh oh!

Uh oh!

max-sixty commented Oct 30, 2020

Uh oh!

mathause commented Oct 30, 2020

Uh oh!

max-sixty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mathause left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mathauseOct 30, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

max-sixty commented Nov 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants