Relax nanosecond datetime restriction in CF time decoding by kmuehlbauer · Pull Request #9618 · pydata/xarray

kmuehlbauer · 2024-10-13T19:55:39Z

ClosesInteroperability with Pandas 2.0 non-nanosecond datetime #7493
ClosesDataArray constructor still coerces to np.datetime64[ns], not cftime in 0.11.0 #2587
Tests added/changed
User visible changes (including notable bug fixes) are documented in whats-new.rst
New functions/methods are listed in api.rst

This is another attempt to resolve#7493. This goes a step further than #9580.

The idea of this PR is to automatically infer the needed resolutions for decoding/~~encoding~~ and only keep the constraints pandas imposes ("s" - lowest resolution, "ns" - highest resolution). There is still the idea of a default resolution, but this should only take precedence if it doesn't clash with the automatic inference. This can be discussed, though. Update: I've implemented time-unit-kwarg ~~a first try to have default resolution~~ on decode, which will override the current inferred resolution only to higher resolution (eg. 's' -> 'ns'). To work towards #4490 the time decoding options (decode_time and use_cftime are bundled within CFDatetimeCoder which is distributed via decode_times kwarg. use_cftime-kwarg is deprecated.

For sanity checking, and also for my own good, I've created a documentation page on time-coding in the internal dev section. Any suggestions (especially grammar) or ideas for enhancements are much appreciated.

There still might be room for consolidation of functions/methods (mostly in coding/times.py), but I have to leave it alone for some days. I went down that rabbit hole and need to relax, too 😬.

Looking forward to get your insights here, @spencerkclark, @ChrisBarker-NOAA, @pydata/xarray.

Todo:

floating point handling
update decoding tests to iterate over time_units (where appropriate)
CFDatetimeCoder as input for decode_times kwarg
...

kmuehlbauer · 2024-10-14T12:13:07Z

Nice, mypy 1.12 is out and breaks our typing, 😭.

TomNicholas · 2024-10-14T15:16:04Z

Nice, mypy 1.12 is out and breaks our typing, 😭

Can we pin it in the CI temporarily?

kmuehlbauer · 2024-10-14T15:28:08Z

Can we pin it in the CI temporarily?

Yes, 1.11.2 was the last version.

kmuehlbauer · 2024-10-14T19:30:32Z

This is now ready for a first round of review. I think this is already in a quite usable state.

But no rush, this should be thoroughly tested.

spencerkclark · 2024-10-18T01:46:23Z

Sounds good @kmuehlbauer! I’ll try and take an initial look this weekend.

…ore/variable.py to use any-precision datetime/timedelta with autmatic inferring of resolution

…ocessing, raise now early

…_ref_date

…o fix mypy

…t resolution, fix code and tests to allow this

* Fix timedelta encoding overflow issue; always decode to ns resolution * Implement time_unit for decode_cf_timedelta * Reduce diff

kmuehlbauer · 2025-01-13T08:31:39Z

Thanks again @spencerkclark! I've merged your kmuehlbauer#3 as it is essential for this PR to keep consistency and backwards compatibility.

Co-authored-by: Kai Mühlbauer <kmuehlbauer@wradlib.org>

kmuehlbauer · 2025-01-15T17:06:45Z

Failing CI looks unrelated. Decision in the dev meeting was to merge fast to have a bit of time to test before release later this month.

kmuehlbauer · 2025-01-15T17:10:23Z

Thanks All! 🚀

spencerkclark · 2025-01-16T01:36:16Z

Awesome, congrats @kmuehlbauer—thank you!

ilan-gold · 2025-01-16T09:58:47Z

Yes @kmuehlbauer - monumental effort!!!

dcherian · 2025-01-16T14:04:58Z

Amazing! 👏 👏 👏

* main: (79 commits) fix mean for datetime-like using the respective time resolution unit (#9977) Add `time_unit` argument to `CFTimeIndex.to_datetimeindex` (#9965) remove gate and add a test (#9958) Remove repetitive that (replace it with the) (#9994) add shxarray to the xarray ecosystem list (#9995) Add `shards` to `valid_encodings` to enable sharded Zarr writing (#9948) Use flox for grouped first, last (#9986) Bump the actions group with 2 updates (#9989) Fix some typing (#9988) Remove unnecessary a article (#9980) Fix test_doc_example on big-endian systems (#9949) fix weighted polyfit for arrays with more than 2 dimensions (#9974) Use zarr-fixture to prevent thread leakage errors (#9967) remove dask-expr from CI runs, fix related tests (#9971) Update time coding tests to assert exact equality (#9961) cast type to PDDatetimeUnitOptions (#9963) Suggest the correct name when no key matches in the dataset (#9943) fix upstream dev issues (#9953) Relax nanosecond datetime restriction in CF time decoding (#9618) Remove outdated quantile test. (#9945) ...

In #9618 we allowed encoding `np.datetime64` values prior to 1582-10-15 using a `"standard"` or `"gregorian"` calendar through cftime. While technically possible, this implicitly introduces a calendar change, and means the values can no longer be round tripped as `np.datetime64`—xarray will choose `cftime.DatetimeGregorian` instances when decoding instead. I am not sure how often this will come up and the behavior may not be the user's intent. This PR switches to raising a `ValueError` in this circumstance, and recommends encoding with a `"proleptic_gregorian"` calendar instead (the calendar that xarray automatically chooses for `np.datetime64` values if provided no user input).

* main: (79 commits) fix mean for datetime-like using the respective time resolution unit (pydata#9977) Add `time_unit` argument to `CFTimeIndex.to_datetimeindex` (pydata#9965) remove gate and add a test (pydata#9958) Remove repetitive that (replace it with the) (pydata#9994) add shxarray to the xarray ecosystem list (pydata#9995) Add `shards` to `valid_encodings` to enable sharded Zarr writing (pydata#9948) Use flox for grouped first, last (pydata#9986) Bump the actions group with 2 updates (pydata#9989) Fix some typing (pydata#9988) Remove unnecessary a article (pydata#9980) Fix test_doc_example on big-endian systems (pydata#9949) fix weighted polyfit for arrays with more than 2 dimensions (pydata#9974) Use zarr-fixture to prevent thread leakage errors (pydata#9967) remove dask-expr from CI runs, fix related tests (pydata#9971) Update time coding tests to assert exact equality (pydata#9961) cast type to PDDatetimeUnitOptions (pydata#9963) Suggest the correct name when no key matches in the dataset (pydata#9943) fix upstream dev issues (pydata#9953) Relax nanosecond datetime restriction in CF time decoding (pydata#9618) Remove outdated quantile test. (pydata#9945) ...

TomNicholas mentioned this pull request Oct 14, 2024
Reimplement Datatree typed ops #9619
Merged
4 tasks

kmuehlbauer force-pushed the any-time-resolution-2 branch from ca5050d to f7396cfCompare October 14, 2024 16:09

kmuehlbauer marked this pull request as ready for review October 14, 2024 18:05

kmuehlbauer mentioned this pull request Oct 15, 2024
Implement default time resolution for CF time encoding/decoding #9580
Closed
4 tasks

kmuehlbauer added topic-CF conventions topic-cftime run-upstream Run upstream CI labels Oct 16, 2024

kmuehlbauerand others added 18 commits October 18, 2024 07:31

implement default_precision_timestamp, refactor coding/times.py and c…
7b5f323
…ore/variable.py to use any-precision datetime/timedelta with autmatic inferring of resolution

align tests with new time resolution behaviour
8784f33

timedelta decoding, fsspec handling
b45ab23

fixes in coding/times.py
39086ef

add docs on time coding
df49a40

attempt fixing doc tests
adb8ca3

fix issue where out-of-bounds floating point values slipped in the pr…
266b1ed
…ocessing, raise now early

convert to UTC first before stripping of tz in _unpack_time_units_and…
6d5f13b
…_ref_date

reorganize pandas compatibility code, remove unneeded code, attempt t…
5d68bfe
…o fix mypy

another attempt to finally fix mypy
07bba69

refactor out _check_date_is_after_shift
6e7f0bb

refactor out _maybe_strip_tz_from_timestamp
b4a49bb

more refactoring in coding.times.py
2e1ff4f

more refactoring in coding.times.py
d5a7da0

minor fix in time-coding.rst
821b68d

set default resolution to "s", which actually means, use pandas lowes…
d066edf
…t resolution, fix code and tests to allow this

Add section for default units, fix options
ed22da1

attempt to fix typing
8bf23f4

spencerkclarkand others added 2 commits January 13, 2025 08:20

Implementtime_unitoption fordecode_cf_timedelta(#3)
ffc1828
* Fix timedelta encoding overflow issue; always decode to ns resolution * Implement time_unit for decode_cf_timedelta * Reduce diff

fix typing
eaf3c73

kmuehlbauer commented Jan 13, 2025
View reviewed changes

Comment threaddoc/whats-new.rst Outdated

use nanmin/nanmax, catch numpy RuntimeWarnings
1e6ba18

spencerkclark reviewed Jan 14, 2025
View reviewed changes

Comment threadxarray/tests/test_coding_times.py Outdated
Comment threadxarray/tests/test_coding_times.py Outdated
Comment threaddoc/whats-new.rst Outdated

Apply suggestions from code review
85a340b
Co-authored-by: Kai Mühlbauer <kmuehlbauer@wradlib.org>

spencerkclark added the plan to merge Final call for comments label Jan 14, 2025

tomvothecoder mentioned this pull request Jan 14, 2025
[Enhancement]: Xarray: Deprecated since version 2025.01.1: Please pass a coders.CFDatetimeCoder instance initialized with use_cftime to the decode_times kwarg instead.xCDAT/xcdat#719
Closed

Merge branch 'main' into any-time-resolution-2
db69b63

kmuehlbauer merged commit 6bea715 into pydata:mainJan 15, 2025

This was referenced Jan 18, 2025
Update time coding tests to assert exact equality #9961
Merged
Add time_unit argument to CFTimeIndex.to_datetimeindex#9965
Merged
Enable passing a CFTimedeltaCoder to decode_timedelta#9966
Merged

This was referenced May 24, 2025
Disallow standard calendar np.datetime64 encoding prior to reform #10352
Merged
Implement cftime vectorization as discussed in PR #8322 #8324
Merged

kmuehlbauer mentioned this pull request Jun 3, 2026
Fix Upstream CI NaT issues #11340
Merged
4 tasks

Uh oh!

Uh oh!

Conversation

kmuehlbauer commented Oct 13, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kmuehlbauer commented Oct 14, 2024

Uh oh!

TomNicholas commented Oct 14, 2024

Uh oh!

kmuehlbauer commented Oct 14, 2024

Uh oh!

kmuehlbauer commented Oct 14, 2024

Uh oh!

spencerkclark commented Oct 18, 2024

Uh oh!

kmuehlbauer commented Jan 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmuehlbauer commented Jan 15, 2025

Uh oh!

kmuehlbauer commented Jan 15, 2025

Uh oh!

spencerkclark commented Jan 16, 2025

Uh oh!

ilan-gold commented Jan 16, 2025

Uh oh!

dcherian commented Jan 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

kmuehlbauer commented Oct 13, 2024•
edited
Loading