Implement cftime vectorization as discussed in PR #8322 by antscloud · Pull Request #8324 · pydata/xarray

antscloud · 2023-10-17T17:01:25Z

As discussed in #8322, here is the test for implementing the vectorization

Only this test seems to fail in test_coding_times.py :

xarray/xarray/tests/test_coding_times.py

Lines 1061 to 1071 in f895dc1

@requires_cftime
@pytest.mark.parametrize("freq", ["U", "L", "S", "T", "H", "D"])
deftest_encode_decode_roundtrip_cftime(freq) ->None: 
initial_time=cftime_range("0001", periods=1) 
times=initial_time.append( 
cftime_range("0001", periods=2, freq=freq) +timedelta(days=291000*365) 
 ) 
variable=Variable(["time"], times) 
encoded=conventions.encode_cf_variable(variable) 
decoded=conventions.decode_cf_variable("time", encoded, use_cftime=True) 
assert_equal(variable, decoded) 

I don't really understand why though if you have an idea

kmuehlbauer

@antscloud Nice catch! I have two suggestion to get the test suite running.

headtr1ck · 2024-07-11T17:05:41Z

@antscloud are you willing to continue this? It would be a good addition.

antscloud · 2024-07-14T15:29:19Z

Hi, sorry i completely forgot about this issue, i've implemented the suggestions and all the tests appear to pass, I hope it works in the CI 🤞

headtr1ck · 2024-07-15T15:05:17Z

BENCHMARKS NOT SIGNIFICANTLY CHANGED.

I didn't check in detail if we have a test that might cover this already and if yes, what the changes are.

spencerkclark · 2024-07-20T15:02:16Z

I don't think we currently do, but I think this new benchmark should cover it: #9262.

spencerkclark

Thanks @antscloud! I think this looks good to me now. Could you add a what's new entry under a new Performance heading?

headtr1ck · 2024-07-20T17:05:20Z

| Change | Before [91a52dc] | After [3512bee] | Ratio | Benchmark (Parameter) |
|----------|----------------------|---------------------|---------|-----------------------------------------------------------|
| - | 139±0.9ms | 42.3±0.7ms | 0.3 | coding.EncodeCFDatetime.time_encode_cf_datetime('noleap') |

3 times faster, not bad

headtr1ck

Looks good to me!

headtr1ck · 2024-07-22T09:55:12Z

+ dates = np.atleast_1d(dates)
+
+ # Find all the None position
+ none_position = np.equal(dates, None)


mypy complains here.
Not sure if this is an issue of wrong typing of numpy, or if this is not a recommended way of calling equals. (I think the None is the problem).

Co-authored-by: Michael Niklas <mick.niklas@gmail.com>

headtr1ck · 2024-10-14T16:34:08Z

The mypy error is still there, not sure how to effectively fix it, maybe use kwargs instead or simply type ignore it.

Otherwise this looks good and should get an entry in whats-new!

kmuehlbauer · 2025-05-23T12:54:27Z

Thanks @antscloud! Still a valuable addition, which should be mentioned in whats-new.rst. @antscloud are you still around to finalize this? Otherwise I could take care of that.

kmuehlbauer · 2025-05-23T13:12:15Z

Great, either I messed up the merge or this fails now, because of other concurring changes in the past time. @spencerkclark Do you spot anything suspicious here?

spencerkclark · 2025-05-24T13:59:34Z

Thanks @kmuehlbauer for reviving this—I think the test failures are a manifestation of the cftime issue you found earlier (Unidata/cftime#354). The difference here from what was implemented in #9618 is that datetime.datetime objects are no longer converted to cftime.DatetimeProlepticGregorian objects prior to being passed to cftime.date2num.

While the issue was fixed in Unidata/cftime#355, a new version of cftime has not yet been released. We could lobby for that, or raise an error in these circumstances. Independent of the cftime issue, there may be a reasonable case for raising (#10352).

kmuehlbauer · 2025-05-27T13:53:36Z

@spencerkclark Looks like this works after merging #10352. 🎉

spencerkclark

Thanks @antscloud! Sorry this took so long to push through. I went ahead and added a what's new entry on your behalf. Feel free to edit as you see fit, but otherwise I will merge on Thursday.

spencerkclark · 2025-05-27T22:27:09Z

@kmuehlbauer I'm guessing you approve now, but giving you the chance to take another look and remove your requested change (which I think is resolved).

spencerkclark · 2025-05-30T01:16:28Z

The benchmark workflow failure is unrelated, so I am going to go ahead and merge. Thanks again @antscloud!

* main: Fix performance regression in interp from pydata#9881 (pydata#10370) html repr: improve style for dropdown sections (pydata#10354) Grouper tweaks. (pydata#10362) Docs: Add links to getting help mermaid diagram (pydata#10324) Enforce ruff/flynt rules (FLY) (pydata#10375) Add missing AbstractWritableDataStore base methods and arguments (pydata#10343) Improve html repr in dark mode (Jupyterlab + Xarray docs) (pydata#10353) Pin Mypy to 1.15 (pydata#10378) use numpy dtype exposed by zarr array instead of metadata.data_type (pydata#10348) Fix doc typo for caption "Interoperability" (pydata#10374) Implement cftime vectorization as discussed in PR pydata#8322 (pydata#8324) Enforce ruff/flake8-pyi rules (PYI) (pydata#10359) Apply assorted ruff/Pylint rules (PL) / Enforce PLE rules (pydata#10366) (fix): pandas extension array repr for int64[pyarrow] (pydata#10317) Enforce ruff/flake8-implicit-str-concat rules (ISC) (pydata#10368) Enforce ruff/refurb rules (FURB) (pydata#10367) Ignore ruff/Pyflakes rule F401 more precisely (pydata#10369) Apply assorted ruff/flake8-simplify rules (SIM) (pydata#10364) Apply assorted ruff/flake8-pytest-style rules (PT) (pydata#10363) Fix "a array" misspelling (pydata#10365)

github-actionsBot added the topic-cftime label Oct 17, 2023

dcherian changed the title ~~Implement vectorization as discussed in PR #8322~~Implement cftime vectorization as discussed in PR #8322Oct 17, 2023

Illviljan added the run-benchmark Run the ASV benchmark workflow label Oct 17, 2023

kmuehlbauer requested changes Oct 18, 2023
View reviewed changes

Comment threadxarray/coding/times.py Outdated
Comment threadxarray/coding/times.py

headtr1ck reviewed Oct 23, 2023
View reviewed changes

Comment threadxarray/coding/times.py Outdated

antscloud added 2 commits July 14, 2024 16:43

feat: implement as discussed in PRpydata#8322
3f6bf49

fix: Stick closer to the original function and add PR suggestions
f28fc7d

antscloud force-pushed the vectorization_of_encode_cftime branch from cff06d4 to f28fc7dCompare July 14, 2024 15:27

spencerkclark mentioned this pull request Jul 20, 2024
Add encode_cf_datetime benchmark #9262
Merged

spencerkclark reviewed Jul 20, 2024
View reviewed changes

Merge branch 'main' into vectorization_of_encode_cftime
dc2609c

headtr1ck approved these changes Jul 20, 2024
View reviewed changes

Comment threadxarray/coding/times.py Outdated

headtr1ck reviewed Jul 22, 2024
View reviewed changes

dcherianand others added 2 commits August 2, 2024 09:29

Update xarray/coding/times.py
06af81a
Co-authored-by: Michael Niklas <mick.niklas@gmail.com>

Merge branch 'main' into vectorization_of_encode_cftime
05428fd

dcherian reviewed Oct 22, 2024
View reviewed changes

Comment threadxarray/coding/times.py Outdated

dcherianand others added 3 commits October 22, 2024 14:28

Update xarray/coding/times.py
80bb5fe

Merge branch 'main' into vectorization_of_encode_cftime
f4828a2

Merge branch 'main' into vectorization_of_encode_cftime
8be73cf

github-actionsBot added the topic-documentation label May 23, 2025

kmuehlbauer reviewed May 23, 2025
View reviewed changes

Comment threadxarray/coding/times.py Outdated

Update xarray/coding/times.py
042638d

spencerkclark mentioned this pull request May 24, 2025
Disallow standard calendar np.datetime64 encoding prior to reform #10352
Merged
2 tasks

Merge branch 'main' into vectorization_of_encode_cftime
147ea67

dcherian requested a review from spencerkclark May 27, 2025 17:34

spencerkclark added 2 commits May 27, 2025 18:14

Add what's new entry
c83f76a

Merge branch 'main' into vectorization_of_encode_cftime
6d4dc3b

spencerkclark approved these changes May 27, 2025
View reviewed changes

spencerkclark requested a review from kmuehlbauer May 27, 2025 22:23

kmuehlbauer approved these changes May 28, 2025
View reviewed changes

spencerkclark added the plan to merge Final call for comments label May 28, 2025

Merge branch 'main' into vectorization_of_encode_cftime
c37f1f1

spencerkclark merged commit 8b961e4 into pydata:mainMay 30, 2025

	@requires_cftime
	@pytest.mark.parametrize("freq", ["U", "L", "S", "T", "H", "D"])
	deftest_encode_decode_roundtrip_cftime(freq) ->None:
	initial_time=cftime_range("0001", periods=1)
	times=initial_time.append(
	cftime_range("0001", periods=2, freq=freq) +timedelta(days=291000*365)
	)
	variable=Variable(["time"], times)
	encoded=conventions.encode_cf_variable(variable)
	decoded=conventions.decode_cf_variable("time", encoded, use_cftime=True)
	assert_equal(variable, decoded)

Uh oh!

Uh oh!

Conversation

antscloud commented Oct 17, 2023

Uh oh!

kmuehlbauer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

headtr1ck commented Jul 11, 2024

Uh oh!

antscloud commented Jul 14, 2024

Uh oh!

headtr1ck commented Jul 15, 2024

Uh oh!

spencerkclark commented Jul 20, 2024

Uh oh!

spencerkclark left a comment

Choose a reason for hiding this comment

Uh oh!

headtr1ck commented Jul 20, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

headtr1ck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

headtr1ckJul 22, 2024

Choose a reason for hiding this comment

Uh oh!

headtr1ck commented Oct 14, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kmuehlbauer commented May 23, 2025

Uh oh!

Uh oh!

kmuehlbauer commented May 23, 2025

Uh oh!

spencerkclark commented May 24, 2025

Uh oh!

kmuehlbauer commented May 27, 2025

Uh oh!

spencerkclark left a comment

Choose a reason for hiding this comment

Uh oh!

spencerkclark commented May 27, 2025

Uh oh!

spencerkclark commented May 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

headtr1ck commented Jul 20, 2024•
edited
Loading

headtr1ck commented Oct 14, 2024•
edited
Loading