clib.conversion._to_numpy: Add tests for PyArrow's timestamp type by seisman · Pull Request #3621 · GenericMappingTools/pygmt

seisman · 2024-11-15T13:43:40Z

pyarrow.timestamp is the datetime dtype in pyarrow. It has two parameters, unit and tz.

This PR adds tests to ensure _to_numpy support pyarrow.timestamp. Please note that, unit can only be s/ms/us/ns, and ~~tz information is discarded when converting to np.datetime64.~~ Actually, when converting to np.datetime64, timezone is corrected to UTC.

In [1]: import pyarrow as pa
In [2]: import datetime
In [3]: import numpy as np
In [4]: x = pa.array([datetime.datetime(2024, 1, 1, 12, 34, 56)], type=pa.timestamp("s"))
In [5]: np.ascontiguousarray(x)
Out[5]: array(['2024-01-01T12:34:56'], dtype='datetime64[s]')
In [6]: x = pa.array([datetime.datetime(2024, 1, 1, 12, 34, 56)], type=pa.timestamp("ms"))
In [7]: np.ascontiguousarray(x)
Out[7]: array(['2024-01-01T12:34:56.000'], dtype='datetime64[ms]')
In [8]: x = pa.array([datetime.datetime(2024, 1, 1, 12, 34, 56)], type=pa.timestamp("ms", tz="UTZ"))
In [9]: np.ascontiguousarray(x)
Out[9]: array(['2024-01-01T12:34:56.000'], dtype='datetime64[ms]')

Related to #3600.

seisman · 2024-11-21T04:21:52Z

 _HAS_PYARROW = True
 except ImportError:
+
+ class pa: # noqa: N801


pa.timestamp() with tz doesn't have string aliases, so we can't use strings like timestamp[s, UTC]. So we have to define a dummy function here.

Could we avoid this dummy class by starting from a pandas.Timestamp instead of Python datetime? The timezone info can be carried over:
importpyarrowaspaimportpandasaspdfromzoneinfoimportZoneInfoarray=pa.array( [ pd.Timestamp(2024, 1, 2, 3, 4, 5, tzinfo=ZoneInfo("America/New_York")), pd.Timestamp(2024, 1, 2, 3, 4, 6, tzinfo=ZoneInfo("America/New_York")), ] ) print(array.type) # timestamp[us, tz=America/New_York]

We can, but I feel it will make the pytest parametrize more complicated.

Ok, let's keep this dummy class then.

weiji14 · 2024-11-26T02:51:50Z

+ pytest.param(
+ pa.timestamp("s", tz="America/New_York"),
+ "datetime64[s]",
+ id="timestamp[s, tz=America/New_York]",
+ ),
+ pytest.param(
+ pa.timestamp("s", tz="+07:30"),
+ "datetime64[s]",
+ id="timestamp[s, tz=+07:30]",
+ ),
+ ],
+)
+def test_to_numpy_pyarrow_timestamp(dtype, expected_dtype):
+ """
+ Test the _to_numpy function with PyArrow arrays of PyArrow datetime types.
+
+ pyarrow.timestamp(unit, tz=None) can accept units "s", "ms", "us", and "ns".
+
+ Reference: https://arrow.apache.org/docs/python/generated/pyarrow.timestamp.html
+ """
+ data = [datetime(2024, 1, 2, 3, 4, 5), datetime(2024, 1, 2, 3, 4, 6)]
+ array = pa.array(data, type=dtype)


Is the timezone part of the pa.timestamp type actually working?
data= [datetime(2024, 1, 2, 3, 4, 5), datetime(2024, 1, 2, 3, 4, 6)] array=pa.array(data, type=pa.timestamp("s", tz="America/New_York")) print(array) # [# 2024-01-02 03:04:05Z,# 2024-01-02 03:04:06Z# ]result=_to_numpy(array) print(result) # ['2024-01-02T03:04:05' '2024-01-02T03:04:06']
If using pd.Timestamp instead
import pandas as pd import pyarrow as pa from datetime import datetime, timezone from zoneinfo import ZoneInfo from pygmt.clib.conversion import _to_numpy array = pa.array( [ pd.Timestamp(2024, 1, 2, 3, 4, 5, tzinfo=ZoneInfo("America/New_York")), pd.Timestamp(2024, 1, 2, 3, 4, 6, tzinfo=ZoneInfo("America/New_York")), ] ) print(array) # [ # 2024-01-02 08:04:05.000000Z, # 2024-01-02 08:04:06.000000Z # ] result = _to_numpy(array) print(result) # ['2024-01-02T08:04:05.000000' '2024-01-02T08:04:06.000000']
For the first one, the timezone offset for New York (UTC-5) doesn't seem to be applied? Whereas for the second one, it is converting from New York timezone (UTC-5) to UTC by adding 5 hours.

It seems the timezone information is applied but not shown when representing the array.
In [1]: fromdatetimeimportdatetimeIn [2]: importpyarrowaspa# For pa.scalar, it's shown with timezone offsetIn [3]: pa.scalar(datetime(2024, 1, 2, 3, 4, 5), type=pa.timestamp("s", tz="America/New_York")) Out[3]: <pyarrow.TimestampScalar: '2024-01-01T22:04:05-0500'>In [4]: array=pa.array([datetime(2024, 1, 2, 3, 4, 5)], type=pa.timestamp("s", tz="America/New_York")) # For pa.array, it's shown without timezone offsetIn [5]: arrayOut[5]: <pyarrow.lib.TimestampArrayobjectat0x7f52feace740> [ 2024-01-0203:04:05Z ] # When converted to pandas, it's shown with timezone offsetIn [6]: array.to_pandas() Out[6]: 02024-01-0122:04:05-05:00dtype: datetime64[s, America/New_York] # When converted to numpy, timezone information is lost, since numpy doesn's have tz support.In [7]: array.to_numpy() Out[7]: array(['2024-01-02T03:04:05'], dtype='datetime64[s]')

Likely related to apache/arrow#40122.

Is the timezone part of the pa.timestamp type actually working?
data= [datetime(2024, 1, 2, 3, 4, 5), datetime(2024, 1, 2, 3, 4, 6)] array=pa.array(data, type=pa.timestamp("s", tz="America/New_York")) print(array) # [# 2024-01-02 03:04:05Z,# 2024-01-02 03:04:06Z# ]result=_to_numpy(array) print(result) # ['2024-01-02T03:04:05' '2024-01-02T03:04:06']
If using pd.Timestamp instead
import pandas as pd import pyarrow as pa from datetime import datetime, timezone from zoneinfo import ZoneInfo from pygmt.clib.conversion import _to_numpy array = pa.array( [ pd.Timestamp(2024, 1, 2, 3, 4, 5, tzinfo=ZoneInfo("America/New_York")), pd.Timestamp(2024, 1, 2, 3, 4, 6, tzinfo=ZoneInfo("America/New_York")), ] ) print(array) # [ # 2024-01-02 08:04:05.000000Z, # 2024-01-02 08:04:06.000000Z # ] result = _to_numpy(array) print(result) # ['2024-01-02T08:04:05.000000' '2024-01-02T08:04:06.000000']
For the first one, the timezone offset for New York (UTC-5) doesn't seem to be applied? Whereas for the second one, it is converting from New York timezone (UTC-5) to UTC by adding 5 hours.
Also need to note the difference between datetime and pd.Timestamp. datetime is timezone-unware, while pd.Timestamp has timezone support.
So, in your first example:
actual datetime: 2024-01-02T03:04:05+00:00
In pyarrow.array, the actual datetime is 2024-01-01T22:04:05-05:00, but UTC time (2024-01-02T03:04:05) is shown
to_numpy: 2024-01-02T03:04:05 (UTC time)
In your 2nd example:
the actual datetime: 2024-01-02T03:04:05-05:00
In pyarrow.array: the UTC time (2024-01-02 08:04:05) is shown
to_numpy: 2024-01-02T08:04:05
So, I think there is no inconsistency. The pyarrow.array has timezone stored internally, but its repr always shows UTC time. And when converting to numpy array (using either its own to_numpy() method or np.ascontinuousarray()), the datetime are always converted to UTC since numpy.datetime64 doesn't have timezone support.

https://stackoverflow.com/a/73276431
Arrow internally stores datetime as UTC + timezone info and will print it as such

Ok, I see now that there is indeed consistency, thanks for clearing this up, the repr on pyarrow.array is indeed confusing 😅 . So in summary:
class/type has timezone support Link
Python datetime ✅ https://docs.python.org/3/library/datetime.html#datetime.datetime.tzinfo
pandas.Timestamp ✅ https://pandas.pydata.org/pandas-docs/version/2.2/reference/api/pandas.Timestamp.html
pyarrow.timestamp (type) ✅ https://arrow.apache.org/docs/17.0/python/generated/pyarrow.timestamp.html
numpy.datetime64 ❌ https://numpy.org/doc/2.1/reference/arrays.scalars.html#numpy.datetime64
Since we're converting to NumPy, the timezone information will always be lost (everything is converted to UTC). What does this mean for PyGMT/GMT? Does it mean that users cannot plot data in a specific timezone, because it will always convert to UTC?

As shown below, GMT itself doesn't have timezone support, similar to numpy.datetime64. The big difference is that GMT simply ignore the TZ infor, while the to_numpy conversion always converts to UTC.
gmt begin map gmt basemap -R2024-01-01T00:00:00+08:00/2024-01-02T00:00:00+08:00/1/10 -JX10c/4c -Baf echo 2024-01-01T08:00:00+08:00 5 | gmt plot -Sc1c -Gblack gmt end show
Does it mean that users cannot plot data in a specific timezone, because it will always convert to UTC?
I think yes, unless we try hard to drop the TZ information before converting to numpy.datetime64. Below is an example, but if we want to support TZ, we need to find a more general way to do it.
In [1]: importpyarrowaspaIn [2]: fromdatetimeimportdatetimeIn [3]: data= [datetime(2024, 1, 2, 3, 4, 5), datetime(2024, 1, 2, 3, 4, 6)] In [4]: array=pa.array(data, type=pa.timestamp("s", tz="America/New_York")) In [5]: arrayOut[5]: <pyarrow.lib.TimestampArrayobjectat0x7f78c07bafe0> [ 2024-01-0203:04:05Z, 2024-01-0203:04:06Z ] In [6]: array.to_pandas().dt.tz_localize(None) Out[6]: 02024-01-0122:04:0512024-01-0122:04:06dtype: datetime64[s] In [7]: importnumpyasnpIn [8]: np.ascontiguousarray(array.to_pandas().dt.tz_localize(None)) Out[8]: array(['2024-01-01T22:04:05', '2024-01-01T22:04:06'], dtype='datetime64[s]')

Maybe we need to open an issue to discuss this - whether to:
Follow GMT (allow users to plot data at a non-UTC timezone, by ignoring the timezone offset)
Follow NumPy, whereby the data will from any non-UTC timezone will be converted to UTC always.
If going with 2, we should at least raise a warning if a non-UTC timezone is used, that a conversion is taking place. If going with 1, we would need to special-case datetime types, that might mean extra logic in the _to_numpy function, or having to keep array_to_datetime (so don't merge #3507 yet).

I'm also debating whether to following option 1 or 2, so a separate issue sounds good.
Since the current behavior is option 2 (as done in array_to_datetime), I think we should go with option 2 first, then revisit the timezone support later, which will be a breaking change anyway.

Ok, I'll open an issue to discuss (edit: see #3656). And sure, we can go with option 2 for now.

… to do conversion

weiji14

Pre-approving, but see optional suggestions below.

weiji14 · 2024-11-28T09:33:39Z

 _HAS_PYARROW = True
 except ImportError:
+
+ class pa: # noqa: N801


Ok, let's keep this dummy class then.

Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>

clib.conversion._to_numpy: Add tests for PyArrow's timestamp type
ce2b830

seisman added maintenance Boring but important stuff for the core devs skip-changelog Skip adding Pull Request to changelog needs review This PR has higher priority and needs review. labels Nov 15, 2024

seisman added this to the 0.14.0 milestone Nov 15, 2024

seisman mentioned this pull request Nov 15, 2024
Supported data types by the private _to_numpy function #3600
Open
35 tasks

seisman removed the needs review This PR has higher priority and needs review. label Nov 16, 2024

seisman marked this pull request as draft November 16, 2024 00:21

Add a dummy function to mimin pyarrow
214f060

seisman marked this pull request as ready for review November 16, 2024 07:39

seisman added the needs review This PR has higher priority and needs review. label Nov 16, 2024

Merge branch 'main' into to_numpy/pyarrow_timestamp
bf16e97

seisman commented Nov 21, 2024
View reviewed changes

seisman requested a review from weiji14 November 21, 2024 04:35

Merge branch 'main' into to_numpy/pyarrow_timestamp
e0c2e7b

weiji14 reviewed Nov 26, 2024
View reviewed changes

seisman added 3 commits November 26, 2024 20:31

Merge branch 'main' into to_numpy/pyarrow_timestamp
95ac538

Merge branch 'main' into to_numpy/pyarrow_timestamp
7cbfeb8

Explicitly check each array element since np.assert_array_equal needs…
c622438
… to do conversion

weiji14 mentioned this pull request Nov 28, 2024
How to handle datetime arrays with timezone information? #3656
Open

weiji14 approved these changes Nov 28, 2024
View reviewed changes

seismanand others added 2 commits November 28, 2024 17:41

Apply suggestions from code review
7bc1f6f
Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>

Improve comments about timestamp types
5c2eb46

seisman removed the needs review This PR has higher priority and needs review. label Nov 28, 2024

seisman enabled auto-merge (squash) November 28, 2024 09:50

seisman merged commit 6766b84 into mainNov 28, 2024

seisman deleted the to_numpy/pyarrow_timestamp branch November 28, 2024 09:57

class/type	has timezone support	Link
Python `datetime`	✅	https://docs.python.org/3/library/datetime.html#datetime.datetime.tzinfo
pandas.Timestamp	✅	https://pandas.pydata.org/pandas-docs/version/2.2/reference/api/pandas.Timestamp.html
pyarrow.timestamp (type)	✅	https://arrow.apache.org/docs/17.0/python/generated/pyarrow.timestamp.html
numpy.datetime64	❌	https://numpy.org/doc/2.1/reference/arrays.scalars.html#numpy.datetime64

Uh oh!

Conversation

seisman commented Nov 15, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seismanNov 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14Nov 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14Nov 28, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

seisman commented Nov 15, 2024•
edited
Loading

seismanNov 26, 2024•
edited
Loading

weiji14Nov 26, 2024•
edited
Loading

weiji14Nov 28, 2024•
edited
Loading