GH-49875: [Python] Fix timezone dropped when converting tz-aware Categorical to Arrow array by AnkitAhlawat7742 · Pull Request #49878 · apache/arrow

AnkitAhlawat7742 · 2026-04-28T05:44:33Z

Rationale for this change

When converting a pandas.Categorical with tz-aware datetime categories to a PyArrow array, the timezone information was silently dropped from the dictionary array's value type. This is a silent data loss bug — no warning or error is raised, but the timezone metadata is lost.

What changes are included in this PR?

In python/pyarrow/array.pxi, the Categorical conversion was using values.categories.values(raw numpy array) which strips timezone metadata since numpy does not support tz-aware datetimes. Changed to values.categories (pandas Index) and added from_pandas=True so PyArrow uses the pandas conversion path, which correctly preserves timezone metadata.

Are these changes tested?

Yes. Verified manually

Are there any user-facing changes?

Yes — this is a bug fix. Users did #49875

This PR contains a "Critical Fix" — timezone information was lost silently during conversion without any warning or error.

GitHub Issue: [Python] Timezone being dropped in dictionary array #49875

… Categorical to Arrow array

github-actions · 2026-04-28T05:44:57Z

⚠️ GitHub issue #49875has been automatically assigned in GitHub to PR creator.

AnkitAhlawat7742 · 2026-04-30T04:13:30Z

Hi @AlenkaF
Please review these changes

AlenkaF

I think this change makes sense but the PR is lacking a test. The reproducible example from the issue can be reused and added to https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_pandas.py mimicking existing categorical tests.

cc @jorisvandenbossche in case of any opinions.

…reservation

AnkitAhlawat7742 · 2026-05-04T12:04:49Z

I think this change makes sense but the PR is lacking a test. The reproducible example from the issue can be reused and added to https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_pandas.py mimicking existing categorical tests.
cc @jorisvandenbossche in case of any opinions.

Thank you for the review!
I have added a regression test to python/pyarrow/tests/test_pandas.py based on the reproducible example from the issue.
Please have a look.

AlenkaF · 2026-05-04T13:20:11Z

CI failures are related, could you have a look?

…esolution

AnkitAhlawat7742 · 2026-05-04T13:47:58Z

CI failures are related, could you have a look?

Fixed:

Resolved the formatting issue.
CI failure was caused by pandas version compatibility in the test assertion:
str(cat.dtype.categories.dtype) == "datetime64[us, US/Eastern]"

Please retrigger the CI

jorisvandenbossche

Looks good!

github-actions · 2026-05-04T14:59:32Z

⚠️ GitHub issue #49875has been automatically assigned in GitHub to PR creator.

…iew comments

AlenkaF · 2026-05-05T09:05:15Z

@github-actions crossbow submit pandas

github-actions · 2026-05-05T09:07:38Z

Revision: 0f1f595

Submitted crossbow builds: ursacomputing/crossbow @ actions-3f725ba4fc

Task	Status
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly

AlenkaF

Thanks! The failing extended builds are not connected. Will try to see if there is an issue opened already to track that.

AlenkaF · 2026-05-05T10:09:15Z

Opened an issue for the failing builds here: #49920

AlenkaF · 2026-05-06T06:31:52Z

@jorisvandenbossche mind having one more look before I merge?

raulcd

This looks good to me

conbench-apache-arrow · 2026-05-06T08:47:29Z

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit ea8cef5.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

conbench-apache-arrow · 2026-05-15T12:37:22Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit ea8cef5.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

…e Categorical to Arrow array (apache#49878) ### Rationale for this change When converting a pandas.Categorical with tz-aware datetime categories to a PyArrow array, the timezone information was silently dropped from the dictionary array's value type. This is a silent data loss bug — no warning or error is raised, but the timezone metadata is lost. ### What changes are included in this PR? In `python/pyarrow/array.pxi`, the Categorical conversion was using `values.categories.values(raw numpy array) `which strips timezone metadata since numpy does not support tz-aware datetimes. Changed to values.categories (pandas Index) and added from_pandas=True so PyArrow uses the pandas conversion path, which correctly preserves timezone metadata. ### Are these changes tested? Yes. Verified manually ### Are there any user-facing changes? Yes — this is a bug fix. Users did apache#49875 This PR contains a **"Critical Fix"** — timezone information was lost silently during conversion without any warning or error. * GitHub Issue: apache#49875 Authored-by: Ankit.Ahlawat@ibm.com <Ankit.Ahlawat@ibm.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com>

apacheGH-49875:[Python] Fix timezone dropped when converting tz-aware…
dcf9a3a
… Categorical to Arrow array

AnkitAhlawat7742 requested review from AlenkaF, raulcd and rok as code owners April 28, 2026 05:44

github-actionsBot added Component: Python awaiting review Awaiting review labels Apr 28, 2026

AlenkaF reviewed May 4, 2026
View reviewed changes

apacheGH-49875: [Python] Add test for tz-aware Categorical timezone p…
fb275d0
…reservation

apacheGH-49875: [Python] Fix test to handle both ns and us datetime r…
aa65de7
…esolution

jorisvandenbossche reviewed May 4, 2026
View reviewed changes

Comment threadpython/pyarrow/tests/test_pandas.py Outdated
Comment threadpython/pyarrow/tests/test_pandas.py Outdated

jorisvandenbossche changed the title ~~GH-49875:[Python] Fix timezone dropped when converting tz-aware Categorical to Arrow array~~GH-49875: [Python] Fix timezone dropped when converting tz-aware Categorical to Arrow arrayMay 4, 2026

jorisvandenbossche reviewed May 4, 2026
View reviewed changes

Comment threadpython/pyarrow/tests/test_pandas.py Outdated

apacheGH-49875: [Python] Update the test case location as per the rev…
0f1f595
…iew comments

github-actionsBot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels May 4, 2026

AlenkaF approved these changes May 5, 2026
View reviewed changes

AnkitAhlawat7742 mentioned this pull request May 5, 2026
[CI][Python] Pandas nightly tests failing due to DatetimeIndex missmatch #49920
Closed

raulcd approved these changes May 6, 2026
View reviewed changes

github-actionsBot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels May 6, 2026

AnkitAhlawat7742 mentioned this pull request May 6, 2026
[Python] Timezone being dropped in dictionary array #49875
Closed

jorisvandenbossche approved these changes May 6, 2026
View reviewed changes

AlenkaF merged commit ea8cef5 into apache:mainMay 6, 2026
22 checks passed

AlenkaF removed the awaiting merge Awaiting merge label May 6, 2026

Uh oh!

Conversation

AnkitAhlawat7742 commented Apr 28, 2026• edited by github-actions BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Apr 28, 2026

Uh oh!

AnkitAhlawat7742 commented Apr 30, 2026

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

AnkitAhlawat7742 commented May 4, 2026

Uh oh!

AlenkaF commented May 4, 2026

Uh oh!

AnkitAhlawat7742 commented May 4, 2026

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented May 4, 2026

Uh oh!

Uh oh!

AlenkaF commented May 5, 2026

Uh oh!

github-actionsBot commented May 5, 2026

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

AlenkaF commented May 5, 2026

Uh oh!

AlenkaF commented May 6, 2026

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

conbench-apache-arrowBot commented May 6, 2026

Uh oh!

conbench-apache-arrowBot commented May 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AnkitAhlawat7742 commented Apr 28, 2026•
edited by github-actions Bot
Loading