Skip to content

(fix): pandas extension array repr for int64[pyarrow] - #10317

Merged
dcherian merged 12 commits into
pydata:mainfrom
ilan-gold:ig/fix_repr
May 29, 2025
Merged

(fix): pandas extension array repr for int64[pyarrow]#10317
dcherian merged 12 commits into
pydata:mainfrom
ilan-gold:ig/fix_repr

Conversation

@ilan-gold

@ilan-goldilan-gold commented May 13, 2025

Copy link
Copy Markdown
Contributor

Similar to #10315, we uncover more unexpected behavior with int64[pyarrow] as the testing data type.

  • We were apparently relying on a call to np.array (via the xp.reshape) internally for the actual printing of the elements but this is not implemented for int64[pyarrow] (unlike other extension array types) and therefore broke without an explicit implementation. So we add that implementation

  • Once this has been handled, the PandasExtensionArray is no longer being converted into a numpy array silently, and thus we need to ensure that the repr can handle the PandasExtensionArray natively.

  • Lastly we add a KeyError to the lookup to ensure that this sort of thing cannot happen again.

  • From (fix): no fill_value on reindex #10304

  • Tests added

  • User visible changes (including notable bug fixes) are documented in whats-new.rst

  • New functions/methods are listed in api.rst

@ilan-gold

Copy link
Copy Markdown
ContributorAuthor

Separately, I am not sure if max_width was meant to be inclusive or exclusive. The line length for the IntervalArray is exactly 100.

@ilan-gold

Copy link
Copy Markdown
ContributorAuthor

Reverted the addition of IntervalArraya23de74 - see failure at https://github.com/pydata/xarray/actions/runs/15000671362/job/42146523169 for test_repr where the line length is exactly 100 instead of less than 100. I'm not clear what the behavior should be but let's keep this as minimal as possible. I will follow up with a more general "more test cases" PR.

Comment threadxarray/tests/test_dataset.py Outdated
Comment threadxarray/tests/test_dataset.py
@ilan-gold

Copy link
Copy Markdown
ContributorAuthor

Is it possible these three tests are flaky? It seems to pass locally for me and on the face of it, has nothing to do with this PR>

@kmuehlbauer

Copy link
Copy Markdown
Contributor

Is it possible these three tests are flaky? It seems to pass locally for me and on the face of it, has nothing to do with this PR>

@ilan-gold This surfaced with the release of dask 2025.5.0. Something has changed on the dask side and/or on the xarray side, which let the error surface.

Here is the stacktrace on my machine:

xarray/tests/test_backends.py:827 (TestDask.test_outer_indexing_reversed)
self=<xarray.tests.test_backends.TestDaskobjectat0x7f2da44c2ff0>deftest_outer_indexing_reversed(self) ->None:
# regression test for GH6560ds=xr.Dataset(
{"z": (("t", "p", "y", "x"), np.ones((1, 1, 31, 40)))},
)
withself.roundtrip(ds) ason_disk:
>subset=on_disk.isel(t=[0], p=0).z[:, ::10, ::10][:, ::-1, :]
/home/kai/python/projects/xarray/xarray/tests/test_backends.py:835: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /home/kai/python/projects/xarray/xarray/core/dataset.py:2764: iniselvar=var.isel(var_indexers)
/home/kai/python/projects/xarray/xarray/core/variable.py:1032: iniselreturnself[key]
/home/kai/python/projects/xarray/xarray/core/variable.py:782: in__getitem__returnself._finalize_indexing_result(dims, data)
/home/kai/python/projects/xarray/xarray/core/variable.py:786: in_finalize_indexing_resultreturnself._replace(dims=dims, data=data)
/home/kai/python/projects/xarray/xarray/core/variable.py:938: in_replacereturntype(self)(dims, data, attrs, encoding, fastpath=True)
/home/kai/python/projects/xarray/xarray/core/variable.py:365: in__init__super().__init__(
/home/kai/python/projects/xarray/xarray/namedarray/core.py:264: in__init__self._dims=self._parse_dimensions(dims)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self=<[AttributeError("'Variable' object has no attribute '_dims'") raisedinrepr()] Variableobjectat0x7f2d5c6c6380>dims= ('t', 'y', 'x')
def_parse_dimensions(self, dims: _DimsLike) ->_Dims:
dims= (dims,) ifisinstance(dims, str) elsetuple(dims)
iflen(dims) !=self.ndim:
>raiseValueError(
f"dimensions {dims} must have the same length as the "f"number of data dimensions, ndim={self.ndim}"
)
EValueError: dimensions ('t', 'y', 'x') musthavethesamelengthasthenumberofdatadimensions, ndim=4/home/kai/python/projects/xarray/xarray/namedarray/core.py:508: ValueError

@ilan-gold

Copy link
Copy Markdown
ContributorAuthor

@kmuehlbauer Thanks for the heads up, sounds like I need to update my env then. Thanks for the info here!

@kmuehlbauer

Copy link
Copy Markdown
Contributor

@kmuehlbauer Thanks for the heads up, sounds like I need to update my env then. Thanks for the info here!

See new issue #10321.

@ilan-gold
ilan-gold requested a review from dcherianMay 22, 2025 14:13
Comment threadxarray/core/formatting.py
Comment threadxarray/core/formatting.py
@dcherian
dcherian merged commit 3cbf960 into pydata:mainMay 29, 2025
dcherian added a commit to dcherian/xarray that referenced this pull request May 30, 2025
* main:
Fix performance regression in interp from pydata#9881 (pydata#10370)
html repr: improve style for dropdown sections (pydata#10354)
Grouper tweaks. (pydata#10362)
Docs: Add links to getting help mermaid diagram (pydata#10324)
Enforce ruff/flynt rules (FLY) (pydata#10375)
Add missing AbstractWritableDataStore base methods and arguments (pydata#10343)
Improve html repr in dark mode (Jupyterlab + Xarray docs) (pydata#10353)
Pin Mypy to 1.15 (pydata#10378)
use numpy dtype exposed by zarr array instead of metadata.data_type (pydata#10348)
Fix doc typo for caption "Interoperability" (pydata#10374)
Implement cftime vectorization as discussed in PR pydata#8322 (pydata#8324)
Enforce ruff/flake8-pyi rules (PYI) (pydata#10359)
Apply assorted ruff/Pylint rules (PL) / Enforce PLE rules (pydata#10366)
(fix): pandas extension array repr for int64[pyarrow] (pydata#10317)
Enforce ruff/flake8-implicit-str-concat rules (ISC) (pydata#10368)
Enforce ruff/refurb rules (FURB) (pydata#10367)
Ignore ruff/Pyflakes rule F401 more precisely (pydata#10369)
Apply assorted ruff/flake8-simplify rules (SIM) (pydata#10364)
Apply assorted ruff/flake8-pytest-style rules (PT) (pydata#10363)
Fix "a array" misspelling (pydata#10365)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@ilan-gold@kmuehlbauer@dcherian