Skip to content

Move absolute path finder from open_mfdataset to own function - #7968

Merged
Illviljan merged 9 commits into
pydata:mainfrom
Illviljan:path_finder
Jul 10, 2023
Merged

Move absolute path finder from open_mfdataset to own function#7968
Illviljan merged 9 commits into
pydata:mainfrom
Illviljan:path_finder

Conversation

@Illviljan

@IllviljanIllviljan commented Jul 9, 2023

Copy link
Copy Markdown
Contributor

A simple refactor to make it easier to retrieve the proper paths that open_mfdataset uses and passes on the engine.

I've been thinking how to make use of DataTree and one idea I wanted to try was:

  • Open file (using _find_absolute_path).
  • Get all groups in the file.
  • For each group run xr.open_mfdataset(..., group=group)

@IllviljanIllviljan changed the title Move absolute path finder to own functionMove absolute path finder from open_mfdataset to own functionJul 9, 2023
@Illviljan
Illviljan marked this pull request as ready for review July 9, 2023 15:16
Comment threadxarray/backends/common.py Outdated
else:
paths = sorted(glob(_normalize_path(paths)))
elif isinstance(paths, os.PathLike):
paths = [os.fspath(paths)]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that this is just moved from the previous location, but is there a reason why we don't convert to absolute path here and below?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, it's a little odd to me as well.

I haven't really felt the need for os.PathLike in my normal usage so I don't have much experience with that part though.

Co-authored-by: Michael Niklas <mick.niklas@gmail.com>
@IllviljanIllviljan added the plan to merge Final call for comments label Jul 9, 2023
@TomNicholas

Copy link
Copy Markdown
Member

@Illviljan see also xarray-contrib/datatree#51

@Illviljan

Copy link
Copy Markdown
ContributorAuthor

@TomNicholas I've found the current open_datatree implementations too focused on zarr and netcdf. Which is fine, it's the most popular after all.
But I want to make use of custom engines so I'll play around on my side for a while.

@Illviljan
Illviljan merged commit 554285b into pydata:mainJul 10, 2023
dcherian added a commit to dcherian/xarray that referenced this pull request Jul 16, 2023
* main:
Remove hue_style from plot1d docstring (pydata#7925)
Add new what's new section (pydata#7986)
Release summary for v2023.07.0 (pydata#7979)
Improve explanation in example "Working with Multidimensional Coordinates" (pydata#7984)
Fix typo in zarr.py (pydata#7983)
Examples added to docstrings (pydata#7936)
[pre-commit.ci] pre-commit autoupdate (pydata#7973)
Skip broken tests on python 3.11 and windows (pydata#7972)
Use another repository for upstream testing (pydata#7970)
Move absolute path finder from open_mfdataset to own function (pydata#7968)
ensure no forward slashes in names for HDF5-based backends (pydata#7953)
Chunked array docs (pydata#7951)
[pre-commit.ci] pre-commit autoupdate (pydata#7959)
manually unshallow the repository on RTD (pydata#7961)
Update minimum version of typing extensions in pre-commit (pydata#7960)
Docstring examples (pydata#7881)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ioplan to mergeFinal call for commentstopic-backends

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Illviljan@TomNicholas@headtr1ck