Skip to content

Add Python version 3.13 to test matrix. - #1377

Closed
JE-Chen wants to merge 54 commits into
apache:mainfrom
JE-Chen:test_matrix
Closed

Add Python version 3.13 to test matrix.#1377
JE-Chen wants to merge 54 commits into
apache:mainfrom
JE-Chen:test_matrix

Conversation

@JE-Chen

@JE-ChenJE-Chen commented Nov 26, 2024•
edited by Fokko
Loading

Copy link
Copy Markdown
Contributor

Fixes#1372
Add Python version 3.13 to matrix.

JE-Chenand others added 2 commits November 26, 2024 09:40
@JE-Chen
Update python-ci.yml
647c710
Add Python version 3.13 to matrix.
@Fokko
Bump Poetry to 1.8.4
8fbdb47
@JE-Chen

JE-Chen commented Nov 26, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Python Rust binding maximum supported version (3.12)

error: the configured Python interpreter version (3.13) is newer than PyO3's maximum supported version (3.12)

@Fokko

Fokko commented Nov 26, 2024

Copy link
Copy Markdown
Contributor

@JE-Chen Thanks for checking. It looks like Py3O is already at 3.13: PyO3/pyo3#4636 I was hoping that maybe bumping the Poetry build might fix it #1379. Could you rebase? Otherwise we have to track which dependency is still locked at this older version of Py3O.

@JE-Chen

JE-Chen commented Nov 27, 2024

Copy link
Copy Markdown
ContributorAuthor

Done

@JE-Chen

JE-Chen commented Nov 27, 2024

Copy link
Copy Markdown
ContributorAuthor

Same error.

error: the configured Python interpreter version (3.13) is newer than PyO3's maximum supported version (3.12)

@JE-Chen

JE-Chen commented Nov 27, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Currently, we are using pyo3 v0.20.3. I think only versions of pyo3 v0.21 or newer support Python 3.13.
Some dependencies require pyo3 v0.20.3, but I don't find pyo3 listed in the Poetry requirements.

kevinjqliuand others added 12 commits November 27, 2024 15:42
@kevinjqliu
use the non-deprecated func (#1326)
06f746b
@kevinjqliu
0.8.0 post release steps (#1334)
2d4b8c8
* add
* fix mkdoc
@sumanth-manchala@kevinjqliu
Drop upper bounds for fsspec and it's implementations (#1341)
2b596a1
* Drop upper bounds for fsspec and it's implementations
* Run poetry lock
@gitzwzWenzhuo Zhao@kevinjqliu
Ignore tables without table_type from Glue and Hive
8643938
* Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (#1331)
* Use TABLE_TYPE
---------
Co-authored-by: Wenzhuo Zhao <zhaowenzhuo01@bilibili.com>
@kevinjqliu
Replace reference ofTable.identifierwithTable.name(#1346)
f8cb1c0
* fix Table.name
* replace Table.identifier with Table.name
* add warning filter
@vincenzon@kevinjqliu
Allow leading underscore in column name used in row filter (#1358)
f3323ab
* Update parser.py
Allow leading underscore in column name used in row filter.
* Update test_parser.py
* Update test_parser.py
* Update test_parser.py
@bigluck@kevinjqliu
Remove Python 3.13 upper bound restriction (#1355)
fe4375a
* Remove Python 3.13 upper bound restriction
* Fix missing poetry.lock file
* Upgrading numpy on the poetry.lock file from v1.26.0 to v1.26.4
@kevinjqliu
Improve documentation for "how to release" (#1359)
c48cf52
* initial update
* edits
* add gpg instructions
* verify artifacts
* add twine not
* grammar
* edits
* remove old artifacts
* update doc workflow action
* and name
* add docs on patch vs major/minor release
@binayakd@kevinjqliu
fix KeyError raised by add_files when parquet file doe not have c…
a89bd73
…olumn stats (#1354)
* fix KeyError, by switching del to pop
* added unit test
* update test
* fix python 3.9 compatibility, and refactor test
* update test
@kevinjqliu
Add instruction for patch release (#1373)
92f36ce
* add instruction for patch release
* create branch from tag
@Fokko@kevinjqliu
Writenullwhen there is no parent-snapshot-id (#1383)
5030d04
@kevinjqliu
bump to 0.8.1
a051584
@Fokko

Fokko commented Dec 20, 2024

Copy link
Copy Markdown
Contributor

@JE-Chen I think it pulls pyo3 in when it can't find pre-build wheels for a combination of the Python version and architecture.

@Fokko

Fokko commented Dec 20, 2024

Copy link
Copy Markdown
Contributor
 at ~/.local/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
160│ 161│ error = ChefBuildError("\n\n".join(message_parts))
162│ 163│ if error is not None:
→ 164│ raise error from None
165│ 166│ return path
167│ 168│ def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with cramjam (2.8.3) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "cramjam (==2.8.3)"'.

Looks like we're still pinning on an older version of cramjam, as 2.9.1 is out already: https://pypi.org/project/cramjam/

@FokkoFokko mentioned this pull request Dec 20, 2024
Merged
dependabotBotand others added 6 commits December 23, 2024 13:00
@dependabot@JE-Chen
Bump coverage from 7.6.4 to 7.6.5 (#1325)
553949e
Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.6.4 to 7.6.5.
- [Release notes](https://github.com/nedbat/coveragepy/releases)
- [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst)
- [Commits](coveragepy/coveragepy@7.6.4...7.6.5)
---
updated-dependencies:
- dependency-name: coverage
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot@JE-Chen
Bump mkdocstrings from 0.26.2 to 0.27.0 (#1324)
9ac0c8b
Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.26.2 to 0.27.0.
- [Release notes](https://github.com/mkdocstrings/mkdocstrings/releases)
- [Changelog](https://github.com/mkdocstrings/mkdocstrings/blob/main/CHANGELOG.md)
- [Commits](mkdocstrings/mkdocstrings@0.26.2...0.27.0)
---
updated-dependencies:
- dependency-name: mkdocstrings
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot@JE-Chen
Bump coverage from 7.6.5 to 7.6.7 (#1329)
e618f9e
Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.6.5 to 7.6.7.
- [Release notes](https://github.com/nedbat/coveragepy/releases)
- [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst)
- [Commits](coveragepy/coveragepy@7.6.5...7.6.7)
---
updated-dependencies:
- dependency-name: coverage
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot@JE-Chen
Bump aiohttp from 3.10.5 to 3.10.11 (#1338)
fef31e5
---
updated-dependencies:
- dependency-name: aiohttp
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot@JE-Chen
Bump moto from 5.0.20 to 5.0.21 (#1339)
c7262e0
Bumps [moto](https://github.com/getmoto/moto) from 5.0.20 to 5.0.21.
- [Release notes](https://github.com/getmoto/moto/releases)
- [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md)
- [Commits](getmoto/moto@5.0.20...5.0.21)
---
updated-dependencies:
- dependency-name: moto
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@hussein-awala@JE-Chen
Remove deprecateddatetimefunctions (#1134)
74b3cbd
* Remove deprecated datetime functions
* remove all usage of deprecated methods
* readd utcnow filter because of boto3
@kevinjqliu

kevinjqliu commented Jan 19, 2025

Copy link
Copy Markdown
Contributor

Blocked on Ray 3.13 ray-project/ray#49738

We can run poetry update after

@Fokko

Fokko commented Apr 2, 2025

Copy link
Copy Markdown
Contributor

It looks like there is some progress: ray-project/ray#49738 (comment)

We should also bump the upper bounds when generating the wheels: https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20%3C3.13&type=code

@FokkoFokko added this to the PyIceberg 0.10.0 milestone Apr 2, 2025
@Fokko

Fokko commented May 3, 2025

Copy link
Copy Markdown
Contributor

@JE-Chen Could you do another attempt? We've bumped to Ray 2.45, but the issue is still open 🤔

@kevinjqliu

kevinjqliu commented May 4, 2025

Copy link
Copy Markdown
Contributor

yep ray 2.45 has artifacts for python 3.13 https://pypi.org/project/ray/2.45.0/#files

@JE-Chen

JE-Chen commented May 4, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Do I need to update this list?

https://github.com/JE-Chen/iceberg-python/blob/7768850d932a700a774719d1f4613ed9dc13ddf3/poetry.lock#L4486C1-L4486C10

image

@kevinjqliu

kevinjqliu commented May 4, 2025

Copy link
Copy Markdown
Contributor

You dont need to change the content of the poetry.lock file directly, its generated by running poetry lock after modifying the pyproject.toml file.

Take a look at #1068 as an example. This is how we added 3.12 to the project

@JE-Chen

JE-Chen commented May 4, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

You dont need to change the content of the poetry.lock file directly, its generated by running poetry lock after modifying the pyproject.toml file.

Take a look at #1068 as an example. This is how we added 3.12 to the project

Thanks for the information. So, I think I need to modify pyproject.toml.

https://github.com/JE-Chen/iceberg-python/blob/7768850d932a700a774719d1f4613ed9dc13ddf3/pyproject.toml#L68

I will run pre-commit and poetry lock later.

JE-Chen added 3 commits May 5, 2025 00:25
@JE-Chen
Add new optional ray version
2f33236
Add a new optional ray version:
- If the Python version is ≥ 3.13, use a ray version that is ≥ 2.45.0.
@JE-Chen
Update pyproject.toml
42cbd38
Fix format (space)
@JE-Chen
Update poetry.lock
533acaf
After running poetry lock command
kevinjqliu
kevinjqliu reviewed May 4, 2025
View reviewed changes
Comment threadpoetry.lock Outdated
kevinjqliu
kevinjqliu reviewed May 4, 2025
View reviewed changes
Comment threadpyproject.toml
kevinjqliu
kevinjqliu reviewed May 4, 2025
View reviewed changes

@kevinjqliukevinjqliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like these are the last 2 changes we'd need to make
https://grep.app/search?f.repo.pattern=iceberg-python&q=3.13
i suggestion <3.13 -> <=3.13 so that its easier for us to change the next time around

JE-Chen added 3 commits May 5, 2025 14:51
@JE-Chen
Update poetry.lock
12e77fa
make install and re-run poetry lock
@JE-Chen
Add python 3.13 to pyproject.toml classifiers
0e717a9
Add python 3.13 to pyproject.toml classifiers:
"Programming Language :: Python :: 3.13",
@JE-Chen
Replace <3.13 with <=3.13
6f38ebd
Replace <3.13 with <=3.13
@kevinjqliu

kevinjqliu commented May 5, 2025

Copy link
Copy Markdown
Contributor

Looks like 3.13 gave us some extra warnings

ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f72ddf1ae30>
==== 52 failed, 2802 passed, 1093 deselected, 12 errors in 80.29s (0:01:20) ====

https://alexwlchan.net/til/2025/python3-13-sqlite-warnings/
pallets-eco/flask-sqlalchemy#1379
pola-rs/polars#20296

@kevinjqliukevinjqliu mentioned this pull request Oct 25, 2025
Closed
@FokkoFokko removed this from the PyIceberg 0.11.0 milestone Dec 22, 2025
@kevinjqliukevinjqliu mentioned this pull request Dec 24, 2025
Merged
Fokko pushed a commit that referenced this pull request Jan 5, 2026
@kevinjqliu@sa1
infra: Add Python 3.13 support (#2863)
dea8078
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Closes#1372, #2530
Based on #1377, #2536, #2658
Adds Python 3.13 throughout the repo
Upgrade `ray` to a version built with python 3.13
Add `filterwarnings` for `ResourceWarnings` coming from `sqlite` and
`ray` for now, will follow up with a proper fix
## Are these changes tested?
Yes
```
PYTHON=3.13 make install
PYTHON=3.13 make test
PYTHON=3.13 make test-integration
```
## Are there any user-facing changes?
<!-- In the case of user-facing changes, please add the changelog label.
-->
---------
Co-authored-by: Saurabh Kumar <github@sa1.me>
Co-authored-by: Saurabh Kumar <557509+sa1@users.noreply.github.com>
@Fokko

Fokko commented Jan 5, 2026

Copy link
Copy Markdown
Contributor

Superseded by #2863

@FokkoFokko closed this Jan 5, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kevinjqliukevinjqliukevinjqliu left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Python 3.13 to the test matrix

10 participants

@JE-Chen@Fokko@kevinjqliu@sumanth-manchala@gitzwz@vincenzon@bigluck@binayakd@hussein-awala@mark-major