GH-37803: [CI][Dev][Python] Release and merge script errors

[AlenkaF]

What changes are included in this PR?

Pin the version of setuptools_scm to <8.0.0.

• Closes: [CI][Dev][Python] Release and merge script errors #37803

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.10

[raulcd]

@AlenkaF is this still a draft? I am fine with merging this as is to fix the merge script + archery crossbow tasks, etcetera and maybe opening an issue to follow up to either fix it for setuptools_scm > 8.0.0 or change how we manage the version.

[github-actions]

parse() missing 1 required positional argument: 'config'
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6259636256

[AlenkaF]

@AlenkaF is this still a draft? I am fine with merging this as is to fix the merge script + archery crossbow tasks, etcetera and maybe opening an issue to follow up to either fix it for setuptools_scm > 8.0.0 or change how we manage the version.

There are two files where I didn't pin setuptools_scm so I need to run the tests first, to see if they also need to be pinned. Then I will merge 👍

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.10

[github-actions]

parse() missing 1 required positional argument: 'config'
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6259681246

[raulcd]

There are two files where I didn't pin setuptools_scm so I need to run the tests first, to see if they also need to be pinned. Then I will merge 👍

Sorry you are correct, we have to pin it on archery too: https://github.com/apache/arrow/blob/main/dev/archery/setup.py#L33

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.10

[github-actions]

parse() missing 1 required positional argument: 'config'
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6259766715

[raulcd]

I just validated we do the checkout from apache/arrow from main for the comment bot. The crossbow task won't work until we merge :(

[AlenkaF]

I just validated we do the checkout from apache/arrow from main for the comment bot. The crossbow task won't work until we merge :(

Got it. Let's give it a go 👍

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 79e49db.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

[jorisvandenbossche]

This might already be fixed in the meantime: pypa/setuptools-scm#919 (the same error message is mentioned on that issue, but didn't check if it actually fixed it)

[cwegener]

opening an issue to follow up to either fix it for setuptools_scm > 8.0.0 or change how we manage the version.

A permanent fix to make pyarrow build on latest setuptools + setuptools_scm would be highly desirable. I had to resort to the exact same version constraint myself in an emergency fix to unblock some automated build system failures.

[jorisvandenbossche]

@cwegener I am not sure there is something to fix on our side. As far as I understand, it's a regression in setuptools_scm that will be fixed there (pypa/setuptools-scm#925, pypa/setuptools-scm#935). Ideally we would update old releases to add this upper pin for setuptools_scm in the meantime. But our release process is quite heavy, and hopefully a release of setuptools_scm fixing this issue will be released more quickly.

Now, it seems that nowadays it is recommended to configure setuptools_scm throught pyproject.toml instead of use_scm_version in setup() in setup.py. So that's something that we should update on our side (but that's also something we will only do for a future release, I think)

[jorisvandenbossche]

I opened a new issue about the problem of having to upper pin setuptools_scm right now with existing releases, so we have an open issue about this that users can find if they run into this -> #37931

And also opened an issue to track that we should move our setuptools_scm configuration to pyproject.toml -> #37929

[cwegener]

@jorisvandenbossche

Thanks so much for the detailed explanation!

I was in a rush to fix the breakage and the huge amount of code in setup.py, plus the distractions of there also being a pyproject.toml that doesn't seem to be used yet was causing a lot of confusion for me.