GH-42111: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock

[amoeba]

Rationale for this change

See #42111.

What changes are included in this PR?

• Sync'd config files from arrow-cpp-feedstock
• Sync'd auxiliary files
• Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

[amoeba]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 6ce51d530139825ec1e73f98f97e0807ce681bde

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[amoeba]

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

[amoeba]

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

[h-vetinari]

So in general, what you'd need to do is:

• sync the recipe contents (meta.yaml plus all relevant build & test scripts), minus the patches and keep pointing to the local sources
  • https://github.com/conda-forge/arrow-cpp-feedstock
  • https://github.com/conda-forge/pyarrow-feedstock (new!)
  • https://github.com/conda-forge/r-arrow-feedstock (currently broken on windows, unfortunately)
• sync the respective variant configs
  • per feedstock; e.g. from here for arrow-cpp
  • ensure that the file names in tasks.yaml are updated where necessary
• (potentially) sync changes from build scripts (example)
  • again per feedstock
  • this is the trickiest part, because there are local changes to make arrow CI work that AFAIU aren't documented anywhere; it basically needs careful study of the diff, and deciding which changes to take or not. Fortunately, this step is rarely if ever necessary.

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637#34652#14832#14102

[amoeba]

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

[raulcd]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[raulcd]

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

[jonkeane]

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[amoeba]

Rationale for this change

See #42111.

What changes are included in this PR?

• Sync'd config files from arrow-cpp-feedstock
• Sync'd auxiliary files
• Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

[amoeba]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 6ce51d530139825ec1e73f98f97e0807ce681bde

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[amoeba]

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

[amoeba]

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

[h-vetinari]

So in general, what you'd need to do is:

• sync the recipe contents (meta.yaml plus all relevant build & test scripts), minus the patches and keep pointing to the local sources
  • https://github.com/conda-forge/arrow-cpp-feedstock
  • https://github.com/conda-forge/pyarrow-feedstock (new!)
  • https://github.com/conda-forge/r-arrow-feedstock (currently broken on windows, unfortunately)
• sync the respective variant configs
  • per feedstock; e.g. from here for arrow-cpp
  • ensure that the file names in tasks.yaml are updated where necessary
• (potentially) sync changes from build scripts (example)
  • again per feedstock
  • this is the trickiest part, because there are local changes to make arrow CI work that AFAIU aren't documented anywhere; it basically needs careful study of the diff, and deciding which changes to take or not. Fortunately, this step is rarely if ever necessary.

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637#34652#14832#14102

[amoeba]

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

[raulcd]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[raulcd]

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

[jonkeane]

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[amoeba]

Rationale for this change

See #42111.

What changes are included in this PR?

• Sync'd config files from arrow-cpp-feedstock
• Sync'd auxiliary files
• Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

[amoeba]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 6ce51d530139825ec1e73f98f97e0807ce681bde

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[amoeba]

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

[amoeba]

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

[h-vetinari]

So in general, what you'd need to do is:

• sync the recipe contents (meta.yaml plus all relevant build & test scripts), minus the patches and keep pointing to the local sources
  • https://github.com/conda-forge/arrow-cpp-feedstock
  • https://github.com/conda-forge/pyarrow-feedstock (new!)
  • https://github.com/conda-forge/r-arrow-feedstock (currently broken on windows, unfortunately)
• sync the respective variant configs
  • per feedstock; e.g. from here for arrow-cpp
  • ensure that the file names in tasks.yaml are updated where necessary
• (potentially) sync changes from build scripts (example)
  • again per feedstock
  • this is the trickiest part, because there are local changes to make arrow CI work that AFAIU aren't documented anywhere; it basically needs careful study of the diff, and deciding which changes to take or not. Fortunately, this step is rarely if ever necessary.

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637#34652#14832#14102

[amoeba]

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

[raulcd]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[raulcd]

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

[jonkeane]

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[amoeba]

Rationale for this change

See #42111.

What changes are included in this PR?

• Sync'd config files from arrow-cpp-feedstock
• Sync'd auxiliary files
• Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

[amoeba]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 6ce51d530139825ec1e73f98f97e0807ce681bde

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[amoeba]

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

[amoeba]

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

[h-vetinari]

So in general, what you'd need to do is:

• sync the recipe contents (meta.yaml plus all relevant build & test scripts), minus the patches and keep pointing to the local sources
  • https://github.com/conda-forge/arrow-cpp-feedstock
  • https://github.com/conda-forge/pyarrow-feedstock (new!)
  • https://github.com/conda-forge/r-arrow-feedstock (currently broken on windows, unfortunately)
• sync the respective variant configs
  • per feedstock; e.g. from here for arrow-cpp
  • ensure that the file names in tasks.yaml are updated where necessary
• (potentially) sync changes from build scripts (example)
  • again per feedstock
  • this is the trickiest part, because there are local changes to make arrow CI work that AFAIU aren't documented anywhere; it basically needs careful study of the diff, and deciding which changes to take or not. Fortunately, this step is rarely if ever necessary.

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637#34652#14832#14102

[amoeba]

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

[raulcd]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[raulcd]

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

[jonkeane]

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[amoeba]

Rationale for this change

See #42111.

What changes are included in this PR?

• Sync'd config files from arrow-cpp-feedstock
• Sync'd auxiliary files
• Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

[amoeba]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 6ce51d530139825ec1e73f98f97e0807ce681bde

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[amoeba]

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

[amoeba]

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

[h-vetinari]

So in general, what you'd need to do is:

• sync the recipe contents (meta.yaml plus all relevant build & test scripts), minus the patches and keep pointing to the local sources
  • https://github.com/conda-forge/arrow-cpp-feedstock
  • https://github.com/conda-forge/pyarrow-feedstock (new!)
  • https://github.com/conda-forge/r-arrow-feedstock (currently broken on windows, unfortunately)
• sync the respective variant configs
  • per feedstock; e.g. from here for arrow-cpp
  • ensure that the file names in tasks.yaml are updated where necessary
• (potentially) sync changes from build scripts (example)
  • again per feedstock
  • this is the trickiest part, because there are local changes to make arrow CI work that AFAIU aren't documented anywhere; it basically needs careful study of the diff, and deciding which changes to take or not. Fortunately, this step is rarely if ever necessary.

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637#34652#14832#14102

[amoeba]

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

[raulcd]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[raulcd]

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

[jonkeane]

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[amoeba]

Rationale for this change

See #42111.

What changes are included in this PR?

• Sync'd config files from arrow-cpp-feedstock
• Sync'd auxiliary files
• Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

[amoeba]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 6ce51d530139825ec1e73f98f97e0807ce681bde

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[amoeba]

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

[amoeba]

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

[h-vetinari]

So in general, what you'd need to do is:

• sync the recipe contents (meta.yaml plus all relevant build & test scripts), minus the patches and keep pointing to the local sources
  • https://github.com/conda-forge/arrow-cpp-feedstock
  • https://github.com/conda-forge/pyarrow-feedstock (new!)
  • https://github.com/conda-forge/r-arrow-feedstock (currently broken on windows, unfortunately)
• sync the respective variant configs
  • per feedstock; e.g. from here for arrow-cpp
  • ensure that the file names in tasks.yaml are updated where necessary
• (potentially) sync changes from build scripts (example)
  • again per feedstock
  • this is the trickiest part, because there are local changes to make arrow CI work that AFAIU aren't documented anywhere; it basically needs careful study of the diff, and deciding which changes to take or not. Fortunately, this step is rarely if ever necessary.

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637#34652#14832#14102

[amoeba]

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

[raulcd]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[raulcd]

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

[jonkeane]

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[amoeba]

Rationale for this change

See #42111.

What changes are included in this PR?

• Sync'd config files from arrow-cpp-feedstock
• Sync'd auxiliary files
• Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

[amoeba]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 6ce51d530139825ec1e73f98f97e0807ce681bde

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[amoeba]

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

[amoeba]

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

[h-vetinari]

So in general, what you'd need to do is:

• sync the recipe contents (meta.yaml plus all relevant build & test scripts), minus the patches and keep pointing to the local sources
  • https://github.com/conda-forge/arrow-cpp-feedstock
  • https://github.com/conda-forge/pyarrow-feedstock (new!)
  • https://github.com/conda-forge/r-arrow-feedstock (currently broken on windows, unfortunately)
• sync the respective variant configs
  • per feedstock; e.g. from here for arrow-cpp
  • ensure that the file names in tasks.yaml are updated where necessary
• (potentially) sync changes from build scripts (example)
  • again per feedstock
  • this is the trickiest part, because there are local changes to make arrow CI work that AFAIU aren't documented anywhere; it basically needs careful study of the diff, and deciding which changes to take or not. Fortunately, this step is rarely if ever necessary.

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637#34652#14832#14102

[amoeba]

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

[raulcd]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[raulcd]

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

[jonkeane]

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[amoeba]

Rationale for this change

See #42111.

What changes are included in this PR?

• Sync'd config files from arrow-cpp-feedstock
• Sync'd auxiliary files
• Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

[amoeba]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 6ce51d530139825ec1e73f98f97e0807ce681bde

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[amoeba]

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

[amoeba]

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

[h-vetinari]

So in general, what you'd need to do is:

• sync the recipe contents (meta.yaml plus all relevant build & test scripts), minus the patches and keep pointing to the local sources
  • https://github.com/conda-forge/arrow-cpp-feedstock
  • https://github.com/conda-forge/pyarrow-feedstock (new!)
  • https://github.com/conda-forge/r-arrow-feedstock (currently broken on windows, unfortunately)
• sync the respective variant configs
  • per feedstock; e.g. from here for arrow-cpp
  • ensure that the file names in tasks.yaml are updated where necessary
• (potentially) sync changes from build scripts (example)
  • again per feedstock
  • this is the trickiest part, because there are local changes to make arrow CI work that AFAIU aren't documented anywhere; it basically needs careful study of the diff, and deciding which changes to take or not. Fortunately, this step is rarely if ever necessary.

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637#34652#14832#14102

[amoeba]

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

[raulcd]

@github-actions crossbow submit -g conda

[github-actions]

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-x64-cpu-py3
conda-win-x64-cpu-py3
conda-win-x64-cuda-py3

[raulcd]

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

[jonkeane]

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.