Add description on the vendoring process we use

[potiuk]

We need to vendor in cgroupspy library in order to make Airflow
compatible with Python 3.10 (see #22050) so this is the right time
to make our vendoring process more organized.

I based in parts on the readme described by bleach package.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Some ideas from how pip does this. pip fully automates the vendoring-patching process so a new version can be pulled immediately if needed. It uses vendoring for this. It takes a requirements file (a la requiurements.txt) and diff files for patches. When invoked, it downloads and extracts the package and applies the patches in the vendor directory (specified in pyproject.toml).

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

[potiuk]

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

[potiuk]

The Fix to cgroupspy submitted: cloudsigma/cgroupspy#14 - but the last change was 7 months ago, so I am not sure how fast it might get merged/released.

[potiuk]

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

I tihnk when we have more than one vendored in dependencies with significant changes and active development of the vendored-library (and when we see that such vendoring-in might be needed for at least few releases - we should consider automating it - in the case we have now with cgroupspy - I can't imagine they release new version without accepting the single abc import change, so I think any automation around it is a waste :)

[potiuk]

First in-line to get finally Python 3.10 support :)

[potiuk]

Hey @kaxil I cherry-picked your process description fixes from #22209 and added more description on the "no-compile".

[potiuk]

Hey @kaxil -> can we merge it ? I hope to start merging the series of PRs: #22204, #22206, #22207, #22208, #22209 leading to Python 3.10 compatibility on Monay (hopefully) when I releae all the provider packages.

[potiuk]

🙏

[potiuk]

I rebased on top of latest changes - I need those #22204, #22206, #22207, #22208, #22209 to complete Python 3.10 support

Looking for approvals on all of those (they are slal small)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

We need to vendor in cgroupspy library in order to make Airflow
compatible with Python 3.10 (see #22050) so this is the right time
to make our vendoring process more organized.

I based in parts on the readme described by bleach package.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Some ideas from how pip does this. pip fully automates the vendoring-patching process so a new version can be pulled immediately if needed. It uses vendoring for this. It takes a requirements file (a la requiurements.txt) and diff files for patches. When invoked, it downloads and extracts the package and applies the patches in the vendor directory (specified in pyproject.toml).

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

[potiuk]

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

[potiuk]

The Fix to cgroupspy submitted: cloudsigma/cgroupspy#14 - but the last change was 7 months ago, so I am not sure how fast it might get merged/released.

[potiuk]

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

I tihnk when we have more than one vendored in dependencies with significant changes and active development of the vendored-library (and when we see that such vendoring-in might be needed for at least few releases - we should consider automating it - in the case we have now with cgroupspy - I can't imagine they release new version without accepting the single abc import change, so I think any automation around it is a waste :)

[potiuk]

First in-line to get finally Python 3.10 support :)

[potiuk]

Hey @kaxil I cherry-picked your process description fixes from #22209 and added more description on the "no-compile".

[potiuk]

Hey @kaxil -> can we merge it ? I hope to start merging the series of PRs: #22204, #22206, #22207, #22208, #22209 leading to Python 3.10 compatibility on Monay (hopefully) when I releae all the provider packages.

[potiuk]

🙏

[potiuk]

I rebased on top of latest changes - I need those #22204, #22206, #22207, #22208, #22209 to complete Python 3.10 support

Looking for approvals on all of those (they are slal small)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

We need to vendor in cgroupspy library in order to make Airflow
compatible with Python 3.10 (see #22050) so this is the right time
to make our vendoring process more organized.

I based in parts on the readme described by bleach package.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Some ideas from how pip does this. pip fully automates the vendoring-patching process so a new version can be pulled immediately if needed. It uses vendoring for this. It takes a requirements file (a la requiurements.txt) and diff files for patches. When invoked, it downloads and extracts the package and applies the patches in the vendor directory (specified in pyproject.toml).

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

[potiuk]

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

[potiuk]

The Fix to cgroupspy submitted: cloudsigma/cgroupspy#14 - but the last change was 7 months ago, so I am not sure how fast it might get merged/released.

[potiuk]

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

I tihnk when we have more than one vendored in dependencies with significant changes and active development of the vendored-library (and when we see that such vendoring-in might be needed for at least few releases - we should consider automating it - in the case we have now with cgroupspy - I can't imagine they release new version without accepting the single abc import change, so I think any automation around it is a waste :)

[potiuk]

First in-line to get finally Python 3.10 support :)

[potiuk]

Hey @kaxil I cherry-picked your process description fixes from #22209 and added more description on the "no-compile".

[potiuk]

Hey @kaxil -> can we merge it ? I hope to start merging the series of PRs: #22204, #22206, #22207, #22208, #22209 leading to Python 3.10 compatibility on Monay (hopefully) when I releae all the provider packages.

[potiuk]

🙏

[potiuk]

I rebased on top of latest changes - I need those #22204, #22206, #22207, #22208, #22209 to complete Python 3.10 support

Looking for approvals on all of those (they are slal small)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

We need to vendor in cgroupspy library in order to make Airflow
compatible with Python 3.10 (see #22050) so this is the right time
to make our vendoring process more organized.

I based in parts on the readme described by bleach package.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Some ideas from how pip does this. pip fully automates the vendoring-patching process so a new version can be pulled immediately if needed. It uses vendoring for this. It takes a requirements file (a la requiurements.txt) and diff files for patches. When invoked, it downloads and extracts the package and applies the patches in the vendor directory (specified in pyproject.toml).

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

[potiuk]

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

[potiuk]

The Fix to cgroupspy submitted: cloudsigma/cgroupspy#14 - but the last change was 7 months ago, so I am not sure how fast it might get merged/released.

[potiuk]

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

I tihnk when we have more than one vendored in dependencies with significant changes and active development of the vendored-library (and when we see that such vendoring-in might be needed for at least few releases - we should consider automating it - in the case we have now with cgroupspy - I can't imagine they release new version without accepting the single abc import change, so I think any automation around it is a waste :)

[potiuk]

First in-line to get finally Python 3.10 support :)

[potiuk]

Hey @kaxil I cherry-picked your process description fixes from #22209 and added more description on the "no-compile".

[potiuk]

Hey @kaxil -> can we merge it ? I hope to start merging the series of PRs: #22204, #22206, #22207, #22208, #22209 leading to Python 3.10 compatibility on Monay (hopefully) when I releae all the provider packages.

[potiuk]

🙏

[potiuk]

I rebased on top of latest changes - I need those #22204, #22206, #22207, #22208, #22209 to complete Python 3.10 support

Looking for approvals on all of those (they are slal small)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

We need to vendor in cgroupspy library in order to make Airflow
compatible with Python 3.10 (see #22050) so this is the right time
to make our vendoring process more organized.

I based in parts on the readme described by bleach package.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Some ideas from how pip does this. pip fully automates the vendoring-patching process so a new version can be pulled immediately if needed. It uses vendoring for this. It takes a requirements file (a la requiurements.txt) and diff files for patches. When invoked, it downloads and extracts the package and applies the patches in the vendor directory (specified in pyproject.toml).

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

[potiuk]

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

[potiuk]

The Fix to cgroupspy submitted: cloudsigma/cgroupspy#14 - but the last change was 7 months ago, so I am not sure how fast it might get merged/released.

[potiuk]

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

I tihnk when we have more than one vendored in dependencies with significant changes and active development of the vendored-library (and when we see that such vendoring-in might be needed for at least few releases - we should consider automating it - in the case we have now with cgroupspy - I can't imagine they release new version without accepting the single abc import change, so I think any automation around it is a waste :)

[potiuk]

First in-line to get finally Python 3.10 support :)

[potiuk]

Hey @kaxil I cherry-picked your process description fixes from #22209 and added more description on the "no-compile".

[potiuk]

Hey @kaxil -> can we merge it ? I hope to start merging the series of PRs: #22204, #22206, #22207, #22208, #22209 leading to Python 3.10 compatibility on Monay (hopefully) when I releae all the provider packages.

[potiuk]

🙏

[potiuk]

I rebased on top of latest changes - I need those #22204, #22206, #22207, #22208, #22209 to complete Python 3.10 support

Looking for approvals on all of those (they are slal small)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

We need to vendor in cgroupspy library in order to make Airflow
compatible with Python 3.10 (see #22050) so this is the right time
to make our vendoring process more organized.

I based in parts on the readme described by bleach package.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Some ideas from how pip does this. pip fully automates the vendoring-patching process so a new version can be pulled immediately if needed. It uses vendoring for this. It takes a requirements file (a la requiurements.txt) and diff files for patches. When invoked, it downloads and extracts the package and applies the patches in the vendor directory (specified in pyproject.toml).

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

[potiuk]

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

[potiuk]

The Fix to cgroupspy submitted: cloudsigma/cgroupspy#14 - but the last change was 7 months ago, so I am not sure how fast it might get merged/released.

[potiuk]

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

I tihnk when we have more than one vendored in dependencies with significant changes and active development of the vendored-library (and when we see that such vendoring-in might be needed for at least few releases - we should consider automating it - in the case we have now with cgroupspy - I can't imagine they release new version without accepting the single abc import change, so I think any automation around it is a waste :)

[potiuk]

First in-line to get finally Python 3.10 support :)

[potiuk]

Hey @kaxil I cherry-picked your process description fixes from #22209 and added more description on the "no-compile".

[potiuk]

Hey @kaxil -> can we merge it ? I hope to start merging the series of PRs: #22204, #22206, #22207, #22208, #22209 leading to Python 3.10 compatibility on Monay (hopefully) when I releae all the provider packages.

[potiuk]

🙏

[potiuk]

I rebased on top of latest changes - I need those #22204, #22206, #22207, #22208, #22209 to complete Python 3.10 support

Looking for approvals on all of those (they are slal small)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

We need to vendor in cgroupspy library in order to make Airflow
compatible with Python 3.10 (see #22050) so this is the right time
to make our vendoring process more organized.

I based in parts on the readme described by bleach package.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Some ideas from how pip does this. pip fully automates the vendoring-patching process so a new version can be pulled immediately if needed. It uses vendoring for this. It takes a requirements file (a la requiurements.txt) and diff files for patches. When invoked, it downloads and extracts the package and applies the patches in the vendor directory (specified in pyproject.toml).

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

[potiuk]

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

[potiuk]

The Fix to cgroupspy submitted: cloudsigma/cgroupspy#14 - but the last change was 7 months ago, so I am not sure how fast it might get merged/released.

[potiuk]

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

I tihnk when we have more than one vendored in dependencies with significant changes and active development of the vendored-library (and when we see that such vendoring-in might be needed for at least few releases - we should consider automating it - in the case we have now with cgroupspy - I can't imagine they release new version without accepting the single abc import change, so I think any automation around it is a waste :)

[potiuk]

First in-line to get finally Python 3.10 support :)

[potiuk]

Hey @kaxil I cherry-picked your process description fixes from #22209 and added more description on the "no-compile".

[potiuk]

Hey @kaxil -> can we merge it ? I hope to start merging the series of PRs: #22204, #22206, #22207, #22208, #22209 leading to Python 3.10 compatibility on Monay (hopefully) when I releae all the provider packages.

[potiuk]

🙏

[potiuk]

I rebased on top of latest changes - I need those #22204, #22206, #22207, #22208, #22209 to complete Python 3.10 support

Looking for approvals on all of those (they are slal small)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

We need to vendor in cgroupspy library in order to make Airflow
compatible with Python 3.10 (see #22050) so this is the right time
to make our vendoring process more organized.

I based in parts on the readme described by bleach package.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Some ideas from how pip does this. pip fully automates the vendoring-patching process so a new version can be pulled immediately if needed. It uses vendoring for this. It takes a requirements file (a la requiurements.txt) and diff files for patches. When invoked, it downloads and extracts the package and applies the patches in the vendor directory (specified in pyproject.toml).

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

[potiuk]

Not sure if Airflow needs this much infrastructure (we only vendor two? dependencies and don’t generally upgrade the existing one), but could be useful if we want to go this route.

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

[potiuk]

The Fix to cgroupspy submitted: cloudsigma/cgroupspy#14 - but the last change was 7 months ago, so I am not sure how fast it might get merged/released.

[potiuk]

Yeah. I considered it too, but let's say we might want to do it when we see if it is needed. For now we have 0 vendored-in dependencies and and the aim is to remove them asap - likely with next release (I am going to submit a PR to fix it in cgroupspy in a moment).

I tihnk when we have more than one vendored in dependencies with significant changes and active development of the vendored-library (and when we see that such vendoring-in might be needed for at least few releases - we should consider automating it - in the case we have now with cgroupspy - I can't imagine they release new version without accepting the single abc import change, so I think any automation around it is a waste :)

[potiuk]

First in-line to get finally Python 3.10 support :)

[potiuk]

Hey @kaxil I cherry-picked your process description fixes from #22209 and added more description on the "no-compile".

[potiuk]

Hey @kaxil -> can we merge it ? I hope to start merging the series of PRs: #22204, #22206, #22207, #22208, #22209 leading to Python 3.10 compatibility on Monay (hopefully) when I releae all the provider packages.

[potiuk]

🙏

[potiuk]

I rebased on top of latest changes - I need those #22204, #22206, #22207, #22208, #22209 to complete Python 3.10 support

Looking for approvals on all of those (they are slal small)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.