Help pip resolver to make better decisions on Pyarrow version by potiuk · Pull Request #25697 · apache/airflow

potiuk · 2022-08-12T17:25:16Z

The pip resolver in eager upgrade mode on Python 3.10 decides
to downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.

Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.

potiuk · 2022-08-12T18:00:24Z

In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4

uranusjr · 2022-08-17T14:57:58Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

potiuk · 2022-08-17T16:02:16Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ?

alexott · 2022-08-17T16:03:16Z

Ok, let me point team to it

susodapop · 2022-08-17T19:23:47Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

…apache#25697)" This reverts commit 1b84048.

potiuk · 2022-08-17T19:54:33Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good.

potiuk · 2022-08-17T20:16:06Z

Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :)

…#25697)" (#25777) This reverts commit 1b84048.

potiuk · 2022-08-18T10:48:48Z

Unfortunately it came back: #25789 so that was not the reason (or maybe not the only reason @uranusjr).

potiuk · 2022-08-18T10:49:46Z

(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)).

…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)

potiuk · 2022-08-12T17:25:16Z

The pip resolver in eager upgrade mode on Python 3.10 decides
to downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.

Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.

potiuk · 2022-08-12T18:00:24Z

In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4

uranusjr · 2022-08-17T14:57:58Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

potiuk · 2022-08-17T16:02:16Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ?

alexott · 2022-08-17T16:03:16Z

Ok, let me point team to it

susodapop · 2022-08-17T19:23:47Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

…apache#25697)" This reverts commit 1b84048.

potiuk · 2022-08-17T19:54:33Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good.

potiuk · 2022-08-17T20:16:06Z

Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :)

…#25697)" (#25777) This reverts commit 1b84048.

potiuk · 2022-08-18T10:48:48Z

Unfortunately it came back: #25789 so that was not the reason (or maybe not the only reason @uranusjr).

potiuk · 2022-08-18T10:49:46Z

(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)).

…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)

potiuk · 2022-08-12T17:25:16Z

The pip resolver in eager upgrade mode on Python 3.10 decides
to downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.

Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.

potiuk · 2022-08-12T18:00:24Z

In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4

uranusjr · 2022-08-17T14:57:58Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

potiuk · 2022-08-17T16:02:16Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ?

alexott · 2022-08-17T16:03:16Z

Ok, let me point team to it

susodapop · 2022-08-17T19:23:47Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

…apache#25697)" This reverts commit 1b84048.

potiuk · 2022-08-17T19:54:33Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good.

potiuk · 2022-08-17T20:16:06Z

Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :)

…#25697)" (#25777) This reverts commit 1b84048.

potiuk · 2022-08-18T10:48:48Z

Unfortunately it came back: #25789 so that was not the reason (or maybe not the only reason @uranusjr).

potiuk · 2022-08-18T10:49:46Z

(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)).

…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)

potiuk · 2022-08-12T17:25:16Z

The pip resolver in eager upgrade mode on Python 3.10 decides
to downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.

Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.

potiuk · 2022-08-12T18:00:24Z

In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4

uranusjr · 2022-08-17T14:57:58Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

potiuk · 2022-08-17T16:02:16Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ?

alexott · 2022-08-17T16:03:16Z

Ok, let me point team to it

susodapop · 2022-08-17T19:23:47Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

…apache#25697)" This reverts commit 1b84048.

potiuk · 2022-08-17T19:54:33Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good.

potiuk · 2022-08-17T20:16:06Z

Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :)

…#25697)" (#25777) This reverts commit 1b84048.

potiuk · 2022-08-18T10:48:48Z

Unfortunately it came back: #25789 so that was not the reason (or maybe not the only reason @uranusjr).

potiuk · 2022-08-18T10:49:46Z

(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)).

…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)

potiuk · 2022-08-12T17:25:16Z

The pip resolver in eager upgrade mode on Python 3.10 decides
to downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.

Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.

potiuk · 2022-08-12T18:00:24Z

In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4

uranusjr · 2022-08-17T14:57:58Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

potiuk · 2022-08-17T16:02:16Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ?

alexott · 2022-08-17T16:03:16Z

Ok, let me point team to it

susodapop · 2022-08-17T19:23:47Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

…apache#25697)" This reverts commit 1b84048.

potiuk · 2022-08-17T19:54:33Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good.

potiuk · 2022-08-17T20:16:06Z

Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :)

…#25697)" (#25777) This reverts commit 1b84048.

potiuk · 2022-08-18T10:48:48Z

Unfortunately it came back: #25789 so that was not the reason (or maybe not the only reason @uranusjr).

potiuk · 2022-08-18T10:49:46Z

(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)).

…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)

potiuk · 2022-08-12T17:25:16Z

The pip resolver in eager upgrade mode on Python 3.10 decides
to downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.

Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.

potiuk · 2022-08-12T18:00:24Z

In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4

uranusjr · 2022-08-17T14:57:58Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

potiuk · 2022-08-17T16:02:16Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ?

alexott · 2022-08-17T16:03:16Z

Ok, let me point team to it

susodapop · 2022-08-17T19:23:47Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

…apache#25697)" This reverts commit 1b84048.

potiuk · 2022-08-17T19:54:33Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good.

potiuk · 2022-08-17T20:16:06Z

Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :)

…#25697)" (#25777) This reverts commit 1b84048.

potiuk · 2022-08-18T10:48:48Z

Unfortunately it came back: #25789 so that was not the reason (or maybe not the only reason @uranusjr).

potiuk · 2022-08-18T10:49:46Z

(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)).

…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)

potiuk · 2022-08-12T17:25:16Z

The pip resolver in eager upgrade mode on Python 3.10 decides
to downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.

Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.

potiuk · 2022-08-12T18:00:24Z

In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4

uranusjr · 2022-08-17T14:57:58Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

potiuk · 2022-08-17T16:02:16Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ?

alexott · 2022-08-17T16:03:16Z

Ok, let me point team to it

susodapop · 2022-08-17T19:23:47Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

…apache#25697)" This reverts commit 1b84048.

potiuk · 2022-08-17T19:54:33Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good.

potiuk · 2022-08-17T20:16:06Z

Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :)

…#25697)" (#25777) This reverts commit 1b84048.

potiuk · 2022-08-18T10:48:48Z

Unfortunately it came back: #25789 so that was not the reason (or maybe not the only reason @uranusjr).

potiuk · 2022-08-18T10:49:46Z

(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)).

…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)

potiuk · 2022-08-12T17:25:16Z

The pip resolver in eager upgrade mode on Python 3.10 decides
to downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.

Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.

potiuk · 2022-08-12T18:00:24Z

In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4

uranusjr · 2022-08-17T14:57:58Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

potiuk · 2022-08-17T16:02:16Z

A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probablymaybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker.

That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ?

alexott · 2022-08-17T16:03:16Z

Ok, let me point team to it

susodapop · 2022-08-17T19:23:47Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

…apache#25697)" This reverts commit 1b84048.

potiuk · 2022-08-17T19:54:33Z

Hey team,databricks-sql-connector maintainer here: we just pushed v2.0.4 with Python 3.10 support. LMK if this fixes things for you.

Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good.

potiuk · 2022-08-17T20:16:06Z

Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :)

…#25697)" (#25777) This reverts commit 1b84048.

potiuk · 2022-08-18T10:48:48Z

Unfortunately it came back: #25789 so that was not the reason (or maybe not the only reason @uranusjr).

potiuk · 2022-08-18T10:49:46Z

(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)).

…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)

potiuk requested review from ashb and mik-laj as code owners August 12, 2022 17:25

boring-cyborgBot added area:dev-tools area:production-image Production image improvements and fixes labels Aug 12, 2022

potiuk requested review from jedcunningham and uranusjr August 12, 2022 17:25

potiuk mentioned this pull request Aug 12, 2022
Bump cattrs version #25689
Merged

potiuk changed the title ~~Help pip resolved to make better decisions on Pyarrow version~~Help pip resolver to make better decisions on Pyarrow versionAug 12, 2022

eladkal approved these changes Aug 12, 2022
View reviewed changes

jedcunningham approved these changes Aug 12, 2022
View reviewed changes

potiuk merged commit 1b84048 into apache:mainAug 12, 2022

potiuk deleted the add-pyarrow-hint-for-pip branch August 12, 2022 17:57

potiuk added this to the Airflow 2.3.4 milestone Aug 12, 2022

kaxil mentioned this pull request Aug 14, 2022
Add @task.kubernetes taskflow decorator #25663
Merged

ephraimbuddy modified the milestones: Airflow 2.3.4, Airflow 2.4.0 Aug 15, 2022

alexott mentioned this pull request Aug 17, 2022
Add support for Python 3.10 databricks/databricks-sql-python#26
Closed

potiuk added a commit to potiuk/airflow that referenced this pull request Aug 17, 2022
Revert "Help pip resolved to make better decisions on Pyarrow version (…
9aac5ee
…apache#25697)" This reverts commit 1b84048.

potiuk mentioned this pull request Aug 17, 2022
Revert "Help pip resolved to make better decisions on Pyarrow version… #25777
Merged

potiuk added a commit that referenced this pull request Aug 17, 2022
Revert "Help pip resolved to make better decisions on Pyarrow version (…
11b6014
…#25697)" (#25777) This reverts commit 1b84048.