Fix new MyPy errors in main by potiuk · Pull Request #22884 · apache/airflow

potiuk · 2022-04-10T08:03:22Z

Those MyPe errors are side effect of some new dependencies.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

potiuk · 2022-04-10T08:08:55Z

BTW. A number of those errors (except the big google Retry) are because MyPy has quite a big problem with recognizing that for example our airflow.providers.grpc module is not the same as grpc . This is because provider's module is implicit and somehow mypy things that grpc is the top-level module (same with jenkins and few others). We will have to solve it better when we split providers from the core.

Those MyPe errors are side effect of some new dependencies.

github-actions · 2022-04-10T14:19:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

malthe · 2022-04-10T14:48:15Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?

If someone wants a deterministic order, then can use a list or tuple as the value.

potiuk · 2022-04-10T14:56:05Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?
If someone wants a deterministic order, then can use a list or tuple as the value.

I just fixed consistency. I think there is no particular reason. I do not think it matters as long as it consistent. You can make a PR to change it ti Iterable -but it has to be consistent in all providers and code.

potiuk · 2022-04-10T14:59:05Z

It also could be that somewhere in the code len() is used and that would make it not good candidate for Iterable.

malthe · 2022-04-10T15:21:40Z

@potiuk from the description of those attributes, I don't see why any code would do len() on it. It seems you've got to iterate until you find a match or keep trying.

potiuk · 2022-04-10T17:32:55Z

Yeah. Could be :) . I do not see any problem with changing it to Iterable if it passes all the tests. I

potiuk · 2022-04-10T08:03:22Z

Those MyPe errors are side effect of some new dependencies.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

potiuk · 2022-04-10T08:08:55Z

BTW. A number of those errors (except the big google Retry) are because MyPy has quite a big problem with recognizing that for example our airflow.providers.grpc module is not the same as grpc . This is because provider's module is implicit and somehow mypy things that grpc is the top-level module (same with jenkins and few others). We will have to solve it better when we split providers from the core.

Those MyPe errors are side effect of some new dependencies.

github-actions · 2022-04-10T14:19:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

malthe · 2022-04-10T14:48:15Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?

If someone wants a deterministic order, then can use a list or tuple as the value.

potiuk · 2022-04-10T14:56:05Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?
If someone wants a deterministic order, then can use a list or tuple as the value.

I just fixed consistency. I think there is no particular reason. I do not think it matters as long as it consistent. You can make a PR to change it ti Iterable -but it has to be consistent in all providers and code.

potiuk · 2022-04-10T14:59:05Z

It also could be that somewhere in the code len() is used and that would make it not good candidate for Iterable.

malthe · 2022-04-10T15:21:40Z

@potiuk from the description of those attributes, I don't see why any code would do len() on it. It seems you've got to iterate until you find a match or keep trying.

potiuk · 2022-04-10T17:32:55Z

Yeah. Could be :) . I do not see any problem with changing it to Iterable if it passes all the tests. I

potiuk · 2022-04-10T08:03:22Z

Those MyPe errors are side effect of some new dependencies.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

potiuk · 2022-04-10T08:08:55Z

BTW. A number of those errors (except the big google Retry) are because MyPy has quite a big problem with recognizing that for example our airflow.providers.grpc module is not the same as grpc . This is because provider's module is implicit and somehow mypy things that grpc is the top-level module (same with jenkins and few others). We will have to solve it better when we split providers from the core.

Those MyPe errors are side effect of some new dependencies.

github-actions · 2022-04-10T14:19:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

malthe · 2022-04-10T14:48:15Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?

If someone wants a deterministic order, then can use a list or tuple as the value.

potiuk · 2022-04-10T14:56:05Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?
If someone wants a deterministic order, then can use a list or tuple as the value.

I just fixed consistency. I think there is no particular reason. I do not think it matters as long as it consistent. You can make a PR to change it ti Iterable -but it has to be consistent in all providers and code.

potiuk · 2022-04-10T14:59:05Z

It also could be that somewhere in the code len() is used and that would make it not good candidate for Iterable.

malthe · 2022-04-10T15:21:40Z

@potiuk from the description of those attributes, I don't see why any code would do len() on it. It seems you've got to iterate until you find a match or keep trying.

potiuk · 2022-04-10T17:32:55Z

Yeah. Could be :) . I do not see any problem with changing it to Iterable if it passes all the tests. I

potiuk · 2022-04-10T08:03:22Z

Those MyPe errors are side effect of some new dependencies.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

potiuk · 2022-04-10T08:08:55Z

BTW. A number of those errors (except the big google Retry) are because MyPy has quite a big problem with recognizing that for example our airflow.providers.grpc module is not the same as grpc . This is because provider's module is implicit and somehow mypy things that grpc is the top-level module (same with jenkins and few others). We will have to solve it better when we split providers from the core.

Those MyPe errors are side effect of some new dependencies.

github-actions · 2022-04-10T14:19:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

malthe · 2022-04-10T14:48:15Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?

If someone wants a deterministic order, then can use a list or tuple as the value.

potiuk · 2022-04-10T14:56:05Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?
If someone wants a deterministic order, then can use a list or tuple as the value.

I just fixed consistency. I think there is no particular reason. I do not think it matters as long as it consistent. You can make a PR to change it ti Iterable -but it has to be consistent in all providers and code.

potiuk · 2022-04-10T14:59:05Z

It also could be that somewhere in the code len() is used and that would make it not good candidate for Iterable.

malthe · 2022-04-10T15:21:40Z

@potiuk from the description of those attributes, I don't see why any code would do len() on it. It seems you've got to iterate until you find a match or keep trying.

potiuk · 2022-04-10T17:32:55Z

Yeah. Could be :) . I do not see any problem with changing it to Iterable if it passes all the tests. I

potiuk · 2022-04-10T08:03:22Z

Those MyPe errors are side effect of some new dependencies.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

potiuk · 2022-04-10T08:08:55Z

BTW. A number of those errors (except the big google Retry) are because MyPy has quite a big problem with recognizing that for example our airflow.providers.grpc module is not the same as grpc . This is because provider's module is implicit and somehow mypy things that grpc is the top-level module (same with jenkins and few others). We will have to solve it better when we split providers from the core.

Those MyPe errors are side effect of some new dependencies.

github-actions · 2022-04-10T14:19:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

malthe · 2022-04-10T14:48:15Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?

If someone wants a deterministic order, then can use a list or tuple as the value.

potiuk · 2022-04-10T14:56:05Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?
If someone wants a deterministic order, then can use a list or tuple as the value.

I just fixed consistency. I think there is no particular reason. I do not think it matters as long as it consistent. You can make a PR to change it ti Iterable -but it has to be consistent in all providers and code.

potiuk · 2022-04-10T14:59:05Z

It also could be that somewhere in the code len() is used and that would make it not good candidate for Iterable.

malthe · 2022-04-10T15:21:40Z

@potiuk from the description of those attributes, I don't see why any code would do len() on it. It seems you've got to iterate until you find a match or keep trying.

potiuk · 2022-04-10T17:32:55Z

Yeah. Could be :) . I do not see any problem with changing it to Iterable if it passes all the tests. I

potiuk · 2022-04-10T08:03:22Z

Those MyPe errors are side effect of some new dependencies.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

potiuk · 2022-04-10T08:08:55Z

BTW. A number of those errors (except the big google Retry) are because MyPy has quite a big problem with recognizing that for example our airflow.providers.grpc module is not the same as grpc . This is because provider's module is implicit and somehow mypy things that grpc is the top-level module (same with jenkins and few others). We will have to solve it better when we split providers from the core.

Those MyPe errors are side effect of some new dependencies.

github-actions · 2022-04-10T14:19:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

malthe · 2022-04-10T14:48:15Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?

If someone wants a deterministic order, then can use a list or tuple as the value.

potiuk · 2022-04-10T14:56:05Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?
If someone wants a deterministic order, then can use a list or tuple as the value.

I just fixed consistency. I think there is no particular reason. I do not think it matters as long as it consistent. You can make a PR to change it ti Iterable -but it has to be consistent in all providers and code.

potiuk · 2022-04-10T14:59:05Z

It also could be that somewhere in the code len() is used and that would make it not good candidate for Iterable.

malthe · 2022-04-10T15:21:40Z

@potiuk from the description of those attributes, I don't see why any code would do len() on it. It seems you've got to iterate until you find a match or keep trying.

potiuk · 2022-04-10T17:32:55Z

Yeah. Could be :) . I do not see any problem with changing it to Iterable if it passes all the tests. I

potiuk · 2022-04-10T08:03:22Z

Those MyPe errors are side effect of some new dependencies.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

potiuk · 2022-04-10T08:08:55Z

BTW. A number of those errors (except the big google Retry) are because MyPy has quite a big problem with recognizing that for example our airflow.providers.grpc module is not the same as grpc . This is because provider's module is implicit and somehow mypy things that grpc is the top-level module (same with jenkins and few others). We will have to solve it better when we split providers from the core.

Those MyPe errors are side effect of some new dependencies.

github-actions · 2022-04-10T14:19:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

malthe · 2022-04-10T14:48:15Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?

If someone wants a deterministic order, then can use a list or tuple as the value.

potiuk · 2022-04-10T14:56:05Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?
If someone wants a deterministic order, then can use a list or tuple as the value.

I just fixed consistency. I think there is no particular reason. I do not think it matters as long as it consistent. You can make a PR to change it ti Iterable -but it has to be consistent in all providers and code.

potiuk · 2022-04-10T14:59:05Z

It also could be that somewhere in the code len() is used and that would make it not good candidate for Iterable.

malthe · 2022-04-10T15:21:40Z

@potiuk from the description of those attributes, I don't see why any code would do len() on it. It seems you've got to iterate until you find a match or keep trying.

potiuk · 2022-04-10T17:32:55Z

Yeah. Could be :) . I do not see any problem with changing it to Iterable if it passes all the tests. I

potiuk · 2022-04-10T08:03:22Z

Those MyPe errors are side effect of some new dependencies.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

potiuk · 2022-04-10T08:08:55Z

BTW. A number of those errors (except the big google Retry) are because MyPy has quite a big problem with recognizing that for example our airflow.providers.grpc module is not the same as grpc . This is because provider's module is implicit and somehow mypy things that grpc is the top-level module (same with jenkins and few others). We will have to solve it better when we split providers from the core.

Those MyPe errors are side effect of some new dependencies.

github-actions · 2022-04-10T14:19:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

malthe · 2022-04-10T14:48:15Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?

If someone wants a deterministic order, then can use a list or tuple as the value.

potiuk · 2022-04-10T14:56:05Z

Why are template_fields and template_ext now sequences and not instead Iterable – ?
If someone wants a deterministic order, then can use a list or tuple as the value.

I just fixed consistency. I think there is no particular reason. I do not think it matters as long as it consistent. You can make a PR to change it ti Iterable -but it has to be consistent in all providers and code.

potiuk · 2022-04-10T14:59:05Z

It also could be that somewhere in the code len() is used and that would make it not good candidate for Iterable.

malthe · 2022-04-10T15:21:40Z

@potiuk from the description of those attributes, I don't see why any code would do len() on it. It seems you've got to iterate until you find a match or keep trying.

potiuk · 2022-04-10T17:32:55Z

Yeah. Could be :) . I do not see any problem with changing it to Iterable if it passes all the tests. I

potiuk requested review from XD-DENG, ashb, jedcunningham, kaxil and turbaszek as code owners April 10, 2022 08:03

boring-cyborgBot added area:CLI area:core-operators area:dev-tools provider:cncf-kubernetes Kubernetes (k8s) provider related issues area:logging area:providers provider:amazon AWS/Amazon - related issues provider:google Google (including GCP) related issues labels Apr 10, 2022

potiuk requested review from eladkal and ephraimbuddy April 10, 2022 08:03

Fix new MyPy errors in main
81c31c9
Those MyPe errors are side effect of some new dependencies.

potiuk force-pushed the fix-newer-mypy-issues branch from 6e627e8 to 81c31c9Compare April 10, 2022 08:40

potiuk mentioned this pull request Apr 10, 2022
Databricks SQL operators are now Python 3.10 compatible #22886
Merged

mik-laj approved these changes Apr 10, 2022
View reviewed changes

github-actionsBot added the full tests needed We need to run full set of tests for this PR to merge label Apr 10, 2022

potiuk merged commit 6933022 into apache:mainApr 10, 2022

potiuk deleted the fix-newer-mypy-issues branch April 10, 2022 14:45

malthe assigned malthe and unassigned malthe Apr 10, 2022

malthe removed their assignment Apr 10, 2022

malthe self-requested a review April 10, 2022 15:28

ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Apr 11, 2022

potiuk mentioned this pull request Apr 30, 2022
Status of testing Providers that were prepared on April 30, 2022 #23375
Closed
9 tasks

potiuk mentioned this pull request May 11, 2022
Status of testing Providers that were prepared on May 12, 2022 #23659
Closed
59 tasks

potiuk mentioned this pull request Jun 7, 2022
Status of testing Providers that were prepared on June 07, 2022 #24289
Closed