Skip to content

Fix conn-fields check crash for nested provider packages - #70224

Merged
Lee-W merged 2 commits into
apache:mainfrom
astronomer:fix-conn-field-check-crash
Jul 23, 2026
Merged

Fix conn-fields check crash for nested provider packages#70224
Lee-W merged 2 commits into
apache:mainfrom
astronomer:fix-conn-field-check-crash

Conversation

@Lee-W

@Lee-WLee-W commented Jul 22, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Why

The conn-fields static check crashed for any change under a namespace provider directory — apache, atlassian, common, cncf, dbt, microsoft. These nest the real provider one level deeper (e.g. the provider lives at apache/beam, not apache), but _resolve_provider_yaml_files() in check_provider_yaml_files.py always took the first path segment as the provider name. A change under apache/beam/... resolved to the nonexistent apache/provider.yaml, so run_provider_yaml_files_check.py raised FileNotFoundError in https://github.com/apache/airflow/actions/runs/29899861660/job/88860560132

What

The fix reuses the existing KNOWN_SECOND_LEVEL_PATHS constant from common_prek_utils — the same list already driving mypy_folder.py and check_providers_subpackages_all_have_init.py — to pick the correct nesting depth, instead of hand-rolling a second copy of that list that would drift.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

amoghrajesh
amoghrajesh approved these changes Jul 22, 2026
View reviewed changes
Comment threadscripts/ci/prek/check_provider_yaml_files.py
@Lee-W
Lee-Wforce-pushed the fix-conn-field-check-crash branch from 50d2287 to 02d5829CompareJuly 22, 2026 10:10
dabla
dabla requested changes Jul 22, 2026
View reviewed changes
Comment threadscripts/ci/prek/check_provider_yaml_files.py
@amoghrajesh

amoghrajesh commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

@dabla did you request changes by mistake?

@Lee-W
Lee-Wforce-pushed the fix-conn-field-check-crash branch from 02d5829 to 3a7c18cCompareJuly 22, 2026 10:49
@Lee-W

Lee-W commented Jul 22, 2026

Copy link
Copy Markdown
MemberAuthor

yep, all fixed

@dabla

dabla commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

@dabla did you request changes by mistake?

Yes sorry for that.

phanikumv
phanikumv reviewed Jul 22, 2026
View reviewed changes
Comment threadscripts/ci/prek/check_provider_yaml_files.py Outdated
@dabladabla mentioned this pull request Jul 22, 2026
Merged
@dabla

dabla commented Jul 22, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

I created a PR which fixes the selective checks issues, so that when a change is done to the static checks of the providers but no providers where changed, it will still run the static checks on the providers.

@Lee-W

Lee-W commented Jul 22, 2026

Copy link
Copy Markdown
MemberAuthor

need another fix related to jira. will update it soon

@Lee-W
Lee-Wforce-pushed the fix-conn-field-check-crash branch from 3a7c18c to f5a352bCompareJuly 22, 2026 13:39
@Lee-W

Lee-W commented Jul 22, 2026

Copy link
Copy Markdown
MemberAuthor

jira fix is up. let's see how CI goes (even though it won't really run without #70230)

@dabla

dabla commented Jul 22, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

jira fix is up. let's see how CI goes (even though it won't really run without #70230)

Great, once this one is merged, I'll rebase into my PR and when mine is also green, will merge it. Thanks for the fix and approval @Lee-W !

gopidesupavan
gopidesupavan reviewed Jul 22, 2026
View reviewed changes
Comment threadscripts/ci/prek/check_provider_yaml_files.py
Lee-W added 2 commits July 22, 2026 22:33
@Lee-W
Fix conn-fields check crash for nested provider packages
119038f
check_provider_yaml_files.py mapped any changed Python file to
"<first-path-segment>/provider.yaml", which breaks for namespace
packages (apache, atlassian, common, cncf, dbt, microsoft) that nest
an extra directory level, e.g. a change under apache/beam/... resolved
to the nonexistent apache/provider.yaml instead of
apache/beam/provider.yaml. Reuse the existing KNOWN_SECOND_LEVEL_PATHS
list from common_prek_utils to resolve the correct nesting depth.
@Lee-W
fixup! Fix conn-fields check crash for nested provider packages
b07ed12
@Lee-W
Lee-Wforce-pushed the fix-conn-field-check-crash branch from f5a352b to b07ed12CompareJuly 22, 2026 14:36
@ramitkatariaramitkataria mentioned this pull request Jul 22, 2026
Merged
1 task
@Lee-W
Lee-W merged commit c578603 into apache:mainJul 23, 2026
80 checks passed
@Lee-W
Lee-W deleted the fix-conn-field-check-crash branch July 23, 2026 00:41
@github-actions

github-actionsBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
❌v3-3-testCommit Link

You can attempt to backport this manually by running:

cherry_picker c578603 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

@Lee-WLee-W mentioned this pull request Jul 23, 2026
Merged
1 task
shahar1 added a commit to shahar1/airflow that referenced this pull request Jul 23, 2026
@shahar1
Add atlassian.jira 3.3.5 to the 2026-07-22 provider wave
e415b3a
apache#70224 merged a few hours after this wave was cut, declaring a Verify SSL
connection field and hidden-field behaviour for the Jira connection. That
metadata ships to users through get_provider_info.py, so the provider needs a
release for the connection form to gain the field.
@shahar1shahar1 mentioned this pull request Jul 23, 2026
Merged
1 task
shahar1 added a commit to shahar1/airflow that referenced this pull request Jul 23, 2026
@shahar1
Add atlassian.jira 3.3.5 to the 2026-07-22 provider wave
fdac8e0
apache#70224 merged a few hours after this wave was cut, declaring a Verify SSL
connection field and hidden-field behaviour for the Jira connection. That
metadata ships to users through get_provider_info.py, so the provider needs a
release for the connection form to gain the field.
@shahar1shahar1 mentioned this pull request Jul 24, 2026
Closed
Lee-W added a commit to astronomer/airflow that referenced this pull request Jul 24, 2026
@Lee-W
[v3-3-test] Fix conn-fields check crash for nested provider packages (a…
6198803
…pache#70224)
@Aaryan123456679Aaryan123456679 mentioned this pull request Jul 24, 2026
Merged
@potiukpotiuk mentioned this pull request Jul 26, 2026
Merged
1 task
potiuk pushed a commit that referenced this pull request Jul 26, 2026
@Lee-W
[v3-3-test] Fix conn-fields check crash for nested provider packages (#…
ce0fd3f
…70224) (#70279)
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.1 milestone Aug 3, 2026
vatsrahul1001 pushed a commit that referenced this pull request Aug 5, 2026
@Lee-W@vatsrahul1001
[v3-3-test] Fix conn-fields check crash for nested provider packages (#…
5cebb65
…70224) (#70279)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gopidesupavangopidesupavangopidesupavan left review comments
@phanikumvphanikumvphanikumv left review comments
@dabladabladabla approved these changes
@amoghrajeshamoghrajeshamoghrajesh approved these changes
@eladkaleladkaleladkal approved these changes
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner
+2 more reviewers
@FrankYang0529FrankYang0529FrankYang0529 approved these changes
@ramitkatariaramitkatariaramitkataria approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:dev-toolsbackport-to-v3-3-testBackport to v3-3-test

Projects

None yet

Milestone

Airflow 3.3.1

Development

Successfully merging this pull request may close these issues.

9 participants

@Lee-W@amoghrajesh@dabla@FrankYang0529@gopidesupavan@ramitkataria@eladkal@phanikumv@vatsrahul1001