Skip to content

Handle Example dags case when checking for missing files - #41856

Merged
ephraimbuddy merged 1 commit into
apache:mainfrom
astronomer:DagParseExamples
Aug 29, 2024
Merged

Handle Example dags case when checking for missing files#41856
ephraimbuddy merged 1 commit into
apache:mainfrom
astronomer:DagParseExamples

Conversation

@utkarsharma2

@utkarsharma2utkarsharma2 commented Aug 29, 2024•
edited by ephraimbuddy
Loading

Copy link
Copy Markdown
Contributor

Earlier PR create to address the issue was not handling the case for the Example Dags, due to which the example dags were marked as stale since they are not present in the dag_directory. This PR handles that scenarios and update the testcase accordingly.

related: #41432
closes: #41852

@boring-cyborgboring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Aug 29, 2024
This was referenced Aug 29, 2024
Closed
Closed
@utkarsharma2

utkarsharma2 commented Aug 29, 2024

Copy link
Copy Markdown
ContributorAuthor

@ephraimbuddy I tested it with standalone dag processor in breeze env and it's working as expected.

@utkarsharma2
utkarsharma2force-pushed the DagParseExamples branch 2 times, most recently from a2f11cf to ecc119eCompareAugust 29, 2024 11:46
@utkarsharma2
Handle Example dags case when checking for missing files
f2ec13e
Earlier PR create to address the issue was not handling the case for the Example Dags, due to which the example dags were marked as stale since they are not present in the dag_directory. This PR handles that scenarios and update the testcase accordingly.
related: apache#41432
@potiuk

potiuk commented Aug 29, 2024

Copy link
Copy Markdown
Member

I believe that was the reason for the Open API test failing ?

@ephraimbuddy

ephraimbuddy commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

I believe that was the reason for the Open API test failing ?

Yes. That's my believe too

@ephraimbuddy
ephraimbuddy merged commit 435e968 into apache:mainAug 29, 2024
@ephraimbuddy
ephraimbuddy deleted the DagParseExamples branch August 29, 2024 13:06
@utkarsharma2

utkarsharma2 commented Aug 29, 2024

Copy link
Copy Markdown
ContributorAuthor

@potiuk / @ephraimbuddy, which PR were the tests failing for?

@utkarsharma2

utkarsharma2 commented Aug 29, 2024

Copy link
Copy Markdown
ContributorAuthor

@potiuk / @ephraimbuddy, which PR were the tests failing for?

Never mind, got it.

@potiuk

potiuk commented Aug 29, 2024

Copy link
Copy Markdown
Member

@potiuk / @ephraimbuddy, which PR were the tests failing for?

Main tests: https://github.com/apache/airflow/actions?query=event%3Aschedule and PRs that got "full tests needed" because for example they modified dependencies. The problem was in this case that there is non-obvious connection between DAGs diseappearing and Open API tests failing - what was failing there was the test that tries to retrieve example DAG information via the API. Those Open API tests are skipped unless you modify API.

@ephraimbuddy

ephraimbuddy commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

@potiuk / @ephraimbuddy, which PR were the tests failing for?

Here: https://github.com/apache/airflow/actions/runs/10613181076/job/29416509993.

An example DAG could not be found

@potiuk

potiuk commented Aug 29, 2024

Copy link
Copy Markdown
Member

Also v2-10-test were failing - because they also run "full tests" https://github.com/apache/airflow/actions?query=branch%3Av2-10-test

@potiukpotiuk mentioned this pull request Aug 29, 2024
Merged
@potiuk

potiuk commented Aug 29, 2024

Copy link
Copy Markdown
Member

Yep. That was it - It's fixed here for example - I rebased failing cloudant provider and it's green now https://github.com/apache/airflow/actions/runs/10615673609/job/29424342526

@utkarsharma2

utkarsharma2 commented Aug 29, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Yes, I think it's related.

@ephraimbuddy

ephraimbuddy commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

@utkarsharma2 , can you backport this to v2-10-test

utkarsharma2 added a commit to astronomer/airflow that referenced this pull request Aug 30, 2024
@utkarsharma2
Handle Example dags case when checking for missing files (apache#41856)
1eb1466
Earlier PR create to address the issue was not handling the case for the Example Dags, due to which the example dags were marked as stale since they are not present in the dag_directory. This PR handles that scenarios and update the testcase accordingly.
related: apache#41432
(cherry picked from commit 435e968)
@shahar1shahar1 added this to the Airflow 2.10.1 milestone Aug 30, 2024
@shahar1shahar1 mentioned this pull request Aug 30, 2024
Merged
shahar1 pushed a commit that referenced this pull request Aug 30, 2024
@utkarsharma2
Handle Example dags case when checking for missing files (#41856) (#4…
5e962ed
…1874)
Earlier PR create to address the issue was not handling the case for the Example Dags, due to which the example dags were marked as stale since they are not present in the dag_directory. This PR handles that scenarios and update the testcase accordingly.
related: #41432
(cherry picked from commit 435e968)
@utkarsharma2utkarsharma2 added the type:bug-fix Changelog: Bug Fixes label Aug 30, 2024
utkarsharma2 added a commit that referenced this pull request Sep 2, 2024
@utkarsharma2
Handle Example dags case when checking for missing files (#41856) (#4…
6fb6fdb
…1874)
Earlier PR create to address the issue was not handling the case for the Example Dags, due to which the example dags were marked as stale since they are not present in the dag_directory. This PR handles that scenarios and update the testcase accordingly.
related: #41432
(cherry picked from commit 435e968)
utkarsharma2 added a commit that referenced this pull request Sep 12, 2024
@utkarsharma2
Revert "Handle Example dags case when checking for missing files (#41856
56d7af6
)"
This reverts commit 435e968.
@utkarsharma2utkarsharma2 mentioned this pull request Sep 12, 2024
Merged
utkarsharma2 added a commit that referenced this pull request Sep 12, 2024
@utkarsharma2
Revert "Handle Example dags case when checking for missing files (#41856
6235cf8
)" (#42193)
This reverts commit 435e968.
ephraimbuddy added a commit to astronomer/airflow that referenced this pull request Sep 13, 2024
@ephraimbuddy
Revert "Handle Example dags case when checking for missing files (apa…
585ab16
…che#41856) (apache#41874)"
This reverts commit 5e962ed.
ephraimbuddy added a commit that referenced this pull request Sep 13, 2024
@ephraimbuddy
Revert "Handle Example dags case when checking for missing files (#41856
103e65b
) (#41874)" (#42217)
This reverts commit 5e962ed.
ephraimbuddy added a commit that referenced this pull request Sep 13, 2024
@ephraimbuddy
Revert "Handle Example dags case when checking for missing files (#41856
0b74a08
) (#41874)" (#42217)
This reverts commit 5e962ed.
joaopamaral pushed a commit to joaopamaral/airflow that referenced this pull request Oct 21, 2024
@utkarsharma2@joaopamaral
Revert "Handle Example dags case when checking for missing files (apa…
8300630
…che#41856)" (apache#42193)
This reverts commit 435e968.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@ephraimbuddyephraimbuddyephraimbuddy approved these changes
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulertype:bug-fixChangelog: Bug Fixes

Projects

None yet

Milestone

Airflow 2.10.1

Development

Successfully merging this pull request may close these issues.

4 participants

@utkarsharma2@potiuk@ephraimbuddy@shahar1