AIP-66: Add support for parsing DAG bundles by jedcunningham · Pull Request #45532 · apache/airflow

jedcunningham · 2025-01-09T23:35:29Z

This is #45371, which had to be reverted.

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

… loop

* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

jedcunningham · 2025-01-09T23:35:29Z

This is #45371, which had to be reverted.

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

… loop

* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

jedcunningham · 2025-01-09T23:35:29Z

This is #45371, which had to be reverted.

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

… loop

* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

jedcunningham · 2025-01-09T23:35:29Z

This is #45371, which had to be reverted.

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

… loop

* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

jedcunningham · 2025-01-09T23:35:29Z

This is #45371, which had to be reverted.

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

… loop

* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

jedcunningham · 2025-01-09T23:35:29Z

This is #45371, which had to be reverted.

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

… loop

* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

jedcunningham · 2025-01-09T23:35:29Z

This is #45371, which had to be reverted.

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

… loop

* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

jedcunningham · 2025-01-09T23:35:29Z

This is #45371, which had to be reverted.

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

… loop

* AIP-66: Add support for parsing DAG bundles Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). * Get existing tests happy again! * Fix AF2 compat tests * Fix all but 1 FAB test * Let manager manage the db session * DagFilePath -> DagFileInfo * Fix OL test * fix tests after main rebase * review feedback & some cleanup * Update airflow/models/dag.py Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> * Add some new tests that cover new bundled functionality * docstring and better consistency in new tests * Remove example_dags support from list_py_file_paths - it's now a bundle! * Test a theory... * less invasive fix * Fix bug where bundles wouldnt be refreshed at least once during first loop --------- Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

jedcunningham added the AIP-66: DAG Bundle/Manifest label Jan 9, 2025

boring-cyborgBot added area:API Airflow's REST/HTTP API area:CLI area:providers area:Scheduler including HA (high availability) scheduler provider:fab provider:openlineage AIP-53 labels Jan 9, 2025

jedcunningham added the full tests needed We need to run full set of tests for this PR to merge label Jan 9, 2025

jedcunningham closed this Jan 9, 2025

jedcunningham reopened this Jan 9, 2025

jedcunningham mentioned this pull request Jan 10, 2025
Add a bundle for example dags when enabled #45533
Merged

jedcunningham closed this Jan 10, 2025

jedcunningham reopened this Jan 10, 2025

jedcunningham closed this Jan 10, 2025

jedcunningham reopened this Jan 10, 2025

jedcunningham closed this Jan 10, 2025

jedcunningham reopened this Jan 10, 2025

jedcunningham closed this Jan 10, 2025

jedcunningham reopened this Jan 10, 2025

jedcunningham closed this Jan 10, 2025

jedcunningham reopened this Jan 10, 2025

jedcunningham closed this Jan 10, 2025

jedcunningham reopened this Jan 10, 2025

jedcunningham force-pushed the bundle-parsing branch from 1e74bff to 37c41a2Compare January 10, 2025 17:45

jedcunningham added 6 commits January 10, 2025 12:43

Get existing tests happy again!
f884cd1

Fix AF2 compat tests
cd8aef8

Fix all but 1 FAB test
002477b

Let manager manage the db session
30cf1ce

DagFilePath -> DagFileInfo
15bbac3

Fix bug where bundles wouldnt be refreshed at least once during first…
25da62c
… loop

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham closed this Jan 11, 2025

jedcunningham reopened this Jan 11, 2025

jedcunningham requested review from dstandish and potiuk January 11, 2025 23:07

jedcunningham mentioned this pull request Jan 13, 2025
Update dags reserialize command to work with DAG bundle #45507
Merged

ephraimbuddy approved these changes Jan 13, 2025
View reviewed changes

ephraimbuddy merged commit 96645b1 into apache:mainJan 13, 2025

ephraimbuddy deleted the bundle-parsing branch January 13, 2025 11:50

yuseok89 mentioned this pull request Apr 17, 2026
DAG Import Errors message is dangling in web-interface #45227
Open
2 tasks