AIP-66: Add support for parsing DAG bundles by jedcunningham · Pull Request #45371 · apache/airflow

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

boring-cyborgBot added area:API Airflow's REST/HTTP API area:CLI area:Scheduler including HA (high availability) scheduler labels Jan 3, 2025

uranusjr reviewed Jan 3, 2025
View reviewed changes

Comment threadairflow/dag_processing/manager.py Outdated

uranusjr reviewed Jan 3, 2025
View reviewed changes

Comment threadairflow/models/dagbundle.py Outdated

jedcunningham force-pushed the bundle-parsing branch 2 times, most recently from 5fa6022 to 5f5b30cCompare January 3, 2025 16:04

jedcunningham mentioned this pull request Jan 4, 2025
Remove some unnecessary FAB test setup code #45400
Merged

ephraimbuddy reviewed Jan 6, 2025
View reviewed changes

Comment threadairflow/dag_processing/manager.py Outdated

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/dag_processing/collection.py

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/models/dag.py Outdated

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/models/dag.py

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/dag_processing/manager.py Outdated

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/dag_processing/manager.py Outdated

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/dag_processing/manager.py Outdated

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/dag_processing/manager.py Outdated

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/dag_processing/manager.py

dstandish reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/dag_processing/manager.py Outdated

dstandish approved these changes Jan 7, 2025
View reviewed changes

jedcunningham force-pushed the bundle-parsing branch 3 times, most recently from dcf0549 to 677db81Compare January 7, 2025 21:10

jedcunningham added 8 commits January 7, 2025 16:23

Get existing tests happy again!
b233a7e

Fix AF2 compat tests
b90d5a4

Fix all but 1 FAB test
3fe2592

Let manager manage the db session
940bc57

DagFilePath -> DagFileInfo
3b0d12b

Fix OL test
9b6ef56

fix tests after main rebase
782e0c4

ephraimbuddy mentioned this pull request Jan 8, 2025
Add bundle_name to ParseImportError #45480
Merged

ephraimbuddy approved these changes Jan 8, 2025
View reviewed changes

jedcunningham added 3 commits January 8, 2025 10:05

Add some new tests that cover new bundled functionality
2878c40

docstring and better consistency in new tests
889dd5f

Merge remote-tracking branch 'upstream/main' into bundle-parsing
e7750be

dstandish approved these changes Jan 8, 2025
View reviewed changes

jedcunningham marked this pull request as ready for review January 8, 2025 18:49

jedcunningham requested review from XD-DENG, ashb and mobuchowski as code owners January 8, 2025 18:49

Merge branch 'main' into bundle-parsing
3911640

jedcunningham added the full tests needed We need to run full set of tests for this PR to merge label Jan 8, 2025

jedcunningham closed this Jan 8, 2025

jedcunningham reopened this Jan 8, 2025

jedcunningham added 2 commits January 8, 2025 15:16

Remove example_dags support from list_py_file_paths - it's now a bundle!
0d831be

Merge remote-tracking branch 'upstream/main' into bundle-parsing
a2b6bd5

dstandish approved these changes Jan 9, 2025
View reviewed changes

jedcunningham merged commit 72ab1d5 into apache:mainJan 9, 2025

jedcunningham deleted the bundle-parsing branch January 9, 2025 16:55

jedcunningham added a commit to astronomer/airflow that referenced this pull request Jan 9, 2025
Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)"
20384d9
This reverts commit 72ab1d5.

jedcunningham mentioned this pull request Jan 9, 2025
AIP-66: Revert bundle parsing #45531
Merged

This was referenced Jan 9, 2025
AIP-66: Add support for parsing DAG bundles #45532
Merged
Move list_py_file_paths test to the right file (#45521) #45617
Merged