Extract all the various DB update calls and processes for Dag Parsing to one place by dimberman · Pull Request #44898 · apache/airflow

dimberman · 2024-12-12T22:40:55Z

The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base.

This change gathers them all in to one place to make it easier to a) find, and b) see what is changed.

The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

… to one place The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

ashb · 2024-12-13T12:12:12Z

Re-triggering with full tests

This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go. Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in #44898 and the "subprocess" machinery introduced in #44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in #44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

… to one place (apache#44898) The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com> * Move ParseImportError formatting/depth tests to dagbag This is a property of dagbag parsing, not the processor. Also convert it to a parameterized test too Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

dimberman · 2024-12-12T22:40:55Z

The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base.

This change gathers them all in to one place to make it easier to a) find, and b) see what is changed.

The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

… to one place The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

ashb · 2024-12-13T12:12:12Z

Re-triggering with full tests

This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go. Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in #44898 and the "subprocess" machinery introduced in #44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in #44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

… to one place (apache#44898) The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com> * Move ParseImportError formatting/depth tests to dagbag This is a property of dagbag parsing, not the processor. Also convert it to a parameterized test too Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

dimberman · 2024-12-12T22:40:55Z

The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base.

This change gathers them all in to one place to make it easier to a) find, and b) see what is changed.

The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

… to one place The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

ashb · 2024-12-13T12:12:12Z

Re-triggering with full tests

This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go. Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in #44898 and the "subprocess" machinery introduced in #44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in #44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

… to one place (apache#44898) The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com> * Move ParseImportError formatting/depth tests to dagbag This is a property of dagbag parsing, not the processor. Also convert it to a parameterized test too Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

dimberman · 2024-12-12T22:40:55Z

The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base.

This change gathers them all in to one place to make it easier to a) find, and b) see what is changed.

The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

… to one place The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

ashb · 2024-12-13T12:12:12Z

Re-triggering with full tests

This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go. Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in #44898 and the "subprocess" machinery introduced in #44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in #44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

… to one place (apache#44898) The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com> * Move ParseImportError formatting/depth tests to dagbag This is a property of dagbag parsing, not the processor. Also convert it to a parameterized test too Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

dimberman · 2024-12-12T22:40:55Z

The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base.

This change gathers them all in to one place to make it easier to a) find, and b) see what is changed.

The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

… to one place The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

ashb · 2024-12-13T12:12:12Z

Re-triggering with full tests

This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go. Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in #44898 and the "subprocess" machinery introduced in #44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in #44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

… to one place (apache#44898) The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com> * Move ParseImportError formatting/depth tests to dagbag This is a property of dagbag parsing, not the processor. Also convert it to a parameterized test too Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

dimberman · 2024-12-12T22:40:55Z

The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base.

This change gathers them all in to one place to make it easier to a) find, and b) see what is changed.

The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

… to one place The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

ashb · 2024-12-13T12:12:12Z

Re-triggering with full tests

This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go. Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in #44898 and the "subprocess" machinery introduced in #44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in #44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

… to one place (apache#44898) The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com> * Move ParseImportError formatting/depth tests to dagbag This is a property of dagbag parsing, not the processor. Also convert it to a parameterized test too Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

dimberman · 2024-12-12T22:40:55Z

The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base.

This change gathers them all in to one place to make it easier to a) find, and b) see what is changed.

The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

… to one place The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

ashb · 2024-12-13T12:12:12Z

Re-triggering with full tests

This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go. Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in #44898 and the "subprocess" machinery introduced in #44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in #44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

… to one place (apache#44898) The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com> * Move ParseImportError formatting/depth tests to dagbag This is a property of dagbag parsing, not the processor. Also convert it to a parameterized test too Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

dimberman · 2024-12-12T22:40:55Z

The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base.

This change gathers them all in to one place to make it easier to a) find, and b) see what is changed.

The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

… to one place The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

ashb · 2024-12-13T12:12:12Z

Re-triggering with full tests

This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMIANTE and END messages over the control socket can go. Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in #44898 and the "subprocess" machinery introduced in #44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in #44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

… to one place (apache#44898) The updates to the various DB tables as a result of DAG file parsing and collection are scattered throughout the code base. This change gathers them all in to one place to make it easier to a) find, and b) see what is changed. The current situation is the result of organic code growth, but it is now almost impossible to find all the things we update as they are not organized, and were scatterd across at least DagBag, DagProcessor. Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com> * Move ParseImportError formatting/depth tests to dagbag This is a property of dagbag parsing, not the processor. Also convert it to a parameterized test too Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

As part of Airflow 3 DAG definition files will have to use the Task SDK for all their classes, and anything involving running user code will need to be de-coupled from the database in the user-code process. This change moves all of the "serialization" change up to the DagFileProcessorManager, using the new function introduced in apache#44898 and the "subprocess" machinery introduced in apache#44874. **Important Note**: this change does not remove the ability for dag processes to access the DB for Variables etc. That will come in a future change. Some key parts of this change: - It builds upon the WatchedSubprocess from the TaskSDK. Right now this puts a nasty/unwanted depenednecy between the Dag Parsing code upon the TaskSDK. This will be addressed before release (we have talked about introducing a new "apache-airflow-base-executor" dist where this subprocess+supervisor could live, as the "execution_time" folder in the Task SDK is more a feature of the executor, not of the TaskSDK itself.) - A number of classes that we need to send between processes have been converted to Pydantic for ease of serialization. - In order to not have to serialize everything in the subprocess and deserialize everything in the parent Manager process, we have created a `LazyDeserializedDAG` class that provides lazy access to much of the properties needed to create update the DAG related DB objects, without needing to fully deserialize the entire DAG structure. - Classes switched to attrs based for less boilerplate in constructors. - Internal timers convert to `time.monotonic` where possible, and `time.time` where not, we only need second diff between two points, not datetime objects. - With the earlier removal of "sync mode" for SQLite in apache#44839 the need for separate TERMINATE and END messages over the control socket can go. --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>

boring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Dec 12, 2024

ashb changed the title ~~Extract all the various DB update calls and processes for Dag Parsing…~~Extract all the various DB update calls and processes for Dag Parsing to one placeDec 13, 2024

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb marked this pull request as ready for review December 13, 2024 11:07

ashb requested review from XD-DENG, ephraimbuddy and jedcunningham as code owners December 13, 2024 11:07

ashb force-pushed the extract-dag-db-update-to-one-place branch from 1a8eeaf to d341351Compare December 13, 2024 11:26

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2024

ashb closed this Dec 13, 2024

ashb reopened this Dec 13, 2024

Move ParseImportError formatting/depth tests to dagbag
cd8addc
This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

kaxil reviewed Dec 13, 2024
View reviewed changes

ashb added 2 commits December 13, 2024 14:29

fixup! Move ParseImportError formatting/depth tests to dagbag
2f698b8

fixup! Move ParseImportError formatting/depth tests to dagbag
84a80f1

kaxil approved these changes Dec 13, 2024
View reviewed changes

ashb merged commit ec893fd into apache:mainDec 13, 2024

ashb deleted the extract-dag-db-update-to-one-place branch December 13, 2024 15:51

ashb mentioned this pull request Dec 16, 2024
Swap Dag Parsing to use the TaskSDK machinery. #44972
Merged

boring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Dec 12, 2024

ashb changed the title ~~Extract all the various DB update calls and processes for Dag Parsing…~~Extract all the various DB update calls and processes for Dag Parsing to one placeDec 13, 2024

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb marked this pull request as ready for review December 13, 2024 11:07

ashb requested review from XD-DENG, ephraimbuddy and jedcunningham as code owners December 13, 2024 11:07

ashb force-pushed the extract-dag-db-update-to-one-place branch from 1a8eeaf to d341351Compare December 13, 2024 11:26

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2024

ashb closed this Dec 13, 2024

ashb reopened this Dec 13, 2024

Move ParseImportError formatting/depth tests to dagbag
cd8addc
This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

kaxil reviewed Dec 13, 2024
View reviewed changes

ashb added 2 commits December 13, 2024 14:29

fixup! Move ParseImportError formatting/depth tests to dagbag
2f698b8

fixup! Move ParseImportError formatting/depth tests to dagbag
84a80f1

kaxil approved these changes Dec 13, 2024
View reviewed changes

ashb merged commit ec893fd into apache:mainDec 13, 2024

ashb deleted the extract-dag-db-update-to-one-place branch December 13, 2024 15:51

ashb mentioned this pull request Dec 16, 2024
Swap Dag Parsing to use the TaskSDK machinery. #44972
Merged

boring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Dec 12, 2024

ashb changed the title ~~Extract all the various DB update calls and processes for Dag Parsing…~~Extract all the various DB update calls and processes for Dag Parsing to one placeDec 13, 2024

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb marked this pull request as ready for review December 13, 2024 11:07

ashb requested review from XD-DENG, ephraimbuddy and jedcunningham as code owners December 13, 2024 11:07

ashb force-pushed the extract-dag-db-update-to-one-place branch from 1a8eeaf to d341351Compare December 13, 2024 11:26

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2024

ashb closed this Dec 13, 2024

ashb reopened this Dec 13, 2024

Move ParseImportError formatting/depth tests to dagbag
cd8addc
This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

kaxil reviewed Dec 13, 2024
View reviewed changes

ashb added 2 commits December 13, 2024 14:29

fixup! Move ParseImportError formatting/depth tests to dagbag
2f698b8

fixup! Move ParseImportError formatting/depth tests to dagbag
84a80f1

kaxil approved these changes Dec 13, 2024
View reviewed changes

ashb merged commit ec893fd into apache:mainDec 13, 2024

ashb deleted the extract-dag-db-update-to-one-place branch December 13, 2024 15:51

ashb mentioned this pull request Dec 16, 2024
Swap Dag Parsing to use the TaskSDK machinery. #44972
Merged

boring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Dec 12, 2024

ashb changed the title ~~Extract all the various DB update calls and processes for Dag Parsing…~~Extract all the various DB update calls and processes for Dag Parsing to one placeDec 13, 2024

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb marked this pull request as ready for review December 13, 2024 11:07

ashb requested review from XD-DENG, ephraimbuddy and jedcunningham as code owners December 13, 2024 11:07

ashb force-pushed the extract-dag-db-update-to-one-place branch from 1a8eeaf to d341351Compare December 13, 2024 11:26

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2024

ashb closed this Dec 13, 2024

ashb reopened this Dec 13, 2024

Move ParseImportError formatting/depth tests to dagbag
cd8addc
This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

kaxil reviewed Dec 13, 2024
View reviewed changes

ashb added 2 commits December 13, 2024 14:29

fixup! Move ParseImportError formatting/depth tests to dagbag
2f698b8

fixup! Move ParseImportError formatting/depth tests to dagbag
84a80f1

kaxil approved these changes Dec 13, 2024
View reviewed changes

ashb merged commit ec893fd into apache:mainDec 13, 2024

ashb deleted the extract-dag-db-update-to-one-place branch December 13, 2024 15:51

ashb mentioned this pull request Dec 16, 2024
Swap Dag Parsing to use the TaskSDK machinery. #44972
Merged

boring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Dec 12, 2024

ashb changed the title ~~Extract all the various DB update calls and processes for Dag Parsing…~~Extract all the various DB update calls and processes for Dag Parsing to one placeDec 13, 2024

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb marked this pull request as ready for review December 13, 2024 11:07

ashb requested review from XD-DENG, ephraimbuddy and jedcunningham as code owners December 13, 2024 11:07

ashb force-pushed the extract-dag-db-update-to-one-place branch from 1a8eeaf to d341351Compare December 13, 2024 11:26

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2024

ashb closed this Dec 13, 2024

ashb reopened this Dec 13, 2024

Move ParseImportError formatting/depth tests to dagbag
cd8addc
This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

kaxil reviewed Dec 13, 2024
View reviewed changes

ashb added 2 commits December 13, 2024 14:29

fixup! Move ParseImportError formatting/depth tests to dagbag
2f698b8

fixup! Move ParseImportError formatting/depth tests to dagbag
84a80f1

kaxil approved these changes Dec 13, 2024
View reviewed changes

ashb merged commit ec893fd into apache:mainDec 13, 2024

ashb deleted the extract-dag-db-update-to-one-place branch December 13, 2024 15:51

ashb mentioned this pull request Dec 16, 2024
Swap Dag Parsing to use the TaskSDK machinery. #44972
Merged

boring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Dec 12, 2024

ashb changed the title ~~Extract all the various DB update calls and processes for Dag Parsing…~~Extract all the various DB update calls and processes for Dag Parsing to one placeDec 13, 2024

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb marked this pull request as ready for review December 13, 2024 11:07

ashb requested review from XD-DENG, ephraimbuddy and jedcunningham as code owners December 13, 2024 11:07

ashb force-pushed the extract-dag-db-update-to-one-place branch from 1a8eeaf to d341351Compare December 13, 2024 11:26

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2024

ashb closed this Dec 13, 2024

ashb reopened this Dec 13, 2024

Move ParseImportError formatting/depth tests to dagbag
cd8addc
This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

kaxil reviewed Dec 13, 2024
View reviewed changes

ashb added 2 commits December 13, 2024 14:29

fixup! Move ParseImportError formatting/depth tests to dagbag
2f698b8

fixup! Move ParseImportError formatting/depth tests to dagbag
84a80f1

kaxil approved these changes Dec 13, 2024
View reviewed changes

ashb merged commit ec893fd into apache:mainDec 13, 2024

ashb deleted the extract-dag-db-update-to-one-place branch December 13, 2024 15:51

ashb mentioned this pull request Dec 16, 2024
Swap Dag Parsing to use the TaskSDK machinery. #44972
Merged

boring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Dec 12, 2024

ashb changed the title ~~Extract all the various DB update calls and processes for Dag Parsing…~~Extract all the various DB update calls and processes for Dag Parsing to one placeDec 13, 2024

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb marked this pull request as ready for review December 13, 2024 11:07

ashb requested review from XD-DENG, ephraimbuddy and jedcunningham as code owners December 13, 2024 11:07

ashb force-pushed the extract-dag-db-update-to-one-place branch from 1a8eeaf to d341351Compare December 13, 2024 11:26

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2024

ashb closed this Dec 13, 2024

ashb reopened this Dec 13, 2024

Move ParseImportError formatting/depth tests to dagbag
cd8addc
This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

kaxil reviewed Dec 13, 2024
View reviewed changes

ashb added 2 commits December 13, 2024 14:29

fixup! Move ParseImportError formatting/depth tests to dagbag
2f698b8

fixup! Move ParseImportError formatting/depth tests to dagbag
84a80f1

kaxil approved these changes Dec 13, 2024
View reviewed changes

ashb merged commit ec893fd into apache:mainDec 13, 2024

ashb deleted the extract-dag-db-update-to-one-place branch December 13, 2024 15:51

ashb mentioned this pull request Dec 16, 2024
Swap Dag Parsing to use the TaskSDK machinery. #44972
Merged

boring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Dec 12, 2024

ashb changed the title ~~Extract all the various DB update calls and processes for Dag Parsing…~~Extract all the various DB update calls and processes for Dag Parsing to one placeDec 13, 2024

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadairflow/dag_processing/processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/models/test_dagbag.py

ashb marked this pull request as ready for review December 13, 2024 11:07

ashb requested review from XD-DENG, ephraimbuddy and jedcunningham as code owners December 13, 2024 11:07

ashb force-pushed the extract-dag-db-update-to-one-place branch from 1a8eeaf to d341351Compare December 13, 2024 11:26

ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2024

ashb closed this Dec 13, 2024

ashb reopened this Dec 13, 2024

Move ParseImportError formatting/depth tests to dagbag
cd8addc
This is a property of dagbag parsing, not the processor. Also convert it to a parameterizes test too

ashb reviewed Dec 13, 2024
View reviewed changes

Comment threadtests/dag_processing/test_processor.py

kaxil reviewed Dec 13, 2024
View reviewed changes

ashb added 2 commits December 13, 2024 14:29

fixup! Move ParseImportError formatting/depth tests to dagbag
2f698b8

fixup! Move ParseImportError formatting/depth tests to dagbag
84a80f1

kaxil approved these changes Dec 13, 2024
View reviewed changes

ashb merged commit ec893fd into apache:mainDec 13, 2024

ashb deleted the extract-dag-db-update-to-one-place branch December 13, 2024 15:51

ashb mentioned this pull request Dec 16, 2024
Swap Dag Parsing to use the TaskSDK machinery. #44972
Merged

Uh oh!

Conversation

dimberman commented Dec 12, 2024• edited by ashb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashb commented Dec 13, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dimberman commented Dec 12, 2024• edited by ashb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashb commented Dec 13, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dimberman commented Dec 12, 2024• edited by ashb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashb commented Dec 13, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimberman commented Dec 12, 2024•
edited by ashb
Loading

dimberman commented Dec 12, 2024•
edited by ashb
Loading

dimberman commented Dec 12, 2024•
edited by ashb
Loading

dimberman commented Dec 12, 2024•
edited by ashb
Loading

dimberman commented Dec 12, 2024•
edited by ashb
Loading