Rename JobRunner modules to *_job_runner and base_job* to job by potiuk · Pull Request #30302 · apache/airflow

potiuk · 2023-03-25T21:27:39Z

The #30255 introduced "JobRunner" concept and decoupled the job logic
from the ORM polymorphic *Job objects. The change was implemented
in the way to minimise the review effort needed, so it avoided renaming
the modules for the runners (from _job to _job_runner).

Also BaseJob lost its "polymorphism" properties so the package, and class name
can be renamed to simply job.

This PR completes the JobRunner concept introduction by applying the
renames.

Closes: #30296

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-03-25T21:29:17Z

Based on #30255, so please check only the last commit.

potiuk · 2023-04-09T22:35:44Z

Not needed any more - theose are not polymorphic an do not need to be registered at init.

potiuk · 2023-04-10T09:19:22Z

Now - this one is pure rename of the modules + claass names to reflect the refactoring from #30255:

*_job.py -> *_job_runner.py
base_job.py -> job.py
BaseJob -> Job
BasJobPydantic -> JobPydantic

potiuk · 2023-04-10T09:34:29Z

This one should be super-easy to review :)

The apache#30255 introduced "JobRunner" concept and decoupled the job logic from the ORM polymorphic *Job objects. The change was implemented in the way to minimise the review effort needed, so it avoided renaming the modules for the runners (from `_job` to `_job_runner`). Also BaseJob lost its "polymorphism" properties so the package, and class name can be renamed to simply job. This PR completes the JobRunner concept introduction by applying the renames. Closes: apache#30296

potiuk · 2023-03-25T21:27:39Z

The #30255 introduced "JobRunner" concept and decoupled the job logic
from the ORM polymorphic *Job objects. The change was implemented
in the way to minimise the review effort needed, so it avoided renaming
the modules for the runners (from _job to _job_runner).

Also BaseJob lost its "polymorphism" properties so the package, and class name
can be renamed to simply job.

This PR completes the JobRunner concept introduction by applying the
renames.

Closes: #30296

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-03-25T21:29:17Z

Based on #30255, so please check only the last commit.

potiuk · 2023-04-09T22:35:44Z

Not needed any more - theose are not polymorphic an do not need to be registered at init.

potiuk · 2023-04-10T09:19:22Z

Now - this one is pure rename of the modules + claass names to reflect the refactoring from #30255:

*_job.py -> *_job_runner.py
base_job.py -> job.py
BaseJob -> Job
BasJobPydantic -> JobPydantic

potiuk · 2023-04-10T09:34:29Z

This one should be super-easy to review :)

The apache#30255 introduced "JobRunner" concept and decoupled the job logic from the ORM polymorphic *Job objects. The change was implemented in the way to minimise the review effort needed, so it avoided renaming the modules for the runners (from `_job` to `_job_runner`). Also BaseJob lost its "polymorphism" properties so the package, and class name can be renamed to simply job. This PR completes the JobRunner concept introduction by applying the renames. Closes: apache#30296

potiuk · 2023-03-25T21:27:39Z

The #30255 introduced "JobRunner" concept and decoupled the job logic
from the ORM polymorphic *Job objects. The change was implemented
in the way to minimise the review effort needed, so it avoided renaming
the modules for the runners (from _job to _job_runner).

Also BaseJob lost its "polymorphism" properties so the package, and class name
can be renamed to simply job.

This PR completes the JobRunner concept introduction by applying the
renames.

Closes: #30296

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-03-25T21:29:17Z

Based on #30255, so please check only the last commit.

potiuk · 2023-04-09T22:35:44Z

Not needed any more - theose are not polymorphic an do not need to be registered at init.

potiuk · 2023-04-10T09:19:22Z

Now - this one is pure rename of the modules + claass names to reflect the refactoring from #30255:

*_job.py -> *_job_runner.py
base_job.py -> job.py
BaseJob -> Job
BasJobPydantic -> JobPydantic

potiuk · 2023-04-10T09:34:29Z

This one should be super-easy to review :)

The apache#30255 introduced "JobRunner" concept and decoupled the job logic from the ORM polymorphic *Job objects. The change was implemented in the way to minimise the review effort needed, so it avoided renaming the modules for the runners (from `_job` to `_job_runner`). Also BaseJob lost its "polymorphism" properties so the package, and class name can be renamed to simply job. This PR completes the JobRunner concept introduction by applying the renames. Closes: apache#30296

potiuk · 2023-03-25T21:27:39Z

The #30255 introduced "JobRunner" concept and decoupled the job logic
from the ORM polymorphic *Job objects. The change was implemented
in the way to minimise the review effort needed, so it avoided renaming
the modules for the runners (from _job to _job_runner).

Also BaseJob lost its "polymorphism" properties so the package, and class name
can be renamed to simply job.

This PR completes the JobRunner concept introduction by applying the
renames.

Closes: #30296

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-03-25T21:29:17Z

Based on #30255, so please check only the last commit.

potiuk · 2023-04-09T22:35:44Z

Not needed any more - theose are not polymorphic an do not need to be registered at init.

potiuk · 2023-04-10T09:19:22Z

Now - this one is pure rename of the modules + claass names to reflect the refactoring from #30255:

*_job.py -> *_job_runner.py
base_job.py -> job.py
BaseJob -> Job
BasJobPydantic -> JobPydantic

potiuk · 2023-04-10T09:34:29Z

This one should be super-easy to review :)

The apache#30255 introduced "JobRunner" concept and decoupled the job logic from the ORM polymorphic *Job objects. The change was implemented in the way to minimise the review effort needed, so it avoided renaming the modules for the runners (from `_job` to `_job_runner`). Also BaseJob lost its "polymorphism" properties so the package, and class name can be renamed to simply job. This PR completes the JobRunner concept introduction by applying the renames. Closes: apache#30296

potiuk · 2023-03-25T21:27:39Z

The #30255 introduced "JobRunner" concept and decoupled the job logic
from the ORM polymorphic *Job objects. The change was implemented
in the way to minimise the review effort needed, so it avoided renaming
the modules for the runners (from _job to _job_runner).

Also BaseJob lost its "polymorphism" properties so the package, and class name
can be renamed to simply job.

This PR completes the JobRunner concept introduction by applying the
renames.

Closes: #30296

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-03-25T21:29:17Z

Based on #30255, so please check only the last commit.

potiuk · 2023-04-09T22:35:44Z

Not needed any more - theose are not polymorphic an do not need to be registered at init.

potiuk · 2023-04-10T09:19:22Z

Now - this one is pure rename of the modules + claass names to reflect the refactoring from #30255:

*_job.py -> *_job_runner.py
base_job.py -> job.py
BaseJob -> Job
BasJobPydantic -> JobPydantic

potiuk · 2023-04-10T09:34:29Z

This one should be super-easy to review :)

The apache#30255 introduced "JobRunner" concept and decoupled the job logic from the ORM polymorphic *Job objects. The change was implemented in the way to minimise the review effort needed, so it avoided renaming the modules for the runners (from `_job` to `_job_runner`). Also BaseJob lost its "polymorphism" properties so the package, and class name can be renamed to simply job. This PR completes the JobRunner concept introduction by applying the renames. Closes: apache#30296

potiuk · 2023-03-25T21:27:39Z

The #30255 introduced "JobRunner" concept and decoupled the job logic
from the ORM polymorphic *Job objects. The change was implemented
in the way to minimise the review effort needed, so it avoided renaming
the modules for the runners (from _job to _job_runner).

Also BaseJob lost its "polymorphism" properties so the package, and class name
can be renamed to simply job.

This PR completes the JobRunner concept introduction by applying the
renames.

Closes: #30296

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-03-25T21:29:17Z

Based on #30255, so please check only the last commit.

potiuk · 2023-04-09T22:35:44Z

Not needed any more - theose are not polymorphic an do not need to be registered at init.

potiuk · 2023-04-10T09:19:22Z

Now - this one is pure rename of the modules + claass names to reflect the refactoring from #30255:

*_job.py -> *_job_runner.py
base_job.py -> job.py
BaseJob -> Job
BasJobPydantic -> JobPydantic

potiuk · 2023-04-10T09:34:29Z

This one should be super-easy to review :)

The apache#30255 introduced "JobRunner" concept and decoupled the job logic from the ORM polymorphic *Job objects. The change was implemented in the way to minimise the review effort needed, so it avoided renaming the modules for the runners (from `_job` to `_job_runner`). Also BaseJob lost its "polymorphism" properties so the package, and class name can be renamed to simply job. This PR completes the JobRunner concept introduction by applying the renames. Closes: apache#30296

potiuk · 2023-03-25T21:27:39Z

The #30255 introduced "JobRunner" concept and decoupled the job logic
from the ORM polymorphic *Job objects. The change was implemented
in the way to minimise the review effort needed, so it avoided renaming
the modules for the runners (from _job to _job_runner).

Also BaseJob lost its "polymorphism" properties so the package, and class name
can be renamed to simply job.

This PR completes the JobRunner concept introduction by applying the
renames.

Closes: #30296

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-03-25T21:29:17Z

Based on #30255, so please check only the last commit.

potiuk · 2023-04-09T22:35:44Z

Not needed any more - theose are not polymorphic an do not need to be registered at init.

potiuk · 2023-04-10T09:19:22Z

Now - this one is pure rename of the modules + claass names to reflect the refactoring from #30255:

*_job.py -> *_job_runner.py
base_job.py -> job.py
BaseJob -> Job
BasJobPydantic -> JobPydantic

potiuk · 2023-04-10T09:34:29Z

This one should be super-easy to review :)

The apache#30255 introduced "JobRunner" concept and decoupled the job logic from the ORM polymorphic *Job objects. The change was implemented in the way to minimise the review effort needed, so it avoided renaming the modules for the runners (from `_job` to `_job_runner`). Also BaseJob lost its "polymorphism" properties so the package, and class name can be renamed to simply job. This PR completes the JobRunner concept introduction by applying the renames. Closes: apache#30296

potiuk · 2023-03-25T21:27:39Z

The #30255 introduced "JobRunner" concept and decoupled the job logic
from the ORM polymorphic *Job objects. The change was implemented
in the way to minimise the review effort needed, so it avoided renaming
the modules for the runners (from _job to _job_runner).

Also BaseJob lost its "polymorphism" properties so the package, and class name
can be renamed to simply job.

This PR completes the JobRunner concept introduction by applying the
renames.

Closes: #30296

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-03-25T21:29:17Z

Based on #30255, so please check only the last commit.

potiuk · 2023-04-09T22:35:44Z

Not needed any more - theose are not polymorphic an do not need to be registered at init.

potiuk · 2023-04-10T09:19:22Z

Now - this one is pure rename of the modules + claass names to reflect the refactoring from #30255:

*_job.py -> *_job_runner.py
base_job.py -> job.py
BaseJob -> Job
BasJobPydantic -> JobPydantic

potiuk · 2023-04-10T09:34:29Z

This one should be super-easy to review :)

The apache#30255 introduced "JobRunner" concept and decoupled the job logic from the ORM polymorphic *Job objects. The change was implemented in the way to minimise the review effort needed, so it avoided renaming the modules for the runners (from `_job` to `_job_runner`). Also BaseJob lost its "polymorphism" properties so the package, and class name can be renamed to simply job. This PR completes the JobRunner concept introduction by applying the renames. Closes: apache#30296

potiuk requested review from XD-DENG, ashb, bbovenzi, dstandish, ephraimbuddy, jedcunningham, kaxil, o-nikolas and ryanahamilton as code owners March 25, 2023 21:27

boring-cyborgBot added area:API Airflow's REST/HTTP API area:CLI area:Scheduler including HA (high availability) scheduler area:webserver Webserver related Issues labels Mar 25, 2023

potiuk changed the title ~~Change module names for job runners~~Rename JobRunner modules to *_job_runnerMar 25, 2023

potiuk closed this Mar 25, 2023

potiuk reopened this Mar 25, 2023

potiuk added the AIP-44 Airflow Internal API label Mar 26, 2023

This was referenced Mar 26, 2023
Separate and split run job method into prepare/execute/complete steps #30308
Merged
AIP-44 Migrate BaseJob.run to Internal API #29316
Open

potiuk force-pushed the change-module-names-for-job-runners branch 3 times, most recently from ff62bc7 to b5fe668Compare March 27, 2023 10:25

This was referenced Mar 27, 2023
AIP-44 Remove job_runner from BaseJob as a final step of decoupling #30325
Closed
Add retry to the scheduler loop to protect against DB hiccups #29815
Closed

potiuk force-pushed the change-module-names-for-job-runners branch from b5fe668 to 2d9ebfdCompare March 30, 2023 17:56

potiuk mentioned this pull request Mar 30, 2023
Remove JobRunners back reference from Job #30376
Merged

potiuk commented Apr 9, 2023
View reviewed changes

potiuk force-pushed the change-module-names-for-job-runners branch from dc3e7c9 to 4a18d1cCompare April 10, 2023 09:17

potiuk force-pushed the change-module-names-for-job-runners branch from 4a18d1c to 6bacc03Compare April 10, 2023 09:22

uranusjr reviewed Apr 10, 2023
View reviewed changes

Comment threadairflow/cli/commands/jobs_command.py Outdated

potiuk force-pushed the change-module-names-for-job-runners branch from 7e8d971 to d76aa13Compare April 10, 2023 12:36

uranusjr reviewed Apr 10, 2023
View reviewed changes

Comment threadairflow/models/__init__.py Outdated

potiuk force-pushed the change-module-names-for-job-runners branch from d76aa13 to 30cad5eCompare April 10, 2023 12:57

jedcunningham approved these changes Apr 10, 2023
View reviewed changes

jedcunningham reviewed Apr 10, 2023
View reviewed changes

Comment threadairflow/serialization/serialized_objects.py Outdated

potiuk force-pushed the change-module-names-for-job-runners branch from 1db3239 to e785922Compare April 10, 2023 20:50

potiuk merged commit a1b99fe into apache:mainApr 10, 2023

potiuk deleted the change-module-names-for-job-runners branch April 10, 2023 21:33

davidspek mentioned this pull request May 2, 2023
Liveness probes don't work for airflow 2.6.0 airflow-helm/charts#737
Closed
2 tasks

This was referenced May 13, 2023
Constant "The scheduler does not appear to be running" warning on the UI following 2.6.0 upgrade #31200
Closed
fix: liveness probes in airflow 2.6.0 airflow-helm/charts#743
Merged

potiuk mentioned this pull request May 14, 2023
Fix calculation of health check threshold for SchedulerJob #31277
Merged

thesuperzapper mentioned this pull request May 14, 2023
fix: liveness probes airflow 2.6.0 support airflow-helm/charts#738
Closed
6 tasks

Jie211 mentioned this pull request Jun 5, 2024
job table missing dag_id and executor_class for BackfillJob type #40055
Open
2 tasks