AIP-72: Port task success overtime to the Supervisor - #44590

Merged

kaxil merged 1 commit into

apache:mainfrom

astronomer:add-overtime

Dec 3, 2024

kaxil commented Dec 3, 2024•
edited
Loading

Member

This PR ports the overtime feature on LocalTaskJob (added in #39890) to the Supervisor. It allows the Task process to terminate when it is exceeding the configured overtime threshold which is useful when we add Listenener to the Task process.

The overtime configured here applies to all states not just success, as was the case with LTJ.

closes#44356

Also added TaskState to update state and send end_date from task process to the supervisor.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

kaxil added the area:task-execution-interface-aip72 label

kaxil requested review from amoghrajesh and ashb

December 3, 2024 02:03

boring-cyborgBot added the area:task-sdk label

kaxil force-pushed the add-overtime branch 6 times, most recently from 1d0afab to 4be85f7Compare

December 3, 2024 03:29

ashb reviewed

View reviewed changes

task_sdk/src/airflow/sdk/execution_time/supervisor.py

ashb reviewed

View reviewed changes

task_sdk/src/airflow/sdk/execution_time/supervisor.py Outdated

ashb reviewed

View reviewed changes

task_sdk/src/airflow/sdk/execution_time/supervisor.py Outdated

ashb reviewed

View reviewed changes

task_sdk/src/airflow/sdk/execution_time/supervisor.py Outdated

ashb reviewed

View reviewed changes

task_sdk/src/airflow/sdk/execution_time/task_runner.py

ashb reviewed

View reviewed changes

task_sdk/tests/execution_time/test_supervisor.py Outdated

ashb reviewed

View reviewed changes

task_sdk/src/airflow/sdk/execution_time/task_runner.py Outdated

kaxil force-pushed the add-overtime branch from 4be85f7 to 752e4f3Compare

December 3, 2024 14:45

kaxil requested a review from ashb

December 3, 2024 14:49

kaxil commented

View reviewed changes

task_sdk/src/airflow/sdk/execution_time/comms.py

ashb reviewed

View reviewed changes

task_sdk/src/airflow/sdk/execution_time/supervisor.py Outdated

AIP-72: Port task success overtime to the Supervisor

5b8fd67

This PR ports the overtime feature on `LocalTaskJob` (added in apache#39890) to the Supervisor.
It allows to terminate Task process to terminate when it exceeding the configured success overtime threshold which is useful when we add Listenener to the Task process.
closesapache#44356
Also added `TaskState` to update state and send end_date from task process to the supervisor.

kaxil force-pushed the add-overtime branch from 752e4f3 to 5b8fd67Compare

December 3, 2024 15:05

ashb reviewed

View reviewed changes

task_sdk/tests/execution_time/test_task_runner.py

ashb approved these changes

View reviewed changes

kaxil mentioned this pull request

Rework SUPERVISOR_COMMS.send_request class/method to have log stored as a attribute #44615

Open

kaxil merged commit d059d4a into apache:main

kaxil deleted the add-overtime branch

December 3, 2024 15:47

got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request

AIP-72: Port task success overtime to the Supervisor (apache#44590)

5b6f468

This PR ports the overtime feature on `LocalTaskJob` (added in apache#39890) to the Supervisor.
It allows to terminate Task process to terminate when it exceeding the configured success overtime threshold which is useful when we add Listenener to the Task process.
closesapache#44356
Also added `TaskState` to update state and send end_date from task process to the supervisor.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:task-execution-interface-aip72area:task-sdk