Skip to content

Fix TypeError clearing a DAG run when a queued-at deadline interval is stored as JSON - #70370

Open
hkc-8010 wants to merge 2 commits into
apache:mainfrom
hkc-8010:fix/deadline-interval-clear-recalc
Open

Fix TypeError clearing a DAG run when a queued-at deadline interval is stored as JSON#70370
hkc-8010 wants to merge 2 commits into
apache:mainfrom
hkc-8010:fix/deadline-interval-clear-recalc

Conversation

@hkc-8010

@hkc-8010hkc-8010 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Clearing a DAG run or task instance whose DAG defines a deadline referencing the run's queued-at time (DeadlineReference.DAGRUN_QUEUED_AT) raised TypeError: unsupported type for timedelta seconds component: dict and surfaced as an HTTP 500 in the API server.

Since 3.3.0 the deadline_alert.interval column is JSON (migration 0117_3_3_0_change_deadline_interval_to_json.py, from #64751), holding a serialized timedelta or VariableInterval, e.g. {"__classname__": "datetime.timedelta", "__version__": 2, "__data__": 3600.0}. _recalculate_dagrun_queued_at_deadlines (called from clear_task_instances) still passed that raw dict into timedelta(seconds=...). Every other consumer (DagRun creation in serialization/definitions/dag.py, the model __repr__) decodes the JSON first; only the clear path was missed.

DagRun creation already had the decode plus interval resolution written out inline, so rather than copy it, both steps move into serialization/decoders.py and both call sites use them.

Changes

  • airflow-core/src/airflow/serialization/decoders.py: add decode_deadline_alert_model() (decode a DeadlineAlert ORM row) and resolve_deadline_alert_interval() (resolve a VariableInterval to a timedelta). VariableInterval moves to a module-level import so the two functions share it, and decode_deadline_alert gets its missing return annotation.
  • airflow-core/src/airflow/models/taskinstance.py: _recalculate_dagrun_queued_at_deadlines calls the two helpers instead of building the encoded dict itself. No airflow.sdk import.
  • airflow-core/src/airflow/serialization/definitions/dag.py: _process_dagrun_deadline_alerts was doing the same two steps inline and now calls the helpers, so its VariableInterval, DeadlineAlertFields, DAT and Encoding imports are gone.
  • generated/known_sdk_imports_in_core.txt: the hook tightened definitions/dag.py from 2 to 1.
  • airflow-core/tests/unit/models/test_taskinstance.py: the existing test stored the interval as a plain float, so it never exercised the JSON path. It now stores the serialized envelope and is parametrized over a fixed timedelta and a VariableInterval. Both cases fail on main with the reported TypeError and pass with the fix.

The core-side SerializedVariableInterval counterpart is being added separately in #71802. resolve_deadline_alert_interval() is the single place that isinstance-checks the SDK dataclass, so that PR changes it in one spot without moving any call site.

closes: #70368

PR Checklist

  • Targeted at the main branch
  • Tests added (fixed-interval and VariableInterval cases, both reproduce the bug on main)
  • prek hooks pass on the changed files (ruff, ruff-format, mypy for airflow-core, check-sdk-imports)

@potiukpotiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 28, 2026
@eladkaleladkal added this to the Airflow 3.3.2 milestone Aug 5, 2026
@eladkaleladkal added type:bug-fix Changelog: Bug Fixes backport-to-v3-3-test Backport to v3-3-test labels Aug 5, 2026
Comment threadairflow-core/src/airflow/models/taskinstance.py Outdated

@amoghrajeshamoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This propagates the same coupling I raised in #64751 - isinstance(interval, VariableInterval) here imports airflow.sdk.definitions.deadline.VariableInterval into core airflow at runtime. See #64751 (review) for details.

…s stored as JSON
Since 3.3.0 the deadline_alert.interval column is JSON (migration 0117, apache#64751),
holding a serialized timedelta or VariableInterval. _recalculate_dagrun_queued_at_deadlines
still passed that raw dict into timedelta(seconds=...), so clearing a DAG run or task
instance whose DAG has a DagRunQueuedAtDeadline failed with
"TypeError: unsupported type for timedelta seconds component: dict" (HTTP 500).
Decode the interval the same way DagRun creation does (decode_deadline_alert), then
resolve a VariableInterval to a timedelta before recalculating. The existing test stored
the interval as a plain float and never exercised the JSON path; it now stores the
serialized envelope and is parametrized over fixed-timedelta and VariableInterval.
closes: apache#70368
Move the decode and interval resolution into decode_deadline_alert_model() and
resolve_deadline_alert_interval() in serialization/decoders.py, so the deadline
recalculation on clear does not need its own copy or an airflow.sdk import.
SerializedDAG._process_dagrun_deadline_alerts was doing the same two steps
inline and now calls the helpers, dropping its VariableInterval import.
closes: apache#70368
@hkc-8010
hkc-8010force-pushed the fix/deadline-interval-clear-recalc branch from d906351 to 92c00e3CompareAugust 23, 2026 09:51
@hkc-8010

Copy link
Copy Markdown
ContributorAuthor

@amoghrajesh Agreed the counterpart is the real fix, and #71802 is already adding SerializedVariableInterval for it. I did not want this bugfix to depend on that landing, so here the SDK type is confined to one function, resolve_deadline_alert_interval() in serialization/decoders.py, which both DagRun creation and the deadline recalculation now call. That is the only isinstance against the SDK dataclass left in this path, so when #71802 lands it changes in one place and no call site moves.

The duplication you flagged is gone too. taskinstance.py has no SDK import, and definitions/dag.py dropped its VariableInterval import, so the hook tightened generated/known_sdk_imports_in_core.txt from 2 to 1 for that file.

@ferruzziferruzzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amoghrajesh looks like the SDK coupling you flagged has been addressed, and I think this is clear to move now if you agree.

models/taskinstance.py no longer imports from the SDK at all, and the single remaining isinstance(..., VariableInterval) is confined to resolve_deadline_alert_interval() in serialization/decoders.py. Worth noting that decoders.py already imported VariableInterval (function-locally, inside decode_deadline_alert), so this consolidates an existing coupling site rather than adding one, and serialization/definitions/dag.py loses its own import in the process. The core to SDK coupling is one file lower than before the PR.

#71802 then replaces that last isinstance in one place, so nothing here has to move again.

This PR is the only one of the three currently touching _recalculate_dagrun_queued_at_deadlines that is milestoned and backport labelled, so it is the only route this fix has to 3.3.x. More context in my comments on #70714 and #71850.

:meta private:
"""
if isinstance(alert.interval, VariableInterval):
return alert.interval.resolve()

@ferruzziferruzziAug 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolve() can raise ValueError (for example VariableInterval.resolve() raises on a non-integer), so on the clear path this can abort the dagrun clear even though there is a valid deadline in the db. Since dagrun create and dagrun clear both filter through here now, I think we should be catching and logging in the taskinstance.py where this gets called IF we're clearing.

import dateutil.relativedelta

from airflow._shared.module_loading import import_string
from airflow.sdk.definitions.deadline import VariableInterval

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this one. We moved this sdk import to the module level but left the other sdk import below it as a local?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v3-3-testBackport to v3-3-testready for maintainer reviewSet after triaging when all criteria pass.type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clearing a DAG run/task instance with a DagRunQueuedAtDeadline fails with TypeError: unsupported type for timedelta seconds component: dict

6 participants

@hkc-8010@ferruzzi@kaxil@amoghrajesh@potiuk@eladkal