Deliver the per-run log ID template to workers via TIRunContext by jason810496 · Pull Request #69688 · apache/airflow

jason810496 · 2026-07-10T05:49:00Z

related: Drop the LogTemplate DB model #69520 (discussion on dropping vs. fixing the LogTemplate flow)
next:
- Make Elasticsearch log handler respect the per-run log ID template #69896
- Make OpenSearch log handler respect the per-run log ID template #69897

Why

The LogTemplate model pins the log ID template to each Dag run so task logs written before a log_id_template config change stay readable, but on the Airflow 3 write path workers render the log ID from the current conf value — they cannot query the metadata DB to fetch the pinned row. This PR adds the core plumbing that hands the pinned template to workers; the Elasticsearch and OpenSearch handler changes that consume it follow in per-provider PRs stacked on this one.

How

The Dag-run-pinned LogTemplate.elasticsearch_id is delivered to the supervisor via a new optional TIRunContext.log_id_template field, populated by the ti_run Execution API endpoint.
upload_to_remote forwards the run context only to handlers whose upload opts into a ti_context keyword, so S3/GCS/third-party RemoteLogIO implementations and old-provider × new-sdk combinations are untouched.

What

airflow-core: add TIRunContext.log_id_template; populate it in the ti_run route from dr.get_log_template(); add Execution API version 2026-09-30 with a Cadwyn migration.
task-sdk: retain the TIRunContext on ActivitySubprocess and pass it to upload_to_remote; regenerate the client datamodels; add the first in-progress supervisor-schema version file plus the regenerated schema.json snapshot.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Fable 5) following the guidelines

The LogTemplate model pins the log_id template to each Dag run so task logs written before a config change stay readable, but the Airflow 3 paths of the Elasticsearch and OpenSearch handlers rendered log_id from the current conf value on both the write (worker) and read (API server) sides, so changing [elasticsearch/opensearch] log_id_template orphaned all previously written logs. Workers cannot query the metadata DB, so the pinned template is delivered to the supervisor through a new optional TIRunContext field populated by the ti_run Execution API endpoint (with a version migration for older clients); the API-server read paths restore the per-run lookup directly. Handlers that do not opt into the new ti_context keyword keep their existing upload signature and behavior.

…r-side TIs Worker-side RuntimeTIs have no get_dagrun and cannot reach the metadata DB; the pinned template reaches them through TIRunContext instead. Opening a session just to fail also trips the DB-access guard (AirflowInternalRuntimeError is a BaseException, so the fallback except clause cannot catch it) in non-DB test runs.

…n supervisor tests The remote-logging supervisor tests drop airflow.sdk.log (and the logging-config modules) from sys.modules to force a fresh import, but the re-import rebinds the attribute on the parent package and monkeypatch only restores the sys.modules entry. On Python < 3.12 mock.patch resolves dotted targets through getattr on the parent package, so later tests on the same worker patched the stale module object while the code under test imported the restored one, failing test_upload_logs_forwards_ti_context and TestUploadToRemoteTIContext::test_no_ti_context_keeps_handler_default in CI.

Keep this branch scoped to the core plumbing (Execution API and Task SDK) that delivers the per-run log ID template to workers. The Elasticsearch and OpenSearch handler changes re-land in per-provider PRs stacked on this branch, so each provider change can be reviewed and released on its own cadence.

…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 · 2026-07-10T05:49:00Z

related: Drop the LogTemplate DB model #69520 (discussion on dropping vs. fixing the LogTemplate flow)
next:
- Make Elasticsearch log handler respect the per-run log ID template #69896
- Make OpenSearch log handler respect the per-run log ID template #69897

Why

The LogTemplate model pins the log ID template to each Dag run so task logs written before a log_id_template config change stay readable, but on the Airflow 3 write path workers render the log ID from the current conf value — they cannot query the metadata DB to fetch the pinned row. This PR adds the core plumbing that hands the pinned template to workers; the Elasticsearch and OpenSearch handler changes that consume it follow in per-provider PRs stacked on this one.

How

The Dag-run-pinned LogTemplate.elasticsearch_id is delivered to the supervisor via a new optional TIRunContext.log_id_template field, populated by the ti_run Execution API endpoint.
upload_to_remote forwards the run context only to handlers whose upload opts into a ti_context keyword, so S3/GCS/third-party RemoteLogIO implementations and old-provider × new-sdk combinations are untouched.

What

airflow-core: add TIRunContext.log_id_template; populate it in the ti_run route from dr.get_log_template(); add Execution API version 2026-09-30 with a Cadwyn migration.
task-sdk: retain the TIRunContext on ActivitySubprocess and pass it to upload_to_remote; regenerate the client datamodels; add the first in-progress supervisor-schema version file plus the regenerated schema.json snapshot.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Fable 5) following the guidelines

The LogTemplate model pins the log_id template to each Dag run so task logs written before a config change stay readable, but the Airflow 3 paths of the Elasticsearch and OpenSearch handlers rendered log_id from the current conf value on both the write (worker) and read (API server) sides, so changing [elasticsearch/opensearch] log_id_template orphaned all previously written logs. Workers cannot query the metadata DB, so the pinned template is delivered to the supervisor through a new optional TIRunContext field populated by the ti_run Execution API endpoint (with a version migration for older clients); the API-server read paths restore the per-run lookup directly. Handlers that do not opt into the new ti_context keyword keep their existing upload signature and behavior.

…r-side TIs Worker-side RuntimeTIs have no get_dagrun and cannot reach the metadata DB; the pinned template reaches them through TIRunContext instead. Opening a session just to fail also trips the DB-access guard (AirflowInternalRuntimeError is a BaseException, so the fallback except clause cannot catch it) in non-DB test runs.

…n supervisor tests The remote-logging supervisor tests drop airflow.sdk.log (and the logging-config modules) from sys.modules to force a fresh import, but the re-import rebinds the attribute on the parent package and monkeypatch only restores the sys.modules entry. On Python < 3.12 mock.patch resolves dotted targets through getattr on the parent package, so later tests on the same worker patched the stale module object while the code under test imported the restored one, failing test_upload_logs_forwards_ti_context and TestUploadToRemoteTIContext::test_no_ti_context_keeps_handler_default in CI.

Keep this branch scoped to the core plumbing (Execution API and Task SDK) that delivers the per-run log ID template to workers. The Elasticsearch and OpenSearch handler changes re-land in per-provider PRs stacked on this branch, so each provider change can be reviewed and released on its own cadence.

…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 · 2026-07-10T05:49:00Z

related: Drop the LogTemplate DB model #69520 (discussion on dropping vs. fixing the LogTemplate flow)
next:
- Make Elasticsearch log handler respect the per-run log ID template #69896
- Make OpenSearch log handler respect the per-run log ID template #69897

Why

The LogTemplate model pins the log ID template to each Dag run so task logs written before a log_id_template config change stay readable, but on the Airflow 3 write path workers render the log ID from the current conf value — they cannot query the metadata DB to fetch the pinned row. This PR adds the core plumbing that hands the pinned template to workers; the Elasticsearch and OpenSearch handler changes that consume it follow in per-provider PRs stacked on this one.

How

The Dag-run-pinned LogTemplate.elasticsearch_id is delivered to the supervisor via a new optional TIRunContext.log_id_template field, populated by the ti_run Execution API endpoint.
upload_to_remote forwards the run context only to handlers whose upload opts into a ti_context keyword, so S3/GCS/third-party RemoteLogIO implementations and old-provider × new-sdk combinations are untouched.

What

airflow-core: add TIRunContext.log_id_template; populate it in the ti_run route from dr.get_log_template(); add Execution API version 2026-09-30 with a Cadwyn migration.
task-sdk: retain the TIRunContext on ActivitySubprocess and pass it to upload_to_remote; regenerate the client datamodels; add the first in-progress supervisor-schema version file plus the regenerated schema.json snapshot.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Fable 5) following the guidelines

The LogTemplate model pins the log_id template to each Dag run so task logs written before a config change stay readable, but the Airflow 3 paths of the Elasticsearch and OpenSearch handlers rendered log_id from the current conf value on both the write (worker) and read (API server) sides, so changing [elasticsearch/opensearch] log_id_template orphaned all previously written logs. Workers cannot query the metadata DB, so the pinned template is delivered to the supervisor through a new optional TIRunContext field populated by the ti_run Execution API endpoint (with a version migration for older clients); the API-server read paths restore the per-run lookup directly. Handlers that do not opt into the new ti_context keyword keep their existing upload signature and behavior.

…r-side TIs Worker-side RuntimeTIs have no get_dagrun and cannot reach the metadata DB; the pinned template reaches them through TIRunContext instead. Opening a session just to fail also trips the DB-access guard (AirflowInternalRuntimeError is a BaseException, so the fallback except clause cannot catch it) in non-DB test runs.

…n supervisor tests The remote-logging supervisor tests drop airflow.sdk.log (and the logging-config modules) from sys.modules to force a fresh import, but the re-import rebinds the attribute on the parent package and monkeypatch only restores the sys.modules entry. On Python < 3.12 mock.patch resolves dotted targets through getattr on the parent package, so later tests on the same worker patched the stale module object while the code under test imported the restored one, failing test_upload_logs_forwards_ti_context and TestUploadToRemoteTIContext::test_no_ti_context_keeps_handler_default in CI.

Keep this branch scoped to the core plumbing (Execution API and Task SDK) that delivers the per-run log ID template to workers. The Elasticsearch and OpenSearch handler changes re-land in per-provider PRs stacked on this branch, so each provider change can be reviewed and released on its own cadence.

…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 · 2026-07-10T05:49:00Z

related: Drop the LogTemplate DB model #69520 (discussion on dropping vs. fixing the LogTemplate flow)
next:
- Make Elasticsearch log handler respect the per-run log ID template #69896
- Make OpenSearch log handler respect the per-run log ID template #69897

Why

The LogTemplate model pins the log ID template to each Dag run so task logs written before a log_id_template config change stay readable, but on the Airflow 3 write path workers render the log ID from the current conf value — they cannot query the metadata DB to fetch the pinned row. This PR adds the core plumbing that hands the pinned template to workers; the Elasticsearch and OpenSearch handler changes that consume it follow in per-provider PRs stacked on this one.

How

The Dag-run-pinned LogTemplate.elasticsearch_id is delivered to the supervisor via a new optional TIRunContext.log_id_template field, populated by the ti_run Execution API endpoint.
upload_to_remote forwards the run context only to handlers whose upload opts into a ti_context keyword, so S3/GCS/third-party RemoteLogIO implementations and old-provider × new-sdk combinations are untouched.

What

airflow-core: add TIRunContext.log_id_template; populate it in the ti_run route from dr.get_log_template(); add Execution API version 2026-09-30 with a Cadwyn migration.
task-sdk: retain the TIRunContext on ActivitySubprocess and pass it to upload_to_remote; regenerate the client datamodels; add the first in-progress supervisor-schema version file plus the regenerated schema.json snapshot.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Fable 5) following the guidelines

The LogTemplate model pins the log_id template to each Dag run so task logs written before a config change stay readable, but the Airflow 3 paths of the Elasticsearch and OpenSearch handlers rendered log_id from the current conf value on both the write (worker) and read (API server) sides, so changing [elasticsearch/opensearch] log_id_template orphaned all previously written logs. Workers cannot query the metadata DB, so the pinned template is delivered to the supervisor through a new optional TIRunContext field populated by the ti_run Execution API endpoint (with a version migration for older clients); the API-server read paths restore the per-run lookup directly. Handlers that do not opt into the new ti_context keyword keep their existing upload signature and behavior.

…r-side TIs Worker-side RuntimeTIs have no get_dagrun and cannot reach the metadata DB; the pinned template reaches them through TIRunContext instead. Opening a session just to fail also trips the DB-access guard (AirflowInternalRuntimeError is a BaseException, so the fallback except clause cannot catch it) in non-DB test runs.

…n supervisor tests The remote-logging supervisor tests drop airflow.sdk.log (and the logging-config modules) from sys.modules to force a fresh import, but the re-import rebinds the attribute on the parent package and monkeypatch only restores the sys.modules entry. On Python < 3.12 mock.patch resolves dotted targets through getattr on the parent package, so later tests on the same worker patched the stale module object while the code under test imported the restored one, failing test_upload_logs_forwards_ti_context and TestUploadToRemoteTIContext::test_no_ti_context_keeps_handler_default in CI.

Keep this branch scoped to the core plumbing (Execution API and Task SDK) that delivers the per-run log ID template to workers. The Elasticsearch and OpenSearch handler changes re-land in per-provider PRs stacked on this branch, so each provider change can be reviewed and released on its own cadence.

…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 · 2026-07-10T05:49:00Z

related: Drop the LogTemplate DB model #69520 (discussion on dropping vs. fixing the LogTemplate flow)
next:
- Make Elasticsearch log handler respect the per-run log ID template #69896
- Make OpenSearch log handler respect the per-run log ID template #69897

Why

The LogTemplate model pins the log ID template to each Dag run so task logs written before a log_id_template config change stay readable, but on the Airflow 3 write path workers render the log ID from the current conf value — they cannot query the metadata DB to fetch the pinned row. This PR adds the core plumbing that hands the pinned template to workers; the Elasticsearch and OpenSearch handler changes that consume it follow in per-provider PRs stacked on this one.

How

The Dag-run-pinned LogTemplate.elasticsearch_id is delivered to the supervisor via a new optional TIRunContext.log_id_template field, populated by the ti_run Execution API endpoint.
upload_to_remote forwards the run context only to handlers whose upload opts into a ti_context keyword, so S3/GCS/third-party RemoteLogIO implementations and old-provider × new-sdk combinations are untouched.

What

airflow-core: add TIRunContext.log_id_template; populate it in the ti_run route from dr.get_log_template(); add Execution API version 2026-09-30 with a Cadwyn migration.
task-sdk: retain the TIRunContext on ActivitySubprocess and pass it to upload_to_remote; regenerate the client datamodels; add the first in-progress supervisor-schema version file plus the regenerated schema.json snapshot.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Fable 5) following the guidelines

The LogTemplate model pins the log_id template to each Dag run so task logs written before a config change stay readable, but the Airflow 3 paths of the Elasticsearch and OpenSearch handlers rendered log_id from the current conf value on both the write (worker) and read (API server) sides, so changing [elasticsearch/opensearch] log_id_template orphaned all previously written logs. Workers cannot query the metadata DB, so the pinned template is delivered to the supervisor through a new optional TIRunContext field populated by the ti_run Execution API endpoint (with a version migration for older clients); the API-server read paths restore the per-run lookup directly. Handlers that do not opt into the new ti_context keyword keep their existing upload signature and behavior.

…r-side TIs Worker-side RuntimeTIs have no get_dagrun and cannot reach the metadata DB; the pinned template reaches them through TIRunContext instead. Opening a session just to fail also trips the DB-access guard (AirflowInternalRuntimeError is a BaseException, so the fallback except clause cannot catch it) in non-DB test runs.

…n supervisor tests The remote-logging supervisor tests drop airflow.sdk.log (and the logging-config modules) from sys.modules to force a fresh import, but the re-import rebinds the attribute on the parent package and monkeypatch only restores the sys.modules entry. On Python < 3.12 mock.patch resolves dotted targets through getattr on the parent package, so later tests on the same worker patched the stale module object while the code under test imported the restored one, failing test_upload_logs_forwards_ti_context and TestUploadToRemoteTIContext::test_no_ti_context_keeps_handler_default in CI.

Keep this branch scoped to the core plumbing (Execution API and Task SDK) that delivers the per-run log ID template to workers. The Elasticsearch and OpenSearch handler changes re-land in per-provider PRs stacked on this branch, so each provider change can be reviewed and released on its own cadence.

…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 · 2026-07-10T05:49:00Z

related: Drop the LogTemplate DB model #69520 (discussion on dropping vs. fixing the LogTemplate flow)
next:
- Make Elasticsearch log handler respect the per-run log ID template #69896
- Make OpenSearch log handler respect the per-run log ID template #69897

Why

The LogTemplate model pins the log ID template to each Dag run so task logs written before a log_id_template config change stay readable, but on the Airflow 3 write path workers render the log ID from the current conf value — they cannot query the metadata DB to fetch the pinned row. This PR adds the core plumbing that hands the pinned template to workers; the Elasticsearch and OpenSearch handler changes that consume it follow in per-provider PRs stacked on this one.

How

The Dag-run-pinned LogTemplate.elasticsearch_id is delivered to the supervisor via a new optional TIRunContext.log_id_template field, populated by the ti_run Execution API endpoint.
upload_to_remote forwards the run context only to handlers whose upload opts into a ti_context keyword, so S3/GCS/third-party RemoteLogIO implementations and old-provider × new-sdk combinations are untouched.

What

airflow-core: add TIRunContext.log_id_template; populate it in the ti_run route from dr.get_log_template(); add Execution API version 2026-09-30 with a Cadwyn migration.
task-sdk: retain the TIRunContext on ActivitySubprocess and pass it to upload_to_remote; regenerate the client datamodels; add the first in-progress supervisor-schema version file plus the regenerated schema.json snapshot.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Fable 5) following the guidelines

The LogTemplate model pins the log_id template to each Dag run so task logs written before a config change stay readable, but the Airflow 3 paths of the Elasticsearch and OpenSearch handlers rendered log_id from the current conf value on both the write (worker) and read (API server) sides, so changing [elasticsearch/opensearch] log_id_template orphaned all previously written logs. Workers cannot query the metadata DB, so the pinned template is delivered to the supervisor through a new optional TIRunContext field populated by the ti_run Execution API endpoint (with a version migration for older clients); the API-server read paths restore the per-run lookup directly. Handlers that do not opt into the new ti_context keyword keep their existing upload signature and behavior.

…r-side TIs Worker-side RuntimeTIs have no get_dagrun and cannot reach the metadata DB; the pinned template reaches them through TIRunContext instead. Opening a session just to fail also trips the DB-access guard (AirflowInternalRuntimeError is a BaseException, so the fallback except clause cannot catch it) in non-DB test runs.

…n supervisor tests The remote-logging supervisor tests drop airflow.sdk.log (and the logging-config modules) from sys.modules to force a fresh import, but the re-import rebinds the attribute on the parent package and monkeypatch only restores the sys.modules entry. On Python < 3.12 mock.patch resolves dotted targets through getattr on the parent package, so later tests on the same worker patched the stale module object while the code under test imported the restored one, failing test_upload_logs_forwards_ti_context and TestUploadToRemoteTIContext::test_no_ti_context_keeps_handler_default in CI.

Keep this branch scoped to the core plumbing (Execution API and Task SDK) that delivers the per-run log ID template to workers. The Elasticsearch and OpenSearch handler changes re-land in per-provider PRs stacked on this branch, so each provider change can be reviewed and released on its own cadence.

…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 · 2026-07-10T05:49:00Z

related: Drop the LogTemplate DB model #69520 (discussion on dropping vs. fixing the LogTemplate flow)
next:
- Make Elasticsearch log handler respect the per-run log ID template #69896
- Make OpenSearch log handler respect the per-run log ID template #69897

Why

The LogTemplate model pins the log ID template to each Dag run so task logs written before a log_id_template config change stay readable, but on the Airflow 3 write path workers render the log ID from the current conf value — they cannot query the metadata DB to fetch the pinned row. This PR adds the core plumbing that hands the pinned template to workers; the Elasticsearch and OpenSearch handler changes that consume it follow in per-provider PRs stacked on this one.

How

The Dag-run-pinned LogTemplate.elasticsearch_id is delivered to the supervisor via a new optional TIRunContext.log_id_template field, populated by the ti_run Execution API endpoint.
upload_to_remote forwards the run context only to handlers whose upload opts into a ti_context keyword, so S3/GCS/third-party RemoteLogIO implementations and old-provider × new-sdk combinations are untouched.

What

airflow-core: add TIRunContext.log_id_template; populate it in the ti_run route from dr.get_log_template(); add Execution API version 2026-09-30 with a Cadwyn migration.
task-sdk: retain the TIRunContext on ActivitySubprocess and pass it to upload_to_remote; regenerate the client datamodels; add the first in-progress supervisor-schema version file plus the regenerated schema.json snapshot.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Fable 5) following the guidelines

The LogTemplate model pins the log_id template to each Dag run so task logs written before a config change stay readable, but the Airflow 3 paths of the Elasticsearch and OpenSearch handlers rendered log_id from the current conf value on both the write (worker) and read (API server) sides, so changing [elasticsearch/opensearch] log_id_template orphaned all previously written logs. Workers cannot query the metadata DB, so the pinned template is delivered to the supervisor through a new optional TIRunContext field populated by the ti_run Execution API endpoint (with a version migration for older clients); the API-server read paths restore the per-run lookup directly. Handlers that do not opt into the new ti_context keyword keep their existing upload signature and behavior.

…r-side TIs Worker-side RuntimeTIs have no get_dagrun and cannot reach the metadata DB; the pinned template reaches them through TIRunContext instead. Opening a session just to fail also trips the DB-access guard (AirflowInternalRuntimeError is a BaseException, so the fallback except clause cannot catch it) in non-DB test runs.

…n supervisor tests The remote-logging supervisor tests drop airflow.sdk.log (and the logging-config modules) from sys.modules to force a fresh import, but the re-import rebinds the attribute on the parent package and monkeypatch only restores the sys.modules entry. On Python < 3.12 mock.patch resolves dotted targets through getattr on the parent package, so later tests on the same worker patched the stale module object while the code under test imported the restored one, failing test_upload_logs_forwards_ti_context and TestUploadToRemoteTIContext::test_no_ti_context_keeps_handler_default in CI.

Keep this branch scoped to the core plumbing (Execution API and Task SDK) that delivers the per-run log ID template to workers. The Elasticsearch and OpenSearch handler changes re-land in per-provider PRs stacked on this branch, so each provider change can be reviewed and released on its own cadence.

…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 · 2026-07-10T05:49:00Z

related: Drop the LogTemplate DB model #69520 (discussion on dropping vs. fixing the LogTemplate flow)
next:
- Make Elasticsearch log handler respect the per-run log ID template #69896
- Make OpenSearch log handler respect the per-run log ID template #69897

Why

The LogTemplate model pins the log ID template to each Dag run so task logs written before a log_id_template config change stay readable, but on the Airflow 3 write path workers render the log ID from the current conf value — they cannot query the metadata DB to fetch the pinned row. This PR adds the core plumbing that hands the pinned template to workers; the Elasticsearch and OpenSearch handler changes that consume it follow in per-provider PRs stacked on this one.

How

The Dag-run-pinned LogTemplate.elasticsearch_id is delivered to the supervisor via a new optional TIRunContext.log_id_template field, populated by the ti_run Execution API endpoint.
upload_to_remote forwards the run context only to handlers whose upload opts into a ti_context keyword, so S3/GCS/third-party RemoteLogIO implementations and old-provider × new-sdk combinations are untouched.

What

airflow-core: add TIRunContext.log_id_template; populate it in the ti_run route from dr.get_log_template(); add Execution API version 2026-09-30 with a Cadwyn migration.
task-sdk: retain the TIRunContext on ActivitySubprocess and pass it to upload_to_remote; regenerate the client datamodels; add the first in-progress supervisor-schema version file plus the regenerated schema.json snapshot.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Fable 5) following the guidelines

The LogTemplate model pins the log_id template to each Dag run so task logs written before a config change stay readable, but the Airflow 3 paths of the Elasticsearch and OpenSearch handlers rendered log_id from the current conf value on both the write (worker) and read (API server) sides, so changing [elasticsearch/opensearch] log_id_template orphaned all previously written logs. Workers cannot query the metadata DB, so the pinned template is delivered to the supervisor through a new optional TIRunContext field populated by the ti_run Execution API endpoint (with a version migration for older clients); the API-server read paths restore the per-run lookup directly. Handlers that do not opt into the new ti_context keyword keep their existing upload signature and behavior.

…r-side TIs Worker-side RuntimeTIs have no get_dagrun and cannot reach the metadata DB; the pinned template reaches them through TIRunContext instead. Opening a session just to fail also trips the DB-access guard (AirflowInternalRuntimeError is a BaseException, so the fallback except clause cannot catch it) in non-DB test runs.

…n supervisor tests The remote-logging supervisor tests drop airflow.sdk.log (and the logging-config modules) from sys.modules to force a fresh import, but the re-import rebinds the attribute on the parent package and monkeypatch only restores the sys.modules entry. On Python < 3.12 mock.patch resolves dotted targets through getattr on the parent package, so later tests on the same worker patched the stale module object while the code under test imported the restored one, failing test_upload_logs_forwards_ti_context and TestUploadToRemoteTIContext::test_no_ti_context_keeps_handler_default in CI.

Keep this branch scoped to the core plumbing (Execution API and Task SDK) that delivers the per-run log ID template to workers. The Elasticsearch and OpenSearch handler changes re-land in per-provider PRs stacked on this branch, so each provider change can be reviewed and released on its own cadence.

…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 self-assigned this Jul 10, 2026

boring-cyborgBot added area:API Airflow's REST/HTTP API area:logging area:providers area:task-sdk provider:elasticsearch provider:opensearch labels Jul 10, 2026

jason810496 added 2 commits July 11, 2026 06:35

jason810496 requested review from eladkal and uranusjr July 13, 2026 03:32

jason810496 marked this pull request as ready for review July 13, 2026 03:32

jason810496 requested review from amoghrajesh, ashb and kaxil as code owners July 13, 2026 03:32

jason810496 mentioned this pull request Jul 13, 2026
Drop the LogTemplate DB model #69520
Closed
1 task

This was referenced Jul 15, 2026
Make Elasticsearch log handler respect the per-run log ID template #69896
Draft
Make OpenSearch log handler respect the per-run log ID template #69897
Draft

jason810496 changed the title ~~Make ES and OpenSearch log handlers respect the per-run log ID template~~Deliver the per-run log ID template to workers via TIRunContextJul 15, 2026

phanikumv reviewed Jul 17, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py

Add a test asserting log_id_template always reflects the newest LogTe…
3bd7711
…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 added this to the Airflow 3.4.0 milestone Jul 18, 2026

jason810496 self-assigned this Jul 10, 2026

boring-cyborgBot added area:API Airflow's REST/HTTP API area:logging area:providers area:task-sdk provider:elasticsearch provider:opensearch labels Jul 10, 2026

jason810496 added 2 commits July 11, 2026 06:35

jason810496 requested review from eladkal and uranusjr July 13, 2026 03:32

jason810496 marked this pull request as ready for review July 13, 2026 03:32

jason810496 requested review from amoghrajesh, ashb and kaxil as code owners July 13, 2026 03:32

jason810496 mentioned this pull request Jul 13, 2026
Drop the LogTemplate DB model #69520
Closed
1 task

This was referenced Jul 15, 2026
Make Elasticsearch log handler respect the per-run log ID template #69896
Draft
Make OpenSearch log handler respect the per-run log ID template #69897
Draft

jason810496 changed the title ~~Make ES and OpenSearch log handlers respect the per-run log ID template~~Deliver the per-run log ID template to workers via TIRunContextJul 15, 2026

phanikumv reviewed Jul 17, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py

Add a test asserting log_id_template always reflects the newest LogTe…
3bd7711
…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 added this to the Airflow 3.4.0 milestone Jul 18, 2026

jason810496 self-assigned this Jul 10, 2026

boring-cyborgBot added area:API Airflow's REST/HTTP API area:logging area:providers area:task-sdk provider:elasticsearch provider:opensearch labels Jul 10, 2026

jason810496 added 2 commits July 11, 2026 06:35

jason810496 requested review from eladkal and uranusjr July 13, 2026 03:32

jason810496 marked this pull request as ready for review July 13, 2026 03:32

jason810496 requested review from amoghrajesh, ashb and kaxil as code owners July 13, 2026 03:32

jason810496 mentioned this pull request Jul 13, 2026
Drop the LogTemplate DB model #69520
Closed
1 task

This was referenced Jul 15, 2026
Make Elasticsearch log handler respect the per-run log ID template #69896
Draft
Make OpenSearch log handler respect the per-run log ID template #69897
Draft

jason810496 changed the title ~~Make ES and OpenSearch log handlers respect the per-run log ID template~~Deliver the per-run log ID template to workers via TIRunContextJul 15, 2026

phanikumv reviewed Jul 17, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py

Add a test asserting log_id_template always reflects the newest LogTe…
3bd7711
…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 added this to the Airflow 3.4.0 milestone Jul 18, 2026

jason810496 self-assigned this Jul 10, 2026

boring-cyborgBot added area:API Airflow's REST/HTTP API area:logging area:providers area:task-sdk provider:elasticsearch provider:opensearch labels Jul 10, 2026

jason810496 added 2 commits July 11, 2026 06:35

jason810496 requested review from eladkal and uranusjr July 13, 2026 03:32

jason810496 marked this pull request as ready for review July 13, 2026 03:32

jason810496 requested review from amoghrajesh, ashb and kaxil as code owners July 13, 2026 03:32

jason810496 mentioned this pull request Jul 13, 2026
Drop the LogTemplate DB model #69520
Closed
1 task

This was referenced Jul 15, 2026
Make Elasticsearch log handler respect the per-run log ID template #69896
Draft
Make OpenSearch log handler respect the per-run log ID template #69897
Draft

jason810496 changed the title ~~Make ES and OpenSearch log handlers respect the per-run log ID template~~Deliver the per-run log ID template to workers via TIRunContextJul 15, 2026

phanikumv reviewed Jul 17, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py

Add a test asserting log_id_template always reflects the newest LogTe…
3bd7711
…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 added this to the Airflow 3.4.0 milestone Jul 18, 2026

jason810496 self-assigned this Jul 10, 2026

boring-cyborgBot added area:API Airflow's REST/HTTP API area:logging area:providers area:task-sdk provider:elasticsearch provider:opensearch labels Jul 10, 2026

jason810496 added 2 commits July 11, 2026 06:35

jason810496 requested review from eladkal and uranusjr July 13, 2026 03:32

jason810496 marked this pull request as ready for review July 13, 2026 03:32

jason810496 requested review from amoghrajesh, ashb and kaxil as code owners July 13, 2026 03:32

jason810496 mentioned this pull request Jul 13, 2026
Drop the LogTemplate DB model #69520
Closed
1 task

This was referenced Jul 15, 2026
Make Elasticsearch log handler respect the per-run log ID template #69896
Draft
Make OpenSearch log handler respect the per-run log ID template #69897
Draft

jason810496 changed the title ~~Make ES and OpenSearch log handlers respect the per-run log ID template~~Deliver the per-run log ID template to workers via TIRunContextJul 15, 2026

phanikumv reviewed Jul 17, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py

Add a test asserting log_id_template always reflects the newest LogTe…
3bd7711
…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 added this to the Airflow 3.4.0 milestone Jul 18, 2026

jason810496 self-assigned this Jul 10, 2026

boring-cyborgBot added area:API Airflow's REST/HTTP API area:logging area:providers area:task-sdk provider:elasticsearch provider:opensearch labels Jul 10, 2026

jason810496 added 2 commits July 11, 2026 06:35

jason810496 requested review from eladkal and uranusjr July 13, 2026 03:32

jason810496 marked this pull request as ready for review July 13, 2026 03:32

jason810496 requested review from amoghrajesh, ashb and kaxil as code owners July 13, 2026 03:32

jason810496 mentioned this pull request Jul 13, 2026
Drop the LogTemplate DB model #69520
Closed
1 task

This was referenced Jul 15, 2026
Make Elasticsearch log handler respect the per-run log ID template #69896
Draft
Make OpenSearch log handler respect the per-run log ID template #69897
Draft

jason810496 changed the title ~~Make ES and OpenSearch log handlers respect the per-run log ID template~~Deliver the per-run log ID template to workers via TIRunContextJul 15, 2026

phanikumv reviewed Jul 17, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py

Add a test asserting log_id_template always reflects the newest LogTe…
3bd7711
…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 added this to the Airflow 3.4.0 milestone Jul 18, 2026

jason810496 self-assigned this Jul 10, 2026

boring-cyborgBot added area:API Airflow's REST/HTTP API area:logging area:providers area:task-sdk provider:elasticsearch provider:opensearch labels Jul 10, 2026

jason810496 added 2 commits July 11, 2026 06:35

jason810496 requested review from eladkal and uranusjr July 13, 2026 03:32

jason810496 marked this pull request as ready for review July 13, 2026 03:32

jason810496 requested review from amoghrajesh, ashb and kaxil as code owners July 13, 2026 03:32

jason810496 mentioned this pull request Jul 13, 2026
Drop the LogTemplate DB model #69520
Closed
1 task

This was referenced Jul 15, 2026
Make Elasticsearch log handler respect the per-run log ID template #69896
Draft
Make OpenSearch log handler respect the per-run log ID template #69897
Draft

jason810496 changed the title ~~Make ES and OpenSearch log handlers respect the per-run log ID template~~Deliver the per-run log ID template to workers via TIRunContextJul 15, 2026

phanikumv reviewed Jul 17, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py

Add a test asserting log_id_template always reflects the newest LogTe…
3bd7711
…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 added this to the Airflow 3.4.0 milestone Jul 18, 2026

jason810496 self-assigned this Jul 10, 2026

boring-cyborgBot added area:API Airflow's REST/HTTP API area:logging area:providers area:task-sdk provider:elasticsearch provider:opensearch labels Jul 10, 2026

jason810496 added 2 commits July 11, 2026 06:35

jason810496 requested review from eladkal and uranusjr July 13, 2026 03:32

jason810496 marked this pull request as ready for review July 13, 2026 03:32

jason810496 requested review from amoghrajesh, ashb and kaxil as code owners July 13, 2026 03:32

jason810496 mentioned this pull request Jul 13, 2026
Drop the LogTemplate DB model #69520
Closed
1 task

This was referenced Jul 15, 2026
Make Elasticsearch log handler respect the per-run log ID template #69896
Draft
Make OpenSearch log handler respect the per-run log ID template #69897
Draft

jason810496 changed the title ~~Make ES and OpenSearch log handlers respect the per-run log ID template~~Deliver the per-run log ID template to workers via TIRunContextJul 15, 2026

phanikumv reviewed Jul 17, 2026
View reviewed changes

Comment threadairflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py

Add a test asserting log_id_template always reflects the newest LogTe…
3bd7711
…mplate Exercises the pinning behavior across several LogTemplate rows in a loop, rather than a single row, to guard against the value being cached or resolved against a stale row.

jason810496 added this to the Airflow 3.4.0 milestone Jul 18, 2026

Uh oh!

Conversation

jason810496 commented Jul 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

What

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jason810496 commented Jul 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

What

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jason810496 commented Jul 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

What

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jason810496 commented Jul 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

What

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jason810496 commented Jul 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

What

Was generative AI tooling used to co-author this PR?

jason810496 commented Jul 10, 2026•
edited
Loading

jason810496 commented Jul 10, 2026•
edited
Loading

jason810496 commented Jul 10, 2026•
edited
Loading

jason810496 commented Jul 10, 2026•
edited
Loading

jason810496 commented Jul 10, 2026•
edited
Loading

jason810496 commented Jul 10, 2026•
edited
Loading

jason810496 commented Jul 10, 2026•
edited
Loading