Drop the LogTemplate DB model

[jason810496]

• related: Fix the write-to-es feature #53821, Drop LogTemplate DB Model #59027 (stale predecessor of this PR), [DISCUSS] Drop LogTemplate DB Model for ElasticsearchTaskHandler in Airflow 3

Why

The LogTemplate model pinned historical values of [logging] log_filename_template / [elasticsearch] log_id_template per DagRun, and task handlers read it via dag_run.get_log_template(session=...) — a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.

Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.

What

• Remove the LogTemplate model, the dag_run.log_template_id column/FK, DagRun.get_log_template(), and synchronize_log_template() with its call sites. FileTaskHandler._render_filename() now reads [logging] log_filename_template from live config at render time.
• Add migration 0126_3_4_0_drop_log_template.py (downgrade recreates the column and table); both directions use disable_sqlite_fkeys(op) so the SQLite dag_run rebuild doesn't cascade-delete task_instance rows.
• Elasticsearch: remove the USE_PER_RUN_LOG_ID flag — defined but unused since ElasticsearchRemoteLogIO, so dead-code cleanup only.
• OpenSearch: keep the hasattr(DagRun, "get_log_template") shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to [opensearch] log_id_template from config.
• Make the shared create_log_template fixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.

Behaviour change (core 3.4.0+): changing log_filename_template / log_id_template now applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Fable 5) following the guidelines

[ashb]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

[ashb]

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

[jason810496]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

Yes, but based on

• a) Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (https://github.com/apache/airflow/pull/61880);
• b) the ES / OS provider for Airflow 3 did respect the log_template (more details as below)

I thought we can introduce the breaking change in 3.4 to drop this feature.

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model.

IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model.

Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc)
WDYT? Thanks.

[ashb]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

[jason810496]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks.

[jason810496]

Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause.

[jason810496]

• related: Fix the write-to-es feature #53821, Drop LogTemplate DB Model #59027 (stale predecessor of this PR), [DISCUSS] Drop LogTemplate DB Model for ElasticsearchTaskHandler in Airflow 3

Why

The LogTemplate model pinned historical values of [logging] log_filename_template / [elasticsearch] log_id_template per DagRun, and task handlers read it via dag_run.get_log_template(session=...) — a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.

Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.

What

• Remove the LogTemplate model, the dag_run.log_template_id column/FK, DagRun.get_log_template(), and synchronize_log_template() with its call sites. FileTaskHandler._render_filename() now reads [logging] log_filename_template from live config at render time.
• Add migration 0126_3_4_0_drop_log_template.py (downgrade recreates the column and table); both directions use disable_sqlite_fkeys(op) so the SQLite dag_run rebuild doesn't cascade-delete task_instance rows.
• Elasticsearch: remove the USE_PER_RUN_LOG_ID flag — defined but unused since ElasticsearchRemoteLogIO, so dead-code cleanup only.
• OpenSearch: keep the hasattr(DagRun, "get_log_template") shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to [opensearch] log_id_template from config.
• Make the shared create_log_template fixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.

Behaviour change (core 3.4.0+): changing log_filename_template / log_id_template now applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Fable 5) following the guidelines

[ashb]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

[ashb]

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

[jason810496]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

Yes, but based on

• a) Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (https://github.com/apache/airflow/pull/61880);
• b) the ES / OS provider for Airflow 3 did respect the log_template (more details as below)

I thought we can introduce the breaking change in 3.4 to drop this feature.

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model.

IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model.

Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc)
WDYT? Thanks.

[ashb]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

[jason810496]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks.

[jason810496]

Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause.

[jason810496]

• related: Fix the write-to-es feature #53821, Drop LogTemplate DB Model #59027 (stale predecessor of this PR), [DISCUSS] Drop LogTemplate DB Model for ElasticsearchTaskHandler in Airflow 3

Why

The LogTemplate model pinned historical values of [logging] log_filename_template / [elasticsearch] log_id_template per DagRun, and task handlers read it via dag_run.get_log_template(session=...) — a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.

Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.

What

• Remove the LogTemplate model, the dag_run.log_template_id column/FK, DagRun.get_log_template(), and synchronize_log_template() with its call sites. FileTaskHandler._render_filename() now reads [logging] log_filename_template from live config at render time.
• Add migration 0126_3_4_0_drop_log_template.py (downgrade recreates the column and table); both directions use disable_sqlite_fkeys(op) so the SQLite dag_run rebuild doesn't cascade-delete task_instance rows.
• Elasticsearch: remove the USE_PER_RUN_LOG_ID flag — defined but unused since ElasticsearchRemoteLogIO, so dead-code cleanup only.
• OpenSearch: keep the hasattr(DagRun, "get_log_template") shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to [opensearch] log_id_template from config.
• Make the shared create_log_template fixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.

Behaviour change (core 3.4.0+): changing log_filename_template / log_id_template now applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Fable 5) following the guidelines

[ashb]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

[ashb]

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

[jason810496]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

Yes, but based on

• a) Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (https://github.com/apache/airflow/pull/61880);
• b) the ES / OS provider for Airflow 3 did respect the log_template (more details as below)

I thought we can introduce the breaking change in 3.4 to drop this feature.

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model.

IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model.

Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc)
WDYT? Thanks.

[ashb]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

[jason810496]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks.

[jason810496]

Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause.

[jason810496]

• related: Fix the write-to-es feature #53821, Drop LogTemplate DB Model #59027 (stale predecessor of this PR), [DISCUSS] Drop LogTemplate DB Model for ElasticsearchTaskHandler in Airflow 3

Why

The LogTemplate model pinned historical values of [logging] log_filename_template / [elasticsearch] log_id_template per DagRun, and task handlers read it via dag_run.get_log_template(session=...) — a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.

Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.

What

• Remove the LogTemplate model, the dag_run.log_template_id column/FK, DagRun.get_log_template(), and synchronize_log_template() with its call sites. FileTaskHandler._render_filename() now reads [logging] log_filename_template from live config at render time.
• Add migration 0126_3_4_0_drop_log_template.py (downgrade recreates the column and table); both directions use disable_sqlite_fkeys(op) so the SQLite dag_run rebuild doesn't cascade-delete task_instance rows.
• Elasticsearch: remove the USE_PER_RUN_LOG_ID flag — defined but unused since ElasticsearchRemoteLogIO, so dead-code cleanup only.
• OpenSearch: keep the hasattr(DagRun, "get_log_template") shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to [opensearch] log_id_template from config.
• Make the shared create_log_template fixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.

Behaviour change (core 3.4.0+): changing log_filename_template / log_id_template now applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Fable 5) following the guidelines

[ashb]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

[ashb]

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

[jason810496]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

Yes, but based on

• a) Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (https://github.com/apache/airflow/pull/61880);
• b) the ES / OS provider for Airflow 3 did respect the log_template (more details as below)

I thought we can introduce the breaking change in 3.4 to drop this feature.

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model.

IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model.

Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc)
WDYT? Thanks.

[ashb]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

[jason810496]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks.

[jason810496]

Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause.

[jason810496]

• related: Fix the write-to-es feature #53821, Drop LogTemplate DB Model #59027 (stale predecessor of this PR), [DISCUSS] Drop LogTemplate DB Model for ElasticsearchTaskHandler in Airflow 3

Why

The LogTemplate model pinned historical values of [logging] log_filename_template / [elasticsearch] log_id_template per DagRun, and task handlers read it via dag_run.get_log_template(session=...) — a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.

Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.

What

• Remove the LogTemplate model, the dag_run.log_template_id column/FK, DagRun.get_log_template(), and synchronize_log_template() with its call sites. FileTaskHandler._render_filename() now reads [logging] log_filename_template from live config at render time.
• Add migration 0126_3_4_0_drop_log_template.py (downgrade recreates the column and table); both directions use disable_sqlite_fkeys(op) so the SQLite dag_run rebuild doesn't cascade-delete task_instance rows.
• Elasticsearch: remove the USE_PER_RUN_LOG_ID flag — defined but unused since ElasticsearchRemoteLogIO, so dead-code cleanup only.
• OpenSearch: keep the hasattr(DagRun, "get_log_template") shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to [opensearch] log_id_template from config.
• Make the shared create_log_template fixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.

Behaviour change (core 3.4.0+): changing log_filename_template / log_id_template now applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Fable 5) following the guidelines

[ashb]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

[ashb]

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

[jason810496]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

Yes, but based on

• a) Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (https://github.com/apache/airflow/pull/61880);
• b) the ES / OS provider for Airflow 3 did respect the log_template (more details as below)

I thought we can introduce the breaking change in 3.4 to drop this feature.

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model.

IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model.

Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc)
WDYT? Thanks.

[ashb]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

[jason810496]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks.

[jason810496]

Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause.

[jason810496]

• related: Fix the write-to-es feature #53821, Drop LogTemplate DB Model #59027 (stale predecessor of this PR), [DISCUSS] Drop LogTemplate DB Model for ElasticsearchTaskHandler in Airflow 3

Why

The LogTemplate model pinned historical values of [logging] log_filename_template / [elasticsearch] log_id_template per DagRun, and task handlers read it via dag_run.get_log_template(session=...) — a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.

Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.

What

• Remove the LogTemplate model, the dag_run.log_template_id column/FK, DagRun.get_log_template(), and synchronize_log_template() with its call sites. FileTaskHandler._render_filename() now reads [logging] log_filename_template from live config at render time.
• Add migration 0126_3_4_0_drop_log_template.py (downgrade recreates the column and table); both directions use disable_sqlite_fkeys(op) so the SQLite dag_run rebuild doesn't cascade-delete task_instance rows.
• Elasticsearch: remove the USE_PER_RUN_LOG_ID flag — defined but unused since ElasticsearchRemoteLogIO, so dead-code cleanup only.
• OpenSearch: keep the hasattr(DagRun, "get_log_template") shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to [opensearch] log_id_template from config.
• Make the shared create_log_template fixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.

Behaviour change (core 3.4.0+): changing log_filename_template / log_id_template now applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Fable 5) following the guidelines

[ashb]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

[ashb]

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

[jason810496]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

Yes, but based on

• a) Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (https://github.com/apache/airflow/pull/61880);
• b) the ES / OS provider for Airflow 3 did respect the log_template (more details as below)

I thought we can introduce the breaking change in 3.4 to drop this feature.

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model.

IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model.

Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc)
WDYT? Thanks.

[ashb]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

[jason810496]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks.

[jason810496]

Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause.

[jason810496]

• related: Fix the write-to-es feature #53821, Drop LogTemplate DB Model #59027 (stale predecessor of this PR), [DISCUSS] Drop LogTemplate DB Model for ElasticsearchTaskHandler in Airflow 3

Why

The LogTemplate model pinned historical values of [logging] log_filename_template / [elasticsearch] log_id_template per DagRun, and task handlers read it via dag_run.get_log_template(session=...) — a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.

Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.

What

• Remove the LogTemplate model, the dag_run.log_template_id column/FK, DagRun.get_log_template(), and synchronize_log_template() with its call sites. FileTaskHandler._render_filename() now reads [logging] log_filename_template from live config at render time.
• Add migration 0126_3_4_0_drop_log_template.py (downgrade recreates the column and table); both directions use disable_sqlite_fkeys(op) so the SQLite dag_run rebuild doesn't cascade-delete task_instance rows.
• Elasticsearch: remove the USE_PER_RUN_LOG_ID flag — defined but unused since ElasticsearchRemoteLogIO, so dead-code cleanup only.
• OpenSearch: keep the hasattr(DagRun, "get_log_template") shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to [opensearch] log_id_template from config.
• Make the shared create_log_template fixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.

Behaviour change (core 3.4.0+): changing log_filename_template / log_id_template now applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Fable 5) following the guidelines

[ashb]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

[ashb]

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

[jason810496]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

Yes, but based on

• a) Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (https://github.com/apache/airflow/pull/61880);
• b) the ES / OS provider for Airflow 3 did respect the log_template (more details as below)

I thought we can introduce the breaking change in 3.4 to drop this feature.

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model.

IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model.

Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc)
WDYT? Thanks.

[ashb]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

[jason810496]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks.

[jason810496]

Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause.

[jason810496]

• related: Fix the write-to-es feature #53821, Drop LogTemplate DB Model #59027 (stale predecessor of this PR), [DISCUSS] Drop LogTemplate DB Model for ElasticsearchTaskHandler in Airflow 3

Why

The LogTemplate model pinned historical values of [logging] log_filename_template / [elasticsearch] log_id_template per DagRun, and task handlers read it via dag_run.get_log_template(session=...) — a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.

Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.

What

• Remove the LogTemplate model, the dag_run.log_template_id column/FK, DagRun.get_log_template(), and synchronize_log_template() with its call sites. FileTaskHandler._render_filename() now reads [logging] log_filename_template from live config at render time.
• Add migration 0126_3_4_0_drop_log_template.py (downgrade recreates the column and table); both directions use disable_sqlite_fkeys(op) so the SQLite dag_run rebuild doesn't cascade-delete task_instance rows.
• Elasticsearch: remove the USE_PER_RUN_LOG_ID flag — defined but unused since ElasticsearchRemoteLogIO, so dead-code cleanup only.
• OpenSearch: keep the hasattr(DagRun, "get_log_template") shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to [opensearch] log_id_template from config.
• Make the shared create_log_template fixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.

Behaviour change (core 3.4.0+): changing log_filename_template / log_id_template now applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Fable 5) following the guidelines

[ashb]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

[ashb]

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

[jason810496]

The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it

Yes, but based on

• a) Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (https://github.com/apache/airflow/pull/61880);
• b) the ES / OS provider for Airflow 3 did respect the log_template (more details as below)

I thought we can introduce the breaking change in 3.4 to drop this feature.

a direct metadata-DB dependency that conflicts with the Airflow 3 execution model

This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant

It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model.

IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model.

Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc)
WDYT? Thanks.

[ashb]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

[jason810496]

LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc.

Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks.

[jason810496]

Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause.