Optimize DAG serialization by excluding schema default values by kaxil · Pull Request #55849 · apache/airflow

kaxil · 2025-09-18T17:32:12Z

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like catchup=False, max_active_runs=16, and fail_fast=False are no longer stored when they have default values.

Follow-up of #54569

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of apache#54569

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of #54569 (cherry picked from commit a582464)

…chema default The serialisation optimisation from apache#55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation.

…chema default (#65310) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument

…lue equals schema default (#65310) (#67097) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument (cherry picked from commit 76eb2a0) Co-authored-by: Selman <seruman@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

kaxil · 2025-09-18T17:32:12Z

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like catchup=False, max_active_runs=16, and fail_fast=False are no longer stored when they have default values.

Follow-up of #54569

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of apache#54569

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of #54569 (cherry picked from commit a582464)

…chema default The serialisation optimisation from apache#55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation.

…chema default (#65310) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument

…lue equals schema default (#65310) (#67097) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument (cherry picked from commit 76eb2a0) Co-authored-by: Selman <seruman@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

kaxil · 2025-09-18T17:32:12Z

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like catchup=False, max_active_runs=16, and fail_fast=False are no longer stored when they have default values.

Follow-up of #54569

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of apache#54569

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of #54569 (cherry picked from commit a582464)

…chema default The serialisation optimisation from apache#55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation.

…chema default (#65310) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument

…lue equals schema default (#65310) (#67097) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument (cherry picked from commit 76eb2a0) Co-authored-by: Selman <seruman@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

kaxil · 2025-09-18T17:32:12Z

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like catchup=False, max_active_runs=16, and fail_fast=False are no longer stored when they have default values.

Follow-up of #54569

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of apache#54569

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of #54569 (cherry picked from commit a582464)

…chema default The serialisation optimisation from apache#55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation.

…chema default (#65310) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument

…lue equals schema default (#65310) (#67097) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument (cherry picked from commit 76eb2a0) Co-authored-by: Selman <seruman@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

kaxil · 2025-09-18T17:32:12Z

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like catchup=False, max_active_runs=16, and fail_fast=False are no longer stored when they have default values.

Follow-up of #54569

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of apache#54569

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of #54569 (cherry picked from commit a582464)

…chema default The serialisation optimisation from apache#55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation.

…chema default (#65310) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument

…lue equals schema default (#65310) (#67097) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument (cherry picked from commit 76eb2a0) Co-authored-by: Selman <seruman@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

kaxil · 2025-09-18T17:32:12Z

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like catchup=False, max_active_runs=16, and fail_fast=False are no longer stored when they have default values.

Follow-up of #54569

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of apache#54569

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of #54569 (cherry picked from commit a582464)

…chema default The serialisation optimisation from apache#55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation.

…chema default (#65310) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument

…lue equals schema default (#65310) (#67097) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument (cherry picked from commit 76eb2a0) Co-authored-by: Selman <seruman@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

kaxil · 2025-09-18T17:32:12Z

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like catchup=False, max_active_runs=16, and fail_fast=False are no longer stored when they have default values.

Follow-up of #54569

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of apache#54569

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of #54569 (cherry picked from commit a582464)

…chema default The serialisation optimisation from apache#55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation.

…chema default (#65310) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument

…lue equals schema default (#65310) (#67097) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument (cherry picked from commit 76eb2a0) Co-authored-by: Selman <seruman@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

kaxil · 2025-09-18T17:32:12Z

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like catchup=False, max_active_runs=16, and fail_fast=False are no longer stored when they have default values.

Follow-up of #54569

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of apache#54569

This change reduces serialized DAG size by automatically excluding fields that match their schema default values, similar to how operator serialization works. Fields like `catchup=False`, `max_active_runs=16`, and `fail_fast=False` are no longer stored when they have default values. Follow-up of #54569 (cherry picked from commit a582464)

…chema default The serialisation optimisation from apache#55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation.

…chema default (#65310) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument

…lue equals schema default (#65310) (#67097) * Fix max_active_runs lost during DAG serialisation when value equals schema default The serialisation optimisation from #55849 strips DAG fields that match their schema.json default. For max_active_runs, max_active_tasks, and max_consecutive_failed_dag_runs this is wrong because their runtime defaults come from airflow.cfg, not the schema. When a user explicitly sets max_active_runs=16 and the config has max_active_runs_per_dag=1, the value gets stripped and the dag table ends up with 1. Skip the schema-default exclusion for these three config-driven fields so they always survive serialisation. * fix: introduce DAG_DEFAULTS * chore: add test for catchup * fix: revert comment change, that wording was better * fix: preserve DAG fields whose defaults match the schema default * fix: update schema default check script * fix: address review comments on serialisation tests * fix: add disable_bundle_versioning case to parametrised serialisation test * fix: improve docstring for config-driven fields serialisation test * fix: use tuple for pytest.mark.parametrize first argument (cherry picked from commit 76eb2a0) Co-authored-by: Selman <seruman@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

kaxil requested review from ashb, bolkedebruin, gopidesupavan and potiuk as code owners September 18, 2025 17:32

boring-cyborgBot added area:dev-tools area:serialization labels Sep 18, 2025

ashb approved these changes Sep 18, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/serialization/serialized_objects.py
Comment threadairflow-core/src/airflow/serialization/serialized_objects.py

kaxil added 2 commits September 18, 2025 18:54

fixup! Optimize DAG serialization by excluding schema default values
a8fd7ea

kaxil force-pushed the dag-defaults branch from 243683c to a8fd7eaCompare September 18, 2025 17:54

kaxil added this to the Airflow 3.1.0 milestone Sep 18, 2025

kaxil merged commit a582464 into apache:mainSep 18, 2025
105 checks passed

kaxil deleted the dag-defaults branch September 18, 2025 18:42

kaxil mentioned this pull request Oct 28, 2025
Remove callbacks from DAG default_args when serializating it #57397
Merged

potiuk mentioned this pull request Nov 8, 2025
Add missing translations for PL for 3.1.3 #58076
Merged

seruman mentioned this pull request Apr 15, 2026
Fix max_active_runs lost during DAG serialisation when value equals schema default #65310
Merged
1 task

github-actionsBot mentioned this pull request May 18, 2026
[v3-2-test] Fix max_active_runs lost during DAG serialisation when value equals schema default (#65310) #67097
Merged

vatsrahul1001 mentioned this pull request May 21, 2026
Status of testing of Apache Airflow 3.2.2rc3 #67282
Closed