Skip to content

Rename execution_date to logical_date across codebase - #43902

Merged
uranusjr merged 15 commits into
apache:mainfrom
astronomer:rename-execution-date-to-logical-date
Nov 15, 2024
Merged

Rename execution_date to logical_date across codebase#43902
uranusjr merged 15 commits into
apache:mainfrom
astronomer:rename-execution-date-to-logical-date

Conversation

@sunank200

@sunank200sunank200 commented Nov 11, 2024

Copy link
Copy Markdown
Collaborator

Motivation

This PR renames execution_date to logical_date across the codebase. The shift towards logical_date helps move away from the limitations of execution_date, particularly with dynamic DAG runs and cases where multiple runs occur at the same time.

Key Changes

  • Replaced all instances of execution_date with logical_date including models.
  • Updated references in database models, templates, and functions.
  • Removed unique constraints on execution_date in the database to allow multiple DAG runs with the same logical time.

How execution_date and logical_date Work

  1. Logical date is equivalent to execution date: The two are just different names for the same value.
  2. Timetable controls logical date: The logical date can be set to any value, not necessarily tied to the data interval's start or end.
  3. Schedules dictate behavior: For value-based schedules (like cron), the logical date is set by the timetable class used.

Transitioning from execution_date

  • Airflow 3.0 will fully remove execution_date.
  • For uniquely identifying runs:
    • Use run_id for human-readable identifiers.
    • Use data_interval_start/data_interval_end for data partitioning in future.
    • id (auto-incremented) is recommended for ordering DAG runs.
  • If your existing logic relies on execution_date, switch to using data_interval_start or data_interval_end for identifying time ranges.

Testing

  • Updated unit tests to reflect the changes.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:CLI area:Scheduler including HA (high availability) scheduler provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Nov 11, 2024
@sunank200sunank200 added legacy api Whether legacy API changes should be allowed in PR legacy ui Whether legacy UI change should be allowed in PR labels Nov 11, 2024
@sunank200
sunank200force-pushed the rename-execution-date-to-logical-date branch 9 times, most recently from ea36b6c to 65f9b8eCompareNovember 12, 2024 02:54
@uranusjr

Copy link
Copy Markdown
Member

This needs a news fragment. Rewording the above summary would work fine. See existing files in newsfragments (the significant ones) for some examples.

@uranusjr

uranusjr commented Nov 12, 2024

Copy link
Copy Markdown
Member

For templates, replace {{ ds }} with {{ data_interval_start | ds }}.

This should probably use logical_date for max compatibility. Also should mention ts (same rewrite).

Oh wait, we’re not removing those in this PR yet, so this should not be mentioned at all.

Comment threadairflow/api_connexion/openapi/v1.yaml
Comment threadairflow/api_connexion/schemas/dag_run_schema.py Outdated
Comment threadairflow/api_connexion/schemas/dag_run_schema.py
Comment threadairflow/cli/cli_config.py
Comment threadairflow/models/taskreschedule.py
Comment threadairflow/models/xcom.py
Comment threadairflow/utils/context.py Outdated
Comment threadairflow/models/taskinstance.py Outdated
Comment threadairflow/utils/context.pyi Outdated
Comment threaddocs/apache-airflow/templates-ref.rst Outdated
Comment threadproviders/src/airflow/providers/cncf/kubernetes/cli/kubernetes_command.py Outdated
@uranusjr
uranusjr merged commit 123dadd into apache:mainNov 15, 2024
@uranusjr
uranusjr deleted the rename-execution-date-to-logical-date branch November 15, 2024 09:15
hardeybisey pushed a commit to hardeybisey/airflow that referenced this pull request Nov 15, 2024
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
@Lee-WLee-W mentioned this pull request Nov 18, 2024
2 tasks
@sunank200sunank200 added the AIP-83 Remove Execution Date Unique Constraint from DAG Run label Nov 25, 2024
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.
In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.
In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.
In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.
In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.
In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.
In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in #43902 so we no longer need this code.
In preparation of #45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jan 13, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.
In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIP-83Remove Execution Date Unique Constraint from DAG Runairflow3.0:breakingCandidates for Airflow 3.0 that contain breaking changesarea:APIAirflow's REST/HTTP APIarea:CLIarea:Schedulerincluding HA (high availability) schedulerlegacy apiWhether legacy API changes should be allowed in PRlegacy uiWhether legacy UI change should be allowed in PRprovider:cncf-kubernetesKubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sunank200@uranusjr@eladkal@potiuk@Lee-W@phanikumv