feat(cli): add partitions clear to reset DagRun partition fields - #66520

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear
May 27, 2026
Merged

feat(cli): add partitions clear to reset DagRun partition fields#66520
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear

Conversation

@Lee-W

@Lee-WLee-W commented May 7, 2026

Copy link
Copy Markdown
Member

part of #65921

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach Dag runs whose partition_date / partition_key is what identifies them (rollup or offset partitions). Today the only ways to bulk-reprocess such runs are N API calls or N UI clicks.

What

  • Add airflow dags clear <dag_id> with three mutually-exclusive selectors:
    • --run-id (single run)
    • --partition-key (every run with that exact key)
    • --partition-date-start / --partition-date-end (inclusive partition_date window; runs with NULL partition_date are never matched;
      inverted window is rejected)
  • Reuse Dag.clear() per matched run_id, so TI reset + Dag run re-queue behave identically to every other clear path.
  • Print a listing (run_id, partition_key, partition_date) before prompting; --yes skips the prompt.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 5 times, most recently from f4a47d7 to af063eaCompareMay 8, 2026 13:06
@Lee-W
Lee-W marked this pull request as ready for review May 8, 2026 13:07
@potiuk

Copy link
Copy Markdown
Member

Rebase is needed I guess.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from b809562 to e842cfaCompareMay 11, 2026 04:27
@Lee-W
Lee-W requested a review from uranusjrMay 11, 2026 06:27
@Lee-WLee-W moved this to In Review in AIP-76 Asset PartitioningMay 11, 2026
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 3481fdb to dc42fcaCompareMay 19, 2026 10:25
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/cli_config.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 961c171 to 1f1e52aCompareMay 21, 2026 14:21
@Lee-W
Lee-W requested a review from uranusjrMay 21, 2026 15:23
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 1f1e52a to 53a4e02CompareMay 22, 2026 00:12

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Not sure do I understand the batch clearing logic correctly.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 53a4e02 to ed18131CompareMay 25, 2026 12:54
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from ed18131 to c501951CompareMay 25, 2026 14:20
@Lee-W
Lee-W requested a review from jason810496May 25, 2026 15:27
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from c501951 to 20b1d9dCompareMay 25, 2026 15:28

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, please feel free to address in follow-up, non-blocking.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from a9e83e6 to 995da42CompareMay 27, 2026 01:40
Lee-W added 9 commits May 27, 2026 17:49
Introduce a new `airflow partitions` command group with a `clear`
action that nulls out `partition_key` and `partition_date` on one or
more DagRuns. Useful when migrating off a partitioned timetable or
fixing a Dag run whose partition metadata was set incorrectly.
Targeting:
- `--dag-id` (required)
- `--run-id` for a single run, or
- `--start-date` / `--end-date` to clear a `partition_date` range
(the two modes are mutually exclusive)
Safety:
- `--dry-run` previews matching runs without writing
- Prints the existing `partition_key` / `partition_date` before
clearing each run, and skips runs already cleared
Augment `airflow partitions clear` with `--clear-task-instances`:
in addition to nulling `partition_key` / `partition_date`, clear
the matched DagRuns' task instances so finished runs go back to
QUEUED and re-execute. Useful after fixing partition metadata when
the runs themselves should also re-run against the corrected logic.
Behavior:
- Default behavior is unchanged — only partition fields are nulled.
- With `--clear-task-instances`, runs whose partition fields are
already None are no longer skipped (their TIs are still cleared).
- `--dry-run` reports both projected actions without writing.
- `--partition-key VALUE`: clear all DagRuns whose partition_key matches this exact value.
- `--date a~b`: shorthand range syntax equivalent to `--start-date a --end-date b`.
The four selection modes (--run-id, --partition-key, --start-date/--end-date, --date) are mutually exclusive.
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 995da42 to 3d73d11CompareMay 27, 2026 09:49
@Lee-W
Lee-W merged commit 07859aa into apache:mainMay 27, 2026
143 checks passed
@Lee-W
Lee-W deleted the partitions-clear branch May 27, 2026 11:54
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

5 participants

@Lee-W@potiuk@uranusjr@jason810496@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(cli): add partitions clear to reset DagRun partition fields - #66520

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear
May 27, 2026
Merged

feat(cli): add partitions clear to reset DagRun partition fields#66520
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear

Conversation

@Lee-W

@Lee-WLee-W commented May 7, 2026

Copy link
Copy Markdown
Member

part of #65921

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach Dag runs whose partition_date / partition_key is what identifies them (rollup or offset partitions). Today the only ways to bulk-reprocess such runs are N API calls or N UI clicks.

What

  • Add airflow dags clear <dag_id> with three mutually-exclusive selectors:
    • --run-id (single run)
    • --partition-key (every run with that exact key)
    • --partition-date-start / --partition-date-end (inclusive partition_date window; runs with NULL partition_date are never matched;
      inverted window is rejected)
  • Reuse Dag.clear() per matched run_id, so TI reset + Dag run re-queue behave identically to every other clear path.
  • Print a listing (run_id, partition_key, partition_date) before prompting; --yes skips the prompt.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 5 times, most recently from f4a47d7 to af063eaCompareMay 8, 2026 13:06
@Lee-W
Lee-W marked this pull request as ready for review May 8, 2026 13:07
@potiuk

Copy link
Copy Markdown
Member

Rebase is needed I guess.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from b809562 to e842cfaCompareMay 11, 2026 04:27
@Lee-W
Lee-W requested a review from uranusjrMay 11, 2026 06:27
@Lee-WLee-W moved this to In Review in AIP-76 Asset PartitioningMay 11, 2026
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 3481fdb to dc42fcaCompareMay 19, 2026 10:25
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/cli_config.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 961c171 to 1f1e52aCompareMay 21, 2026 14:21
@Lee-W
Lee-W requested a review from uranusjrMay 21, 2026 15:23
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 1f1e52a to 53a4e02CompareMay 22, 2026 00:12

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Not sure do I understand the batch clearing logic correctly.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 53a4e02 to ed18131CompareMay 25, 2026 12:54
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from ed18131 to c501951CompareMay 25, 2026 14:20
@Lee-W
Lee-W requested a review from jason810496May 25, 2026 15:27
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from c501951 to 20b1d9dCompareMay 25, 2026 15:28

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, please feel free to address in follow-up, non-blocking.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from a9e83e6 to 995da42CompareMay 27, 2026 01:40
Lee-W added 9 commits May 27, 2026 17:49
Introduce a new `airflow partitions` command group with a `clear`
action that nulls out `partition_key` and `partition_date` on one or
more DagRuns. Useful when migrating off a partitioned timetable or
fixing a Dag run whose partition metadata was set incorrectly.
Targeting:
- `--dag-id` (required)
- `--run-id` for a single run, or
- `--start-date` / `--end-date` to clear a `partition_date` range
(the two modes are mutually exclusive)
Safety:
- `--dry-run` previews matching runs without writing
- Prints the existing `partition_key` / `partition_date` before
clearing each run, and skips runs already cleared
Augment `airflow partitions clear` with `--clear-task-instances`:
in addition to nulling `partition_key` / `partition_date`, clear
the matched DagRuns' task instances so finished runs go back to
QUEUED and re-execute. Useful after fixing partition metadata when
the runs themselves should also re-run against the corrected logic.
Behavior:
- Default behavior is unchanged — only partition fields are nulled.
- With `--clear-task-instances`, runs whose partition fields are
already None are no longer skipped (their TIs are still cleared).
- `--dry-run` reports both projected actions without writing.
- `--partition-key VALUE`: clear all DagRuns whose partition_key matches this exact value.
- `--date a~b`: shorthand range syntax equivalent to `--start-date a --end-date b`.
The four selection modes (--run-id, --partition-key, --start-date/--end-date, --date) are mutually exclusive.
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 995da42 to 3d73d11CompareMay 27, 2026 09:49
@Lee-W
Lee-W merged commit 07859aa into apache:mainMay 27, 2026
143 checks passed
@Lee-W
Lee-W deleted the partitions-clear branch May 27, 2026 11:54
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

5 participants

@Lee-W@potiuk@uranusjr@jason810496@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): add partitions clear to reset DagRun partition fields - #66520

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear
May 27, 2026
Merged

feat(cli): add partitions clear to reset DagRun partition fields#66520
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear

Conversation

@Lee-W

@Lee-WLee-W commented May 7, 2026

Copy link
Copy Markdown
Member

part of #65921

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach Dag runs whose partition_date / partition_key is what identifies them (rollup or offset partitions). Today the only ways to bulk-reprocess such runs are N API calls or N UI clicks.

What

  • Add airflow dags clear <dag_id> with three mutually-exclusive selectors:
    • --run-id (single run)
    • --partition-key (every run with that exact key)
    • --partition-date-start / --partition-date-end (inclusive partition_date window; runs with NULL partition_date are never matched;
      inverted window is rejected)
  • Reuse Dag.clear() per matched run_id, so TI reset + Dag run re-queue behave identically to every other clear path.
  • Print a listing (run_id, partition_key, partition_date) before prompting; --yes skips the prompt.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 5 times, most recently from f4a47d7 to af063eaCompareMay 8, 2026 13:06
@Lee-W
Lee-W marked this pull request as ready for review May 8, 2026 13:07
@potiuk

Copy link
Copy Markdown
Member

Rebase is needed I guess.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from b809562 to e842cfaCompareMay 11, 2026 04:27
@Lee-W
Lee-W requested a review from uranusjrMay 11, 2026 06:27
@Lee-WLee-W moved this to In Review in AIP-76 Asset PartitioningMay 11, 2026
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 3481fdb to dc42fcaCompareMay 19, 2026 10:25
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/cli_config.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 961c171 to 1f1e52aCompareMay 21, 2026 14:21
@Lee-W
Lee-W requested a review from uranusjrMay 21, 2026 15:23
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 1f1e52a to 53a4e02CompareMay 22, 2026 00:12

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Not sure do I understand the batch clearing logic correctly.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 53a4e02 to ed18131CompareMay 25, 2026 12:54
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from ed18131 to c501951CompareMay 25, 2026 14:20
@Lee-W
Lee-W requested a review from jason810496May 25, 2026 15:27
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from c501951 to 20b1d9dCompareMay 25, 2026 15:28

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, please feel free to address in follow-up, non-blocking.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from a9e83e6 to 995da42CompareMay 27, 2026 01:40
Lee-W added 9 commits May 27, 2026 17:49
Introduce a new `airflow partitions` command group with a `clear`
action that nulls out `partition_key` and `partition_date` on one or
more DagRuns. Useful when migrating off a partitioned timetable or
fixing a Dag run whose partition metadata was set incorrectly.
Targeting:
- `--dag-id` (required)
- `--run-id` for a single run, or
- `--start-date` / `--end-date` to clear a `partition_date` range
(the two modes are mutually exclusive)
Safety:
- `--dry-run` previews matching runs without writing
- Prints the existing `partition_key` / `partition_date` before
clearing each run, and skips runs already cleared
Augment `airflow partitions clear` with `--clear-task-instances`:
in addition to nulling `partition_key` / `partition_date`, clear
the matched DagRuns' task instances so finished runs go back to
QUEUED and re-execute. Useful after fixing partition metadata when
the runs themselves should also re-run against the corrected logic.
Behavior:
- Default behavior is unchanged — only partition fields are nulled.
- With `--clear-task-instances`, runs whose partition fields are
already None are no longer skipped (their TIs are still cleared).
- `--dry-run` reports both projected actions without writing.
- `--partition-key VALUE`: clear all DagRuns whose partition_key matches this exact value.
- `--date a~b`: shorthand range syntax equivalent to `--start-date a --end-date b`.
The four selection modes (--run-id, --partition-key, --start-date/--end-date, --date) are mutually exclusive.
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 995da42 to 3d73d11CompareMay 27, 2026 09:49
@Lee-W
Lee-W merged commit 07859aa into apache:mainMay 27, 2026
143 checks passed
@Lee-W
Lee-W deleted the partitions-clear branch May 27, 2026 11:54
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

5 participants

@Lee-W@potiuk@uranusjr@jason810496@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): add partitions clear to reset DagRun partition fields - #66520

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear
May 27, 2026
Merged

feat(cli): add partitions clear to reset DagRun partition fields#66520
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear

Conversation

@Lee-W

@Lee-WLee-W commented May 7, 2026

Copy link
Copy Markdown
Member

part of #65921

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach Dag runs whose partition_date / partition_key is what identifies them (rollup or offset partitions). Today the only ways to bulk-reprocess such runs are N API calls or N UI clicks.

What

  • Add airflow dags clear <dag_id> with three mutually-exclusive selectors:
    • --run-id (single run)
    • --partition-key (every run with that exact key)
    • --partition-date-start / --partition-date-end (inclusive partition_date window; runs with NULL partition_date are never matched;
      inverted window is rejected)
  • Reuse Dag.clear() per matched run_id, so TI reset + Dag run re-queue behave identically to every other clear path.
  • Print a listing (run_id, partition_key, partition_date) before prompting; --yes skips the prompt.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 5 times, most recently from f4a47d7 to af063eaCompareMay 8, 2026 13:06
@Lee-W
Lee-W marked this pull request as ready for review May 8, 2026 13:07
@potiuk

Copy link
Copy Markdown
Member

Rebase is needed I guess.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from b809562 to e842cfaCompareMay 11, 2026 04:27
@Lee-W
Lee-W requested a review from uranusjrMay 11, 2026 06:27
@Lee-WLee-W moved this to In Review in AIP-76 Asset PartitioningMay 11, 2026
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 3481fdb to dc42fcaCompareMay 19, 2026 10:25
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/cli_config.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 961c171 to 1f1e52aCompareMay 21, 2026 14:21
@Lee-W
Lee-W requested a review from uranusjrMay 21, 2026 15:23
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 1f1e52a to 53a4e02CompareMay 22, 2026 00:12

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Not sure do I understand the batch clearing logic correctly.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 53a4e02 to ed18131CompareMay 25, 2026 12:54
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from ed18131 to c501951CompareMay 25, 2026 14:20
@Lee-W
Lee-W requested a review from jason810496May 25, 2026 15:27
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from c501951 to 20b1d9dCompareMay 25, 2026 15:28

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, please feel free to address in follow-up, non-blocking.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from a9e83e6 to 995da42CompareMay 27, 2026 01:40
Lee-W added 9 commits May 27, 2026 17:49
Introduce a new `airflow partitions` command group with a `clear`
action that nulls out `partition_key` and `partition_date` on one or
more DagRuns. Useful when migrating off a partitioned timetable or
fixing a Dag run whose partition metadata was set incorrectly.
Targeting:
- `--dag-id` (required)
- `--run-id` for a single run, or
- `--start-date` / `--end-date` to clear a `partition_date` range
(the two modes are mutually exclusive)
Safety:
- `--dry-run` previews matching runs without writing
- Prints the existing `partition_key` / `partition_date` before
clearing each run, and skips runs already cleared
Augment `airflow partitions clear` with `--clear-task-instances`:
in addition to nulling `partition_key` / `partition_date`, clear
the matched DagRuns' task instances so finished runs go back to
QUEUED and re-execute. Useful after fixing partition metadata when
the runs themselves should also re-run against the corrected logic.
Behavior:
- Default behavior is unchanged — only partition fields are nulled.
- With `--clear-task-instances`, runs whose partition fields are
already None are no longer skipped (their TIs are still cleared).
- `--dry-run` reports both projected actions without writing.
- `--partition-key VALUE`: clear all DagRuns whose partition_key matches this exact value.
- `--date a~b`: shorthand range syntax equivalent to `--start-date a --end-date b`.
The four selection modes (--run-id, --partition-key, --start-date/--end-date, --date) are mutually exclusive.
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 995da42 to 3d73d11CompareMay 27, 2026 09:49
@Lee-W
Lee-W merged commit 07859aa into apache:mainMay 27, 2026
143 checks passed
@Lee-W
Lee-W deleted the partitions-clear branch May 27, 2026 11:54
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

5 participants

@Lee-W@potiuk@uranusjr@jason810496@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(cli): add partitions clear to reset DagRun partition fields - #66520

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear
May 27, 2026
Merged

feat(cli): add partitions clear to reset DagRun partition fields#66520
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear

Conversation

@Lee-W

@Lee-WLee-W commented May 7, 2026

Copy link
Copy Markdown
Member

part of #65921

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach Dag runs whose partition_date / partition_key is what identifies them (rollup or offset partitions). Today the only ways to bulk-reprocess such runs are N API calls or N UI clicks.

What

  • Add airflow dags clear <dag_id> with three mutually-exclusive selectors:
    • --run-id (single run)
    • --partition-key (every run with that exact key)
    • --partition-date-start / --partition-date-end (inclusive partition_date window; runs with NULL partition_date are never matched;
      inverted window is rejected)
  • Reuse Dag.clear() per matched run_id, so TI reset + Dag run re-queue behave identically to every other clear path.
  • Print a listing (run_id, partition_key, partition_date) before prompting; --yes skips the prompt.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 5 times, most recently from f4a47d7 to af063eaCompareMay 8, 2026 13:06
@Lee-W
Lee-W marked this pull request as ready for review May 8, 2026 13:07
@potiuk

Copy link
Copy Markdown
Member

Rebase is needed I guess.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from b809562 to e842cfaCompareMay 11, 2026 04:27
@Lee-W
Lee-W requested a review from uranusjrMay 11, 2026 06:27
@Lee-WLee-W moved this to In Review in AIP-76 Asset PartitioningMay 11, 2026
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 3481fdb to dc42fcaCompareMay 19, 2026 10:25
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/cli_config.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 961c171 to 1f1e52aCompareMay 21, 2026 14:21
@Lee-W
Lee-W requested a review from uranusjrMay 21, 2026 15:23
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 1f1e52a to 53a4e02CompareMay 22, 2026 00:12

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Not sure do I understand the batch clearing logic correctly.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 53a4e02 to ed18131CompareMay 25, 2026 12:54
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from ed18131 to c501951CompareMay 25, 2026 14:20
@Lee-W
Lee-W requested a review from jason810496May 25, 2026 15:27
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from c501951 to 20b1d9dCompareMay 25, 2026 15:28

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, please feel free to address in follow-up, non-blocking.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from a9e83e6 to 995da42CompareMay 27, 2026 01:40
Lee-W added 9 commits May 27, 2026 17:49
Introduce a new `airflow partitions` command group with a `clear`
action that nulls out `partition_key` and `partition_date` on one or
more DagRuns. Useful when migrating off a partitioned timetable or
fixing a Dag run whose partition metadata was set incorrectly.
Targeting:
- `--dag-id` (required)
- `--run-id` for a single run, or
- `--start-date` / `--end-date` to clear a `partition_date` range
(the two modes are mutually exclusive)
Safety:
- `--dry-run` previews matching runs without writing
- Prints the existing `partition_key` / `partition_date` before
clearing each run, and skips runs already cleared
Augment `airflow partitions clear` with `--clear-task-instances`:
in addition to nulling `partition_key` / `partition_date`, clear
the matched DagRuns' task instances so finished runs go back to
QUEUED and re-execute. Useful after fixing partition metadata when
the runs themselves should also re-run against the corrected logic.
Behavior:
- Default behavior is unchanged — only partition fields are nulled.
- With `--clear-task-instances`, runs whose partition fields are
already None are no longer skipped (their TIs are still cleared).
- `--dry-run` reports both projected actions without writing.
- `--partition-key VALUE`: clear all DagRuns whose partition_key matches this exact value.
- `--date a~b`: shorthand range syntax equivalent to `--start-date a --end-date b`.
The four selection modes (--run-id, --partition-key, --start-date/--end-date, --date) are mutually exclusive.
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 995da42 to 3d73d11CompareMay 27, 2026 09:49
@Lee-W
Lee-W merged commit 07859aa into apache:mainMay 27, 2026
143 checks passed
@Lee-W
Lee-W deleted the partitions-clear branch May 27, 2026 11:54
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

5 participants

@Lee-W@potiuk@uranusjr@jason810496@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): add partitions clear to reset DagRun partition fields - #66520

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear
May 27, 2026
Merged

feat(cli): add partitions clear to reset DagRun partition fields#66520
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear

Conversation

@Lee-W

@Lee-WLee-W commented May 7, 2026

Copy link
Copy Markdown
Member

part of #65921

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach Dag runs whose partition_date / partition_key is what identifies them (rollup or offset partitions). Today the only ways to bulk-reprocess such runs are N API calls or N UI clicks.

What

  • Add airflow dags clear <dag_id> with three mutually-exclusive selectors:
    • --run-id (single run)
    • --partition-key (every run with that exact key)
    • --partition-date-start / --partition-date-end (inclusive partition_date window; runs with NULL partition_date are never matched;
      inverted window is rejected)
  • Reuse Dag.clear() per matched run_id, so TI reset + Dag run re-queue behave identically to every other clear path.
  • Print a listing (run_id, partition_key, partition_date) before prompting; --yes skips the prompt.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 5 times, most recently from f4a47d7 to af063eaCompareMay 8, 2026 13:06
@Lee-W
Lee-W marked this pull request as ready for review May 8, 2026 13:07
@potiuk

Copy link
Copy Markdown
Member

Rebase is needed I guess.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from b809562 to e842cfaCompareMay 11, 2026 04:27
@Lee-W
Lee-W requested a review from uranusjrMay 11, 2026 06:27
@Lee-WLee-W moved this to In Review in AIP-76 Asset PartitioningMay 11, 2026
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 3481fdb to dc42fcaCompareMay 19, 2026 10:25
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/cli_config.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 961c171 to 1f1e52aCompareMay 21, 2026 14:21
@Lee-W
Lee-W requested a review from uranusjrMay 21, 2026 15:23
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 1f1e52a to 53a4e02CompareMay 22, 2026 00:12

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Not sure do I understand the batch clearing logic correctly.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 53a4e02 to ed18131CompareMay 25, 2026 12:54
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from ed18131 to c501951CompareMay 25, 2026 14:20
@Lee-W
Lee-W requested a review from jason810496May 25, 2026 15:27
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from c501951 to 20b1d9dCompareMay 25, 2026 15:28

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, please feel free to address in follow-up, non-blocking.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from a9e83e6 to 995da42CompareMay 27, 2026 01:40
Lee-W added 9 commits May 27, 2026 17:49
Introduce a new `airflow partitions` command group with a `clear`
action that nulls out `partition_key` and `partition_date` on one or
more DagRuns. Useful when migrating off a partitioned timetable or
fixing a Dag run whose partition metadata was set incorrectly.
Targeting:
- `--dag-id` (required)
- `--run-id` for a single run, or
- `--start-date` / `--end-date` to clear a `partition_date` range
(the two modes are mutually exclusive)
Safety:
- `--dry-run` previews matching runs without writing
- Prints the existing `partition_key` / `partition_date` before
clearing each run, and skips runs already cleared
Augment `airflow partitions clear` with `--clear-task-instances`:
in addition to nulling `partition_key` / `partition_date`, clear
the matched DagRuns' task instances so finished runs go back to
QUEUED and re-execute. Useful after fixing partition metadata when
the runs themselves should also re-run against the corrected logic.
Behavior:
- Default behavior is unchanged — only partition fields are nulled.
- With `--clear-task-instances`, runs whose partition fields are
already None are no longer skipped (their TIs are still cleared).
- `--dry-run` reports both projected actions without writing.
- `--partition-key VALUE`: clear all DagRuns whose partition_key matches this exact value.
- `--date a~b`: shorthand range syntax equivalent to `--start-date a --end-date b`.
The four selection modes (--run-id, --partition-key, --start-date/--end-date, --date) are mutually exclusive.
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 995da42 to 3d73d11CompareMay 27, 2026 09:49
@Lee-W
Lee-W merged commit 07859aa into apache:mainMay 27, 2026
143 checks passed
@Lee-W
Lee-W deleted the partitions-clear branch May 27, 2026 11:54
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

5 participants

@Lee-W@potiuk@uranusjr@jason810496@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): add partitions clear to reset DagRun partition fields - #66520

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear
May 27, 2026
Merged

feat(cli): add partitions clear to reset DagRun partition fields#66520
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear

Conversation

@Lee-W

@Lee-WLee-W commented May 7, 2026

Copy link
Copy Markdown
Member

part of #65921

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach Dag runs whose partition_date / partition_key is what identifies them (rollup or offset partitions). Today the only ways to bulk-reprocess such runs are N API calls or N UI clicks.

What

  • Add airflow dags clear <dag_id> with three mutually-exclusive selectors:
    • --run-id (single run)
    • --partition-key (every run with that exact key)
    • --partition-date-start / --partition-date-end (inclusive partition_date window; runs with NULL partition_date are never matched;
      inverted window is rejected)
  • Reuse Dag.clear() per matched run_id, so TI reset + Dag run re-queue behave identically to every other clear path.
  • Print a listing (run_id, partition_key, partition_date) before prompting; --yes skips the prompt.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 5 times, most recently from f4a47d7 to af063eaCompareMay 8, 2026 13:06
@Lee-W
Lee-W marked this pull request as ready for review May 8, 2026 13:07
@potiuk

Copy link
Copy Markdown
Member

Rebase is needed I guess.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from b809562 to e842cfaCompareMay 11, 2026 04:27
@Lee-W
Lee-W requested a review from uranusjrMay 11, 2026 06:27
@Lee-WLee-W moved this to In Review in AIP-76 Asset PartitioningMay 11, 2026
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 3481fdb to dc42fcaCompareMay 19, 2026 10:25
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/cli_config.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 961c171 to 1f1e52aCompareMay 21, 2026 14:21
@Lee-W
Lee-W requested a review from uranusjrMay 21, 2026 15:23
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 1f1e52a to 53a4e02CompareMay 22, 2026 00:12

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Not sure do I understand the batch clearing logic correctly.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 53a4e02 to ed18131CompareMay 25, 2026 12:54
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from ed18131 to c501951CompareMay 25, 2026 14:20
@Lee-W
Lee-W requested a review from jason810496May 25, 2026 15:27
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from c501951 to 20b1d9dCompareMay 25, 2026 15:28

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, please feel free to address in follow-up, non-blocking.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from a9e83e6 to 995da42CompareMay 27, 2026 01:40
Lee-W added 9 commits May 27, 2026 17:49
Introduce a new `airflow partitions` command group with a `clear`
action that nulls out `partition_key` and `partition_date` on one or
more DagRuns. Useful when migrating off a partitioned timetable or
fixing a Dag run whose partition metadata was set incorrectly.
Targeting:
- `--dag-id` (required)
- `--run-id` for a single run, or
- `--start-date` / `--end-date` to clear a `partition_date` range
(the two modes are mutually exclusive)
Safety:
- `--dry-run` previews matching runs without writing
- Prints the existing `partition_key` / `partition_date` before
clearing each run, and skips runs already cleared
Augment `airflow partitions clear` with `--clear-task-instances`:
in addition to nulling `partition_key` / `partition_date`, clear
the matched DagRuns' task instances so finished runs go back to
QUEUED and re-execute. Useful after fixing partition metadata when
the runs themselves should also re-run against the corrected logic.
Behavior:
- Default behavior is unchanged — only partition fields are nulled.
- With `--clear-task-instances`, runs whose partition fields are
already None are no longer skipped (their TIs are still cleared).
- `--dry-run` reports both projected actions without writing.
- `--partition-key VALUE`: clear all DagRuns whose partition_key matches this exact value.
- `--date a~b`: shorthand range syntax equivalent to `--start-date a --end-date b`.
The four selection modes (--run-id, --partition-key, --start-date/--end-date, --date) are mutually exclusive.
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 995da42 to 3d73d11CompareMay 27, 2026 09:49
@Lee-W
Lee-W merged commit 07859aa into apache:mainMay 27, 2026
143 checks passed
@Lee-W
Lee-W deleted the partitions-clear branch May 27, 2026 11:54
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

5 participants

@Lee-W@potiuk@uranusjr@jason810496@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(cli): add partitions clear to reset DagRun partition fields - #66520

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear
May 27, 2026
Merged

feat(cli): add partitions clear to reset DagRun partition fields#66520
Lee-W merged 9 commits into
apache:mainfrom
astronomer:partitions-clear

Conversation

@Lee-W

@Lee-WLee-W commented May 7, 2026

Copy link
Copy Markdown
Member

part of #65921

Why

airflow tasks clear --start-date / --end-date filters by logical_date, so it cannot reach Dag runs whose partition_date / partition_key is what identifies them (rollup or offset partitions). Today the only ways to bulk-reprocess such runs are N API calls or N UI clicks.

What

  • Add airflow dags clear <dag_id> with three mutually-exclusive selectors:
    • --run-id (single run)
    • --partition-key (every run with that exact key)
    • --partition-date-start / --partition-date-end (inclusive partition_date window; runs with NULL partition_date are never matched;
      inverted window is rejected)
  • Reuse Dag.clear() per matched run_id, so TI reset + Dag run re-queue behave identically to every other clear path.
  • Print a listing (run_id, partition_key, partition_date) before prompting; --yes skips the prompt.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 5 times, most recently from f4a47d7 to af063eaCompareMay 8, 2026 13:06
@Lee-W
Lee-W marked this pull request as ready for review May 8, 2026 13:07
@potiuk

Copy link
Copy Markdown
Member

Rebase is needed I guess.

@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from b809562 to e842cfaCompareMay 11, 2026 04:27
@Lee-W
Lee-W requested a review from uranusjrMay 11, 2026 06:27
@Lee-WLee-W moved this to In Review in AIP-76 Asset PartitioningMay 11, 2026
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 3481fdb to dc42fcaCompareMay 19, 2026 10:25
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/cli_config.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 3 times, most recently from 961c171 to 1f1e52aCompareMay 21, 2026 14:21
@Lee-W
Lee-W requested a review from uranusjrMay 21, 2026 15:23
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 1f1e52a to 53a4e02CompareMay 22, 2026 00:12

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Not sure do I understand the batch clearing logic correctly.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 53a4e02 to ed18131CompareMay 25, 2026 12:54
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from ed18131 to c501951CompareMay 25, 2026 14:20
@Lee-W
Lee-W requested a review from jason810496May 25, 2026 15:27
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from c501951 to 20b1d9dCompareMay 25, 2026 15:28

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, please feel free to address in follow-up, non-blocking.

Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py Outdated
Comment threadairflow-core/src/airflow/cli/commands/partition_command.py
@Lee-W
Lee-Wforce-pushed the partitions-clear branch 2 times, most recently from a9e83e6 to 995da42CompareMay 27, 2026 01:40
Lee-W added 9 commits May 27, 2026 17:49
Introduce a new `airflow partitions` command group with a `clear`
action that nulls out `partition_key` and `partition_date` on one or
more DagRuns. Useful when migrating off a partitioned timetable or
fixing a Dag run whose partition metadata was set incorrectly.
Targeting:
- `--dag-id` (required)
- `--run-id` for a single run, or
- `--start-date` / `--end-date` to clear a `partition_date` range
(the two modes are mutually exclusive)
Safety:
- `--dry-run` previews matching runs without writing
- Prints the existing `partition_key` / `partition_date` before
clearing each run, and skips runs already cleared
Augment `airflow partitions clear` with `--clear-task-instances`:
in addition to nulling `partition_key` / `partition_date`, clear
the matched DagRuns' task instances so finished runs go back to
QUEUED and re-execute. Useful after fixing partition metadata when
the runs themselves should also re-run against the corrected logic.
Behavior:
- Default behavior is unchanged — only partition fields are nulled.
- With `--clear-task-instances`, runs whose partition fields are
already None are no longer skipped (their TIs are still cleared).
- `--dry-run` reports both projected actions without writing.
- `--partition-key VALUE`: clear all DagRuns whose partition_key matches this exact value.
- `--date a~b`: shorthand range syntax equivalent to `--start-date a --end-date b`.
The four selection modes (--run-id, --partition-key, --start-date/--end-date, --date) are mutually exclusive.
@Lee-W
Lee-Wforce-pushed the partitions-clear branch from 995da42 to 3d73d11CompareMay 27, 2026 09:49
@Lee-W
Lee-W merged commit 07859aa into apache:mainMay 27, 2026
143 checks passed
@Lee-W
Lee-W deleted the partitions-clear branch May 27, 2026 11:54
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

5 participants

@Lee-W@potiuk@uranusjr@jason810496@vatsrahul1001