Add partition_key to Metadata so yield matches outlet_events - #71993

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity
Closed

Add partition_key to Metadata so yield matches outlet_events#71993
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Add partition_key to Metadata so yield matches outlet_events

closes: #69117

What I did

Added optional Metadata.partition_key. Sync and async runners call add_partitions when it is set. Extra still merges and never becomes a key. Docs say the two APIs agree for extra, and that a partitioned consumer needs an explicit key. Newsfragment 71993.bugfix.rst.

yield Metadata(asset, extra={"row_count": 1}, partition_key="us")
# same as
outlet_events[asset].extra = {"row_count": 1}
outlet_events[asset].add_partitions("us")

Why I did

Docs said the two APIs were the same. For extra they are. For partitions they were not. outlet_events[ASSET].extra = {"section": "XY"} does not select a partition, so a partitioned consumer gets no APDR. Reporter also hit outlet_events=None as a positional parameter. Injection already worked; the test keeps it that way.

How I did

Metadata.partition_key: str | None = None is the last field, so Metadata(asset, extra) still works. After extra.update, outlet_events[asset].add_partitions(partition_key). Invalid keys fail the TI. Alias emit does not fan out keys; partition_key is recorded on the concrete asset accessor only. Alias-only emit still uses the producing Dag run's key.

Same partition key still batches onto one APDR. Distinct keys still make distinct runs. No scheduler change.

What's the impact

Users can yield a partition key the same way they call add_partitions. Extra-only emit to a partitioned consumer still warns and does not create an empty Dag run.

What's the testing

task-sdk/tests/task_sdk/execution_time/test_callback_runner.py

  • test_two_yields_merge_extra_and_collect_partition_keys
  • test_add_partitions_list_matches_two_metadata_yields
  • test_extra_does_not_imply_partition_key
  • test_invalid_partition_key_raises
  • test_alias_and_partition_key_records_key_on_asset
  • test_alias_as_metadata_asset_with_partition_key_raises_type_error

task-sdk/tests/task_sdk/execution_time/test_task_runner.py

  • test_emits_shared_extra_on_each_partition_key

airflow-core/tests/unit/models/test_taskinstance.py

  • test_mapped_outlet_events_extra_injection
  • test_mapped_metadata_partition_key_and_extra_per_ti
  • test_metadata_invalid_partition_key_fails_task_without_event

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_producer_partition_keys_match_between_apis
  • test_extra_only_mapped_emit_does_not_create_apdr
  • test_runtime_partition_key_without_partition_date_still_queues
  • test_partitioned_event_does_not_queue_non_partitioned_consumer
  • test_multi_partition_payloads_share_merged_extra
uv run --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_runner.py \
task-sdk/tests/task_sdk/execution_time/test_task_runner.py -k 'serialize_outlet or partition or Metadata or add_partitions' -q
uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py \
airflow-core/tests/unit/jobs/test_scheduler_job.py \
-k 'outlet_events or partition_key or Metadata or mapped_producer_partition' -q
Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

@Vamsi-klu
Add partition_key to Metadata so yield matches outlet_events
b2510c3
Docs treated yield Metadata and outlet_events as the same API. Extra
already merged onto the same accessor, but partitioned consumers need
an explicit key and extra was never a substitute.
closes: apache#69117
@Vamsi-klu
Address review feedback for Metadata partition_key
f525aaa
The upgrade note pointed at an unmerged batching PR, alias docs did not
say where partition_key is recorded, and one ADRQ assertion counted the
whole table.
@boring-cyborgboring-cyborgBot added area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation labels Aug 23, 2026
@Vamsi-klu

Vamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr — Metadata.partition_key so yield matches outlet_events.add_partitions.

Drafted-by: Cursor Grok 4.6

@Vamsi-klu@cursoragent
Name the newsfragment after the pull request
3b66bd6
The check rejects issue-numbered fragments so the changelog entry
links to the change that actually merged.
Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-kluVamsi-klu mentioned this pull request Aug 23, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Inconsistency between yielding Metadata and outlet_events extras for partitioned Assets

1 participant

@Vamsi-klu
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Add partition_key to Metadata so yield matches outlet_events - #71993

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity
Closed

Add partition_key to Metadata so yield matches outlet_events#71993
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Add partition_key to Metadata so yield matches outlet_events

closes: #69117

What I did

Added optional Metadata.partition_key. Sync and async runners call add_partitions when it is set. Extra still merges and never becomes a key. Docs say the two APIs agree for extra, and that a partitioned consumer needs an explicit key. Newsfragment 71993.bugfix.rst.

yield Metadata(asset, extra={"row_count": 1}, partition_key="us")
# same as
outlet_events[asset].extra = {"row_count": 1}
outlet_events[asset].add_partitions("us")

Why I did

Docs said the two APIs were the same. For extra they are. For partitions they were not. outlet_events[ASSET].extra = {"section": "XY"} does not select a partition, so a partitioned consumer gets no APDR. Reporter also hit outlet_events=None as a positional parameter. Injection already worked; the test keeps it that way.

How I did

Metadata.partition_key: str | None = None is the last field, so Metadata(asset, extra) still works. After extra.update, outlet_events[asset].add_partitions(partition_key). Invalid keys fail the TI. Alias emit does not fan out keys; partition_key is recorded on the concrete asset accessor only. Alias-only emit still uses the producing Dag run's key.

Same partition key still batches onto one APDR. Distinct keys still make distinct runs. No scheduler change.

What's the impact

Users can yield a partition key the same way they call add_partitions. Extra-only emit to a partitioned consumer still warns and does not create an empty Dag run.

What's the testing

task-sdk/tests/task_sdk/execution_time/test_callback_runner.py

  • test_two_yields_merge_extra_and_collect_partition_keys
  • test_add_partitions_list_matches_two_metadata_yields
  • test_extra_does_not_imply_partition_key
  • test_invalid_partition_key_raises
  • test_alias_and_partition_key_records_key_on_asset
  • test_alias_as_metadata_asset_with_partition_key_raises_type_error

task-sdk/tests/task_sdk/execution_time/test_task_runner.py

  • test_emits_shared_extra_on_each_partition_key

airflow-core/tests/unit/models/test_taskinstance.py

  • test_mapped_outlet_events_extra_injection
  • test_mapped_metadata_partition_key_and_extra_per_ti
  • test_metadata_invalid_partition_key_fails_task_without_event

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_producer_partition_keys_match_between_apis
  • test_extra_only_mapped_emit_does_not_create_apdr
  • test_runtime_partition_key_without_partition_date_still_queues
  • test_partitioned_event_does_not_queue_non_partitioned_consumer
  • test_multi_partition_payloads_share_merged_extra
uv run --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_runner.py \
task-sdk/tests/task_sdk/execution_time/test_task_runner.py -k 'serialize_outlet or partition or Metadata or add_partitions' -q
uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py \
airflow-core/tests/unit/jobs/test_scheduler_job.py \
-k 'outlet_events or partition_key or Metadata or mapped_producer_partition' -q
Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

@Vamsi-klu
Add partition_key to Metadata so yield matches outlet_events
b2510c3
Docs treated yield Metadata and outlet_events as the same API. Extra
already merged onto the same accessor, but partitioned consumers need
an explicit key and extra was never a substitute.
closes: apache#69117
@Vamsi-klu
Address review feedback for Metadata partition_key
f525aaa
The upgrade note pointed at an unmerged batching PR, alias docs did not
say where partition_key is recorded, and one ADRQ assertion counted the
whole table.
@boring-cyborgboring-cyborgBot added area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation labels Aug 23, 2026
@Vamsi-klu

Vamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr — Metadata.partition_key so yield matches outlet_events.add_partitions.

Drafted-by: Cursor Grok 4.6

@Vamsi-klu@cursoragent
Name the newsfragment after the pull request
3b66bd6
The check rejects issue-numbered fragments so the changelog entry
links to the change that actually merged.
Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-kluVamsi-klu mentioned this pull request Aug 23, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Inconsistency between yielding Metadata and outlet_events extras for partitioned Assets

1 participant

@Vamsi-klu
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add partition_key to Metadata so yield matches outlet_events - #71993

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity
Closed

Add partition_key to Metadata so yield matches outlet_events#71993
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Add partition_key to Metadata so yield matches outlet_events

closes: #69117

What I did

Added optional Metadata.partition_key. Sync and async runners call add_partitions when it is set. Extra still merges and never becomes a key. Docs say the two APIs agree for extra, and that a partitioned consumer needs an explicit key. Newsfragment 71993.bugfix.rst.

yield Metadata(asset, extra={"row_count": 1}, partition_key="us")
# same as
outlet_events[asset].extra = {"row_count": 1}
outlet_events[asset].add_partitions("us")

Why I did

Docs said the two APIs were the same. For extra they are. For partitions they were not. outlet_events[ASSET].extra = {"section": "XY"} does not select a partition, so a partitioned consumer gets no APDR. Reporter also hit outlet_events=None as a positional parameter. Injection already worked; the test keeps it that way.

How I did

Metadata.partition_key: str | None = None is the last field, so Metadata(asset, extra) still works. After extra.update, outlet_events[asset].add_partitions(partition_key). Invalid keys fail the TI. Alias emit does not fan out keys; partition_key is recorded on the concrete asset accessor only. Alias-only emit still uses the producing Dag run's key.

Same partition key still batches onto one APDR. Distinct keys still make distinct runs. No scheduler change.

What's the impact

Users can yield a partition key the same way they call add_partitions. Extra-only emit to a partitioned consumer still warns and does not create an empty Dag run.

What's the testing

task-sdk/tests/task_sdk/execution_time/test_callback_runner.py

  • test_two_yields_merge_extra_and_collect_partition_keys
  • test_add_partitions_list_matches_two_metadata_yields
  • test_extra_does_not_imply_partition_key
  • test_invalid_partition_key_raises
  • test_alias_and_partition_key_records_key_on_asset
  • test_alias_as_metadata_asset_with_partition_key_raises_type_error

task-sdk/tests/task_sdk/execution_time/test_task_runner.py

  • test_emits_shared_extra_on_each_partition_key

airflow-core/tests/unit/models/test_taskinstance.py

  • test_mapped_outlet_events_extra_injection
  • test_mapped_metadata_partition_key_and_extra_per_ti
  • test_metadata_invalid_partition_key_fails_task_without_event

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_producer_partition_keys_match_between_apis
  • test_extra_only_mapped_emit_does_not_create_apdr
  • test_runtime_partition_key_without_partition_date_still_queues
  • test_partitioned_event_does_not_queue_non_partitioned_consumer
  • test_multi_partition_payloads_share_merged_extra
uv run --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_runner.py \
task-sdk/tests/task_sdk/execution_time/test_task_runner.py -k 'serialize_outlet or partition or Metadata or add_partitions' -q
uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py \
airflow-core/tests/unit/jobs/test_scheduler_job.py \
-k 'outlet_events or partition_key or Metadata or mapped_producer_partition' -q
Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

@Vamsi-klu
Add partition_key to Metadata so yield matches outlet_events
b2510c3
Docs treated yield Metadata and outlet_events as the same API. Extra
already merged onto the same accessor, but partitioned consumers need
an explicit key and extra was never a substitute.
closes: apache#69117
@Vamsi-klu
Address review feedback for Metadata partition_key
f525aaa
The upgrade note pointed at an unmerged batching PR, alias docs did not
say where partition_key is recorded, and one ADRQ assertion counted the
whole table.
@boring-cyborgboring-cyborgBot added area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation labels Aug 23, 2026
@Vamsi-klu

Vamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr — Metadata.partition_key so yield matches outlet_events.add_partitions.

Drafted-by: Cursor Grok 4.6

@Vamsi-klu@cursoragent
Name the newsfragment after the pull request
3b66bd6
The check rejects issue-numbered fragments so the changelog entry
links to the change that actually merged.
Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-kluVamsi-klu mentioned this pull request Aug 23, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Inconsistency between yielding Metadata and outlet_events extras for partitioned Assets

1 participant

@Vamsi-klu
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add partition_key to Metadata so yield matches outlet_events - #71993

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity
Closed

Add partition_key to Metadata so yield matches outlet_events#71993
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Add partition_key to Metadata so yield matches outlet_events

closes: #69117

What I did

Added optional Metadata.partition_key. Sync and async runners call add_partitions when it is set. Extra still merges and never becomes a key. Docs say the two APIs agree for extra, and that a partitioned consumer needs an explicit key. Newsfragment 71993.bugfix.rst.

yield Metadata(asset, extra={"row_count": 1}, partition_key="us")
# same as
outlet_events[asset].extra = {"row_count": 1}
outlet_events[asset].add_partitions("us")

Why I did

Docs said the two APIs were the same. For extra they are. For partitions they were not. outlet_events[ASSET].extra = {"section": "XY"} does not select a partition, so a partitioned consumer gets no APDR. Reporter also hit outlet_events=None as a positional parameter. Injection already worked; the test keeps it that way.

How I did

Metadata.partition_key: str | None = None is the last field, so Metadata(asset, extra) still works. After extra.update, outlet_events[asset].add_partitions(partition_key). Invalid keys fail the TI. Alias emit does not fan out keys; partition_key is recorded on the concrete asset accessor only. Alias-only emit still uses the producing Dag run's key.

Same partition key still batches onto one APDR. Distinct keys still make distinct runs. No scheduler change.

What's the impact

Users can yield a partition key the same way they call add_partitions. Extra-only emit to a partitioned consumer still warns and does not create an empty Dag run.

What's the testing

task-sdk/tests/task_sdk/execution_time/test_callback_runner.py

  • test_two_yields_merge_extra_and_collect_partition_keys
  • test_add_partitions_list_matches_two_metadata_yields
  • test_extra_does_not_imply_partition_key
  • test_invalid_partition_key_raises
  • test_alias_and_partition_key_records_key_on_asset
  • test_alias_as_metadata_asset_with_partition_key_raises_type_error

task-sdk/tests/task_sdk/execution_time/test_task_runner.py

  • test_emits_shared_extra_on_each_partition_key

airflow-core/tests/unit/models/test_taskinstance.py

  • test_mapped_outlet_events_extra_injection
  • test_mapped_metadata_partition_key_and_extra_per_ti
  • test_metadata_invalid_partition_key_fails_task_without_event

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_producer_partition_keys_match_between_apis
  • test_extra_only_mapped_emit_does_not_create_apdr
  • test_runtime_partition_key_without_partition_date_still_queues
  • test_partitioned_event_does_not_queue_non_partitioned_consumer
  • test_multi_partition_payloads_share_merged_extra
uv run --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_runner.py \
task-sdk/tests/task_sdk/execution_time/test_task_runner.py -k 'serialize_outlet or partition or Metadata or add_partitions' -q
uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py \
airflow-core/tests/unit/jobs/test_scheduler_job.py \
-k 'outlet_events or partition_key or Metadata or mapped_producer_partition' -q
Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

@Vamsi-klu
Add partition_key to Metadata so yield matches outlet_events
b2510c3
Docs treated yield Metadata and outlet_events as the same API. Extra
already merged onto the same accessor, but partitioned consumers need
an explicit key and extra was never a substitute.
closes: apache#69117
@Vamsi-klu
Address review feedback for Metadata partition_key
f525aaa
The upgrade note pointed at an unmerged batching PR, alias docs did not
say where partition_key is recorded, and one ADRQ assertion counted the
whole table.
@boring-cyborgboring-cyborgBot added area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation labels Aug 23, 2026
@Vamsi-klu

Vamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr — Metadata.partition_key so yield matches outlet_events.add_partitions.

Drafted-by: Cursor Grok 4.6

@Vamsi-klu@cursoragent
Name the newsfragment after the pull request
3b66bd6
The check rejects issue-numbered fragments so the changelog entry
links to the change that actually merged.
Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-kluVamsi-klu mentioned this pull request Aug 23, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Inconsistency between yielding Metadata and outlet_events extras for partitioned Assets

1 participant

@Vamsi-klu
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Add partition_key to Metadata so yield matches outlet_events - #71993

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity
Closed

Add partition_key to Metadata so yield matches outlet_events#71993
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Add partition_key to Metadata so yield matches outlet_events

closes: #69117

What I did

Added optional Metadata.partition_key. Sync and async runners call add_partitions when it is set. Extra still merges and never becomes a key. Docs say the two APIs agree for extra, and that a partitioned consumer needs an explicit key. Newsfragment 71993.bugfix.rst.

yield Metadata(asset, extra={"row_count": 1}, partition_key="us")
# same as
outlet_events[asset].extra = {"row_count": 1}
outlet_events[asset].add_partitions("us")

Why I did

Docs said the two APIs were the same. For extra they are. For partitions they were not. outlet_events[ASSET].extra = {"section": "XY"} does not select a partition, so a partitioned consumer gets no APDR. Reporter also hit outlet_events=None as a positional parameter. Injection already worked; the test keeps it that way.

How I did

Metadata.partition_key: str | None = None is the last field, so Metadata(asset, extra) still works. After extra.update, outlet_events[asset].add_partitions(partition_key). Invalid keys fail the TI. Alias emit does not fan out keys; partition_key is recorded on the concrete asset accessor only. Alias-only emit still uses the producing Dag run's key.

Same partition key still batches onto one APDR. Distinct keys still make distinct runs. No scheduler change.

What's the impact

Users can yield a partition key the same way they call add_partitions. Extra-only emit to a partitioned consumer still warns and does not create an empty Dag run.

What's the testing

task-sdk/tests/task_sdk/execution_time/test_callback_runner.py

  • test_two_yields_merge_extra_and_collect_partition_keys
  • test_add_partitions_list_matches_two_metadata_yields
  • test_extra_does_not_imply_partition_key
  • test_invalid_partition_key_raises
  • test_alias_and_partition_key_records_key_on_asset
  • test_alias_as_metadata_asset_with_partition_key_raises_type_error

task-sdk/tests/task_sdk/execution_time/test_task_runner.py

  • test_emits_shared_extra_on_each_partition_key

airflow-core/tests/unit/models/test_taskinstance.py

  • test_mapped_outlet_events_extra_injection
  • test_mapped_metadata_partition_key_and_extra_per_ti
  • test_metadata_invalid_partition_key_fails_task_without_event

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_producer_partition_keys_match_between_apis
  • test_extra_only_mapped_emit_does_not_create_apdr
  • test_runtime_partition_key_without_partition_date_still_queues
  • test_partitioned_event_does_not_queue_non_partitioned_consumer
  • test_multi_partition_payloads_share_merged_extra
uv run --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_runner.py \
task-sdk/tests/task_sdk/execution_time/test_task_runner.py -k 'serialize_outlet or partition or Metadata or add_partitions' -q
uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py \
airflow-core/tests/unit/jobs/test_scheduler_job.py \
-k 'outlet_events or partition_key or Metadata or mapped_producer_partition' -q
Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

@Vamsi-klu
Add partition_key to Metadata so yield matches outlet_events
b2510c3
Docs treated yield Metadata and outlet_events as the same API. Extra
already merged onto the same accessor, but partitioned consumers need
an explicit key and extra was never a substitute.
closes: apache#69117
@Vamsi-klu
Address review feedback for Metadata partition_key
f525aaa
The upgrade note pointed at an unmerged batching PR, alias docs did not
say where partition_key is recorded, and one ADRQ assertion counted the
whole table.
@boring-cyborgboring-cyborgBot added area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation labels Aug 23, 2026
@Vamsi-klu

Vamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr — Metadata.partition_key so yield matches outlet_events.add_partitions.

Drafted-by: Cursor Grok 4.6

@Vamsi-klu@cursoragent
Name the newsfragment after the pull request
3b66bd6
The check rejects issue-numbered fragments so the changelog entry
links to the change that actually merged.
Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-kluVamsi-klu mentioned this pull request Aug 23, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Inconsistency between yielding Metadata and outlet_events extras for partitioned Assets

1 participant

@Vamsi-klu
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add partition_key to Metadata so yield matches outlet_events - #71993

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity
Closed

Add partition_key to Metadata so yield matches outlet_events#71993
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Add partition_key to Metadata so yield matches outlet_events

closes: #69117

What I did

Added optional Metadata.partition_key. Sync and async runners call add_partitions when it is set. Extra still merges and never becomes a key. Docs say the two APIs agree for extra, and that a partitioned consumer needs an explicit key. Newsfragment 71993.bugfix.rst.

yield Metadata(asset, extra={"row_count": 1}, partition_key="us")
# same as
outlet_events[asset].extra = {"row_count": 1}
outlet_events[asset].add_partitions("us")

Why I did

Docs said the two APIs were the same. For extra they are. For partitions they were not. outlet_events[ASSET].extra = {"section": "XY"} does not select a partition, so a partitioned consumer gets no APDR. Reporter also hit outlet_events=None as a positional parameter. Injection already worked; the test keeps it that way.

How I did

Metadata.partition_key: str | None = None is the last field, so Metadata(asset, extra) still works. After extra.update, outlet_events[asset].add_partitions(partition_key). Invalid keys fail the TI. Alias emit does not fan out keys; partition_key is recorded on the concrete asset accessor only. Alias-only emit still uses the producing Dag run's key.

Same partition key still batches onto one APDR. Distinct keys still make distinct runs. No scheduler change.

What's the impact

Users can yield a partition key the same way they call add_partitions. Extra-only emit to a partitioned consumer still warns and does not create an empty Dag run.

What's the testing

task-sdk/tests/task_sdk/execution_time/test_callback_runner.py

  • test_two_yields_merge_extra_and_collect_partition_keys
  • test_add_partitions_list_matches_two_metadata_yields
  • test_extra_does_not_imply_partition_key
  • test_invalid_partition_key_raises
  • test_alias_and_partition_key_records_key_on_asset
  • test_alias_as_metadata_asset_with_partition_key_raises_type_error

task-sdk/tests/task_sdk/execution_time/test_task_runner.py

  • test_emits_shared_extra_on_each_partition_key

airflow-core/tests/unit/models/test_taskinstance.py

  • test_mapped_outlet_events_extra_injection
  • test_mapped_metadata_partition_key_and_extra_per_ti
  • test_metadata_invalid_partition_key_fails_task_without_event

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_producer_partition_keys_match_between_apis
  • test_extra_only_mapped_emit_does_not_create_apdr
  • test_runtime_partition_key_without_partition_date_still_queues
  • test_partitioned_event_does_not_queue_non_partitioned_consumer
  • test_multi_partition_payloads_share_merged_extra
uv run --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_runner.py \
task-sdk/tests/task_sdk/execution_time/test_task_runner.py -k 'serialize_outlet or partition or Metadata or add_partitions' -q
uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py \
airflow-core/tests/unit/jobs/test_scheduler_job.py \
-k 'outlet_events or partition_key or Metadata or mapped_producer_partition' -q
Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

@Vamsi-klu
Add partition_key to Metadata so yield matches outlet_events
b2510c3
Docs treated yield Metadata and outlet_events as the same API. Extra
already merged onto the same accessor, but partitioned consumers need
an explicit key and extra was never a substitute.
closes: apache#69117
@Vamsi-klu
Address review feedback for Metadata partition_key
f525aaa
The upgrade note pointed at an unmerged batching PR, alias docs did not
say where partition_key is recorded, and one ADRQ assertion counted the
whole table.
@boring-cyborgboring-cyborgBot added area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation labels Aug 23, 2026
@Vamsi-klu

Vamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr — Metadata.partition_key so yield matches outlet_events.add_partitions.

Drafted-by: Cursor Grok 4.6

@Vamsi-klu@cursoragent
Name the newsfragment after the pull request
3b66bd6
The check rejects issue-numbered fragments so the changelog entry
links to the change that actually merged.
Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-kluVamsi-klu mentioned this pull request Aug 23, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Inconsistency between yielding Metadata and outlet_events extras for partitioned Assets

1 participant

@Vamsi-klu
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add partition_key to Metadata so yield matches outlet_events - #71993

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity
Closed

Add partition_key to Metadata so yield matches outlet_events#71993
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Add partition_key to Metadata so yield matches outlet_events

closes: #69117

What I did

Added optional Metadata.partition_key. Sync and async runners call add_partitions when it is set. Extra still merges and never becomes a key. Docs say the two APIs agree for extra, and that a partitioned consumer needs an explicit key. Newsfragment 71993.bugfix.rst.

yield Metadata(asset, extra={"row_count": 1}, partition_key="us")
# same as
outlet_events[asset].extra = {"row_count": 1}
outlet_events[asset].add_partitions("us")

Why I did

Docs said the two APIs were the same. For extra they are. For partitions they were not. outlet_events[ASSET].extra = {"section": "XY"} does not select a partition, so a partitioned consumer gets no APDR. Reporter also hit outlet_events=None as a positional parameter. Injection already worked; the test keeps it that way.

How I did

Metadata.partition_key: str | None = None is the last field, so Metadata(asset, extra) still works. After extra.update, outlet_events[asset].add_partitions(partition_key). Invalid keys fail the TI. Alias emit does not fan out keys; partition_key is recorded on the concrete asset accessor only. Alias-only emit still uses the producing Dag run's key.

Same partition key still batches onto one APDR. Distinct keys still make distinct runs. No scheduler change.

What's the impact

Users can yield a partition key the same way they call add_partitions. Extra-only emit to a partitioned consumer still warns and does not create an empty Dag run.

What's the testing

task-sdk/tests/task_sdk/execution_time/test_callback_runner.py

  • test_two_yields_merge_extra_and_collect_partition_keys
  • test_add_partitions_list_matches_two_metadata_yields
  • test_extra_does_not_imply_partition_key
  • test_invalid_partition_key_raises
  • test_alias_and_partition_key_records_key_on_asset
  • test_alias_as_metadata_asset_with_partition_key_raises_type_error

task-sdk/tests/task_sdk/execution_time/test_task_runner.py

  • test_emits_shared_extra_on_each_partition_key

airflow-core/tests/unit/models/test_taskinstance.py

  • test_mapped_outlet_events_extra_injection
  • test_mapped_metadata_partition_key_and_extra_per_ti
  • test_metadata_invalid_partition_key_fails_task_without_event

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_producer_partition_keys_match_between_apis
  • test_extra_only_mapped_emit_does_not_create_apdr
  • test_runtime_partition_key_without_partition_date_still_queues
  • test_partitioned_event_does_not_queue_non_partitioned_consumer
  • test_multi_partition_payloads_share_merged_extra
uv run --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_runner.py \
task-sdk/tests/task_sdk/execution_time/test_task_runner.py -k 'serialize_outlet or partition or Metadata or add_partitions' -q
uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py \
airflow-core/tests/unit/jobs/test_scheduler_job.py \
-k 'outlet_events or partition_key or Metadata or mapped_producer_partition' -q
Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

@Vamsi-klu
Add partition_key to Metadata so yield matches outlet_events
b2510c3
Docs treated yield Metadata and outlet_events as the same API. Extra
already merged onto the same accessor, but partitioned consumers need
an explicit key and extra was never a substitute.
closes: apache#69117
@Vamsi-klu
Address review feedback for Metadata partition_key
f525aaa
The upgrade note pointed at an unmerged batching PR, alias docs did not
say where partition_key is recorded, and one ADRQ assertion counted the
whole table.
@boring-cyborgboring-cyborgBot added area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation labels Aug 23, 2026
@Vamsi-klu

Vamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr — Metadata.partition_key so yield matches outlet_events.add_partitions.

Drafted-by: Cursor Grok 4.6

@Vamsi-klu@cursoragent
Name the newsfragment after the pull request
3b66bd6
The check rejects issue-numbered fragments so the changelog entry
links to the change that actually merged.
Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-kluVamsi-klu mentioned this pull request Aug 23, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Inconsistency between yielding Metadata and outlet_events extras for partitioned Assets

1 participant

@Vamsi-klu
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Add partition_key to Metadata so yield matches outlet_events - #71993

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity
Closed

Add partition_key to Metadata so yield matches outlet_events#71993
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:pr3/69117-metadata-partition-parity

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Add partition_key to Metadata so yield matches outlet_events

closes: #69117

What I did

Added optional Metadata.partition_key. Sync and async runners call add_partitions when it is set. Extra still merges and never becomes a key. Docs say the two APIs agree for extra, and that a partitioned consumer needs an explicit key. Newsfragment 71993.bugfix.rst.

yield Metadata(asset, extra={"row_count": 1}, partition_key="us")
# same as
outlet_events[asset].extra = {"row_count": 1}
outlet_events[asset].add_partitions("us")

Why I did

Docs said the two APIs were the same. For extra they are. For partitions they were not. outlet_events[ASSET].extra = {"section": "XY"} does not select a partition, so a partitioned consumer gets no APDR. Reporter also hit outlet_events=None as a positional parameter. Injection already worked; the test keeps it that way.

How I did

Metadata.partition_key: str | None = None is the last field, so Metadata(asset, extra) still works. After extra.update, outlet_events[asset].add_partitions(partition_key). Invalid keys fail the TI. Alias emit does not fan out keys; partition_key is recorded on the concrete asset accessor only. Alias-only emit still uses the producing Dag run's key.

Same partition key still batches onto one APDR. Distinct keys still make distinct runs. No scheduler change.

What's the impact

Users can yield a partition key the same way they call add_partitions. Extra-only emit to a partitioned consumer still warns and does not create an empty Dag run.

What's the testing

task-sdk/tests/task_sdk/execution_time/test_callback_runner.py

  • test_two_yields_merge_extra_and_collect_partition_keys
  • test_add_partitions_list_matches_two_metadata_yields
  • test_extra_does_not_imply_partition_key
  • test_invalid_partition_key_raises
  • test_alias_and_partition_key_records_key_on_asset
  • test_alias_as_metadata_asset_with_partition_key_raises_type_error

task-sdk/tests/task_sdk/execution_time/test_task_runner.py

  • test_emits_shared_extra_on_each_partition_key

airflow-core/tests/unit/models/test_taskinstance.py

  • test_mapped_outlet_events_extra_injection
  • test_mapped_metadata_partition_key_and_extra_per_ti
  • test_metadata_invalid_partition_key_fails_task_without_event

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_producer_partition_keys_match_between_apis
  • test_extra_only_mapped_emit_does_not_create_apdr
  • test_runtime_partition_key_without_partition_date_still_queues
  • test_partitioned_event_does_not_queue_non_partitioned_consumer
  • test_multi_partition_payloads_share_merged_extra
uv run --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_runner.py \
task-sdk/tests/task_sdk/execution_time/test_task_runner.py -k 'serialize_outlet or partition or Metadata or add_partitions' -q
uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py \
airflow-core/tests/unit/jobs/test_scheduler_job.py \
-k 'outlet_events or partition_key or Metadata or mapped_producer_partition' -q
Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

@Vamsi-klu
Add partition_key to Metadata so yield matches outlet_events
b2510c3
Docs treated yield Metadata and outlet_events as the same API. Extra
already merged onto the same accessor, but partitioned consumers need
an explicit key and extra was never a substitute.
closes: apache#69117
@Vamsi-klu
Address review feedback for Metadata partition_key
f525aaa
The upgrade note pointed at an unmerged batching PR, alias docs did not
say where partition_key is recorded, and one ADRQ assertion counted the
whole table.
@boring-cyborgboring-cyborgBot added area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation labels Aug 23, 2026
@Vamsi-klu

Vamsi-klu commented Aug 23, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr — Metadata.partition_key so yield matches outlet_events.add_partitions.

Drafted-by: Cursor Grok 4.6

@Vamsi-klu@cursoragent
Name the newsfragment after the pull request
3b66bd6
The check rejects issue-numbered fragments so the changelog entry
links to the change that actually merged.
Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-kluVamsi-klu mentioned this pull request Aug 23, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Inconsistency between yielding Metadata and outlet_events extras for partitioned Assets

1 participant

@Vamsi-klu