Uh oh!
There was an error while loading. Please reload this page.
Add partition_key to Metadata so yield matches outlet_events - #71999
Open
Vamsi-klu wants to merge 6 commits into
Open
Add partition_key to Metadata so yield matches outlet_events#71999Vamsi-klu wants to merge 6 commits into
Vamsi-klu wants to merge 6 commits into
Conversation
Vamsi-klu
commented
Aug 23, 2026
ContributorAuthor
cc @dstandish@jroachgolf84@ashb@kaxil@Lee-W@uranusjr Metadata.partition_key so yield matches outlet_events.add_partitions. Drafted-by: Cursor Grok 4.6; reviewed by @Vamsi-klu before posting |
Docs treated yield Metadata and outlet_events as the same API. Extra already merged onto the same accessor, but partitioned consumers need an explicit key and extra was never a substitute. closes: apache#69117
The upgrade note pointed at an unmerged batching PR, alias docs did not say where partition_key is recorded, and one ADRQ assertion counted the whole table.
The check rejects issue-numbered fragments so the changelog entry links to the change that actually merged. Co-authored-by: Cursor <cursoragent@cursor.com>
The check rejects a fragment named for an older number so the changelog entry links to the change that actually merged. Co-authored-by: Cursor <cursoragent@cursor.com>
cursorBotforce-pushed
the
pr3/69117-metadata-partition-parity
branch
from
August 24, 2026 00:18
eba6c0d to
b192b4aCompareLee-W
reviewed
Aug 24, 2026
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
The two emit APIs should take the same keys argument. A single-key field still forced one yield per partition, and the changelog called that a bug when it is just the missing side of the API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the change?
Metadatagains an optionalpartition_keysfield that takes one key or a list, the same valuesoutlet_events[asset].add_partitionsaccepts. The sync and async callback runners calladd_partitionswhen it is set, so yieldingMetadatarecords partitions exactly like the accessor API.Why did I do it?
closes: #69117
The docs said the two APIs were equivalent. For
extrathey are; for partitions they were not. Settingextranever selects a partition, so a partitioned consumer got noAssetPartitionDagRunand users had no yield-based way to attach a key.How did I do it?
partition_keys: str | list[str] | None = Noneis the last field, so existing positionalMetadata(asset, extra)calls keep working. After mergingextra, the runner callsadd_partitions. Invalid keys raise and fail the task before any event is emitted. Keys land on the concrete asset accessor only; alias-resolved events keep the producing Dag run's key. I updated the assets docs with the equivalence and the partitioned-consumer rules.What's the impact?
Task authors can attach partition keys from a yield. Extra-only emits to a partitioned consumer still log a missing-key warning and create no run. No scheduler change.
What's the test plan?
New
test_callback_runner.pycovers merging, list keys, alias interaction, and invalid keys for both runners. Parity tests intest_scheduler_job.pyrun both APIs and compare events,AssetPartitionDagRunrows, and consumer runs.test_taskinstance.pyadds mapped injection and invalid-key tests;test_task_runner.pypins serialization.Was generative AI tooling used to co-author this PR?
Generated-by: Grok 4.6 following the guidelines