Skip to content

Add example plugins and expand docs for asset partitions - #68889

Merged
Lee-W merged 2 commits into
apache:mainfrom
astronomer:extend-partition-docs
Jun 26, 2026
Merged

Add example plugins and expand docs for asset partitions#68889
Lee-W merged 2 commits into
apache:mainfrom
astronomer:extend-partition-docs

Conversation

@Lee-W

@Lee-WLee-W commented Jun 23, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Why

Partition related docs and examples can be improved

What

  • Add custom_partition_mapper.py and custom_partition_timetable.py example plugins showing how to ship a custom PartitionMapper and a partitioned Timetable via AirflowPlugin.
  • Wire all three partition plugin examples (mapper, window, timetable) into the asset-partition docs.
  • Document FanOutMapper + max_downstream_keys, Window direction (FORWARD/BACKWARD), wait policies (WaitForAll/MinimumCount), and the partitioned vs partitioned_at_runtime distinction.
Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the extend-partition-docs branch from 4ecd0b9 to 781eacbCompareJune 23, 2026 10:44
@Lee-W
Lee-W marked this pull request as ready for review June 23, 2026 10:56
@Lee-WLee-W added the backport-to-v3-3-test Backport to v3-3-test label Jun 23, 2026
@Lee-WLee-W added this to the Airflow 3.3.0 milestone Jun 23, 2026
@Lee-W
Lee-Wforce-pushed the extend-partition-docs branch 2 times, most recently from e573484 to 4d2ecc8CompareJune 24, 2026 06:05
@Lee-WLee-W self-assigned this Jun 24, 2026
phanikumv
phanikumv reviewed Jun 25, 2026
View reviewed changes
Comment threadairflow-core/tests/unit/plugins/test_plugins_manager.py
@Lee-W
Lee-Wforce-pushed the extend-partition-docs branch from 4d2ecc8 to 38d922fCompareJune 25, 2026 02:51
uranusjr
uranusjr reviewed Jun 25, 2026
View reviewed changes
Comment threadairflow-core/docs/authoring-and-scheduling/assets.rst Outdated
uranusjr
uranusjr reviewed Jun 25, 2026
View reviewed changes
Comment threadairflow-core/docs/authoring-and-scheduling/assets.rst Outdated
@Lee-WLee-W mentioned this pull request Jun 25, 2026
Merged
1 task
@Lee-W
Lee-Wforce-pushed the extend-partition-docs branch 6 times, most recently from 7c0531c to 789a51aCompareJune 26, 2026 00:55
uranusjr
uranusjr reviewed Jun 26, 2026
View reviewed changes
Comment threadairflow-core/docs/authoring-and-scheduling/assets.rst Outdated
Lee-W added 2 commits June 26, 2026 14:45
@Lee-W
Add example plugins and expand docs for asset partitions
e74bdbc
- Add custom_partition_mapper.py and custom_partition_timetable.py example
plugins showing how to ship a custom PartitionMapper and a partitioned
Timetable via AirflowPlugin.
- Wire all three partition plugin examples (mapper, window, timetable) into
the asset-partition docs.
- Document FanOutMapper + max_downstream_keys, Window direction
(FORWARD/BACKWARD), wait policies (WaitForAll/MinimumCount), and the
partitioned vs partitioned_at_runtime distinction.
@Lee-W
Reframe asset partitioning docs as predetermined vs runtime kinds
b464cf1
Rewrite the partitioning section so the two approaches read as kinds
of one concept — predetermined (the key is worked out before the run)
and runtime (the task records it), and are mutually exclusive — with
the section header naming them rather than labelling them generically.
Align the custom timetable plugin docstring, and fix prose "DAG" to
"Dag" in the @asset section.
@Lee-W
Lee-Wforce-pushed the extend-partition-docs branch from 789a51a to b464cf1CompareJune 26, 2026 07:05
@Lee-W
Lee-W merged commit 79e9628 into apache:mainJun 26, 2026
82 checks passed
@Lee-W
Lee-W deleted the extend-partition-docs branch June 26, 2026 07:54
@github-actions

github-actionsBot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
✅v3-3-testPR Link

Lee-W added a commit that referenced this pull request Jun 26, 2026
@github-actions@Lee-W
[v3-3-test] Add example plugins and expand docs for asset partitions (#…
a807b28
…68889) (#69017)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jun 30, 2026
@Lee-W@karenbraganz
Add example plugins and expand docs for asset partitions (apache#68889)
41af722
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@uranusjruranusjruranusjr left review comments
@phanikumvphanikumvphanikumv approved these changes
@jason810496jason810496jason810496 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajesh
@vatsrahul1001vatsrahul1001Awaiting requested review from vatsrahul1001

Assignees

@Lee-WLee-W

Labels

backport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects

None yet

Milestone

Airflow 3.3.0

Development

Successfully merging this pull request may close these issues.

4 participants

@Lee-W@uranusjr@jason810496@phanikumv