Skip to content

AIP-103: Documentation for Task and Asset Store - #67299

Merged
amoghrajesh merged 27 commits into
apache:mainfrom
jroachgolf84:docs/issue-65782
Jun 6, 2026
Merged

AIP-103: Documentation for Task and Asset Store#67299
amoghrajesh merged 27 commits into
apache:mainfrom
jroachgolf84:docs/issue-65782

Conversation

@jroachgolf84

@jroachgolf84jroachgolf84 commented May 21, 2026•
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

This PR adds documentation for AIP-103.

closes: #65782

Was generative AI tooling used to co-author this PR?

Claude Sonnet 4.6 was used to review this PR.

@jroachgolf84

jroachgolf84 commented May 21, 2026

Copy link
Copy Markdown
CollaboratorAuthor

cc: @amoghrajesh

@jroachgolf84

jroachgolf84 commented May 26, 2026

Copy link
Copy Markdown
CollaboratorAuthor

I'm going to need to re-work the assets docs once I get feedback on this PR: #67248. Other than that, I'm curious to see what you think of the docs @amoghrajesh.

jroachgolf84
jroachgolf84 commented May 26, 2026
View reviewed changes
Comment threadairflow-core/docs/core-concepts/asset-state.rst Outdated
amoghrajesh
amoghrajesh reviewed May 27, 2026
View reviewed changes

@amoghrajeshamoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for taking this on @jroachgolf84. I spent some time reviewing the docs and playing around with it.

Comment threadairflow-core/docs/core-concepts/task-and-asset-state.rst Outdated
Comment threadairflow-core/docs/administration-and-deployment/state-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-state.rst Outdated
Comment threadairflow-core/docs/administration-and-deployment/state-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-store.rst
@amoghrajesh

amoghrajesh commented May 27, 2026

Copy link
Copy Markdown
Contributor

And a few things that I think are missing.

@jroachgolf84

jroachgolf84 commented May 27, 2026•
edited
Loading

Copy link
Copy Markdown
CollaboratorAuthor

And a few things that I think are missing.

Where do you think that the REST API page should live?

@jroachgolf84
jroachgolf84 marked this pull request as ready for review May 27, 2026 19:51
kaxil
kaxil reviewed May 29, 2026
View reviewed changes
Comment threadairflow-core/docs/administration-and-deployment/state-api.rst Outdated
@jroachgolf84jroachgolf84 changed the title docs/issue-65782: Initial framework for AIP-103 documentationAIP-103: Initial framework for Task State and Asset State documentationMay 29, 2026
@jroachgolf84

jroachgolf84 commented May 29, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Removed state-api docs, added docs for cleanup! Ready for re-review (I think it's GTG).

@jroachgolf84
jroachgolf84 requested review from amoghrajesh and kaxil and removed request for kaxilMay 29, 2026 19:40
amoghrajesh
amoghrajesh reviewed Jun 2, 2026
View reviewed changes

@amoghrajeshamoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took another pass at it, looking good.

Comment threadairflow-core/docs/administration-and-deployment/state-cleanup.rst Outdated
Comment threadairflow-core/docs/administration-and-deployment/state-cleanup.rst Outdated
Comment threadairflow-core/docs/administration-and-deployment/state-cleanup.rst
Comment threadairflow-core/docs/administration-and-deployment/state-cleanup.rst Outdated
Comment threadairflow-core/docs/administration-and-deployment/state-cleanup.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-state.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-state.rst Outdated
jroachgolf84and others added 4 commits June 2, 2026 08:17
@jroachgolf84@amoghrajesh
Update `airflow-core/docs/administration-and-deployment/state-cleanup…
24ec3ed
….rst` ...
... to use `default state backend`.
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@jroachgolf84@amoghrajesh
Update `airflow-core/docs/administration-and-deployment/state-cleanup…
213af79
….rst` ...
... to improve verbiage.
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@jroachgolf84@amoghrajesh
Update `airflow-core/docs/administration-and-deployment/state-cleanup…
b980539
….rst` ...
... to properly document negative retention days.
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@jroachgolf84@amoghrajesh
Update `airflow-core/docs/administration-and-deployment/state-cleanup…
e577831
….rst` ...
... to clarify verbiage.
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@jroachgolf84

jroachgolf84 commented Jun 3, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@amoghrajesh - do you mind taking one last look?

@potiukpotiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 3, 2026
@amoghrajesh

amoghrajesh commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Yep will be looking at it soon

amoghrajesh
amoghrajesh reviewed Jun 4, 2026
View reviewed changes

@amoghrajeshamoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pushing through the reviews @jroachgolf84! I am mostly ok with the state of this now, few more comments

Comment threadairflow-core/docs/core-concepts/task-and-asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-and-asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-and-asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-and-asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/task-and-asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
Comment threadairflow-core/docs/administration-and-deployment/task-and-asset-store.rst Outdated
@tatianatatiana mentioned this pull request Jun 4, 2026
Open
5 tasks
@jroachgolf84jroachgolf84 changed the title AIP-103: Initial framework for Task State and Asset State documentationAIP-103: Documentation for Task and Asset StoreJun 4, 2026
jroachgolf84and others added 2 commits June 4, 2026 11:13
@jroachgolf84@amoghrajesh
Apply suggestions from code review
cb6e564
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@jroachgolf84
docs/issue-65782: Adding feedback
979a4e3
@jroachgolf84

jroachgolf84 commented Jun 4, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Feedback implemented - should be good to go!

amoghrajesh
amoghrajesh reviewed Jun 5, 2026
View reviewed changes
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
amoghrajesh
amoghrajesh approved these changes Jun 5, 2026
View reviewed changes

@amoghrajeshamoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's it from me, LGTM. Please resolve the CI and apply suggestion from this review, and we are good, we can always improve it over time than aiming for perfection

Comment threadairflow-core/docs/core-concepts/task-store.rst Outdated
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
Comment threadairflow-core/docs/administration-and-deployment/task-and-asset-store.rst Outdated
Comment threadairflow-core/docs/administration-and-deployment/task-and-asset-store.rst Outdated
jroachgolf84and others added 2 commits June 5, 2026 14:48
@jroachgolf84@amoghrajesh
Apply suggestions from code review
1dee3a3
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@jroachgolf84
docs/issue-65782: Fixing CI failures
8f65dd3
@jroachgolf84

jroachgolf84 commented Jun 5, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@amoghrajesh - final feedback implement, and checks passing!

kaxil
kaxil reviewed Jun 6, 2026
View reviewed changes
Comment threadairflow-core/docs/administration-and-deployment/task-and-asset-store.rst Outdated
kaxil
kaxil reviewed Jun 6, 2026
View reviewed changes
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
kaxil
kaxil reviewed Jun 6, 2026
View reviewed changes
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
kaxil
kaxil reviewed Jun 6, 2026
View reviewed changes
Comment threadairflow-core/docs/core-concepts/task-store.rst Outdated
kaxil
kaxil reviewed Jun 6, 2026
View reviewed changes
Comment threadairflow-core/docs/core-concepts/asset-store.rst Outdated
kaxil
kaxil reviewed Jun 6, 2026
View reviewed changes
Comment threadairflow-core/docs/administration-and-deployment/task-and-asset-store.rst Outdated
kaxil
kaxil reviewed Jun 6, 2026
View reviewed changes
Comment threadairflow-core/docs/core-concepts/task-and-asset-store.rst Outdated
kaxil
kaxil reviewed Jun 6, 2026
View reviewed changes
Comment threadairflow-core/docs/administration-and-deployment/task-and-asset-store.rst
@jroachgolf84

jroachgolf84 commented Jun 6, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@kaxil - feedback has been implemented!

amoghrajesh
amoghrajesh approved these changes Jun 6, 2026
View reviewed changes

@amoghrajeshamoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good as it is now, we can revise further in later iterations if we want / need. Thanks for pushing through @jroachgolf84! Much appreciated.

@amoghrajesh
amoghrajesh merged commit 85843a3 into apache:mainJun 6, 2026
69 checks passed
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@amoghrajeshamoghrajeshamoghrajesh approved these changes

Assignees

No one assigned

Labels

kind:documentationready for maintainer reviewSet after triaging when all criteria pass.

Projects

None yet

Milestone

Airflow 3.3.0

Development

Successfully merging this pull request may close these issues.

Documentation for task and asset states

5 participants

@jroachgolf84@amoghrajesh@kaxil@potiuk@vatsrahul1001