Skip to content

Document when to use common.ai vs vendor-specific AI providers - #69551

Merged
Lee-W merged 1 commit into
apache:mainfrom
astronomer:ai-provider-usage-docs
Jul 8, 2026
Merged

Document when to use common.ai vs vendor-specific AI providers#69551
Lee-W merged 1 commit into
apache:mainfrom
astronomer:ai-provider-usage-docs

Conversation

@Lee-W

@Lee-WLee-W commented Jul 7, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Why

  • The AI provider landing pages didn't explain how to choose between the vendor-neutral common.ai operators and each vendor's native API wrappers — readers had to infer it.

What

  • common.ai, openai, and anthropic index pages now each carry a "When to use this provider" section with the decision rule and a runnable example snippet.
  • Frames the choice around where the AI step runs — orchestrated in the Airflow worker (common.ai, model stays swappable via connection) vs submitted to a vendor-managed service (native Batch / Responses / Managed Agents endpoints).
Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Lee-W
Lee-Wforce-pushed the ai-provider-usage-docs branch 2 times, most recently from df9954b to feac1fdCompareJuly 8, 2026 04:40
@Lee-W
Lee-W marked this pull request as ready for review July 8, 2026 06:05
@Lee-W
Lee-Wforce-pushed the ai-provider-usage-docs branch from feac1fd to 36aa507CompareJuly 8, 2026 06:05
gopidesupavan
gopidesupavan approved these changes Jul 8, 2026
View reviewed changes

@gopidesupavangopidesupavan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding..

@Lee-W
Document when to use common.ai vs vendor-specific AI providers
d07db20
The common.ai, openai, and anthropic docs landing pages now explain how to
choose between the vendor-neutral operators and the native API wrappers —
including where the agent loop runs (Airflow worker vs vendor-managed service).
@Lee-W
Lee-Wforce-pushed the ai-provider-usage-docs branch from 36aa507 to d07db20CompareJuly 8, 2026 08:50
@Lee-W
Lee-W merged commit b15c485 into apache:mainJul 8, 2026
70 checks passed
@Lee-W
Lee-W deleted the ai-provider-usage-docs branch July 8, 2026 09:54
kaxil
kaxil reviewed Jul 8, 2026
View reviewed changes
Comment threadproviders/common/ai/docs/index.rst
Comment threadproviders/common/ai/docs/index.rst
When to use this provider
--------------------------

``common.ai`` is the vendor-neutral way to put LLM and agent steps in a Dag. It is built on

@kaxilkaxilJul 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A reader who already has LangChain or LlamaIndex code may read "built on pydantic-ai" as "not for me" and move on. Since the provider ships a LangChain toolset (the langchain extra), one sentence noting that existing LangChain tools can be called from a common.ai agent would keep that reader engaged, without overstating support.

@Lee-WLee-WJul 9, 2026

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

I added it to the middle of this paragraph in https://github.com/apache/airflow/pull/69649/changes. let me know if it's better to move it or extend it

Comment threadproviders/common/ai/docs/index.rst
Comment threadproviders/openai/docs/index.rst
Comment threadproviders/openai/docs/index.rst
Comment threadproviders/anthropic/docs/index.rst
Comment threadproviders/anthropic/docs/index.rst
@Lee-WLee-W mentioned this pull request Jul 9, 2026
Merged
1 task
Lee-W added a commit to astronomer/airflow that referenced this pull request Jul 9, 2026
@Lee-W
Add feature-comparison table and toolset links to common.ai provider …
810d921
…docs
Reviewer feedback on apache#69551 (merged) asked for a scannable
package-choice table linking to the vendor provider pages, a note that
existing LangChain tools work via the langchain extra, and a link to the
toolsets doc from the agent bullet.
Lee-W added a commit to astronomer/airflow that referenced this pull request Jul 9, 2026
@Lee-W
Fix openai provider docs: orient readers before the comparison, clari…
26c7998
…fy batch tradeoff
Reviewer feedback on apache#69551 (merged) noted the comparison
section ran before readers knew what the provider was, that OpenAIHook
isn't batch-specific, and that the batch bullet didn't explain why one
would pick it over a synchronous call.
Lee-W added a commit to astronomer/airflow that referenced this pull request Jul 9, 2026
@Lee-W
Fix anthropic provider docs: orient readers before the comparison, ad…
7de47a9
…d batch cost/turnaround rationale
Reviewer feedback on apache#69551 (merged) noted the comparison
section ran before readers knew what the provider was, and that the batch
bullet didn't explain why one would pick it over a synchronous call. The
new clause reuses the cost/turnaround figures already documented in
operators/anthropic.rst for consistency.
@shahar1shahar1 mentioned this pull request Jul 24, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxilkaxilkaxil left review comments
@gopidesupavangopidesupavangopidesupavan approved these changes

Assignees

No one assigned

Labels

area:providerskind:documentationprovider:anthropicprovider:common-aiprovider:openai

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Lee-W@kaxil@gopidesupavan