Decouple S3 ObjectStorageProvider from common-sql into amazon provider

[cruseakshay]

Summary

Decouples S3ObjectStorageProvider from apache-airflow-providers-common-sql into apache-airflow-providers-amazon.

Architecture:

• common-sql defines the ObjectStorageProvider abstraction and all object storage providers (LocalStorage, S3, future GCS/Azure) are resolved at runtime via ProvidersManager.

Key changes:

• Add object-storage-providers registration mechanism to ProvidersManager (schema, discovery, property)
• Move S3ObjectStorageProvider to providers/amazon/aws/datafusion/object_storage.py
• Update common-sql to dynamically resolve storage providers via the ProvidersManager registry
• Add backward-compatible deprecation shim for old S3ObjectStorageProvider import path
• Decouple amazon from common-sql hard imports — amazon remains as an optional extra (for pip install common-sql[amazon]), but common-sql no longer statically imports from the amazon provider at runtime

Follow-up PRs:

• decouple LocalStorage
• Add Azure Blob Storage (az://).
• Add Google Cloud Storage (gs://)

Testing:

• E2E integration test: verified with AWS S3 bucket using AnalyticsOperator

related: #62867

---

Was generative AI tooling used to co-author this PR?

• Yes — Cursor following the guidelines

[Copilot]

Pull request overview

Decouples DataFusion S3 object storage support from apache-airflow-providers-common-sql by moving the S3 ObjectStorageProvider implementation into apache-airflow-providers-amazon, and adding a new ProvidersManager registry for resolving object storage providers at runtime.

Changes:

• Add object-storage-providers discovery/registry to ProvidersManager (schema + discovery + access property).
• Move S3ObjectStorageProvider into the Amazon provider and register it via provider metadata.
• Update common-sql to resolve non-local storage providers via the registry and add a deprecation shim for the old import path.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
providers/common/sql/src/airflow/providers/common/sql/datafusion/object_storage_provider.py	Replaces hardcoded S3 provider with ProvidersManager registry lookup + deprecation shim.
providers/common/sql/src/airflow/providers/common/sql/datafusion/engine.py	Delegates connection/credential resolution to the storage provider (ConnectionConfig now only carries conn_id).
airflow-core/src/airflow/providers_manager.py	Adds ObjectStorageProviderInfo + registry discovery and property for object storage providers.
airflow-core/src/airflow/provider.yaml.schema.json	Extends provider.yaml schema to allow object-storage-providers.
airflow-core/src/airflow/provider_info.schema.json	Extends provider-info schema to include object-storage-providers.
airflow-core/tests/unit/always/test_providers_manager.py	Adds unit tests for object storage provider discovery/override behavior.
providers/amazon/src/airflow/providers/amazon/aws/datafusion/object_storage.py	New S3 DataFusion-backed S3ObjectStorageProvider implementation in amazon provider.
providers/amazon/provider.yaml	Registers S3 object storage provider under object-storage-providers.
providers/amazon/src/airflow/providers/amazon/get_provider_info.py	Adds generated provider-info entry for object-storage-providers.
providers/amazon/pyproject.toml	Adds a datafusion optional extra for the amazon provider.
providers/amazon/tests/unit/amazon/aws/datafusion/test_object_storage.py	Adds unit tests for the moved S3 provider implementation.
providers/amazon/tests/unit/amazon/aws/datafusion/init.py	New test package init for amazon datafusion tests.
providers/amazon/src/airflow/providers/amazon/aws/datafusion/init.py	New package init for amazon datafusion module.
providers/common/sql/tests/unit/common/sql/datafusion/test_object_storage_provider.py	Updates tests to validate registry resolution and deprecation shim behavior.
providers/common/sql/tests/unit/common/sql/datafusion/test_engine.py	Updates tests to reflect provider-resolved credentials and simplified ConnectionConfig.
providers/common/sql/docs/operators.rst	Documents that S3 support requires apache-airflow-providers-amazon[datafusion].
providers/common/sql/pyproject.toml	Adjusts optional deps ordering; still includes an amazon optional dependency.

[potiuk]

@cruseakshay This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Merge conflicts: this branch has conflicts with main and cannot be merged as-is. Rebase: git fetch upstream main && git rebase upstream/main, resolve, force-push.
• ❌ Unresolved review comments (10 threads): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Once you believe a thread is resolved — whether by pushing a fix or by explaining why the suggestion doesn't apply — please mark it as resolved yourself by clicking the 'Resolve conversation' button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as 'still waiting on the author' and keeps the PR from moving forward.

What to do next:

• Address the items above.
• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Mark the PR as "Ready for review" when you are done.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[dabla]

Really like this implementation, great job @cruseakshay ! Looking good to me, WDYT @potiuk?

[dabla]

Looking good to me except the tests in provider manager, nice work!

[potiuk]

@cruseakshay — There are 1 unresolved review thread on this PR from @dabla. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? Once you believe the feedback is addressed, mark the thread as resolved so the reviewer isn't re-pinged needlessly. Thanks!

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.