Skip to content

Add explicit [tool.flit.sdist] sections to flit-based pyproject.tomls - #65861

Merged
potiuk merged 2 commits into
apache:mainfrom
potiuk:flit-sdist-explicit-includes
Apr 27, 2026
Merged

Add explicit [tool.flit.sdist] sections to flit-based pyproject.tomls#65861
potiuk merged 2 commits into
apache:mainfrom
potiuk:flit-sdist-explicit-includes

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

Prepares the flit-based packages for flit 4 making --no-use-vcs the
default (see pypa/flit#782 and pypa/flit#673).
Each flit-based pyproject.toml now declares an explicit
[tool.flit.sdist] include list, so sdist contents no longer depend on
git.

  • dev/breeze/.../pyproject_TEMPLATE.toml.jinja2 — adds a
    [tool.flit.sdist] block under the flit_core branch with Jinja
    logic to emit the right namespace __init__.py lines depending on
    provider-id depth (e.g. microsoft.azure adds
    airflow/providers/microsoft/__init__.py).
  • 97 provider pyproject.toml files — regenerated from the updated
    template.
  • dev/breeze, devel-common, performance — explicit include lists
    matching what --use-vcs shipped today.

Sdist parity with --use-vcs was verified for 9 representative
providers (single + nested IDs) and dev/breeze + devel-common. The
only delta is .gitignore (only *.iml, irrelevant) plus one stale
newsfragment in the azure provider.

The flit_core==3.12.0 pin is unchanged; bumping it (and switching
breeze's build command to --no-use-vcs, which will also fix the
worktree case from #65771) is left for a follow-up once flit 4 ships
— dependabot will surface that.

Test plan

  • pytest dev/breeze/tests/test_packages.py dev/breeze/tests/test_provider_documentation.py (133 passed)
  • Built sdists with flit --use-vcs and flit --no-use-vcs for 9
    providers + dev/breeze + devel-common and confirmed contents
    match (modulo .gitignore)
  • prek run --stage pre-commit on the modified files

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing i do worry about with this is the chance of locally untracked files from the RMs making it to the release artifacts.

Is flit only changing the default, ) in which case can't we explicitly opt in?) or working towards removing vcs support?

@potiuk

Copy link
Copy Markdown
MemberAuthor

One thing i do worry about with this is the chance of locally untracked files from the RMs making it to the release artifacts.

Is flit only changing the default, ) in which case can't we explicitly opt in?) or working towards removing vcs support?

It's only for sdist - not wheel but yes that is a valid concern.

But in our release process we actualy do cleanup with git clean -fd (I will double check it). I will double-check, but we were supposed to do it.

And yes - it's just changing the default so if someone wants to use flit wiht --use-vcs and it will continue to work.

@potiuk

Copy link
Copy Markdown
MemberAuthor

Checked - and added explicit (but also rather gentle - skipping .venv, .idea and .vscode) cleanup when preparing the packages. git clean -fdx with those exclusions should work as expected.

Now --use-vcs should be equivalent with building it with breeze.

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the discussion, looks "good enough" for now.

…files
Prepares for flit 4 making --no-use-vcs the default
(pypa/flit#782, pypa/flit#673). With explicit
sdist include lists, sdist contents no longer depend on git, which also
fixes the worktree fail-fast case from apache#65771.
The provider template emits namespace __init__.py lines based on
provider-id depth (e.g. microsoft.azure also includes
airflow/providers/microsoft/__init__.py). The 97 flit-based provider
pyproject.toml files were regenerated from the updated template.
dev/breeze, devel-common, and performance got their own include lists.
The flit_core==3.12.0 pin is unchanged - bumping to allow flit 4 is
left to dependabot once flit 4 ships.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providersprovider:airbyteprovider:alibabaprovider:amazonAWS/Amazon - related issuesprovider:apache-beamprovider:apache-cassandraprovider:apache-drillprovider:apache-druidprovider:apache-flinkprovider:apache-hdfsprovider:apache-hiveprovider:apache-icebergprovider:apache-impalaprovider:apache-kafkaprovider:apache-kylinprovider:apache-livyprovider:apache-pigprovider:apache-pinotprovider:apache-sparkprovider:apache-tinkerpopprovider:appriseprovider:arangodbprovider:asanaprovider:atlassian-jiraprovider:celeryprovider:cloudantprovider:cncf-kubernetesKubernetes (k8s) provider related issuesprovider:cohereprovider:common-compatprovider:common-ioprovider:common-messagingprovider:common-sqlprovider:databricksprovider:datadogprovider:dbt-cloudprovider:dingdingprovider:discordprovider:dockerprovider:elasticsearchprovider:exasolprovider:facebookprovider:ftpprovider:gitprovider:githubprovider:googleGoogle (including GCP) related issuesprovider:grpcprovider:hashicorpHashicorp provider related issuesprovider:httpprovider:imapprovider:influxdbprovider:informaticaprovider:jdbcprovider:jenkinsprovider:keycloakprovider:microsoft-azureAzure-related issuesprovider:microsoft-mssqlprovider:microsoft-psrpprovider:microsoft-winrmprovider:mongoprovider:mysqlprovider:neo4jprovider:odbcprovider:openaiprovider:openfaasprovider:openlineageAIP-53provider:opensearchprovider:opsgenieprovider:oracleprovider:pagerdutyprovider:papermillprovider:pgvectorprovider:pineconeprovider:postgresprovider:prestoprovider:qdrantprovider:redisprovider:salesforceprovider:sambaprovider:segmentprovider:sendgridprovider:sftpprovider:singularityprovider:slackprovider:smtpprovider:snowflakeIssues related to Snowflake providerprovider:sqliteprovider:sshprovider:standardprovider:tableauprovider:telegramprovider:teradataprovider:trinoprovider:verticaprovider:vespaprovider:weaviateprovider:yandexprovider:ydbprovider:zendeskskip common compat checkSkips common compat provider modification check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@potiuk@ashb@jscheffl