Reduce Github Action Usage

[alamb]

We received a note on the mailing list today:

Mailing list item: https://lists.apache.org/thread/07rr90dpwdc0mqk2mxd4drkco50d334x

n 2024, the ASF introduced the policy for GitHub Actions usage
across the foundation1. The ASF Github shared pool of
Github-hosted runners has been at, or very close to the limit of
900 jobs most of the time in the past few weeks and this is the
case again today.

Your project has been identified as being among the top 5 consumers of
build time over the past 7 days and we request that you bring your
usage down by stream-lining long-running builds. Contact Infra for
a consultation if you are unable to streamline your builds further.

You can use the infra reporting tool2 to monitor your GHA usage as you
work on stream-lining, as well as locate any bottlenecks in the workflows.

Infra will allow you two weeks time (till the 8th of June, 2026) to
progress this, but should you still be above the limits by then,
without a viable path forward, we will be limiting your GHA usage.

Kind regards,
Bob Thomson, on behalf of ASF Infrastructure.

[alamb]

Here is a snapshot of the report from this morning (need a login to see)

[mbutrovich]

Yikes, thanks for opening up the issue here. 😬

[alamb]

It looks like 78% or more of this usage is from Comet;

• 35% Spark SQL Test: https://github.com/apache/datafusion-comet/blob/main/.github/workflows/spark_sql_test.yml
• 20% Iceberg Spark SQL Tests: https://github.com/apache/datafusion-comet/blob/main/.github/workflows/iceberg_spark_test.yml
• 12% PR Build (Linux): https://github.com/apache/datafusion-comet/blob/7cd0aaf93d3c8c1b937b13463c3fca4117ebb21f/.github/workflows/pr_build_linux.yml#L18
• 11% PR Build (Mac): https://github.com/apache/datafusion-comet/blob/7cd0aaf93d3c8c1b937b13463c3fca4117ebb21f/.github/workflows/pr_build_macos.yml#L18

Any ideas how to reduce this usage @andygrove or @mbutrovich ?

The biggest DataFusion core contributor looks like

• 12%: Rust: https://github.com/apache/datafusion/blob/main/.github/workflows/rust.yml
• 3%: extended tests

[mbutrovich]

Any ideas how to reduce this usage @andygrove or @mbutrovich ?

We've been discussing in apache/datafusion-comet#4389 and apache/datafusion-comet#4289 but this will definitely light a fire under the Comet project to get it sorted out.

[andygrove]

In the short term, we can consider reducing the matrix that runs on PRs and run the full matrix on main - not ideal, since regressions will slip through and create extra work for us - but will buy us time until we can implement apache/datafusion-comet#4289, which seems like the best long term solution to me.

[alamb]

In the short term, we can consider reducing the matrix that runs on PRs and run the full matrix on main - not ideal, since regressions will slip through and create extra work for us - but will buy us time until we can implement apache/datafusion-comet#4289, which seems like the best long term solution to me.

One way to mitigate the extra work would be to have an automatic action that makes a revert PR of you detect a failure 🤔

[alamb]

Also @blaginin has the (very sweet) merge queue setup in the main datafusion repo

I wonder if we could move some of the heavier tests from running on PR push / creation and instead only run when they are put into the merge queue (right before merge to main)

This should reduce the number of heavy test runs (basically 2x per commit to main, rather than per push to each PR)

🤔 I wonder if we could do this for extended tests on datafusion main too 🤔

[mbutrovich]

One way to mitigate the extra work would be to have an automatic action that makes a revert PR of you detect a failure 🤔

Spark SQL tests like to fail spuriously, so this might not be something we want to automate yet. Maybe the action can at least file an issue to bring attention to the failure.

[andygrove]

I created apache/datafusion-comet#4406 to track the Comet-specific discussions

[blaginin]

I wonder if we could move some of the heavier tests from running on PR push / creation and instead only run when they are put into the merge queue (right before merge to main)

🤔 I wonder if we could do this for extended tests on datafusion main too 🤔

Will do!

[kumarUjjawal]

Some of our ci jobs are failing regularly:

• https://github.com/apache/datafusion/actions/runs/26333321641/job/77522875542
• https://github.com/apache/datafusion/actions/runs/26334235111/job/77525452721?pr=22444
• https://github.com/apache/datafusion/actions/runs/26329506986/job/77513261122

I asked codex about it:

  the job  failed in actions/checkout, before Setup
  Rust toolchain or any Rust tests started. The errors are submodule fetch
  failures for parquet-testing and apache/arrow-testing: git exit 128, “did not
  contain <commit>”, “the remote end hung up unexpectedly”, and “could not read
  Username for https://github.com: terminal prompts disabled”. I sampled several
  other failed Rust workflow runs from May 23, 2026 across unrelated PRs and saw
  the same pattern in different jobs, including Linux and macOS. I also checked
  one rerun-success case where attempt 1 failed in checkout and attempt 2 passed
  with no code change.

  Why this keeps happening: the Rust workflow uses actions/checkout@v6 with
  submodule checkout enabled in many jobs, and GitHub’s checkout docs say
  fetch-depth defaults to 1, submodules are fetched when that input is enabled,
  and the auth token is persisted in local git config by default. That means
  these jobs are doing shallow submodule fetches during checkout, which is
  exactly the phase that is flaking here. GitHub Status also shows recent GitHub
  Actions degradation incidents on May 20, 2026 and May 15, 2026, so
  platform-side instability is a plausible contributor. (github.com
  (https://github.com/actions/checkout/blob/main/README.md))

  My conclusion: the real issue is flaky submodule checkout against GitHub-
  hosted repos, not failing DataFusion tests and not a bad PR. The “could not
  read Username” message looks like a fallback symptom after the fetch breaks,
  not the main cause. The shared risk factors are actions/checkout, submodules:
  true, shallow fetch, and GitHub-side transport instability.

[andygrove]

New snapshot - not directly comparable to previous one since we're into the weekend, but we've made good progress in Comet to reduce Spark SQL runs.

[kevinjqliu]

The ceiling per https://infra.apache.org/github-actions-policy.html is

• 250,000 minutes over 7 days (calendar week)
• 216,000 minutes over 5 days

The average number of minutes a project uses per calendar week MUST NOT exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours).

The average number of minutes a project uses in any consecutive five-day period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes, or 3,600 hours).

Datafusion seems to be sitting right below that 😄

• Past 7 day usage is 216,110 minutes
• Past 5 day usage is 120,929 minutes

[andygrove]

In Comet we are working on making some CI jobs run in contributors accounts (same approach as Spark takes). We have done about as much as we can for what runs on each PR.