Skip to content

acc: support DATABRICKS_TEST_SKIPLOCAL=withchanged - #5844

Merged
denik merged 16 commits into
mainfrom
denik/skiplocal-except-new
Jul 8, 2026
Merged

acc: support DATABRICKS_TEST_SKIPLOCAL=withchanged#5844
denik merged 16 commits into
mainfrom
denik/skiplocal-except-new

Conversation

@denik

@denikdenik commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Changes

Add withchanged mode: skips Local=true tests like true does, but re-enables any test added or changed on this branch.

Wire into integration-short-skiplocal so PR cloud runs exercise the tests they touch.

Why

We should run relevant tests on cloud on PRs. The tests that were added by PR are highly relevant. The tests that were modified by PR are probably relevant as well.

denik added 6 commits July 7, 2026 15:44
New mode re-enables Local tests that are added or changed on the current
branch, so a skiplocal cloud run still exercises the tests this PR touches.
Accepted values: "true" (skip all Local tests, previous behaviour),
"withchanged" (skip Local except changed/added on this branch), absent/empty
(disabled). Any other value hard-fails the test run instead of being silently
ignored.
Detects changed tests via git diff --name-status --merge-base (one call, same
approach as tools/lintdiff.py), caps at 50 tests (added-first), and batches
the script-existence check at the merge base into a single git ls-tree call.
Taskfile: test-update-aws uses =true, integration-short-skiplocal uses =withchanged.
Co-authored-by: Isaac
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 18:56 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 18:56 — with GitHub Actions Inactive
@denikdenik changed the title acceptance: DATABRICKS_TEST_SKIPLOCAL=withchanged re-enables changed tests on cloud runsacc: support DATABRICKS_TEST_SKIPLOCAL=withchangedJul 7, 2026
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 18:59 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 18:59 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 19:00 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 19:00 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 19:12 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 19:12 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 19:27 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 19:27 — with GitHub Actions Inactive
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: e6f0b57

Run: 28941239267

Env🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
💚​aws linux4423010646:21
💚​aws windows4423210627:25
💚​aws-ucws linux443149826:45
💚​aws-ucws windows443169807:46
💚​azure linux4423010635:58
🟨​azure windows211423210619:06
💚​azure-ucws linux443169798:05
🔄​azure-ucws windows22431897710:57
💚​gcp linux4422910655:13
💚​gcp windows4423110637:15
8 interesting tests: 4 SKIP, 2 KNOWN, 1 RECOVERED, 1 flaky
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
💚​TestAccept💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
🙈​TestAccept/bundle/invariant/no_drift🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/ssh/connection🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestFetchRepositoryInfoAPI_FromRepo💚​R💚​R💚​R💚​R💚​R🟨​K💚​R💚​R💚​R💚​R
🟨​TestFetchRepositoryInfoAPI_FromRepo/root💚​R💚​R💚​R💚​R💚​R🟨​K💚​R🔄​f💚​R💚​R
🔄​TestFetchRepositoryInfoAPI_FromRepo/subdir💚​R💚​R💚​R💚​R💚​R🔄​f💚​R🔄​f💚​R💚​R
Top 10 slowest tests (at least 2 minutes):
durationenvtestname
6:33azure windowsTestAccept
6:31aws-ucws windowsTestAccept
6:30azure-ucws windowsTestAccept
6:15aws windowsTestAccept
6:12gcp windowsTestAccept
2:56azure-ucws linuxTestAccept
2:56azure linuxTestAccept
2:56gcp linuxTestAccept
2:54aws linuxTestAccept
2:52aws-ucws linuxTestAccept

@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 20:01 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 20:01 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 20:19 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 7, 2026 20:19 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 8, 2026 06:44 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 8, 2026 06:44 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 8, 2026 09:37 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 8, 2026 09:37 — with GitHub Actions Inactive
@denik
denik enabled auto-merge July 8, 2026 09:51
// touched but not yet committed, which breaks the "touch a config, run the
// test" local dev workflow (same reason lintdiff.py uses --merge-base).
func selectChangedLocalTests(testDirs map[string]bool) map[string][]string {
out, _ := exec.Command("git", "diff", "--name-status", "--merge-base", "-M", "origin/main").Output()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does this work in local with committed, staged, and unstaged changes?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all except untracked will be included


for line := range strings.SplitSeq(diff, "\n") {
fields := strings.Split(line, "\t")
if len(fields) < 2 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you extract 2 as magic constant with name that makes sense? Also used above in testDirForFile

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'2' here and '2' above are two different twos.

This 2 is very local to code below, making a constant somewhere else won't improve readability.

@denik
deniktemporarily deployed to test-trigger-is July 8, 2026 12:04 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is July 8, 2026 12:04 — with GitHub Actions Inactive
@denik
denik added this pull request to the merge queueJul 8, 2026
Merged via the queue into main with commit 985f8f8Jul 8, 2026
34 checks passed
@denik
denik deleted the denik/skiplocal-except-new branch July 8, 2026 13:06
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 985f8f8

Run: 28944997777

Env❌​FAIL🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
💚​aws linux4452999440:21
🔄​aws windows334484100753:11
🔄​aws-ucws linux562983809192:03
❌​aws-ucws windows2852921827229:57
💚​azure linux4452399548:03
🔄​azure windows514478100858:18
🔄​azure-ucws linux752888840178:26
🟨​azure-ucws windows3432832858197:58
💚​gcp linux44514100046:00
🔄​gcp windows444467101361:16
30 interesting tests: 21 flaky, 3 KNOWN, 2 FAIL, 2 RECOVERED, 2 SKIP
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🔄​TestAccept💚​R🔄​f💚​R🔄​f💚​R🔄​f💚​R💚​R💚​R💚​R
❌​TestAccept/bundle/deploy/spark-jar-task✅​p✅​p✅​p❌​F✅​p✅​p✅​p✅​p✅​p✅​p
❌​TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p❌​F✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/generate/pipeline_and_deploy✅​p✅​p✅​p🔄​f✅​p🔄​f🔄​f🔄​f✅​p🔄​f
🔄​TestAccept/bundle/generate/pipeline_and_deploy/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p🔄​f
🔄​TestAccept/bundle/generate/pipeline_and_deploy/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p🔄​f✅​p🔄​f✅​p🔄​f✅​p✅​p
🔄​TestAccept/bundle/generate/python_job_and_deploy✅​p🔄​f✅​p🔄​f✅​p✅​p✅​p✅​p✅​p🔄​f
🔄​TestAccept/bundle/generate/python_job_and_deploy/DATABRICKS_BUNDLE_ENGINE=direct✅​p🔄​f✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/generate/python_job_and_deploy/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f
💚​TestAccept/bundle/invariant/no_drift🙈​S🙈​S💚​R💚​R🙈​S🙈​S💚​R💚​R🙈​S🙈​S
🔄​TestAccept/bundle/resources/dashboards/change-name✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/resources/dashboards/change-name/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/resources/dashboards/change-serialized-dashboard✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p
🔄​TestAccept/bundle/resources/dashboards/change-serialized-dashboard/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p
🔄​TestAccept/bundle/resources/permissions/jobs/delete_one🙈​s🙈​s✅​p✅​p🙈​s🙈​s🔄​f✅​p🙈​s🙈​s
🔄​TestAccept/bundle/resources/permissions/jobs/delete_one/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p🔄​f✅​p
🔄​TestAccept/bundle/resources/postgres_databases/basic🙈​s🙈​s🔄​f✅​p🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
🔄​TestAccept/bundle/resources/postgres_databases/basic/DATABRICKS_BUNDLE_ENGINE=terraform🔄​f✅​p
🔄​TestAccept/bundle/resources/postgres_synced_tables/recreate🙈​s🙈​s✅​p🔄​f🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
🔄​TestAccept/bundle/resources/postgres_synced_tables/recreate/DATABRICKS_BUNDLE_ENGINE=direct✅​p🔄​f
🔄​TestAccept/bundle/resources/quality_monitors/change_assets_dir🙈​s🙈​s✅​p✅​p🙈​s🙈​s🔄​f✅​p🙈​s🙈​s
🔄​TestAccept/bundle/resources/quality_monitors/change_assets_dir/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p🔄​f✅​p
💚​TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name🙈​S🙈​S💚​R💚​R🙈​S🙈​S💚​R💚​R🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/ssh/connection🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🔄​TestSyncFullFileSync✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestSyncIncrementalFileSync✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
🟨​TestFetchRepositoryInfoAPI_FromRepo💚​R💚​R💚​R💚​R💚​R💚​R🔄​f🟨​K💚​R💚​R
🟨​TestFetchRepositoryInfoAPI_FromRepo/root💚​R💚​R💚​R💚​R💚​R🔄​f💚​R🟨​K💚​R💚​R
🟨​TestFetchRepositoryInfoAPI_FromRepo/subdir💚​R💚​R💚​R💚​R💚​R🔄​f💚​R🟨​K💚​R💚​R
Top 50 slowest tests (at least 2 minutes):
durationenvtestname
13:15gcp linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
11:27azure-ucws windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
11:08gcp windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
10:48aws-ucws linuxTestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
10:28aws-ucws linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
10:25azure-ucws windowsTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
9:53aws-ucws windowsTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
9:52aws-ucws linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
9:52azure-ucws linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
9:36aws-ucws windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
9:34aws-ucws windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:53aws-ucws linuxTestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
8:48aws-ucws linuxTestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
8:14azure-ucws windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:11aws windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:08azure-ucws windowsTestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
7:57azure windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:32azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
7:31azure linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:26aws-ucws linuxTestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
7:17azure-ucws linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:12aws linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:04gcp windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:01gcp linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:47gcp windowsTestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
6:28azure-ucws linuxTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:19azure-ucws windowsTestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes/READPLAN=
6:07azure windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:01azure-ucws linuxTestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
5:57aws windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
5:57aws linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
5:55azure-ucws windowsTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
5:54azure-ucws linuxTestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct
5:54azure-ucws windowsTestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=no/READPLAN=
5:53aws-ucws linuxTestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct
5:47azure-ucws windowsTestAccept/bundle/apps/job_permissions/DATABRICKS_BUNDLE_ENGINE=terraform
5:46aws-ucws linuxTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
5:45aws-ucws windowsTestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
5:41aws-ucws linuxTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
5:36azure-ucws windowsTestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=no/READPLAN=1
5:35aws-ucws linuxTestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=no/READPLAN=
5:25aws-ucws windowsTestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes/READPLAN=
5:23azure linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
5:20aws-ucws windowsTestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=no/READPLAN=
5:19aws-ucws linuxTestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=yes/READPLAN=
5:13gcp linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
5:02aws-ucws linuxTestAccept/bundle/select/basic/DATABRICKS_BUNDLE_ENGINE=direct/READPLAN=
5:02gcp windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
5:01gcp linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
5:01azure linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@denik@eng-dev-ecosystem-bot@janniklasrose