Skip to content

direct: Support configuration for unordered slices; use in permissions to ignore order - #4014

Merged
denik merged 16 commits into
mainfrom
denik/path-key-syntax
Nov 27, 2025
Merged

direct: Support configuration for unordered slices; use in permissions to ignore order#4014
denik merged 16 commits into
mainfrom
denik/path-key-syntax

Conversation

@denik

@denikdenik commented Nov 27, 2025

Copy link
Copy Markdown
Contributor

Changes

  • Extend structpath to support [key=‘value’] syntax.
  • Extend structdiff with extra parameter that maps path to keyFunc which is a function that returns key/value pair for a given slice element. They pair is used to identify the slice element.

  • Use this functionality on permissions resource in direct to solve drift due to backend reordering permissions.

Why

  • Prevent remote drift due to backend reordering slice elements.
  • Make diffs more readable / precise.

This is an alternative to always sorting the input, which destroys the references ($resources.jobs.foo.tasks[1] is not valid today because of sorting). In a follow up PR I’ll revert job tasks sorting #3212 and implement KeyedSlices instead.

@denikdenik changed the title Treat some slices as maps; ignore permissions order changeTreat some slices as maps; ignore permissions orderNov 27, 2025
@denik
denikforce-pushed the denik/path-key-syntax branch from 61a55e6 to 9ee67c5CompareNovember 27, 2025 10:02
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 27, 2025

Copy link
Copy Markdown
Collaborator

Commit: 575f2ba

Run: 19741050772

Env🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux7237162816:20
🟨​aws windows72237162618:05
💚​aws-ucws linux7251451321:39
💚​aws-ucws windows7251651121:52
💚​azure linux1437162714:22
💚​azure windows1437362515:15
💚​azure-ucws linux1451051223:14
💚​azure-ucws windows1451251019:02
💚​gcp linux1436463116:52
🔄​gcp windows3436462915:12
13 failing tests:
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K💚​R💚​R💚​R💚​R💚​R💚​R💚​R🔄​f
🔄​TestAccept/bundle/resources/experiments/basic✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f
🔄​TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🙈​TestAccept/bundle/run/app-with-job🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🔄​TestAccept/bundle/templates/default-python/integration_classic✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.11✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p
Top 20 slowest tests (at least 2 minutes):
durationenvtestname
7:40azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
7:34azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
6:07gcp windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
6:05aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:43gcp linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:42aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:38aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:32azure windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:32gcp linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:29aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:19aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:04gcp windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:51aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:38azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:04azure windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:03azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:28azure-ucws linuxTestAccept/bundle/resources/synced_database_tables/basic
2:26aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:19aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:18azure-ucws windowsTestAccept/bundle/resources/synced_database_tables/basic

@denikdenik changed the title Treat some slices as maps; ignore permissions orderdirect: Support configuration for unordered slices; use in permissions to ignor orderNov 27, 2025
@denikdenik changed the title direct: Support configuration for unordered slices; use in permissions to ignor orderdirect: Support configuration for unordered slices; use in permissions to ignore orderNov 27, 2025
@denik
denik marked this pull request as ready for review November 27, 2025 10:40
Comment threadbundle/direct/dresources/adapter.go Outdated

// [Optional] KeyedSlices returns a map from path patterns to KeyFunc for comparing slices by key instead of by index.
// Example: func (*ResourcePermissions) KeyedSlices(state *PermissionsState) map[string]any
KeyedSlices(state any) map[string]any

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the implementation does not use the state parameter; shall it be removed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, cleaned up 9cb6261

@denik
denik enabled auto-merge November 27, 2025 15:23
@denik
denik added this pull request to the merge queueNov 27, 2025
Merged via the queue into main with commit 0bc7fe3Nov 27, 2025
13 checks passed
@denik
denik deleted the denik/path-key-syntax branch November 27, 2025 16:05
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Commit: 0bc7fe3

Run: 19742129063

Env🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux73140461547:51
🟨​aws windows723140461347:07
💚​aws-ucws linux10156549465:00
💚​aws-ucws windows10156749265:19
🔄​azure linux33340261449:12
🔄​azure windows33340461250:12
💚​azure-ucws linux4356149373:53
🔄​azure-ucws windows33356149179:43
💚​gcp linux4338862149:58
🔄​gcp windows33338861952:04
16 failing tests:
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K💚​R💚​R🔄​f🔄​f💚​R🔄​f💚​R🔄​f
🔄​TestAccept/bundle/resources/clusters/run/spark_python_task✅​p✅​p✅​p✅​p🔄​f🔄​f✅​p🔄​f✅​p🔄​f
🔄​TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p🔄​f🔄​f✅​p🔄​f✅​p✅​p
🔄​TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f
🔄​TestAccept/bundle/resources/dashboards/detect-change✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/resources/dashboards/detect-change/DATABRICKS_BUNDLE_ENGINE=direct✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
💚​TestAccept/bundle/run/app-with-job💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
💚​TestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=direct💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
💚​TestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=terraform💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
Top 50 slowest tests (at least 2 minutes):
durationenvtestname
15:11azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
14:38azure-ucws linuxTestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
13:50azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
13:19azure-ucws windowsTestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform
13:13gcp linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
13:13gcp linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
12:04aws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
10:56azure-ucws windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
10:41azure linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
10:19gcp windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
10:15aws-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
9:52gcp windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
9:36azure-ucws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
9:31azure-ucws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
9:27azure windowsTestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
8:45azure-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
8:40aws windowsTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform
8:27azure-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
8:26aws-ucws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:24aws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:16gcp windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:08aws windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
8:08gcp linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
8:05azure-ucws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:58azure windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:57gcp windowsTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform
7:54gcp linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:48azure windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:47azure-ucws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
7:45gcp windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:44gcp windowsTestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=terraform
7:40aws-ucws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:39aws linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
7:36azure linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:36aws-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
7:35gcp linuxTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct
7:34aws-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
7:32gcp linuxTestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=direct
7:30gcp windowsTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct
7:28gcp linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:28azure-ucws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:23aws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:15gcp windowsTestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=direct
7:11azure windowsTestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform
7:10azure-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
7:07aws-ucws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:06aws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:05azure linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
7:05aws-ucws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:05aws-ucws linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform

github-merge-queueBot pushed a commit that referenced this pull request Dec 1, 2025
## Changes
Configure "task_key" as a way to identify tasks in jobs.
Builds on top of #4014
## Why
Makes diffs more precise. Ignores reordering of slice without need of
sorting (#3212). Note, we still
sort, this is to be reverted in separate PR.
## Tests
Existing tests. New test that reorders tasks locally.
github-merge-queueBot pushed a commit that referenced this pull request Dec 1, 2025
## Why
Otherwise it is also applied to key-value pairs which is not intended.
This is related and fixes bug introduced in
#4014
## Tests
Regression acceptance test that shows how old behaviour resulted in
ignoring extra permissions in remote state on direct.
denik added a commit that referenced this pull request May 20, 2026
…s to ignore order (#4014)
## Changes
- Extend structpath to support [key=‘value’] syntax.
- Extend structdiff with extra parameter that maps path to keyFunc which
is a function that returns key/value pair for a given slice element.
They pair is used to identify the slice element.
- Use this functionality on permissions resource in direct to solve
drift due to backend reordering permissions.
## Why
- Prevent remote drift due to backend reordering slice elements.
- Make diffs more readable / precise.
This is an alternative to always sorting the input, which destroys the
references ($resources.jobs.foo.tasks[1] is not valid today because of
sorting). In a follow up PR I’ll revert job tasks sorting
#3212 and implement KeyedSlices
instead.
denik added a commit that referenced this pull request May 20, 2026
## Changes
Configure "task_key" as a way to identify tasks in jobs.
Builds on top of #4014
## Why
Makes diffs more precise. Ignores reordering of slice without need of
sorting (#3212). Note, we still
sort, this is to be reverted in separate PR.
## Tests
Existing tests. New test that reorders tasks locally.
denik added a commit that referenced this pull request May 20, 2026
## Why
Otherwise it is also applied to key-value pairs which is not intended.
This is related and fixes bug introduced in
#4014
## Tests
Regression acceptance test that shows how old behaviour resulted in
ignoring extra permissions in remote state on direct.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@denik@eng-dev-ecosystem-bot@andrewnester