Skip to content

direct: Detailed bundle plan - #3636

Merged
denik merged 30 commits into
mainfrom
denik/structvar
Sep 30, 2025
Merged

direct: Detailed bundle plan#3636
denik merged 30 commits into
mainfrom
denik/structvar

Conversation

@denik

@denikdenik commented Sep 22, 2025

Copy link
Copy Markdown
Contributor

Changes

The JSON plan (output of “bundle debug plan”) is now input to “bundle deploy” internally. This means it contains full information about resources about to be deployed.

The JSON plan stores configs as well as all unresolved fields as map (path -> reference). There is new module, structvar, that supports resolving these references. It does it quickly and without copying the full config for every resolution, like the previous implementation did.

The JSON plan also stores per-field action. This is purely informational, it gives an idea why a certain action for the resource was chosen.

During plan and apply, variable resolution is now done in the node that contains the reference. Previously, it was updated by the node that was referenced. This results in better error messages when resolution does not work and it does not fail resources that themselves have no issues (but have incorrect references pointing to them).

References can now refer to missing field and it works like in terraform (missing_string_field test).

There is slight simplification in how $resources are resolved. All local resolutions (from config) are now done in plan phase. All remote resolutions (from remote state) are now done in apply phase. Previously we could resolve some remote references in plan stage if action was “skip”. This might result in slower deploys, so this is a potential perf regression, to be revisited in the future.

Depends on #3650 & #3646

Other changes:

  • dagrun.Graph now exports its fields. dagrun.OutgoingLabels() is removed.
  • new function structaccess.Set() to set any field in a struct by path (used by structvar).
  • dresources.ClassifyByTriggers([]Change) is replaced by dresources.ClassifyByTriggers(Change), aggregation logic now lives in bundle_plan.go

Why

Having the plan as input to deploy provides a view into deployment that we did not have before, allowing to see exactly what is going to be deployed and why.

This is useful for:

  • debugging (why is my pipeline being recreated?)
  • see diff between deployed infra and config
  • deploying exactly what was in the diff and nothing else: "bundle deploy -readplan plan.json" can be implemented easily after this change.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Sep 22, 2025

Copy link
Copy Markdown
Collaborator

Run: 18126676416

Env✅​pass🔄​flaky🙈​skip
✅​aws linux315533
✅​aws windows316532
✅​aws-ucws linux428430
✅​aws-ucws windows429429
✅​azure linux315532
✅​azure windows316531
✅​azure-ucws linux428429
✅​azure-ucws windows429428
🔄​gcp linux3113534
✅​gcp windows315533
Test Namegcp linux
TestAccept🔄​flaky
TestAccept/bundle/deploy/jobs/check-metadata🔄​flaky
TestAccept/bundle/deploy/jobs/check-metadata/DATABRICKS_BUNDLE_ENGINE=direct-exp🔄​flaky

@denik
denikforce-pushed the denik/structvar branch 2 times, most recently from b15d5b4 to 066c861CompareSeptember 24, 2025 14:57
@denik
denik changed the base branch from denik/structaccess-structpath2 to denik/update-get-resource-configSeptember 24, 2025 14:57
@denik
denikforce-pushed the denik/update-get-resource-config branch from f79fdac to 57a975bCompareSeptember 25, 2025 12:32
denik added a commit that referenced this pull request Sep 25, 2025
## Changes
Include 'skip' nodes in JSON plan.
## Why
We need to process 'skip' resources as well (e.g. to resolve
references). Today, we have these nodes in the graph but not in the
plan. Once we move to serialized plan
#3636 we will build graph from the
plan so they must match. This prepares for that (and minimizes that PR).
This also gives more information to users of JSON plan, e.g. they can
count unchanged count.
## Tests
Existing tests.
@denik
denikforce-pushed the denik/update-get-resource-config branch from 57a975b to 55ce74fCompareSeptember 25, 2025 15:07
@denikdenik changed the title WIP serialized bundle plandirect: Detailed bundle planSep 25, 2025
@denik
denik marked this pull request as ready for review September 25, 2025 15:13
@denik
denik added this pull request to the merge queueSep 30, 2025
Merged via the queue into main with commit 0f6ea36Sep 30, 2025
13 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@denik@eng-dev-ecosystem-bot@pietern@andrewnester