Skip to content

Add bundle debug terraform command - #1294

Merged
andrewnester merged 29 commits into
mainfrom
terraform-dependencies
Apr 2, 2024
Merged

Add bundle debug terraform command#1294
andrewnester merged 29 commits into
mainfrom
terraform-dependencies

Conversation

@ilia-db

@ilia-dbilia-db commented Mar 18, 2024

Copy link
Copy Markdown
Contributor
  • Add bundle debug terraform command. It prints versions of the Terraform and the Databricks Terraform provider. In the text mode it also explains how to setup the CLI in environments with restricted internet access.
  • Use DATABRICKS_TF_EXEC_PATH env var to point Databricks CLI to the Terraform binary. The CLI only uses it if DATABRICKS_TF_VERSION matches the currently used terraform version.
  • Use DATABRICKS_TF_CLI_CONFIG_FILE env var to point Terraform CLI config that points to the filesystem mirror for the Databricks provider. The CLI only uses it if DATABRICKS_TF_PROVIDER_VERSION matches the currently used provider version.

Relevant PR on the VSCode extension side: databricks/databricks-vscode#1147

Example output of the databricks bundle debug terraform:

Terraform version: 1.5.5
Terraform URL: https://releases.hashicorp.com/terraform/1.5.5
Databricks Terraform Provider version: 1.38.0
Databricks Terraform Provider URL: https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.38.0
Databricks CLI downloads its Terraform dependencies automatically.
If you run the CLI in an air-gapped environment, you can download the dependencies manually and set these environment variables:
DATABRICKS_TF_VERSION=1.5.5
DATABRICKS_TF_EXEC_PATH=/path/to/terraform/binary
DATABRICKS_TF_PROVIDER_VERSION=1.38.0
DATABRICKS_TF_CLI_CONFIG_FILE=/path/to/terraform/cli/config.tfrc
Here is an example *.tfrc configuration file:
disable_checkpoint = true
provider_installation {
filesystem_mirror {
path = "/path/to/a/folder/with/databricks/terraform/provider"
}
}
The filesystem mirror path should point to the folder with the Databricks Terraform Provider. The folder should have this structure: /registry.terraform.io/databricks/databricks/terraform-provider-databricks_1.38.0_ARCH.zip
For more information about filesystem mirrors, see the Terraform documentation: https://developer.hashicorp.com/terraform/cli/config/config-file#filesystem_mirror

- Add `bundle dependencies` command. It prints versions of terraform and
databricks terraform provider.
- Use DATABRICKS_TF_EXEC_PATH env var to point databricks CLI to terraform bin
- Use DATABRICKS_TF_PLUGIN_CACHE_DIR env var to point terraform CLI to the
plugin cache dir
@ilia-db
ilia-db marked this pull request as ready for review March 19, 2024 10:51
Comment threadbundle/deploy/terraform/init.go Outdated
Comment threadcmd/bundle/dependencies.go Outdated
Comment threadbundle/deploy/terraform/init.go Outdated
Comment threadbundle/deploy/terraform/init.go
Comment threadbundle/deploy/terraform/init.go Outdated
Comment threadbundle/deploy/terraform/init_test.go Outdated
Comment threadbundle/internal/tf/schema/root.go
Comment threadcmd/bundle/dependencies.go Outdated
Comment threadcmd/bundle/dependencies.go Outdated
@ilia-db
ilia-db requested a review from pieternMarch 22, 2024 13:32
@ilia-dbilia-db changed the title Add bundle dependencies commandAdd bundle internal-dependencies commandMar 22, 2024
Comment threadcmd/bundle/dependencies.go Outdated
Comment threadcmd/bundle/dependencies.go Outdated
Comment threadcmd/bundle/dependencies.go Outdated
Comment threadcmd/bundle/dependencies.go Outdated
Comment threadbundle/deploy/terraform/pkg.go Outdated
@pietern

Copy link
Copy Markdown
Contributor

Please update the PR summary to be up to date with the most recent revision.

Everything else LGTM.

@ilia-db
ilia-db requested a review from pieternApril 2, 2024 12:30
@pietern
pietern requested review from andrewnester and juliacrawf-db and removed request for juliacrawf-dbApril 2, 2024 12:43
@ilia-db
ilia-db added this pull request to the merge queueApr 2, 2024
@andrewnester
andrewnester removed this pull request from the merge queue due to a manual request Apr 2, 2024
@andrewnester
andrewnester added this pull request to the merge queueApr 2, 2024
Merged via the queue into main with commit 079c416Apr 2, 2024
@andrewnester
andrewnester deleted the terraform-dependencies branch April 2, 2024 13:03
andrewnester added a commit that referenced this pull request Apr 3, 2024
CLI:
* Added `auth describe` command ([#1244](#1244)).
* Fixed message for successful auth describe run ([#1336](#1336)).
Bundles:
* Make bundle validation print text output by default ([#1335](#1335)).
* Use UserName field to identify if service principal is used ([#1310](#1310)).
* Allow unknown properties in the config file for template initialization ([#1315](#1315)).
* Remove support for DATABRICKS_BUNDLE_INCLUDES ([#1317](#1317)).
* Make `bundle.deployment` optional in the bundle schema ([#1321](#1321)).
* Add allow list for resources when bundle `run_as` is set ([#1233](#1233)).
* Fix the generated DABs JSON schema ([#1322](#1322)).
* Make bundle loaders return diagnostics ([#1319](#1319)).
* Add `bundle debug terraform` command ([#1294](#1294)).
* Allow specifying CLI version constraints required to run the bundle ([#1320](#1320)).
Internal:
* Retain location information of variable reference ([#1333](#1333)).
* Define `dyn.Mapping` to represent maps ([#1301](#1301)).
* Return `diag.Diagnostics` from mutators ([#1305](#1305)).
* Fix flaky test in `libs/process` ([#1314](#1314)).
* Move path field to bundle type ([#1316](#1316)).
* Load bundle configuration from mutator ([#1318](#1318)).
* Return diagnostics from `config.Load` ([#1324](#1324)).
* Return warning for nil primitive types during normalization ([#1329](#1329)).
* Include `dyn.Path` in normalization warnings and errors ([#1332](#1332)).
* Make normalization return warnings instead of errors ([#1334](#1334)).
API Changes:
* Added `databricks lakeview migrate` command.
* Added `databricks lakeview unpublish` command.
* Changed `databricks ip-access-lists get` command . New request type is .
OpenAPI commit e316cc3d78d087522a74650e26586088da9ac8cb (2024-04-03)
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.36.0 to 0.37.0 ([#1326](#1326)).
@andrewnesterandrewnester mentioned this pull request Apr 3, 2024
github-merge-queueBot pushed a commit that referenced this pull request Apr 3, 2024
Breaking Change:
* Add allow list for resources when bundle `run_as` is set
([#1233](#1233)).
* Make bundle validation print text output by default
([#1335](#1335)).
CLI:
* Added `auth describe` command
([#1244](#1244)).
* Fixed message for successful auth describe run
([#1336](#1336)).
Bundles:
* Use UserName field to identify if service principal is used
([#1310](#1310)).
* Allow unknown properties in the config file for template
initialization ([#1315](#1315)).
* Remove support for DATABRICKS_BUNDLE_INCLUDES
([#1317](#1317)).
* Make `bundle.deployment` optional in the bundle schema
([#1321](#1321)).
* Fix the generated DABs JSON schema
([#1322](#1322)).
* Make bundle loaders return diagnostics
([#1319](#1319)).
* Add `bundle debug terraform` command
([#1294](#1294)).
* Allow specifying CLI version constraints required to run the bundle
([#1320](#1320)).
Internal:
* Retain location information of variable reference
([#1333](#1333)).
* Define `dyn.Mapping` to represent maps
([#1301](#1301)).
* Return `diag.Diagnostics` from mutators
([#1305](#1305)).
* Fix flaky test in `libs/process`
([#1314](#1314)).
* Move path field to bundle type
([#1316](#1316)).
* Load bundle configuration from mutator
([#1318](#1318)).
* Return diagnostics from `config.Load`
([#1324](#1324)).
* Return warning for nil primitive types during normalization
([#1329](#1329)).
* Include `dyn.Path` in normalization warnings and errors
([#1332](#1332)).
* Make normalization return warnings instead of errors
([#1334](#1334)).
API Changes:
* Added `databricks lakeview migrate` command.
* Added `databricks lakeview unpublish` command.
* Changed `databricks ip-access-lists get` command . New request type is
.
OpenAPI commit e316cc3d78d087522a74650e26586088da9ac8cb (2024-04-03)
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.36.0 to 0.37.0
([#1326](#1326)).
ilia-db added a commit to databricks/databricks-vscode that referenced this pull request Apr 8, 2024
Bundle terraform binaries with the extension and expose them with env
vars to the CLI and the terminal
Depends on databricks/cli#1294
github-merge-queueBot pushed a commit that referenced this pull request Apr 12, 2024
## Changes
This PR makes changes to support creating a docker image for the CLI
with the `terraform` dependencies built in. This is useful for customers
that operate in a network-restricted environment. Normally DABs makes
API calls to registry.terraform.io to setup the terraform dependencies,
with this setup the CLI/DABs will rely on the provider binaries bundled
in the docker image.
### Specifically this PR makes the following changes:
----------------
Modifies the CLI release workflow to publish the docker images in the
Github Container Registry. URL:
https://github.com/databricks/cli/pkgs/container/cli.
We use docker support in `goreleaser` to build and publish the images.
Using goreleaser ensures the CLI packaged in the docker image is the
same release artifact as the normal releases. For more information see:
1. https://goreleaser.com/cookbooks/multi-platform-docker-images
2. https://goreleaser.com/customization/docker/
Other choices made include:
1. Using `alpine` as the base image. The reason is `alpine` is a small
and lightweight linux distribution (~5MB) and an industry standard.
2. Not using [docker
manifest](https://docs.docker.com/reference/cli/docker/manifest) to
create a multi-arch build. This is because the functionality is still
experimental.
------------------
Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION`
environment variables optional for using the terraform file mirror.
While it's not strictly necessary to make the docker image work, it's
the "right" behaviour and reduces complexity. The rationale is:
- These environment variables here are needed so the Databricks CLI does
not accidentally use the file mirror bundled with VSCode if it's
incompatible. This does not require the env vars to be mandatory.
context: #1294
- This makes the `Dockerfile` and `setup.sh` simpler. We don't need an
[entrypoint.sh script to set the version environment
variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d).
This also makes using an interactive terminal with `docker run -it ...`
work out of the box.
## Tests
Tested manually. --------------------
To test the release pipeline I triggered a couple of dummy releases and
verified that the images are built successfully and uploaded to Github.
1. https://github.com/databricks/cli/pkgs/container/cli
3. workflow for release:
https://github.com/databricks/cli/actions/runs/8646106333
--------------------
I tested the docker container itself by setting up
[Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying
that no HTTP requests are made to `registry.terraform.io`
Before:
FYI, The Charles web proxy is hosted at localhost:8888.
```
shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy
Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```
<img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM"
src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b">
After:
This time bundle deploy is run from inside the docker container. We use
`host.docker.internal` to map to localhost on the host machine, and -v
to mount the host file system as a volume.
```
shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64 / # cd /bundle/
/bundle # rm -r .databricks/
/bundle # databricks bundle deploy
Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```
<img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM"
src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">
pietern added a commit that referenced this pull request Jun 18, 2025
## Changes
Previously, the CLI would use the Terraform binary at
`DATABRICKS_TF_EXEC_PATH` only if the `DATABRICKS_TF_VERSION`
environment variable was set and matched the _built-in default version_.
It was built such that we could ship all binaries with the VS Code
extension to avoid re-downloading (see #1294).
This change expands the scope of these variables and makes them
composable. Now, you can:
1. Configure `DATABRICKS_TF_VERSION` to override the Terraform version
to use
2. Configure `DATABRICKS_TF_EXEC_PATH` to configure the path to an
existing Terraform binary
If 1 is used alone, the CLI will download the specified version. If 2 is
used alone, the CLI will execute the specified binary to confirm that it
matches the _built-in default version_. If both are used, the CLI will
execute the specified binary to confirm that it matches the specified
version.
## Why
To allow users to use newer version of the Terraform CLI.
## Tests
* Unit tests pass
* Integration tests with the latest Terraform version (#3040)
denik pushed a commit that referenced this pull request May 20, 2026
- Add `bundle debug terraform` command. It prints versions of the
Terraform and the Databricks Terraform provider. In the text mode it
also explains how to setup the CLI in environments with restricted
internet access.
- Use `DATABRICKS_TF_EXEC_PATH` env var to point Databricks CLI to the
Terraform binary. The CLI only uses it if `DATABRICKS_TF_VERSION`
matches the currently used terraform version.
- Use `DATABRICKS_TF_CLI_CONFIG_FILE` env var to point Terraform CLI
config that points to the filesystem mirror for the Databricks provider.
The CLI only uses it if `DATABRICKS_TF_PROVIDER_VERSION` matches the
currently used provider version.
Relevant PR on the VSCode extension side:
databricks/databricks-vscode#1147
Example output of the `databricks bundle debug terraform`:
```
Terraform version: 1.5.5
Terraform URL: https://releases.hashicorp.com/terraform/1.5.5
Databricks Terraform Provider version: 1.38.0
Databricks Terraform Provider URL: https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.38.0
Databricks CLI downloads its Terraform dependencies automatically.
If you run the CLI in an air-gapped environment, you can download the dependencies manually and set these environment variables:
DATABRICKS_TF_VERSION=1.5.5
DATABRICKS_TF_EXEC_PATH=/path/to/terraform/binary
DATABRICKS_TF_PROVIDER_VERSION=1.38.0
DATABRICKS_TF_CLI_CONFIG_FILE=/path/to/terraform/cli/config.tfrc
Here is an example *.tfrc configuration file:
disable_checkpoint = true
provider_installation {
filesystem_mirror {
path = "/path/to/a/folder/with/databricks/terraform/provider"
}
}
The filesystem mirror path should point to the folder with the Databricks Terraform Provider. The folder should have this structure: /registry.terraform.io/databricks/databricks/terraform-provider-databricks_1.38.0_ARCH.zip
For more information about filesystem mirrors, see the Terraform documentation: https://developer.hashicorp.com/terraform/cli/config/config-file#filesystem_mirror
```
---------
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
denik pushed a commit that referenced this pull request May 20, 2026
Breaking Change:
* Add allow list for resources when bundle `run_as` is set
([#1233](#1233)).
* Make bundle validation print text output by default
([#1335](#1335)).
CLI:
* Added `auth describe` command
([#1244](#1244)).
* Fixed message for successful auth describe run
([#1336](#1336)).
Bundles:
* Use UserName field to identify if service principal is used
([#1310](#1310)).
* Allow unknown properties in the config file for template
initialization ([#1315](#1315)).
* Remove support for DATABRICKS_BUNDLE_INCLUDES
([#1317](#1317)).
* Make `bundle.deployment` optional in the bundle schema
([#1321](#1321)).
* Fix the generated DABs JSON schema
([#1322](#1322)).
* Make bundle loaders return diagnostics
([#1319](#1319)).
* Add `bundle debug terraform` command
([#1294](#1294)).
* Allow specifying CLI version constraints required to run the bundle
([#1320](#1320)).
Internal:
* Retain location information of variable reference
([#1333](#1333)).
* Define `dyn.Mapping` to represent maps
([#1301](#1301)).
* Return `diag.Diagnostics` from mutators
([#1305](#1305)).
* Fix flaky test in `libs/process`
([#1314](#1314)).
* Move path field to bundle type
([#1316](#1316)).
* Load bundle configuration from mutator
([#1318](#1318)).
* Return diagnostics from `config.Load`
([#1324](#1324)).
* Return warning for nil primitive types during normalization
([#1329](#1329)).
* Include `dyn.Path` in normalization warnings and errors
([#1332](#1332)).
* Make normalization return warnings instead of errors
([#1334](#1334)).
API Changes:
* Added `databricks lakeview migrate` command.
* Added `databricks lakeview unpublish` command.
* Changed `databricks ip-access-lists get` command . New request type is
.
OpenAPI commit e316cc3d78d087522a74650e26586088da9ac8cb (2024-04-03)
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.36.0 to 0.37.0
([#1326](#1326)).
denik pushed a commit that referenced this pull request May 20, 2026
## Changes
This PR makes changes to support creating a docker image for the CLI
with the `terraform` dependencies built in. This is useful for customers
that operate in a network-restricted environment. Normally DABs makes
API calls to registry.terraform.io to setup the terraform dependencies,
with this setup the CLI/DABs will rely on the provider binaries bundled
in the docker image.
### Specifically this PR makes the following changes:
----------------
Modifies the CLI release workflow to publish the docker images in the
Github Container Registry. URL:
https://github.com/databricks/cli/pkgs/container/cli.
We use docker support in `goreleaser` to build and publish the images.
Using goreleaser ensures the CLI packaged in the docker image is the
same release artifact as the normal releases. For more information see:
1. https://goreleaser.com/cookbooks/multi-platform-docker-images
2. https://goreleaser.com/customization/docker/
Other choices made include:
1. Using `alpine` as the base image. The reason is `alpine` is a small
and lightweight linux distribution (~5MB) and an industry standard.
2. Not using [docker
manifest](https://docs.docker.com/reference/cli/docker/manifest) to
create a multi-arch build. This is because the functionality is still
experimental.
------------------
Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION`
environment variables optional for using the terraform file mirror.
While it's not strictly necessary to make the docker image work, it's
the "right" behaviour and reduces complexity. The rationale is:
- These environment variables here are needed so the Databricks CLI does
not accidentally use the file mirror bundled with VSCode if it's
incompatible. This does not require the env vars to be mandatory.
context: #1294
- This makes the `Dockerfile` and `setup.sh` simpler. We don't need an
[entrypoint.sh script to set the version environment
variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d).
This also makes using an interactive terminal with `docker run -it ...`
work out of the box.
## Tests
Tested manually. --------------------
To test the release pipeline I triggered a couple of dummy releases and
verified that the images are built successfully and uploaded to Github.
1. https://github.com/databricks/cli/pkgs/container/cli
3. workflow for release:
https://github.com/databricks/cli/actions/runs/8646106333
--------------------
I tested the docker container itself by setting up
[Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying
that no HTTP requests are made to `registry.terraform.io`
Before:
FYI, The Charles web proxy is hosted at localhost:8888.
```
shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy
Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```
<img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM"
src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b">
After:
This time bundle deploy is run from inside the docker container. We use
`host.docker.internal` to map to localhost on the host machine, and -v
to mount the host file system as a volume.
```
shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64 / # cd /bundle/
/bundle # rm -r .databricks/
/bundle # databricks bundle deploy
Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```
<img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM"
src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">
denik pushed a commit that referenced this pull request May 20, 2026
## Changes
Previously, the CLI would use the Terraform binary at
`DATABRICKS_TF_EXEC_PATH` only if the `DATABRICKS_TF_VERSION`
environment variable was set and matched the _built-in default version_.
It was built such that we could ship all binaries with the VS Code
extension to avoid re-downloading (see #1294).
This change expands the scope of these variables and makes them
composable. Now, you can:
1. Configure `DATABRICKS_TF_VERSION` to override the Terraform version
to use
2. Configure `DATABRICKS_TF_EXEC_PATH` to configure the path to an
existing Terraform binary
If 1 is used alone, the CLI will download the specified version. If 2 is
used alone, the CLI will execute the specified binary to confirm that it
matches the _built-in default version_. If both are used, the CLI will
execute the specified binary to confirm that it matches the specified
version.
## Why
To allow users to use newer version of the Terraform CLI.
## Tests
* Unit tests pass
* Integration tests with the latest Terraform version (#3040)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@ilia-db@pietern@andrewnester@kartikgupta-db@shreyas-goenka@juliacrawf-db