Remove unused JSON progress logging mode by pietern · Pull Request #3812 · databricks/cli

pietern · 2025-10-24T08:51:15Z

Changes

The progress logger now only supports append mode.

Why

JSON mode was added in #276 (March 2023). The intent was to make progress events machine readable but this didn't materialize (we didn't end up using it in the VS Code extension).

This functionality would print events to stderr if the user specified --progress-format json, or set the an equivalent environment variable. Because the flag is hidden, and there are no online references to the functionality, I believe it is safe to remove. If users take a dependency on JSON output, it should be enabled via the existing --output json flag and be written to stdout.

After this change is merged, the remaining functionality can be moved into the core cmdio types, and the "progress logger" can be removed. Once there is a single type for all I/O, we have a better shot at improving it.

Related change: #3811.

Tests

Tests pass.

In-place mode was designed to update job progress in place using ANSI escape codes (e.g., showing 'PENDING' → 'RUNNING' → 'TERMINATED' on the same line). However, acceptance tests show jobs typically output only a single state transition: 'Run URL: <url>' followed by '[TIMESTAMP] "job-name" TERMINATED', suggesting the job completes before multiple states can be observed. The default mode selection logic required log-file to not be stderr AND stderr to be a terminal to enable in-place mode, which is an uncommon configuration. Additionally, only JobProgressEvent supported in-place updates while all other events (URLs, errors, pipeline events) fell back to append mode, making the feature inconsistent. The implementation added complexity with ANSI escape codes, terminal detection, and an IsInplaceSupported() interface method across all event types. Since the feature provided minimal practical value and likely was rarely (if ever) enabled by default, it has been removed in favor of the simpler append mode.

The JSON mode for progress logging was designed to output structured JSON events for machine parsing. However, this mode had several limitations: it prevented interactive prompts (Ask/AskSelect methods would error), required --auto-approve for destroy commands, and added complexity with JSON marshaling in the logger. The feature provided minimal practical value as most CLI usage is interactive, and the default append mode is sufficient for both human and machine consumption. Since the mode added unnecessary complexity without clear benefits, it has been removed in favor of the simpler append-only mode.

eng-dev-ecosystem-bot · 2025-10-24T09:04:16Z

Run: 18779344613

	Env	❌FAIL	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip
💚	aws linux				1	1	319	580
💚	aws windows				1	1	320	579
💚	aws-ucws linux				1	1	440	475
💚	aws-ucws windows				1	1	441	474
🔄	azure linux			3		1	317	579
🔄	azure windows			3		1	318	578
💚	azure-ucws linux				1	1	438	474
💚	azure-ucws windows				1	1	439	473
🔄	gcp linux			13	1	1	305	581
❌	gcp windows	24	1			1	293	577

40 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚R	💚R	💚R	💚R	🔄f	🔄f	💚R	💚R	💚R	🟨K
TestAccept/bundle/resources/models/basic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/run/app-with-job	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
TestAccept/bundle/templates/default-python/combinations/classic	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=no/NBOOK=no/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=no/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=no/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=yes/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=yes/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=yes	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p
TestAccept/bundle/templates/default-python/combinations/serverless	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=yes/NBOOK=no/PY=no	🙈s	🙈s	✅p	✅p	🙈s	🙈s	✅p	✅p	🙈s	❌F
TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes	🙈s	🙈s	✅p	✅p	🙈s	🙈s	✅p	✅p	🙈s	❌F
TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes	🙈s	🙈s	✅p	✅p	🙈s	🙈s	✅p	✅p	🙈s	❌F
TestAccept/bundle/templates/default-python/integration_classic	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	🔄f	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/UV_PYTHON=3.10	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/UV_PYTHON=3.9	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.10	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.11	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.12	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.13	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
TestFsCpDir	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestFsCpDir/dbfs_to_dbfs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestFsCpSourceIsDirectoryButTargetIsFile	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_dbfs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestSyncEnsureRemotePathIsUsableIfRepoExists	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestImportDirWithOverwriteFlag	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestWorkpaceExportPrintsContents	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerRecursiveDelete	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerRecursiveDelete/dbfs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerRecursiveDelete/workspace_files	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerWorkspaceFilesExtensionsRead	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerWorkspaceFilesExtensionsReadDir	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerWorkspaceNotebook	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestLock	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestTagKeyGCP	🙈s	🙈s	🙈s	🙈s	🙈s	🙈s	🙈s	🙈s	✅p	❌F
TestTagKeyGCP/unicode									✅p	❌F
TestTagKeyGCP/valid									✅p	❌F

pietern · 2025-10-24T13:08:03Z

Integration test fails with:

The service at /api/2.0/workspace/mkdirs is temporarily unavailable. Please try again later.

## Changes This continues the progress logger simplification by migrating from the `Logger.Log()` event-based system to direct cmdio calls. This builds on #3811 and #3812, which made the progress logger effectively only write strings to stderr. ## Why This simplifies the codebase by removing the (unused) event abstraction layer while maintaining functionality. The compatibility layer provides a clean migration path and will eventually be removed once the functionality in those functions have a dedicated new home. ## Tests Tests pass. Analysis of the diff suggests that before/after is functionally equivalent. A minor note is that we no longer have a mutex around `.Log()` calls, but there are no concurrent calls to the function.

…3820) ## Changes This change removes remaining references to the "progress logger". All terminal I/O now goes through a single type in `cmdio`. Builds on #3811, #3812, and #3818.

## Release v0.275.0 ### Notable Changes * Python support for Databricks Asset Bundles is now generally available. ### CLI * Remove `inplace` mode for the `--progress-format` flag ([#3811](#3811)) * Remove `json` mode for the `--progress-format` flag ([#3812](#3812)) * Deprecate the `--progress-format` flag ([#3819](#3819)) ### Bundles * Add support for `--bind` flag in `bundle generate` ([#3782](#3782)) * Add `pydabs` template replacing `experimental-jobs-as-code` template ([#3806](#3806)) * You can now use the top-level `python` section instead of `experimental/python` ([#3540](#3540))

## Changes The progress logger now only supports `append` mode. ## Why JSON mode was added in #276 (March 2023). The intent was to make progress events machine readable but this didn't materialize (we didn't end up using it in the VS Code extension). This functionality would print events to stderr if the user specified `--progress-format json`, or set the an equivalent environment variable. Because the flag is hidden, and there are no online references to the functionality, I believe it is safe to remove. If users take a dependency on JSON output, it should be enabled via the existing `--output json` flag and be written to stdout. After this change is merged, the remaining functionality can be moved into the core `cmdio` types, and the "progress logger" can be removed. Once there is a single type for all I/O, we have a better shot at improving it. Related change: #3811. ## Tests Tests pass.

## Changes This continues the progress logger simplification by migrating from the `Logger.Log()` event-based system to direct cmdio calls. This builds on #3811 and #3812, which made the progress logger effectively only write strings to stderr. ## Why This simplifies the codebase by removing the (unused) event abstraction layer while maintaining functionality. The compatibility layer provides a clean migration path and will eventually be removed once the functionality in those functions have a dedicated new home. ## Tests Tests pass. Analysis of the diff suggests that before/after is functionally equivalent. A minor note is that we no longer have a mutex around `.Log()` calls, but there are no concurrent calls to the function.

…3820) ## Changes This change removes remaining references to the "progress logger". All terminal I/O now goes through a single type in `cmdio`. Builds on #3811, #3812, and #3818.

## Release v0.275.0 ### Notable Changes * Python support for Databricks Asset Bundles is now generally available. ### CLI * Remove `inplace` mode for the `--progress-format` flag ([#3811](#3811)) * Remove `json` mode for the `--progress-format` flag ([#3812](#3812)) * Deprecate the `--progress-format` flag ([#3819](#3819)) ### Bundles * Add support for `--bind` flag in `bundle generate` ([#3782](#3782)) * Add `pydabs` template replacing `experimental-jobs-as-code` template ([#3806](#3806)) * You can now use the top-level `python` section instead of `experimental/python` ([#3540](#3540))

pietern added 3 commits October 23, 2025 15:49

Add entry to NEXT_CHANGELOG.md
8c3f4b7

pietern requested review from andrewnester, anton-107, denik, jefferycheng1, kanterov, lennartkats-db and shreyas-goenka as code owners October 24, 2025 08:51

pietern temporarily deployed to test-trigger-is October 24, 2025 08:51 — with GitHub Actions Inactive

denik approved these changes Oct 24, 2025
View reviewed changes

shreyas-goenka approved these changes Oct 24, 2025
View reviewed changes

andrewnester approved these changes Oct 24, 2025
View reviewed changes

Base automatically changed from remove-inplace to mainOctober 24, 2025 12:06

Merge branch 'main' into remove-json-mode
1ba889d

pietern temporarily deployed to test-trigger-is October 24, 2025 12:11 — with GitHub Actions Inactive

pietern enabled auto-merge October 24, 2025 12:11

pietern mentioned this pull request Oct 24, 2025
Migrate progress logger functions to use cmdio directly #3818
Merged

pietern disabled auto-merge October 24, 2025 13:08

pietern merged commit afe399e into mainOct 24, 2025
12 of 13 checks passed

pietern deleted the remove-json-mode branch October 24, 2025 13:08

pietern mentioned this pull request Oct 24, 2025
Deprecate --progress-format flag and remove progress logger types #3820
Merged

github-actionsBot mentioned this pull request Jan 3, 2026
chore(nix): update flake.lock (act, actionlint, awscli2 +25 more) i9wa4/dotfiles#79
Closed

Navigation Menu

Uh oh!

Remove unused JSON progress logging mode - #3812

Remove unused JSON progress logging mode#3812
pietern merged 4 commits into
mainfrom
remove-json-mode

pietern commented Oct 24, 2025•
edited
Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 24, 2025•
edited
Loading

Uh oh!

pietern commented Oct 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

pietern commented Oct 24, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Oct 24, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pietern commented Oct 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pietern commented Oct 24, 2025•
edited
Loading

eng-dev-ecosystem-bot commented Oct 24, 2025•
edited
Loading