direct: retry 504 errors by denik · Pull Request #5349 · databricks/cli

denik · 2026-05-27T15:03:04Z

Changes

Retry resource methods that return an error that has http_code equal to 504 but that has not been retried by SDK.

This affects all DoRead, DoUpdate(WithID), DoDelete. This affects DoCreate for grants and permissions.

Note, for DoCreate the retry functionality is opt-in - implementations need to wrap error with retrySafe(). For other methods the retry is always enabled.

Why

We've seen reports where deploy fails with

Error: cannot create resources.pipelines..permissions: The service at /api/2.0/permissions/pipelines/<pipeline_id> is taking too long to process your request. (504 TEMPORARILY_UNAVAILABLE)

We also saw that terraform does custom retries for 504/GET databricks/terraform-provider-databricks#4355

Note, the two cases are different - the first one is "cannot create" so it refers to PUT.

Tests

New testserver feature that allows injecting expiring faults in a given endpoint. See fault.py.
New acceptance tests make use of fault.py to check failures in plan/create/update for permissions.

Wraps adapter methods DoRead, DoDelete, DoUpdate, DoUpdateWithID, DoResize, WaitAfterCreate, and WaitAfterUpdate with retry logic that retries on HTTP 408/500/502/503/504 up to 2 times with a 30s interval (overridable via DATABRICKS_BUNDLE_RETRY_INTERVAL_MS for tests). DoCreate is intentionally not retried. Adds fault injection support to testserver (POST /__testserver/fault) so acceptance tests can inject transient errors dynamically, and two acceptance tests verifying retry on permissions PUT (update deploy) and GET (plan). Co-authored-by: Isaac

…DoCreate - Narrow retry condition to 504 errors SDK did not already handle - Add retrySafeError/retrySafe: resource impls wrap DoCreate errors to opt in to transient retries when the operation is idempotent - Wire DoCreate in adapter to retry only when both retrySafe and isTransient - permissions and grants DoCreate opt in (both delegate to a PUT/PATCH) - Update 504/create acceptance test: now expects retry success + 2 PUTs Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

pietern · 2026-05-28T07:53:45Z

+// IsRetrySafe reports whether err was marked as safe to retry from DoCreate.
+func IsRetrySafe(err error) bool {
+	var safe *retrySafeError
+	return errors.As(err, &safe)


I added a linter earlier this week to suggest errors.AsType[T]. Maybe the base SHA is not up to date?

that was it, updated.

pietern · 2026-05-28T07:55:44Z

 	if err != nil {
- return fmt.Errorf("waiting after creating id=%s: %w", newID, err)
+ if isTransient(ctx, err) {
+ log.Warnf(ctx, "waiting after creating id=%s: %s", newID, err)


waitRemoteState is not up to date but this falls through. Must either retry or fail hard.

good catch. changed to retry.

…an.go Adapter is a type-adaptation layer; retries are an operational concern. - New bundle/direct/retry.go: isTransient, retryWith/retryOnTransient/retryErr - dresources/retry.go: only retrySafe/IsRetrySafe/UnwrapRetrySafe (opt-in signal) - adapter.go: stripped to pure type adaptation - apply.go, bundle_plan.go: retries applied at each adapter call site Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

…Update The create/update already succeeded; a 504 during status polling should not abort the deployment. Non-transient errors still fail hard. Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

…site Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

…warn+fallthrough Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

eng-dev-ecosystem-bot · 2026-05-28T10:45:26Z

Commit: 42f0cc1

Run: 26568534085

Follow up to #5349

## Changes Retry resource methods that return an error that has http_code equal to 504 but that has not been retried by SDK. This affects all DoRead, DoUpdate(WithID), DoDelete. This affects DoCreate for grants and permissions. Note, for DoCreate the retry functionality is opt-in - implementations need to wrap error with retrySafe(). For other methods the retry is always enabled. ## Why We've seen reports where deploy fails with > Error: cannot create resources.pipelines.<pipeline>.permissions: The service at /api/2.0/permissions/pipelines/<pipeline_id> is taking too long to process your request. (504 TEMPORARILY_UNAVAILABLE) We also saw that terraform does custom retries for 504/GET databricks/terraform-provider-databricks#4355 Note, the two cases are different - the first one is "cannot create" so it refers to PUT. ## Tests New testserver feature that allows injecting expiring faults in a given endpoint. See fault.py. New acceptance tests make use of fault.py to check failures in plan/create/update for permissions.

eng-dev-ecosystem-bot · 2026-05-28T13:01:28Z

Commit: 813b754

Run: 26570032233

Follow up to #5349

Follow up to #5349 Analysis shows 500 and 408 errors show up as well (although what we saw is those errors happen during file upload, but no reason it cannot happen on other endpoints).

## Release v1.2.0 ### CLI * `experimental open` now opens every DABs resource type that has a workspace URL, picking up `catalogs`, `schemas`, `volumes`, `database_instances`, `database_catalogs`, `synced_database_tables`, `postgres_catalogs`, `postgres_synced_tables`, `quality_monitors`, `vector_search_endpoints`, and `vector_search_indexes` ([#5346](#5346)). ### Bundles * Retry transient HTTP 5xx and 408 errors in direct deployment engine ([#5349](#5349), [#5364](#5364)). * Preserve `.designer.ipynb` suffix when translating notebook task paths so Lakeflow Designer files referenced from a `notebook_task` resolve correctly in the workspace ([#5370](#5370)). * Fix script output dropping last line without trailing newline ([#4995](#4995)). * engine/direct: Add `--select` flag to `bundle plan` and `bundle deploy` to plan/deploy a subset of resources (e.g. `--select my_job` or `--select jobs.my_job`); resources referenced by the selection are included transitively ([#5413](#5413)). * Support `purge_on_delete: true` on `postgres_projects` so bundles can hard-delete a Lakebase project on destroy (skipping the soft-delete retention window) ([#5414](#5414)). * Support terraform references in direct engine ([#5392](#5392)) * Support lifecycle.started for SQL warehouses ([#5348](#5348)) ### Dependency updates * Bump Go toolchain to 1.26.4 ([#5420](#5420)). * Bump `github.com/databricks/databricks-sdk-go` from v0.136.0 to v0.141.0 ([#5361](#5361)) * Bump Terraform provider from v1.115.0 to v1.117.0 ([#5421](#5421))

denik changed the title ~~WIP retry 504~~direct: retry 504 errorsMay 27, 2026

denik marked this pull request as ready for review May 27, 2026 16:40

denik added 6 commits May 28, 2026 10:01

update tests
c955b5c

tests
a176d26

clean up
4edfda4

rename doRefresh to doRead
d5e1599

pietern reviewed May 28, 2026
View reviewed changes

denik added 5 commits May 28, 2026 10:03

Log and fall through on transient errors in WaitAfterCreate/WaitAfter…
05f5bd7
…Update The create/update already succeeded; a 504 during status polling should not abort the deployment. Non-transient errors still fail hard. Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Rename retryErr to retryOnTransientErr
82e3e2c
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Fix log prefix in Destroy: "destroying" not "deploying"
73d0b05
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Reduce default retry interval from 30s to 15s
c68f966
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik force-pushed the denik/retrier branch from 0752251 to c68f966Compare May 28, 2026 08:16

denik temporarily deployed to test-trigger-is May 28, 2026 08:17 — with GitHub Actions Inactive

Use errors.AsType[T] instead of errors.As for Go 1.26+ compatibility
1720072
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:20 — with GitHub Actions Inactive

Revert unrelated cosmetic changes in adapter.go
dd2b0e4
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:25 — with GitHub Actions Inactive

Inline IsRetrySafe: export RetrySafeError, use errors.AsType at call …
fe5f124
…site Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:32 — with GitHub Actions Inactive

Retry WaitAfterCreate/WaitAfterUpdate on transient errors instead of …
5219571
…warn+fallthrough Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:33 — with GitHub Actions Inactive

Guard apiErr nil in retryWith log line
564f68e
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 09:08 — with GitHub Actions Inactive

denik requested a review from pietern May 28, 2026 09:09

Mark acceptance/bin/fault.py as executable
bf19721
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 09:30 — with GitHub Actions Inactive

Fix uniq -c count width portability in 504/plan acceptance test
80a9f58
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 09:44 — with GitHub Actions Inactive

Add NEXT_CHANGELOG entry for 504 retry
e7b037f
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 10:02 — with GitHub Actions Inactive

pietern approved these changes May 28, 2026
View reviewed changes

Comment threadbundle/direct/apply.go
Comment threadbundle/direct/apply.go

add a comment about DoCreate opt-in
42f0cc1

denik temporarily deployed to test-trigger-is May 28, 2026 10:12 — with GitHub Actions Inactive

denik enabled auto-merge May 28, 2026 10:17

denik disabled auto-merge May 28, 2026 10:45

denik merged commit 813b754 into mainMay 28, 2026
24 of 26 checks passed

denik deleted the denik/retrier branch May 28, 2026 10:45

denik added a commit that referenced this pull request May 28, 2026
Extend retries to 5xx and 408
29dc23b
Follow up to #5349

denik mentioned this pull request May 28, 2026
Extend retries to 5xx and 408 #5364
Merged

denik mentioned this pull request May 29, 2026
update output in jobs/destroy_without_mgmtperms #5374
Merged

denik added a commit that referenced this pull request May 29, 2026
update output in jobs/destroy_without_mgmtperms (#5374)
6795f16
Follow up to #5349

denik added a commit that referenced this pull request Jun 1, 2026
Extend retries to 5xx and 408
b7d9b48
Follow up to #5349

Navigation Menu

Uh oh!

direct: retry 504 errors - #5349

direct: retry 504 errors#5349
denik merged 22 commits into
mainfrom
denik/retrier

denik commented May 27, 2026•
edited
Loading

Uh oh!

pieternMay 28, 2026

Uh oh!

denikMay 28, 2026

Uh oh!

pieternMay 28, 2026

Uh oh!

denikMay 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

denik commented May 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

pieternMay 28, 2026

Choose a reason for hiding this comment

Uh oh!

denikMay 28, 2026

Choose a reason for hiding this comment

Uh oh!

pieternMay 28, 2026

Choose a reason for hiding this comment

Uh oh!

denikMay 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

denik commented May 27, 2026•
edited
Loading