Uh oh!
There was an error while loading. Please reload this page.
ssh: increase server startup timeout to 45m for GPU accelerators - #5569
Merged
Conversation
Serverless GPU compute (--accelerator) can take much longer to provision than CPU compute, and the fixed 10-minute wait for the SSH server job to start frequently expires before the GPU node is up. Use a 45-minute startup timeout when an accelerator is requested. Co-authored-by: Isaac
Co-authored-by: Isaac
anton-107force-pushed
the
ssh-gpu-startup-timeout
branch
from
June 12, 2026 10:27
a71119d to
45cfdd4Compareeng-dev-ecosystem-bot
commented
Jun 12, 2026
Collaborator
Integration test reportCommit: 45cfdd4
22 interesting tests: 15 SKIP, 7 RECOVERED
Top 30 slowest tests (at least 2 minutes):
|
rugpanov
commented
Jun 15, 2026
Contributor
NIT: 45m is a magic constant, not configurable, this is consistent with the existing taskStartupTimeout constant style. If GPU provisioning times prove variable in practice, a flag (e.g. --startup-timeout) might be worth it later, but baking in a constant is fine and matches the surrounding code. |
rclarey
approved these changes
Jun 15, 2026
rclarey
left a comment
Contributor
There was a problem hiding this comment.
LGTM, but I second Grigory's nit about 45min seeming arbitrary. Curious how 45min was chosen
eng-dev-ecosystem-bot
commented
Jun 15, 2026
Collaborator
Integration test reportCommit: 3fd9620
561 interesting tests: 513 MISS, 43 FAIL, 2 PANIC, 2 SKIP, 1 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
deco-sdk-taggingBot
added a commit
that referenced
this pull request
Jun 17, 2026
## Release v1.4.0 ### CLI * Improved error messages for `ssh connect`: when an SSH connection attempt fails, the client now fetches and prints the server's recent error logs ([#5555](#5555)). * Increase the SSH server startup timeout from 10 to 45 minutes when a GPU accelerator is requested via `databricks ssh connect --accelerator` ([#5569](#5569)). * Fix authentication falling back to the default profile in `.databrickscfg` when a host is already configured via the environment (e.g. `DATABRICKS_HOST` with `DATABRICKS_TOKEN`) ([#5616](#5616)). * ssh: fix opening remote environment in Cursor, which previously hung on default-extension install and never opened the editor ([#5619](#5619)). * Improve the error shown when `databricks labs install` cannot find a project's `labs.yml`: the message now explains that either the requested version does not exist or the project is not installable with the CLI, and links to the repository ([#5559](#5559)). ### Bundles * Remove API enum values and types that are still in development from the `databricks-bundles` Python package; these were never accepted by the backend ([#5484](#5484)). * direct: Fix resolving a resource reference that is used more than once within the same field ([#5558](#5558)). * Bundle variable references now accept Unicode letters in path segments (e.g. `${var.变量}`). ([#5532](#5532)) * Ignore remote changes for vector search direct_access_index_spec.schema_json to prevent drift when the backend normalizes the schema ([#5481](#5481)). * Remove hidden, never-functional `--existing-dashboard-id`, `--existing-dashboard-path`, `--existing-alert-id`, and `--existing-genie-space-id` alias flags from `bundle generate`; use the documented `--existing-id` / `--existing-path` flags instead ([#5591](#5591)). * engine/direct: Fix WAL corruption after two consecutive failed deploys ([#5606](#5606)). * engine/direct: Don't open the deployment state WAL when a deploy's plan fails ([#5607](#5607)). * Ignore unity catalog managed schema property defaults to avoid unnecessary drift ([#5195](#5195)). * Add `postgres_roles` and `postgres_databases` resources to create Postgres roles and databases on a Lakebase branch ([#5467](#5467), [#5627](#5627)). * direct: Stop spurious recreate/rename on redeploy when the backend normalizes a resource's name-based ID (e.g. Unity Catalog lowercasing a schema or volume name) ([#5599](#5599)). * Fix the generated pipeline README to suggest `databricks bundle run <pipeline> --refresh <table>` for running a single transformation; the previously documented `--select` flag is not supported by `bundle run` ([#5252](#5252)).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
databricks ssh connectwaits for the bootstrap SSH server job run to reach RUNNING state, bounded by a fixed 10-minutetaskStartupTimeout. Serverless GPU compute (requested via--accelerator GPU_1xA10/GPU_8xH100) can take much longer to provision than CPU compute, so the wait frequently expires before the GPU node is up.When
--acceleratoris set, use a 45-minute startup timeout (gpuTaskStartupTimeout) instead. The CPU/dedicated-cluster path keeps the existing 10-minute timeout.Why
GPU serverless provisioning routinely exceeds 10 minutes, causing
ssh connect --accelerator ...to fail with a startup timeout even though the job would have come up shortly after.Tests
go build ./experimental/ssh/...andgo test ./experimental/ssh/...pass.ClientOptions.TaskStartupTimeout; no behavior change without--accelerator.This pull request and its description were written by Isaac.