Uh oh!
There was an error while loading. Please reload this page.
Translate command_path and code_source_path for AI runtime task to remote paths - #5922
Merged
Conversation
… paths DABs now rewrites the two local path fields on an `ai_runtime_task` job task to their uploaded remote locations, so a bundle can reference local files and an artifact-built code archive the same way other task types do. - command_path: registered as a translated task file path (job_paths_visitor), so a local script path resolves to its workspace or immutable-snapshot location like spark_python_task.python_file. - code_source_path: collected as a local library (remote_path/expand_glob_ references), so a local archive is uploaded and the field is rewritten to the uploaded remote path exactly like a wheel or jar. This honors both a UC-Volume artifact_path and the immutable-folder src/artifacts/.internal destination, letting an `artifacts` build command produce the code_source tarball. No new artifact type is needed: a typeless artifact with build + files already builds and uploads generically, and the upload/remote-path machinery does not branch on artifact.Type. Covered by a new acceptance test (bundle/artifacts/ai_runtime_code_source) that deploys and asserts both fields are rewritten and the tarball is uploaded, plus the immutable_workspace_paths validate test and a job_paths_visitor unit case. Co-authored-by: Isaac
command_path and code_source_path for AI runtime task to remote pathspietern
enabled auto-merge
July 14, 2026 09:25
andrewnester
approved these changes
Jul 14, 2026
eng-dev-ecosystem-bot
commented
Jul 14, 2026
Collaborator
Integration test reportCommit: 4f7294e
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 10 slowest tests (at least 2 minutes):
|
Uh oh!
There was an error while loading. Please reload this page.
eng-dev-ecosystem-bot
commented
Jul 14, 2026
Collaborator
Integration test reportCommit: d6af43c
28 interesting tests: 17 flaky, 6 FAIL, 2 SKIP, 2 KNOWN, 1 RECOVERED
Top 50 slowest tests (at least 2 minutes):
|
vinchenzo-db added a commit
that referenced
this pull request
Jul 20, 2026
jobs.AiRuntimeTask.CodeSourcePath is a private-preview SDK field that is added/removed across releases, so referencing it directly makes the package fail to compile whenever the field is absent. Read code_source_path through dyn.MapByPattern instead (the same mechanism PackageAndUpload and #5922 already use), so the mutator compiles regardless of the SDK's typed struct. Tests set the field on the dyn value rather than the struct literal. Co-authored-by: Isaac
vinchenzo-db added a commit
that referenced
this pull request
Aug 4, 2026
…erlay paths - validate: gate the snapshot-dir guards on an existing local *directory*, not any local path. A pre-built tarball (a local file, the #5922 flow) is uploaded by the artifact path and packages no snapshot, so a natural `sync.exclude` of "*.tar.gz" was wrongly rejected. Guarded in the file-based acceptance test. - codeSourceFiles: drop entries outside the code directory. sync.include is force- added regardless of the scoped walk, so an unrelated include made an all-filtered directory look non-empty and shipped an empty archive. The empty_code_source test now covers that shape and uses musterr. - vfs.Overlay: reject names that aren't relative in-root files. An absolute name hung the ancestor walk (path.Dir("/") is never "."), and "" resolved to the root. - print_requests.py: add --del-field for top-level request fields instead of overloading --del-body, which only edits the parsed JSON body. - Rename package_upload.go to package_code_source.go (it performs no upload), trim the changelog fragment, and document that entry modes are not cross-platform identical (Windows has no execute bit). Co-authored-by: Isaac
tanishgupta-db pushed a commit
to tanishgupta-db/cli
that referenced
this pull request
Aug 5, 2026
…bricks#6110) Let ai_runtime_task.code_source_path point at a local directory, not just a pre-built tarball. A new aicode mutator (run in the build phase, before libraries.ExpandGlobReferences/ReplaceWithRemotePath) detects a local-directory value, packages it into a reproducible, content-addressed tarball — honoring .gitignore and the top-level sync.include/exclude globs — uploads it to the user's ~/.air/repo_snapshots directory, and rewrites code_source_path to the uploaded (de-/Workspace-prefixed) path. Because it runs first, PR databricks#5922's artifact-style collection then sees an already-remote path and skips it, so the two compose: a directory is packaged by the CLI, a pre-built .tgz still flows through the artifact path. Also synthesizes a requirements.yaml next to the task's command_path from the job's serverless environments[] spec, so the AI Runtime harness sets up the workload environment. command_path translation itself is provided by databricks#5922. Verified end-to-end on a live workspace: a plain bundle deploy of a directory code_source_path yields a job whose run terminates SUCCESS; sync.exclude and .gitignore entries are absent from the uploaded snapshot; unchanged code skips re-upload. Co-authored-by: Isaac ## Changes Adds an aicode bundle mutator that packages a local code directory referenced by an AI Runtime task's code_source_path and uploads it during bundle deploy, then wires it into the build/initialize phases. - bundle/config/mutator/aicode/package_upload.go — PackageAndUpload (build-phase mutator): finds every ai_runtime_task.code_source_path (direct and under for_each_task) that points at a local directory, packages it into a reproducible, content-addressed .tar.gz (.git/gitignored files excluded, honoring the top-level sync.include/exclude), uploads it to the user's ~/.air/repo_snapshots directory, and rewrites code_source_path to the uploaded remote path. The archive name embeds the SHA-256 of the tarball, so an unchanged directory resolves to the same remote path and re-uploads are skipped. - bundle/config/mutator/aicode/snapshot_package.go — the reproducible tar builder (fixed epoch on every entry so identical content ⇒ identical bytes ⇒ identical hash). - bundle/config/mutator/aicode/requirements.go — SynthesizeRequirements: writes a requirements.yaml next to each task's translated command_path, derived from the job's serverless environments[] spec, so the AI Runtime harness can set up the workload environment. - bundle/config/mutator/aicode/validate.go — Validate (read-only, initialize-phase): surfaces misconfigurations early (local code_source_path combined with git_source, or with experimental.immutable_folder) with actionable messages, performing no uploads. - Directory-vs-file discrimination (package_upload.go + validate.go): both mutators only claim a code_source_path that resolves to an existing directory. A local file (a pre-built tarball delivered via an artifacts block) or a not-yet-existing path is left untouched so it flows through the standard artifact-upload path. This is what lets the two paths compose. - bundle/phases/{build,initialize}.go — runs PackageAndUpload + SynthesizeRequirements in the build phase before libraries.ExpandGlobReferences/ReplaceWithRemotePath, and Validate in the initialize phase. - Acceptance test bundle/ai_runtime_task/local_code_source covering the full package → upload → rewrite → requirements-synthesis flow, plus unit tests for each mutator (including the directory/file/remote skip cases). ## Why The SDK jobs.AiRuntimeTask.code_source_path field expects a workspace or UC-volume path to an uploaded code archive, and its doc comment states the CLI is responsible for packaging the user's local code into that archive. Nothing in DABs did that yet: a user pointing code_source_path at a local directory had no way to bundle deploy a runnable AI Runtime job. This mutator implements that contract for bundles. It deliberately reuses DABs' existing local-artifact upload plumbing and runs before ReplaceWithRemotePath, so the two upload paths compose without special-casing: a directory is packaged and uploaded by this mutator (which then presents an already-remote path that the artifact collector skips), while a pre-built tarball delivered via an artifacts block still flows through the artifact path as a file. Content-addressing keeps redeploys of unchanged code a no-op. SynthesizeRequirements closes the loop so the deployed workload actually has its environment set up — without it the run fails during setup. ## Tests End-to-end validation Verified on a live A10 staging workspace (dbc-04ac0685-8857) that the aicode mutator packages a local-directory code_source_path at bundle deploy and that the deployed AI Runtime workload runs to SUCCESS on real GPU. Covered all three code-source shapes the mutator must handle, plus the artifact-tarball path it must not claim. What was validated Seam: PackageAndUpload claims a local directory Expected: dir → content-addressed .tar.gz, uploaded to ~/.air/repo_snapshots, code_source_path rewritten to the remote archive Result: ✅ ──────────────────────────────────────── Seam: PackageAndUpload skips a local file Expected: a pre-built tarball (artifacts block) flows through the standard artifact upload, not aicode packaging Result: ✅ (bundle/artifacts/ai_runtime_code_source acceptance test) ──────────────────────────────────────── Seam: command_path translation Expected: rewritten to its absolute synced workspace path Result: ✅ ──────────────────────────────────────── Seam: SynthesizeRequirements Expected: writes requirements.yaml next to command_path, derived from environments[].spec Result: ✅ ──────────────────────────────────────── Seam: GPU workload Expected: extracts code, installs deps, runs on A10, exits SUCCESS Result: ✅ Deployed ai_runtime_task after bundle deploy (mutator output) { "code_source_path": "/Users/<me>/.air/repo_snapshots/code_source/code_source_5659f5b55c3427b1.tar.gz", "command_path": "/Workspace/Users/<me>/.bundle/convert-e2e-smoke/dev/files/command.sh" } code_source_path was ./code_source (a local directory) in the bundle — the mutator packaged it, uploaded the content-addressed archive, and rewrote the field to the remote path. ``` Live GPU runs (all SUCCESS on NVIDIA A10G) ┌────────────────────┬───────────────────────────┬───────────┬─────────────────────────────────────────┐ │ Scenario │ code_source shape │ Run │ Workload log evidence │ │ │ │ result │ │ ├────────────────────┼───────────────────────────┼───────────┼─────────────────────────────────────────┤ │ │ │ ✅ │ cuda available: True · device: NVIDIA │ │ Working tree │ local dir │ SUCCESS │ A10G · matmul checksum 1131894.0 · === │ │ │ │ (242s) │ SUCCESS === │ ├────────────────────┼───────────────────────────┼───────────┼─────────────────────────────────────────┤ │ │ local dir (materialized │ ✅ │ ran the committed code, not the dirty │ │ Git-pinned commit │ from a pinned SHA) │ SUCCESS │ working tree · === SUCCESS === │ │ │ │ (192s) │ │ ├────────────────────┼───────────────────────────┼───────────┼─────────────────────────────────────────┤ │ Requirements-file │ local dir + deps in │ ✅ │ torch 2.13.0+cu130, numpy 2.1.3 │ │ deps │ environments[].spec │ SUCCESS │ installed via synthesized │ │ │ │ (272s) │ requirements.yaml · === SUCCESS === │ └────────────────────┴───────────────────────────┴───────────┴─────────────────────────────────────────┘ ``` Reproduce ``` # 1. Author a run YAML with a local code_source directory cat > train.yaml <<'YAML' experiment_name: aicode-e2e command: cd "$CODE_SOURCE_PATH" && python train.py compute: {accelerator_type: GPU_1xA10, num_accelerators: 1} environment: {version: 5, dependencies: [torch]} code_source: {type: snapshot, snapshot: {root_path: ./src}} YAML mkdir -p src && echo 'import torch; print("cuda:", torch.cuda.is_available()); print("=== SUCCESS ===")' > src/train.py # 2. Convert to a bundle (emits code_source_path: ./code_source, a local directory) databricks experimental air convert-to-dabs train.yaml --output-dir bundle cd bundle # 3. Deploy — the aicode mutator packages the directory, uploads it, rewrites code_source_path, # and synthesizes requirements.yaml next to command_path databricks bundle deploy # 4. Run on GPU and confirm SUCCESS databricks bundle run aicode-e2e ``` <!-- If your PR needs to be included in the release notes for next release, add a changelog fragment: create .nextchanges/<section>/<name>.md with a one-line description (e.g. .nextchanges/cli/quickstart.md). See .nextchanges/README.md. --> --------- Co-authored-by: ben.hansen <ben.hansen@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DABs now rewrites the two local path fields on an
ai_runtime_taskjob task to their uploaded remote locations:command_path: translated likespark_python_task.python_file.code_source_path: uploaded and rewritten like other artifacts. Here the artifact is a gzipped tarball (a.tgz), which can be produced by anartifactsbuild block. Works with both UC-Volume and workspaceartifact_pathdestinations.No new artifact type needed. A typeless artifact builds and uploads generically.
This pull request and its description were written by Isaac.