feat: Lakeflow Jobs Plugin

[atilafassina]

Important

to maintain contribution history this PR should be rebased when merged
(don't merge it if it has more than 2 commits)

Summary

Resource-scoped jobs plugin following the files plugin pattern. Jobs are configured as named resources discovered from environment variables at startup.

Design

• Resource-scoped: Only configured jobs are accessible — this is not an open SDK wrapper
• Env-var discovery: Jobs are discovered from DATABRICKS_JOB_<KEY> env vars (e.g. DATABRICKS_JOB_ETL=123)
• Single-job shorthand: DATABRICKS_JOB_ID maps to the "default" key
• Manifest declares job resources with CAN_MANAGE_RUN permission
• Works with databricks apps init --features jobs

API

// Trigger a configured jobconst{ run_id }=awaitappkit.jobs("etl").runNow();// Trigger and wait for completionconstrun=awaitappkit.jobs("etl").runNowAndWait();// OBO accessawaitappkit.jobs("etl").asUser(req).runNow();// List recent runsconstruns=awaitappkit.jobs("etl").listRuns({limit: 10});// Single-job shorthandawaitappkit.jobs("default").runNow();

Files changed

• plugins/jobs/manifest.json — declares job resource with CAN_MANAGE_RUN permission
• plugins/jobs/types.ts — JobAPI, JobHandle, JobsExport, IJobsConfig types
• plugins/jobs/plugin.ts — JobsPlugin with discoverJobs(), getResourceRequirements(), resource-scoped createJobAPI()
• plugins/jobs/index.ts — barrel exports
• connectors/jobs/client.ts — listRuns now respects limit parameter
• plugins/jobs/tests/plugin.test.ts — 32 tests covering discovery, resource requirements, exports, OBO, multi-job, and auto-fill

Documentation safety checklist

• Examples use least-privilege permissions
• Sensitive values are obfuscated
• No insecure patterns introduced

Reopened from #223 — moved head branch from fork to upstream to fix CI secrets access.

demo

lakeflow-jobs.mp4

[Copilot]

Pull request overview

Adds a new resource-scoped “jobs” plugin to @databricks/appkit, following the existing “files” plugin pattern: jobs are discovered from environment variables at startup and exposed via a keyed accessor API, with HTTP routes for triggering and monitoring runs.

Changes:

• Introduces plugins/jobs (manifest, defaults, params mapping, types, plugin implementation, and extensive tests).
• Adds connectors/jobs with telemetry + cancellation support and updates exports/docs to surface the new plugin and types.
• Updates templates and generated API docs/sidebars to include the new Jobs plugin.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
template/appkit.plugins.json	Adds “jobs” to plugin template metadata + resource description.
pnpm-lock.yaml	Locks new dependency (zod@4.3.6) and related lockfile updates.
packages/appkit/src/plugins/jobs/types.ts	Public types for Jobs plugin API/config (JobAPI/JobHandle/IJobsConfig).
packages/appkit/src/plugins/jobs/plugin.ts	Core JobsPlugin: env discovery, dynamic resource requirements, API methods, HTTP routes.
packages/appkit/src/plugins/jobs/params.ts	TaskType-based param mapping into SDK request fields.
packages/appkit/src/plugins/jobs/defaults.ts	Execution defaults for read/write/stream operations.
packages/appkit/src/plugins/jobs/manifest.json	Plugin manifest + config schema + baseline resource definition.
packages/appkit/src/plugins/jobs/index.ts	Barrel exports for Jobs plugin/types.
packages/appkit/src/plugins/jobs/tests/plugin.test.ts	Comprehensive unit tests for discovery, API, routes, and validation.
packages/appkit/src/plugins/index.ts	Exposes jobs from the plugins barrel.
packages/appkit/src/index.ts	Exposes jobs and related public types/configs from package root.
packages/appkit/src/connectors/jobs/client.ts	JobsConnector SDK wrapper with telemetry instrumentation + limit handling.
packages/appkit/src/connectors/jobs/types.ts	Connector config type (timeout/telemetry).
packages/appkit/src/connectors/jobs/index.ts	Connector barrel exports.
packages/appkit/src/connectors/index.ts	Exposes jobs connector from connectors barrel.
packages/appkit/package.json	Adds zod dependency for runtime param schemas + JSON schema generation.
docs/docs/api/appkit/typedoc-sidebar.ts	Adds new Jobs docs entries to sidebar.
docs/docs/api/appkit/*.md	Adds generated API docs pages for Jobs plugin types.
docs/docs/api/appkit/index.md	Adds Jobs-related types to API index list.
docs/docs/api/appkit/Interface.BasePluginConfig.md	Updates “Extended by” list to include IJobsConfig.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pkosiec]

How it is different from @keugenek PR here: #221?

If you based on top of Evgenii's work, maybe it's worth to recognize his contribution? What I'm thinking is either:

• have 2 commits on this PR: Evgenii's base one and yours (with all changes, PR review fixes etc.), and temporarily enable the "Rebase & Merge" option to have 2 commits on main
• or, merge Evgenii's PR as is, right before this PR (so we'd need to wait for this PR to be approved first)

WDYT?

[atilafassina]

@pkosiec

How it is different from @keugenek PR here: #221?
If you based on top of Evgenii's work, maybe it's worth to recognize his contribution? What I'm thinking is either:

• have 2 commits on this PR: Evgenii's base one and yours (with all changes, PR review fixes etc.), and temporarily enable the "Rebase & Merge" option to have 2 commits on main
• or, merge Evgenii's PR as is, right before this PR (so we'd need to wait for this PR to be approved first)

WDYT?

I got stuck trying to update his PR because it was opened from a fork, it's noted in the PR description:

Reopened from #223 — moved head branch from fork to upstream to fix CI secrets access.

So, I couldn't get a clean CI to merge it, and if I merged to main, it would trigger a release.
You'll notice that in the commit history the very first commit is his, that's his work that I iterated on top of.

It's also marked as a comment on the original PR

#223 (comment)

[pkosiec]

So, I couldn't get a clean CI to merge it, and if I merged to main, it would trigger a release.

It wouldn't trigger a release currently (finalizing releases is manual) so there shouldn't be a problem with that 👍 but I'd wait until your PR (on top of this one) is ready to merge, and I'd merge both one after another. What do you think?

[MarioCadenas]

I added some comments, but I'm actually wondering if this should just be a "jobs" plugin or if we should extend this and make it support both jobs and pipelines tbh 😅

[MarioCadenas]

a few more things I noticed going through the core package:

---

1. listRuns cache key probably never hits cache

in plugin.ts, listRuns uses options ?? {} in the cache key:

self._readSettings(["jobs:listRuns",jobKey,options??{}])

when options is undefined (which is common), this creates a new {} every time — so if the cache compares by reference, every call is a miss and the 60s TTL does nothing. the other read methods get this right with primitives (["jobs:getRun", jobKey, runId]). can we normalize this to something like ["jobs:listRuns", jobKey, options?.limit ?? "default"]?

---

2. mapParams silently coerces non-primitives to garbage

in params.ts, the notebook/python_wheel/sql cases do String(v) on every value:

notebook_params: Object.fromEntries(Object.entries(params).map(([k,v])=>[k,String(v)]),),

if someone passes an object or array through (e.g. with a loose zod schema like z.record(z.unknown())), String(v) produces "[object Object]" or "1,2,3" — the job would run with garbage params and no error. can we at least add a guard that rejects non-primitive values here, or coerce more defensively?

---

3. polling in runAndWait — no backoff or jitter

I know I already asked if we can simplify runAndWait, but specifically the poll interval is fixed at 5s forever. with the default 10min timeout that's 120 API calls per waiter, and if you have multiple SSE clients polling the same run they all hit the Jobs API at the same cadence with no jitter. should we add some basic exponential backoff (e.g. 2s → 5s → 10s → cap at 15s) and a small random jitter so concurrent pollers don't align?

[pkosiec]

Just a few comments 👍 Nice work!

[atilafassina]

I added some comments, but I'm actually wondering if this should just be a "jobs" plugin or if we should extend this and make it support both jobs and pipelines tbh 😅

Adding this here just for historical reference.
We synced on this and decided to ship them as separate plugins initially and observe usage first.

[pkosiec]

Works now! Didn't notice CI jobs weren't triggered for the latest run and probably tested an outdated version.

LGTM, as the OBO approach wouldn't be possible currently anyway because of lack of the jobs scope in Apps (unless I missed it)