Uh oh!
There was an error while loading. Please reload this page.
feat(appkit): mlflow tracing for agents (stack 1/5) - #477
Merged
Conversation
Contributor
📦 Bundle size reportCompared against
|
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 869 KB (+6.4 KB) | 303 KB (+2.4 KB) |
| Type declarations | 314 KB | 109 KB (+9 B) |
| Source maps | 1.7 MB (+12 KB) | 566 KB (+4.2 KB) |
| Other | 11 KB | 3.7 KB |
| Total | 2.9 MB (+18 KB) | 981 KB (+6.6 KB) |
Per-entry composition (own code — deps external (as shipped))
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
. | 88 KB | 2.5 KB | 91 KB | external | 288 KB |
./beta | 49 KB (+702 B) | 459 B | 49 KB (+702 B) | external | 143 KB (+2.1 KB) |
./type-generator | 21 KB | 0 B | 21 KB | external | 60 KB |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
. | index.js | initial | 84 KB |
. | utils.js | initial | 4.0 KB |
. | remote-tunnel-manager.js | lazy | 2.5 KB |
./beta | beta.js | initial | 33 KB |
./beta | stream-manager.js | initial | 5.8 KB |
./beta | wide-event-emitter.js | initial | 3.2 KB |
./beta | databricks.js | initial | 3.0 KB |
./beta | configuration.js | initial | 2.1 KB |
./beta | service-context.js | initial | 1.3 KB |
./beta | client.js | initial | 431 B |
./beta | client-options.js | initial | 220 B |
./beta | supervisor-api.js | lazy | 194 B |
./beta | databricks.js | lazy | 142 B |
./beta | index.js | lazy | 123 B |
./type-generator | index.js | initial | 21 KB |
@databricks/appkit-ui
npm tarball (packed): 342 KB (-1 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 390 KB | 130 KB |
| Type declarations | 228 KB | 83 KB (+1 B) |
| Source maps | 753 KB | 248 KB |
| CSS | 16 KB | 3.3 KB |
| Total | 1.4 MB | 464 KB (+1 B) |
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
./js | 5.3 KB | 49 KB | 55 KB | 208 KB | 14 KB |
./js/beta | 20 B | 0 B | 20 B | 0 B | 0 B |
./react | 432 KB | 49 KB | 480 KB | 1.3 MB | 175 KB |
./react/beta | 1.0 KB | 0 B | 1.0 KB | 0 B | 1.9 KB |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
./js | index.js | initial | 5.2 KB |
./js | chunk | initial | 120 B |
./js | apache-arrow | lazy | 49 KB |
./js/beta | beta.js | initial | 20 B |
./react | index.js | initial | 430 KB |
./react | tslib | initial | 2.1 KB |
./react | apache-arrow | lazy | 49 KB |
./react/beta | beta.js | initial | 1.0 KB |
Trace agent turns and tool calls to MLflow via the mlflow-tracing SDK. Adds an optional 'experiment' resource to the agents plugin; when bound (MLFLOW_EXPERIMENT_ID), each turn opens an AGENT span and each tool call a nested TOOL span, with auth resolved from the app's Databricks credentials. A turn's trace can be linked to an evaluation run via mlflow.sourceRun. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
currentTraceId() used getLastActiveTraceId(), which is only set when a root span ends (on export). Read mid-turn it returned the previous turn's id — or another turn's under concurrency. Read the context-active span instead, matching linkTraceToRun. Also read mlflowRunId from the validated schema (capped at 64 chars) instead of casting raw req.body, and drop two unused imports. Adds tracing tests covering trace-id source and output auto-capture. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
/invocations and /responses drove tool calls through traceTool but had no traceAgent wrapper, so with tracing enabled each tool call became an orphan root TOOL trace instead of nesting under a turn. Wrap the adapter run in traceAgent, mirroring the streaming path, so both surfaces produce the same trace shape. Also drop a dead flushTraces mock left from the flushAgentTraces removal. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
MarioCadenasforce-pushed
the
pr/agent-evals-1-tracing
branch
from
August 5, 2026 15:22
412b08d to
f517306CompareContributor
🤖 AppKit PR bot🔬 Run evalsStart an eval for this PR from the evals-monitor app: Go to Evals Monitor → 📦 Try this PR's app templateScaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh run download 31686134891 -R databricks/appkit -n appkit-template-0.59.0-pr.a52e202-pr-agent-evals-1-tracing-477 -D appkit-pr-477 \
&& unzip -o "appkit-pr-477/appkit-template-0.59.0-pr.a52e202-pr-agent-evals-1-tracing-477.zip" -d "appkit-pr-477" \
&& databricks apps init --template "appkit-pr-477"The template pins |
MarioCadenas
enabled auto-merge (squash)
August 11, 2026 09:33
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
/invocations and /responses already produced full agent+tool traces, but unlike /chat they never linked the trace to an MLflow run or surfaced the trace id. Add mlflowRunId to invocationsRequestSchema and, in _runAgentNonStreaming, call linkTraceToRun and return mlflow_trace_id in the response envelope (only when tracing is enabled) so an eval runner driving the non-streaming surface can group traces under a run and attach assessments. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Trim verbose comments to their load-bearing fact, drop duplicated rationale, and reuse the existing seedPlugin helper for the invoke tracing tests. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
atilafassina
approved these changes
Aug 13, 2026
Uh oh!
There was an error while loading. Please reload this page.
IamGalymzhan added a commit
that referenced
this pull request
Aug 19, 2026
**Restore three upstream mlflow tests dropped by the merge.** Resolving the route-handler-errors conflict with `--ours` took the whole file from this branch, discarding main's non-conflicting additions from PR #477: the `vi.mock("../mlflow")` hoist, the linkTraceToRun/mockTraceId resets, the parameterised seedPlugin(adapter), the seedEchoPlugin/invoke helpers, and the three trace tests. 15 upstream tests, 12 here, and nothing failed to say so. Restored from 9538d58 alongside this branch's createTestPluginContext rewrite of the aliases test — the two changes are independent. **close() now memoizes itself, not just the phases.** runOnce() guaranteed the teardown body ran once, but resetCoreSingletons() sat outside it, so every call reset again: `await a.close(); createApp(); await a.close()` dropped the second app's singletons. The reset is also skipped when the budget expired, because the phases are still running and still own those instances. Only reachable through the raw AppHandle — createTestApp's wrapper memoizes, which is why the harness-level test could not see it and the regression test lives in app-close.integration. **Refcount singleton ownership.** The env baseline was already refcounted so overlapping harness apps compose, while the singleton layer reset on every boot and every close — so booting B rebound A's ServiceContext and CacheManager, and closing A while B was live left B with none at all. claimCoreSingletons/ releaseCoreSingletons now follow the same model as the env baseline: first boot claims, last close drops. **Fake the on-behalf-of client.** The kit promised "no workspace, no credentials, no network", but createApp({ client }) installs only the service principal; an `obo` request reached ServiceContext.createUserContext, which builds a real SDK client from process.env.DATABRICKS_HOST. The harness now stubs that for the app's lifetime and restores it on close, mirroring fixtures.ts's createUserContextSpy. Every fix has a test verified by reintroducing the bug. Two of those tests needed a second attempt: asserting the OBO client's host does not discriminate, since a real client carries the same DATABRICKS_HOST string — the test now asserts the harness's mock recorded the call. The probe plugin gained a route that calls the client under asUser, because the existing /as-user route only reads ctx.userId, which is how this escaped notice. 4517 tests pass; typecheck, lint, and format clean. Signed-off-by: Galymzhan <zhangazy2004@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 1/5 · targets
main.Adds MLflow tracing to the agents plugin. Agent turns and tool calls are traced to a bound MLflow experiment via the
mlflow-tracingSDK (OpenTelemetry under the hood). Tracing is a no-op unless the plugin's optionalexperimentresource is set (MLFLOW_EXPERIMENT_ID); auth is resolved from the app's own Databricks credentials, so no tokens or OTLP headers are wired by hand.withAgentSpanwraps each turn (AGENT span) and tool dispatch (TOOL span, auto-nested).mlflow.traceNametag.experimentoptional resource in the agents manifest.This is the base of a 5-PR stack that builds out an agent evaluation framework. Reviewable on its own — touches only the agents plugin.