Uh oh!
There was an error while loading. Please reload this page.
feat(appkit): managed eval datasets + turn semantics (stack 3/5) - #479
Open
MarioCadenas wants to merge 7 commits into
Open
feat(appkit): managed eval datasets + turn semantics (stack 3/5)#479MarioCadenas wants to merge 7 commits into
MarioCadenas wants to merge 7 commits into
Conversation
Trace agent turns and tool calls to MLflow via the mlflow-tracing SDK. Adds an optional 'experiment' resource to the agents plugin; when bound (MLFLOW_EXPERIMENT_ID), each turn opens an AGENT span and each tool call a nested TOOL span, with auth resolved from the app's Databricks credentials. A turn's trace can be linked to an evaluation run via mlflow.sourceRun. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…runs eve-style eval authoring (defineEval + t-context + matchers) discovered from config/agents/<id>/evals/*.eval.ts and run via 'appkit agent eval' against a running app. Streams per-eval progress and gates CI via exit code. When Databricks creds + an experiment are set, it creates a real MLflow evaluation run (mlflow.runType=genai_evaluate): each eval's trace links to the run, pass/fail is written as feedback assessments, and aggregate metrics are logged. All via the MLflow REST API. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Extend the agent eval framework and tighten MLflow output to match the native `mlflow.genai.evaluate` experience: - LLM-as-judge via autoevals (factuality, closedQA, custom), pointed at a Databricks serving endpoint; exposed through `t.judge.*`. - One Feedback assessment per assertion (judges as LLM_JUDGE with score + rationale) plus an overall `appkit_eval`; assessment names sanitized to `[A-Za-z0-9_-]` since the API rejects dots. - Trace-table parity: set Request/Response previews and the `mlflow.traceName` tag (the Trace-name column reads the tag, not the span name). - Eval runs carry `mlflow.source.name`/`type` tags so linked traces show Source and Run name; live chat traces have no run so those stay empty. - Example judge eval under config/agents/query/evals. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Introduce connectors/mlflow as the shared REST + auth layer for MLflow,
so the eval runner (and future callers) stop threading host/token and
hand-rolling fetch/URL logic:
- MlflowClient owns {host, token}: normalizes the host once, exposes
post() (throws) for runs/* and postResult() (structured failure) for
best-effort assessment writes, plus servingEndpointsUrl() for the judge.
- resolveDatabricksAuth() mints an OAuth bearer from a CLI profile via the
SDK WorkspaceClient (the AppKit-native path), so `agent eval` no longer
requires a hand-set DATABRICKS_TOKEN. Adds an `--profile` flag.
- Eval run create/finish, assessment reporting, and the judge take the
client; the agents plugin's host normalization now delegates to the
connector's normalizeHost.
The mlflow-tracing SDK wrapper stays in the agents plugin: it manages a
process-global provider (like TelemetryManager) and has an agent-shaped
API, so it isn't a connector.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>Let an agent eval sweep a Databricks managed evaluation dataset instead of a
single hardcoded turn, and make one-shot vs multi-turn conversations explicit:
- defineEval gains `dataset: { table, limit? }`. The runner reads the UC table
(via SQLWarehouseConnector over the SQL Statement Execution API — the Python
datasets API needs Spark), then runs the test once per row with the row bound
to `t.input`/`t.expected`.
- readEvalDataset reuses the SQL connector's result transform, which already
JSON-parses `inputs`/`expectations` columns whether stored as JSON or structs.
- resolveWorkspaceClient builds the WorkspaceClient the connector needs, from
the same profile/host/token as resolveDatabricksAuth. CLI: `--warehouse`.
- Each dataset row gets a fresh driver so rows are independent conversations
(no thread/context bleed between rows).
- Turn semantics: consecutive `t.send`s share one thread (multi-turn); new
`t.reset()` starts a fresh conversation for independent one-shot checks in a
single test.
- Example dataset.eval.ts + unit tests for the reader and reset.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>…al_dataset Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
LLM-judge assertions (`t.judge.*`) now fail the eval on a miss, like the deterministic assertions — previously they were soft-by-default and a failing guideline left the run green. `.atLeast(n)` sets the pass threshold without demoting; chain `.soft()` to keep a judge as a tracked-only metric. Surfaces real signal: e.g. a dataset row whose guidelines the agent can't satisfy now fails the run (non-zero exit) instead of passing silently. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Contributor
This pull request has had no activity for 23 days and has been marked as stale. It will be closed in 7 days if there is no further activity. Add a comment, push a commit, or apply the |
Contributor
This pull request has been automatically closed because it had no activity for a month. Feel free to reopen it if you would like to continue the work. |
MarioCadenasforce-pushed
the
pr/agent-evals-2-framework
branch
3 times, most recently
from
August 21, 2026 10:37
e32afdd to
65de478Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 3/5 · targets
pr/agent-evals-2-framework(review after #2).Managed datasets and explicit conversation semantics.
dataset: { table }sweeps a Databricks managed evaluation dataset (a Unity Catalogcatalog.schema.tableread via the SQL Statement Execution API — the Python datasets API needs Spark). One run per row;t.input/t.expectedbind each row.--warehouse+ auth required.send), multi-turn (consecutivesends share a thread), andt.reset()for an independent turn in one test. Each dataset row gets a fresh driver so rows don't bleed..atLeast(n)sets the threshold,.soft()opts out.