Skip to content

feat(examples): add validated Foundry-native custom evaluation - #352

Draft
Chang Liu (changliu2) wants to merge 1 commit into
mainfrom
examples/foundry-custom-evaluation
Draft

Chang Liu (changliu2) wants to merge 1 commit into
mainfrom
examples/foundry-custom-evaluation

Conversation

@changliu2

Copy link
Copy Markdown
Collaborator

Summary

Adds a Foundry-native evaluation adapter under
examples/langgraph-foundry-hosted/evaluation/.

The example supports:

  1. registered Foundry agent targets, with Prompt and Scenario runs separated;
  2. existing invoke_agent traces at turn or conversation level;
  3. precomputed ASSERT test_set.jsonl + inference_set.jsonl datasets.

Each route compiles the resolved boolean ASSERT judge dimensions into versioned
Foundry custom prompt evaluators.

Safety and correctness

  • Never reads or uploads scores.jsonl.
  • Recursively sanitizes structured tool arguments/results, credential-bearing
    URLs, and Windows/Unix absolute paths before upload.
  • Rejects stale joins when type, behavior, dimensions, or available source
    identity/hash differs despite a matching test-case ID.
  • Makes the boolean mapping explicit: ASSERT true means a bad event; Foundry
    true means the criterion passed.
  • Applies disabled judge dimensions after built-in, preset, and inline
    resolution.
  • Uses source-specific ISO-8601 and Unix-epoch trace time arguments.
  • Supports separate judge and Scenario-simulator model deployments.
  • Uses service-compatible content-addressed dataset versions while preserving
    the full SHA-256 in lineage metadata.
  • Uses DefaultAzureCredential, creates fresh evaluation definitions, and
    never deletes cloud assets.

Historical lineage

This carries forward the strict validation, deterministic joins, hashing,
evaluator-version drift checks, dry-run behavior, fake clients, and no-delete
lessons from #267. That PR uploaded precomputed ASSERT
scores and reasons; this adapter does not use that architecture. It recompiles
the ASSERT judge contract and runs it as Foundry custom evaluators.

Validation

  • 52 tests passed with azure-ai-projects 2.7.0 and openai 3.13.0.
  • 45 SDK-independent core tests passed.
  • Ruff, compileall, git diff --check, dependency resolution, dry runs, and
    adversarial secret/path scans passed.
  • Live Agent, Scenario, Dataset, and Trace evaluations completed in Foundry.
  • Repeated runs reused all dataset and evaluator versions while creating fresh
    evaluation runs.

Preview boundaries

Conversation simulation and conversation trace evaluation use Foundry preview
contracts. Scenario output retains full lineage in the source dataset and run
metadata, although the downloadable simulated item currently exposes only a
subset of custom source fields.

Add Agent, Trace, and precomputed Dataset routes that compile ASSERT boolean judge dimensions into versioned Foundry prompt evaluators. Preserve artifact lineage, sanitize uploaded evidence, and never read scores.jsonl.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant