Skip to content

feat(candidate): measure exact candidates before activation#547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
drewstone merged 14 commits into from
Jul 17, 2026
Merged

feat(candidate): measure exact candidates before activation #547

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
00e85d2
feat(candidate): execute exact approved outcomes
drewstone Jul 13, 2026
ddeab85
Merge remote-tracking branch 'origin/main' into feat/universal-candid…
drewstone Jul 14, 2026
9a2553d
feat(candidate): adopt exact improvement contracts
drewstone Jul 14, 2026
caa1c79
Merge remote-tracking branch 'origin/main' into feat/universal-candid…
drewstone Jul 14, 2026
2cab99c
refactor(contract): remove candidate schema versions
drewstone Jul 15, 2026
b08bd50
Merge remote-tracking branch 'origin/main' into test/unversion-candid…
drewstone Jul 15, 2026
cff5376
Merge remote-tracking branch 'origin/main' into test/unversion-candid…
drewstone Jul 15, 2026
e81b7c1
fix(candidate): materialize approved knowledge
drewstone Jul 15, 2026
52985b5
Merge remote-tracking branch 'origin/main' into test/unversion-candid…
drewstone Jul 15, 2026
72ebca0
fix(package): verify compatible runtime and bench releases
drewstone Jul 15, 2026
f4aa979
feat(intelligence): bind product metadata to measured comparisons
drewstone Jul 15, 2026
ef4c0b5
fix(intelligence): stabilize measured comparisons
drewstone Jul 15, 2026
908a4c9
Merge remote-tracking branch 'origin/main' into fix/pr547-review
drewstone Jul 17, 2026
0ca7af6
fix(candidate): bind exact experiments before activation
drewstone Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 6 additions & 12 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -42,9 +42,6 @@ jobs:

agent-bench:
runs-on: ubuntu-latest
defaults:
run:
working-directory: bench
steps:
- uses: actions/checkout@v4

Expand All@@ -54,16 +51,13 @@ jobs:
with:
node-version: 22
cache: pnpm
cache-dependency-path: bench/pnpm-lock.yaml
cache-dependency-path: pnpm-lock.yaml

- name: Install published dependencies
- name: Install package set
run: pnpm install --frozen-lockfile

- name: Typecheck public entrypoint against installed packages
run: pnpm run typecheck:public

- name: Test deterministic package paths
run: pnpm test
- name: Build current agent-runtime
run: pnpm run build

- name: Verify packed package in a clean consumer
run: pnpm run verify:package
- name: Verify agent-bench against current agent-runtime
run: pnpm run verify:bench
35 changes: 15 additions & 20 deletions .github/workflows/publish.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -40,6 +40,9 @@ jobs:
- name: Verify packed package exports
run: pnpm run verify:package

- name: Verify agent-bench against this release
run: pnpm run verify:bench

- name: Verify tag/version lock
run: |
NPM_VERSION=$(node -p "require('./package.json').version")
Expand DownExpand Up@@ -98,9 +101,6 @@ jobs:
# tag/version locking stays intact and the root package is never co-published.
if: startsWith(github.ref, 'refs/tags/agent-bench-v')
runs-on: ubuntu-latest
defaults:
run:
working-directory: bench
steps:
- uses: actions/checkout@v4

Expand All@@ -110,23 +110,17 @@ jobs:
with:
node-version: 22
cache: pnpm
cache-dependency-path: bench/pnpm-lock.yaml
cache-dependency-path: pnpm-lock.yaml

- name: Install deps
run: pnpm install --frozen-lockfile

- name: Typecheck public entrypoint against installed packages
run: pnpm run typecheck:public

- name: Test deterministic package paths
run: pnpm test

- name: Verify packed package in a clean consumer
run: pnpm run verify:package
- name: Verify packed agent-bench against published dependencies
run: pnpm run verify:bench:published

- name: Verify tag/version lock
run: |
NPM_VERSION=$(node -p "require('./package.json').version")
NPM_VERSION=$(node -p "require('./bench/package.json').version")
TAG_VERSION="${GITHUB_REF#refs/tags/agent-bench-v}"
if [ "$TAG_VERSION" != "$NPM_VERSION" ]; then
echo "::error::Tag/version mismatch: tag=$TAG_VERSION package=$NPM_VERSION."
Expand All@@ -141,9 +135,6 @@ jobs:
permissions:
contents: read
id-token: write
defaults:
run:
working-directory: bench
steps:
- uses: actions/checkout@v4

Expand All@@ -153,7 +144,7 @@ jobs:
with:
node-version: 22
cache: pnpm
cache-dependency-path: bench/pnpm-lock.yaml
cache-dependency-path: pnpm-lock.yaml

- run: pnpm install --frozen-lockfile

Expand All@@ -162,10 +153,14 @@ jobs:
# Requires the npmjs Trusted Publisher on @tangle-network/agent-bench:
# org tangle-network, repo agent-runtime, workflow publish.yml.
npm install -g npm@11
NAME=$(node -p "require('./package.json').name")
VERSION=$(node -p "require('./package.json').version")
NAME=$(node -p "require('./bench/package.json').name")
VERSION=$(node -p "require('./bench/package.json').version")
if npm view "$NAME@$VERSION" version >/dev/null 2>&1; then
echo "$NAME@$VERSION already on registry; skipping publish"
else
npm publish --provenance --access public
mkdir -p "$RUNNER_TEMP/agent-bench-package"
pnpm --dir bench pack --pack-destination "$RUNNER_TEMP/agent-bench-package"
package="$(find "$RUNNER_TEMP/agent-bench-package" -maxdepth 1 -name '*.tgz' -print -quit)"
test -n "$package"
npm publish "$package" --provenance --access public
fi
20 changes: 8 additions & 12 deletions bench/package.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -24,23 +24,19 @@
"test": "node scripts/run-package-tests.mjs",
"typecheck:public": "tsc -p tsconfig.public.json",
"verify:package": "node scripts/verify-packed-consumer.mjs",
"verify:package:local-runtime": "node scripts/verify-packed-consumer.mjs --local-runtime",
"verify:pier": "tsx scripts/verify-pier-pair.mts"
},
"dependencies": {
"@tangle-network/agent-eval": "^0.117.1",
"@tangle-network/agent-interface": "^0.25.0",
"@tangle-network/agent-runtime": "0.94.9",
"@tangle-network/sandbox": "^0.10.3"
"@tangle-network/agent-eval": "0.122.1",
"@tangle-network/agent-interface": "0.30.0",
"@tangle-network/agent-runtime": "workspace:*",
"@tangle-network/sandbox": "^0.11.1"
},
"devDependencies": {
"@types/node": "^25.0.0",
"tsx": "^4.19.0",
"typescript": "^6.0.3"
},
"pnpm": {
"overrides": {
"@tangle-network/agent-eval": "0.117.1"
}
"@types/node": "^25.9.3",
"tsx": "^4.22.4",
"typescript": "^5.7.0"
},
"files": [
"src",
Expand Down
76 changes: 42 additions & 34 deletions bench/pier_agents/candidate_contract.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -107,6 +107,17 @@ def _object(value: Any, label: str) -> dict[str, Any]:
return value


def _contract_object(
value: Any, label: str, *, kind: str | None = None
) -> dict[str, Any]:
obj = _object(value, label)
if "schemaVersion" in obj or "version" in obj:
raise CandidateContractError(f"{label} contains obsolete version fields")
if kind is not None and obj.get("kind") != kind:
raise CandidateContractError(f"{label} has the wrong kind")
return obj


def _string(value: Any, label: str) -> str:
if not isinstance(value, str) or not value or "\0" in value:
raise CandidateContractError(f"{label} must be a non-empty string without NUL")
Expand DownExpand Up@@ -236,26 +247,22 @@ def _embedded_bytes(value: Any, label: str) -> bytes:


def _workspace_files(value: Any, label: str) -> tuple[WorkspaceFile, ...]:
snapshot = _object(value, label)
if (
snapshot.get("schemaVersion") != 1
or snapshot.get("kind") != "agent-candidate-workspace-snapshot"
):
raise CandidateContractError(f"{label} has the wrong snapshot kind")
material = _object(snapshot.get("material"), f"{label}.material")
if (
material.get("schemaVersion") != 1
or material.get("kind") != "agent-candidate-workspace-manifest"
):
raise CandidateContractError(f"{label}.material has the wrong manifest kind")
snapshot = _contract_object(
value, label, kind="agent-candidate-workspace-snapshot"
)
material = _contract_object(
snapshot.get("material"),
f"{label}.material",
kind="agent-candidate-workspace-manifest",
)
values = material.get("files")
if not isinstance(values, list):
raise CandidateContractError(f"{label}.material.files must be an array")
files: list[WorkspaceFile] = []
for index, value in enumerate(values):
obj = _object(value, f"{label}.material.files[{index}]")
mode = _integer(obj.get("mode"), f"{label}.material.files[{index}].mode")
if mode not in {0o644, 0o755}:
if mode > 0o777:
raise CandidateContractError(
f"{label} contains unsupported file mode {mode:o}"
)
Expand DownExpand Up@@ -287,7 +294,7 @@ def _profile_files(value: Any) -> tuple[ProfileFile, ...]:
for index, value in enumerate(values):
obj = _object(value, f"profilePlan.material.files[{index}]")
mode = _integer(obj.get("mode"), f"profilePlan.material.files[{index}].mode")
if mode not in {0o644, 0o755}:
if mode > 0o777:
raise CandidateContractError(
f"profile plan contains unsupported mode {mode:o}"
)
Expand DownExpand Up@@ -359,26 +366,23 @@ def load_prepared_candidate_contract(
raise CandidateContractError(
"materialization receipt bytes do not match the expected digest"
)
if plan.get("schemaVersion") != 1 or plan.get("kind") != _PLAN_KIND:
raise CandidateContractError("execution plan has the wrong schema or kind")
if receipt.get("schemaVersion") != 1 or receipt.get("kind") != _RECEIPT_KIND:
raise CandidateContractError(
"materialization receipt has the wrong schema or kind"
)
plan = _contract_object(plan, "execution plan", kind=_PLAN_KIND)
receipt = _contract_object(
receipt, "materialization receipt", kind=_RECEIPT_KIND
)
if receipt.get("digestAlgorithm") != "rfc8785-sha256" or "digest" in receipt:
raise CandidateContractError(
"materialization receipt must be exact digest-free canonical material"
)

execution_evidence = _object(receipt.get("executionPlan"), "receipt.executionPlan")
execution_evidence = _contract_object(
receipt.get("executionPlan"),
"receipt.executionPlan",
kind=_EXECUTION_EVIDENCE_KIND,
)
plan_digest = _digest(
execution_evidence.get("digest"), "receipt.executionPlan.digest"
)
if (
execution_evidence.get("schemaVersion") != 1
or execution_evidence.get("kind") != _EXECUTION_EVIDENCE_KIND
):
raise CandidateContractError("receipt execution evidence has the wrong kind")
if (
sha256_bytes(plan_raw) != plan_digest
or execution_evidence.get("material") != plan
Expand All@@ -402,11 +406,16 @@ def load_prepared_candidate_contract(
execution_id = _string(plan.get("executionId"), "plan.executionId")

task = _object(plan.get("task"), "plan.task")
outcome = _object(task.get("outcome"), "plan.task.outcome")
if outcome.get("kind") != "workspace":
raise CandidateContractError("Pier requires a workspace task outcome")
repository = _object(task.get("repository"), "plan.task.repository")
base_commit = _git_object(
repository.get("baseCommit"), "plan.task.repository.baseCommit"
)
base_tree = _git_object(repository.get("baseTree"), "plan.task.repository.baseTree")
base_tree = _git_object(
repository.get("baseTree"), "plan.task.repository.baseTree"
)
if len(base_commit) != len(base_tree):
raise CandidateContractError("task Git objects use different hash formats")
instruction = _instruction(task.get("instruction"))
Expand DownExpand Up@@ -471,12 +480,11 @@ def load_prepared_candidate_contract(
if receipt.get("candidateWorkspace") != candidate_snapshot:
raise CandidateContractError("receipt and plan candidate workspaces differ")

profile_evidence = _object(receipt.get("profilePlan"), "receipt.profilePlan")
if (
profile_evidence.get("schemaVersion") != 1
or profile_evidence.get("kind") != _PROFILE_PLAN_KIND
):
raise CandidateContractError("receipt profile evidence has the wrong kind")
profile_evidence = _contract_object(
receipt.get("profilePlan"),
"receipt.profilePlan",
kind=_PROFILE_PLAN_KIND,
)
profile_digest = _digest(
profile_evidence.get("digest"), "receipt.profilePlan.digest"
)
Expand All@@ -489,7 +497,7 @@ def load_prepared_candidate_contract(
profile_artifact_material = json.loads(profile_raw)
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
raise CandidateContractError("profile artifact is not UTF-8 JSON") from exc
profile_material = _object(
profile_material = _contract_object(
profile_evidence.get("material"), "receipt.profilePlan.material"
)
if profile_artifact_material != profile_material:
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
feat(candidate): measure exact candidates before activation by drewstone · Pull Request #547 · tangle-network/agent-runtime · GitHub
Skip to content
feat(candidate): measure exact candidates before activation#547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

drewstone
merged 14 commits into

from

Jul 17, 2026
 Merged


feat(candidate): measure exact candidates before activation
#547
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
00e85d2

feat(candidate): execute exact approved outcomes


drewstone Jul 13, 2026
ddeab85

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
9a2553d

feat(candidate): adopt exact improvement contracts


drewstone Jul 14, 2026
caa1c79

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
2cab99c

refactor(contract): remove candidate schema versions


drewstone Jul 15, 2026
b08bd50

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
cff5376

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
e81b7c1

fix(candidate): materialize approved knowledge


drewstone Jul 15, 2026
52985b5

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
72ebca0

fix(package): verify compatible runtime and bench releases


drewstone Jul 15, 2026
f4aa979

feat(intelligence): bind product metadata to measured comparisons


drewstone Jul 15, 2026
ef4c0b5

fix(intelligence): stabilize measured comparisons


drewstone Jul 15, 2026
908a4c9

Merge remote-tracking branch 'origin/main' into fix/pr547-review


drewstone Jul 17, 2026
0ca7af6

fix(candidate): bind exact experiments before activation


drewstone Jul 17, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


18 changes: 6 additions & 12 deletions
.github/workflows/ci.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,9 +42,6 @@ jobs:

 agent-bench:
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -54,16 +51,13 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install published dependencies
 - name: Install package set
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test
 - name: Build current agent-runtime
 run: pnpm run build

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify agent-bench against current agent-runtime
 run: pnpm run verify:bench

35 changes: 15 additions & 20 deletions
.github/workflows/publish.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -40,6 +40,9 @@ jobs:
 - name: Verify packed package exports
 run: pnpm run verify:package

 - name: Verify agent-bench against this release
 run: pnpm run verify:bench

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
Expand DownExpand Up@@ -98,9 +101,6 @@ jobs:
 # tag/version locking stays intact and the root package is never co-published.
 if: startsWith(github.ref, 'refs/tags/agent-bench-v')
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -110,23 +110,17 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install deps
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify packed agent-bench against published dependencies
 run: pnpm run verify:bench:published

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
 NPM_VERSION=$(node -p "require('./bench/package.json').version")
 TAG_VERSION="${GITHUB_REF#refs/tags/agent-bench-v}"
 if [ "$TAG_VERSION" != "$NPM_VERSION" ]; then
 echo "::error::Tag/version mismatch: tag=$TAG_VERSION package=$NPM_VERSION."
Expand All@@ -141,9 +135,6 @@ jobs:
 permissions:
 contents: read
 id-token: write
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -153,7 +144,7 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - run: pnpm install --frozen-lockfile

Expand All@@ -162,10 +153,14 @@ jobs:
 # Requires the npmjs Trusted Publisher on @tangle-network/agent-bench:
 # org tangle-network, repo agent-runtime, workflow publish.yml.
 npm install -g npm@11
 NAME=$(node -p "require('./package.json').name")
 VERSION=$(node -p "require('./package.json').version")
 NAME=$(node -p "require('./bench/package.json').name")
 VERSION=$(node -p "require('./bench/package.json').version")
 if npm view "$NAME@$VERSION" version >/dev/null 2>&1; then
 echo "$NAME@$VERSION already on registry; skipping publish"
 else
 npm publish --provenance --access public
 mkdir -p "$RUNNER_TEMP/agent-bench-package"
 pnpm --dir bench pack --pack-destination "$RUNNER_TEMP/agent-bench-package"
 package="$(find "$RUNNER_TEMP/agent-bench-package" -maxdepth 1 -name '*.tgz' -print -quit)"
 test -n "$package"
 npm publish "$package" --provenance --access public
 fi

20 changes: 8 additions & 12 deletions
bench/package.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -24,23 +24,19 @@
 "test": "node scripts/run-package-tests.mjs",
 "typecheck:public": "tsc -p tsconfig.public.json",
 "verify:package": "node scripts/verify-packed-consumer.mjs",
 "verify:package:local-runtime": "node scripts/verify-packed-consumer.mjs --local-runtime",
 "verify:pier": "tsx scripts/verify-pier-pair.mts"
 },
 "dependencies": {
 "@tangle-network/agent-eval": "^0.117.1",
 "@tangle-network/agent-interface": "^0.25.0",
 "@tangle-network/agent-runtime": "0.94.9",
 "@tangle-network/sandbox": "^0.10.3"
 "@tangle-network/agent-eval": "0.122.1",
 "@tangle-network/agent-interface": "0.30.0",
 "@tangle-network/agent-runtime": "workspace:*",
 "@tangle-network/sandbox": "^0.11.1"
 },
 "devDependencies": {
 "@types/node": "^25.0.0",
 "tsx": "^4.19.0",
 "typescript": "^6.0.3"
 },
 "pnpm": {
 "overrides": {
 "@tangle-network/agent-eval": "0.117.1"
 }
 "@types/node": "^25.9.3",
 "tsx": "^4.22.4",
 "typescript": "^5.7.0"
 },
 "files": [
 "src",
Expand Down

76 changes: 42 additions & 34 deletions
bench/pier_agents/candidate_contract.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -107,6 +107,17 @@ def _object(value: Any, label: str) -> dict[str, Any]:
 return value


def _contract_object(
 value: Any, label: str, *, kind: str | None = None
) -> dict[str, Any]:
 obj = _object(value, label)
 if "schemaVersion" in obj or "version" in obj:
 raise CandidateContractError(f"{label} contains obsolete version fields")
 if kind is not None and obj.get("kind") != kind:
 raise CandidateContractError(f"{label} has the wrong kind")
 return obj


def _string(value: Any, label: str) -> str:
 if not isinstance(value, str) or not value or "\0" in value:
 raise CandidateContractError(f"{label} must be a non-empty string without NUL")
Expand DownExpand Up@@ -236,26 +247,22 @@ def _embedded_bytes(value: Any, label: str) -> bytes:


def _workspace_files(value: Any, label: str) -> tuple[WorkspaceFile, ...]:
 snapshot = _object(value, label)
 if (
 snapshot.get("schemaVersion") != 1
 or snapshot.get("kind") != "agent-candidate-workspace-snapshot"
 ):
 raise CandidateContractError(f"{label} has the wrong snapshot kind")
 material = _object(snapshot.get("material"), f"{label}.material")
 if (
 material.get("schemaVersion") != 1
 or material.get("kind") != "agent-candidate-workspace-manifest"
 ):
 raise CandidateContractError(f"{label}.material has the wrong manifest kind")
 snapshot = _contract_object(
 value, label, kind="agent-candidate-workspace-snapshot"
 )
 material = _contract_object(
 snapshot.get("material"),
 f"{label}.material",
 kind="agent-candidate-workspace-manifest",
 )
 values = material.get("files")
 if not isinstance(values, list):
 raise CandidateContractError(f"{label}.material.files must be an array")
 files: list[WorkspaceFile] = []
 for index, value in enumerate(values):
 obj = _object(value, f"{label}.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"{label}.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"{label} contains unsupported file mode {mode:o}"
 )
Expand DownExpand Up@@ -287,7 +294,7 @@ def _profile_files(value: Any) -> tuple[ProfileFile, ...]:
 for index, value in enumerate(values):
 obj = _object(value, f"profilePlan.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"profilePlan.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"profile plan contains unsupported mode {mode:o}"
 )
Expand DownExpand Up@@ -359,26 +366,23 @@ def load_prepared_candidate_contract(
 raise CandidateContractError(
 "materialization receipt bytes do not match the expected digest"
 )
 if plan.get("schemaVersion") != 1 or plan.get("kind") != _PLAN_KIND:
 raise CandidateContractError("execution plan has the wrong schema or kind")
 if receipt.get("schemaVersion") != 1 or receipt.get("kind") != _RECEIPT_KIND:
 raise CandidateContractError(
 "materialization receipt has the wrong schema or kind"
 )
 plan = _contract_object(plan, "execution plan", kind=_PLAN_KIND)
 receipt = _contract_object(
 receipt, "materialization receipt", kind=_RECEIPT_KIND
 )
 if receipt.get("digestAlgorithm") != "rfc8785-sha256" or "digest" in receipt:
 raise CandidateContractError(
 "materialization receipt must be exact digest-free canonical material"
 )

 execution_evidence = _object(receipt.get("executionPlan"), "receipt.executionPlan")
 execution_evidence = _contract_object(
 receipt.get("executionPlan"),
 "receipt.executionPlan",
 kind=_EXECUTION_EVIDENCE_KIND,
 )
 plan_digest = _digest(
 execution_evidence.get("digest"), "receipt.executionPlan.digest"
 )
 if (
 execution_evidence.get("schemaVersion") != 1
 or execution_evidence.get("kind") != _EXECUTION_EVIDENCE_KIND
 ):
 raise CandidateContractError("receipt execution evidence has the wrong kind")
 if (
 sha256_bytes(plan_raw) != plan_digest
 or execution_evidence.get("material") != plan
Expand All@@ -402,11 +406,16 @@ def load_prepared_candidate_contract(
 execution_id = _string(plan.get("executionId"), "plan.executionId")

 task = _object(plan.get("task"), "plan.task")
 outcome = _object(task.get("outcome"), "plan.task.outcome")
 if outcome.get("kind") != "workspace":
 raise CandidateContractError("Pier requires a workspace task outcome")
 repository = _object(task.get("repository"), "plan.task.repository")
 base_commit = _git_object(
 repository.get("baseCommit"), "plan.task.repository.baseCommit"
 )
 base_tree = _git_object(repository.get("baseTree"), "plan.task.repository.baseTree")
 base_tree = _git_object(
 repository.get("baseTree"), "plan.task.repository.baseTree"
 )
 if len(base_commit) != len(base_tree):
 raise CandidateContractError("task Git objects use different hash formats")
 instruction = _instruction(task.get("instruction"))
Expand DownExpand Up@@ -471,12 +480,11 @@ def load_prepared_candidate_contract(
 if receipt.get("candidateWorkspace") != candidate_snapshot:
 raise CandidateContractError("receipt and plan candidate workspaces differ")

 profile_evidence = _object(receipt.get("profilePlan"), "receipt.profilePlan")
 if (
 profile_evidence.get("schemaVersion") != 1
 or profile_evidence.get("kind") != _PROFILE_PLAN_KIND
 ):
 raise CandidateContractError("receipt profile evidence has the wrong kind")
 profile_evidence = _contract_object(
 receipt.get("profilePlan"),
 "receipt.profilePlan",
 kind=_PROFILE_PLAN_KIND,
 )
 profile_digest = _digest(
 profile_evidence.get("digest"), "receipt.profilePlan.digest"
 )
Expand All@@ -489,7 +497,7 @@ def load_prepared_candidate_contract(
 profile_artifact_material = json.loads(profile_raw)
 except (UnicodeDecodeError, json.JSONDecodeError) as exc:
 raise CandidateContractError("profile artifact is not UTF-8 JSON") from exc
 profile_material = _object(
 profile_material = _contract_object(
 profile_evidence.get("material"), "receipt.profilePlan.material"
 )
 if profile_artifact_material != profile_material:
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat(candidate): measure exact candidates before activation by drewstone · Pull Request #547 · tangle-network/agent-runtime · GitHub
Skip to content
feat(candidate): measure exact candidates before activation#547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

drewstone
merged 14 commits into

from

Jul 17, 2026
 Merged


feat(candidate): measure exact candidates before activation
#547
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
00e85d2

feat(candidate): execute exact approved outcomes


drewstone Jul 13, 2026
ddeab85

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
9a2553d

feat(candidate): adopt exact improvement contracts


drewstone Jul 14, 2026
caa1c79

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
2cab99c

refactor(contract): remove candidate schema versions


drewstone Jul 15, 2026
b08bd50

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
cff5376

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
e81b7c1

fix(candidate): materialize approved knowledge


drewstone Jul 15, 2026
52985b5

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
72ebca0

fix(package): verify compatible runtime and bench releases


drewstone Jul 15, 2026
f4aa979

feat(intelligence): bind product metadata to measured comparisons


drewstone Jul 15, 2026
ef4c0b5

fix(intelligence): stabilize measured comparisons


drewstone Jul 15, 2026
908a4c9

Merge remote-tracking branch 'origin/main' into fix/pr547-review


drewstone Jul 17, 2026
0ca7af6

fix(candidate): bind exact experiments before activation


drewstone Jul 17, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


18 changes: 6 additions & 12 deletions
.github/workflows/ci.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,9 +42,6 @@ jobs:

 agent-bench:
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -54,16 +51,13 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install published dependencies
 - name: Install package set
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test
 - name: Build current agent-runtime
 run: pnpm run build

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify agent-bench against current agent-runtime
 run: pnpm run verify:bench

35 changes: 15 additions & 20 deletions
.github/workflows/publish.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -40,6 +40,9 @@ jobs:
 - name: Verify packed package exports
 run: pnpm run verify:package

 - name: Verify agent-bench against this release
 run: pnpm run verify:bench

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
Expand DownExpand Up@@ -98,9 +101,6 @@ jobs:
 # tag/version locking stays intact and the root package is never co-published.
 if: startsWith(github.ref, 'refs/tags/agent-bench-v')
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -110,23 +110,17 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install deps
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify packed agent-bench against published dependencies
 run: pnpm run verify:bench:published

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
 NPM_VERSION=$(node -p "require('./bench/package.json').version")
 TAG_VERSION="${GITHUB_REF#refs/tags/agent-bench-v}"
 if [ "$TAG_VERSION" != "$NPM_VERSION" ]; then
 echo "::error::Tag/version mismatch: tag=$TAG_VERSION package=$NPM_VERSION."
Expand All@@ -141,9 +135,6 @@ jobs:
 permissions:
 contents: read
 id-token: write
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -153,7 +144,7 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - run: pnpm install --frozen-lockfile

Expand All@@ -162,10 +153,14 @@ jobs:
 # Requires the npmjs Trusted Publisher on @tangle-network/agent-bench:
 # org tangle-network, repo agent-runtime, workflow publish.yml.
 npm install -g npm@11
 NAME=$(node -p "require('./package.json').name")
 VERSION=$(node -p "require('./package.json').version")
 NAME=$(node -p "require('./bench/package.json').name")
 VERSION=$(node -p "require('./bench/package.json').version")
 if npm view "$NAME@$VERSION" version >/dev/null 2>&1; then
 echo "$NAME@$VERSION already on registry; skipping publish"
 else
 npm publish --provenance --access public
 mkdir -p "$RUNNER_TEMP/agent-bench-package"
 pnpm --dir bench pack --pack-destination "$RUNNER_TEMP/agent-bench-package"
 package="$(find "$RUNNER_TEMP/agent-bench-package" -maxdepth 1 -name '*.tgz' -print -quit)"
 test -n "$package"
 npm publish "$package" --provenance --access public
 fi

20 changes: 8 additions & 12 deletions
bench/package.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -24,23 +24,19 @@
 "test": "node scripts/run-package-tests.mjs",
 "typecheck:public": "tsc -p tsconfig.public.json",
 "verify:package": "node scripts/verify-packed-consumer.mjs",
 "verify:package:local-runtime": "node scripts/verify-packed-consumer.mjs --local-runtime",
 "verify:pier": "tsx scripts/verify-pier-pair.mts"
 },
 "dependencies": {
 "@tangle-network/agent-eval": "^0.117.1",
 "@tangle-network/agent-interface": "^0.25.0",
 "@tangle-network/agent-runtime": "0.94.9",
 "@tangle-network/sandbox": "^0.10.3"
 "@tangle-network/agent-eval": "0.122.1",
 "@tangle-network/agent-interface": "0.30.0",
 "@tangle-network/agent-runtime": "workspace:*",
 "@tangle-network/sandbox": "^0.11.1"
 },
 "devDependencies": {
 "@types/node": "^25.0.0",
 "tsx": "^4.19.0",
 "typescript": "^6.0.3"
 },
 "pnpm": {
 "overrides": {
 "@tangle-network/agent-eval": "0.117.1"
 }
 "@types/node": "^25.9.3",
 "tsx": "^4.22.4",
 "typescript": "^5.7.0"
 },
 "files": [
 "src",
Expand Down

76 changes: 42 additions & 34 deletions
bench/pier_agents/candidate_contract.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -107,6 +107,17 @@ def _object(value: Any, label: str) -> dict[str, Any]:
 return value


def _contract_object(
 value: Any, label: str, *, kind: str | None = None
) -> dict[str, Any]:
 obj = _object(value, label)
 if "schemaVersion" in obj or "version" in obj:
 raise CandidateContractError(f"{label} contains obsolete version fields")
 if kind is not None and obj.get("kind") != kind:
 raise CandidateContractError(f"{label} has the wrong kind")
 return obj


def _string(value: Any, label: str) -> str:
 if not isinstance(value, str) or not value or "\0" in value:
 raise CandidateContractError(f"{label} must be a non-empty string without NUL")
Expand DownExpand Up@@ -236,26 +247,22 @@ def _embedded_bytes(value: Any, label: str) -> bytes:


def _workspace_files(value: Any, label: str) -> tuple[WorkspaceFile, ...]:
 snapshot = _object(value, label)
 if (
 snapshot.get("schemaVersion") != 1
 or snapshot.get("kind") != "agent-candidate-workspace-snapshot"
 ):
 raise CandidateContractError(f"{label} has the wrong snapshot kind")
 material = _object(snapshot.get("material"), f"{label}.material")
 if (
 material.get("schemaVersion") != 1
 or material.get("kind") != "agent-candidate-workspace-manifest"
 ):
 raise CandidateContractError(f"{label}.material has the wrong manifest kind")
 snapshot = _contract_object(
 value, label, kind="agent-candidate-workspace-snapshot"
 )
 material = _contract_object(
 snapshot.get("material"),
 f"{label}.material",
 kind="agent-candidate-workspace-manifest",
 )
 values = material.get("files")
 if not isinstance(values, list):
 raise CandidateContractError(f"{label}.material.files must be an array")
 files: list[WorkspaceFile] = []
 for index, value in enumerate(values):
 obj = _object(value, f"{label}.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"{label}.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"{label} contains unsupported file mode {mode:o}"
 )
Expand DownExpand Up@@ -287,7 +294,7 @@ def _profile_files(value: Any) -> tuple[ProfileFile, ...]:
 for index, value in enumerate(values):
 obj = _object(value, f"profilePlan.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"profilePlan.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"profile plan contains unsupported mode {mode:o}"
 )
Expand DownExpand Up@@ -359,26 +366,23 @@ def load_prepared_candidate_contract(
 raise CandidateContractError(
 "materialization receipt bytes do not match the expected digest"
 )
 if plan.get("schemaVersion") != 1 or plan.get("kind") != _PLAN_KIND:
 raise CandidateContractError("execution plan has the wrong schema or kind")
 if receipt.get("schemaVersion") != 1 or receipt.get("kind") != _RECEIPT_KIND:
 raise CandidateContractError(
 "materialization receipt has the wrong schema or kind"
 )
 plan = _contract_object(plan, "execution plan", kind=_PLAN_KIND)
 receipt = _contract_object(
 receipt, "materialization receipt", kind=_RECEIPT_KIND
 )
 if receipt.get("digestAlgorithm") != "rfc8785-sha256" or "digest" in receipt:
 raise CandidateContractError(
 "materialization receipt must be exact digest-free canonical material"
 )

 execution_evidence = _object(receipt.get("executionPlan"), "receipt.executionPlan")
 execution_evidence = _contract_object(
 receipt.get("executionPlan"),
 "receipt.executionPlan",
 kind=_EXECUTION_EVIDENCE_KIND,
 )
 plan_digest = _digest(
 execution_evidence.get("digest"), "receipt.executionPlan.digest"
 )
 if (
 execution_evidence.get("schemaVersion") != 1
 or execution_evidence.get("kind") != _EXECUTION_EVIDENCE_KIND
 ):
 raise CandidateContractError("receipt execution evidence has the wrong kind")
 if (
 sha256_bytes(plan_raw) != plan_digest
 or execution_evidence.get("material") != plan
Expand All@@ -402,11 +406,16 @@ def load_prepared_candidate_contract(
 execution_id = _string(plan.get("executionId"), "plan.executionId")

 task = _object(plan.get("task"), "plan.task")
 outcome = _object(task.get("outcome"), "plan.task.outcome")
 if outcome.get("kind") != "workspace":
 raise CandidateContractError("Pier requires a workspace task outcome")
 repository = _object(task.get("repository"), "plan.task.repository")
 base_commit = _git_object(
 repository.get("baseCommit"), "plan.task.repository.baseCommit"
 )
 base_tree = _git_object(repository.get("baseTree"), "plan.task.repository.baseTree")
 base_tree = _git_object(
 repository.get("baseTree"), "plan.task.repository.baseTree"
 )
 if len(base_commit) != len(base_tree):
 raise CandidateContractError("task Git objects use different hash formats")
 instruction = _instruction(task.get("instruction"))
Expand DownExpand Up@@ -471,12 +480,11 @@ def load_prepared_candidate_contract(
 if receipt.get("candidateWorkspace") != candidate_snapshot:
 raise CandidateContractError("receipt and plan candidate workspaces differ")

 profile_evidence = _object(receipt.get("profilePlan"), "receipt.profilePlan")
 if (
 profile_evidence.get("schemaVersion") != 1
 or profile_evidence.get("kind") != _PROFILE_PLAN_KIND
 ):
 raise CandidateContractError("receipt profile evidence has the wrong kind")
 profile_evidence = _contract_object(
 receipt.get("profilePlan"),
 "receipt.profilePlan",
 kind=_PROFILE_PLAN_KIND,
 )
 profile_digest = _digest(
 profile_evidence.get("digest"), "receipt.profilePlan.digest"
 )
Expand All@@ -489,7 +497,7 @@ def load_prepared_candidate_contract(
 profile_artifact_material = json.loads(profile_raw)
 except (UnicodeDecodeError, json.JSONDecodeError) as exc:
 raise CandidateContractError("profile artifact is not UTF-8 JSON") from exc
 profile_material = _object(
 profile_material = _contract_object(
 profile_evidence.get("material"), "receipt.profilePlan.material"
 )
 if profile_artifact_material != profile_material:
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat(candidate): measure exact candidates before activation by drewstone · Pull Request #547 · tangle-network/agent-runtime · GitHub
Skip to content
feat(candidate): measure exact candidates before activation#547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

drewstone
merged 14 commits into

from

Jul 17, 2026
 Merged


feat(candidate): measure exact candidates before activation
#547
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
00e85d2

feat(candidate): execute exact approved outcomes


drewstone Jul 13, 2026
ddeab85

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
9a2553d

feat(candidate): adopt exact improvement contracts


drewstone Jul 14, 2026
caa1c79

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
2cab99c

refactor(contract): remove candidate schema versions


drewstone Jul 15, 2026
b08bd50

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
cff5376

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
e81b7c1

fix(candidate): materialize approved knowledge


drewstone Jul 15, 2026
52985b5

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
72ebca0

fix(package): verify compatible runtime and bench releases


drewstone Jul 15, 2026
f4aa979

feat(intelligence): bind product metadata to measured comparisons


drewstone Jul 15, 2026
ef4c0b5

fix(intelligence): stabilize measured comparisons


drewstone Jul 15, 2026
908a4c9

Merge remote-tracking branch 'origin/main' into fix/pr547-review


drewstone Jul 17, 2026
0ca7af6

fix(candidate): bind exact experiments before activation


drewstone Jul 17, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


18 changes: 6 additions & 12 deletions
.github/workflows/ci.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,9 +42,6 @@ jobs:

 agent-bench:
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -54,16 +51,13 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install published dependencies
 - name: Install package set
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test
 - name: Build current agent-runtime
 run: pnpm run build

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify agent-bench against current agent-runtime
 run: pnpm run verify:bench

35 changes: 15 additions & 20 deletions
.github/workflows/publish.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -40,6 +40,9 @@ jobs:
 - name: Verify packed package exports
 run: pnpm run verify:package

 - name: Verify agent-bench against this release
 run: pnpm run verify:bench

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
Expand DownExpand Up@@ -98,9 +101,6 @@ jobs:
 # tag/version locking stays intact and the root package is never co-published.
 if: startsWith(github.ref, 'refs/tags/agent-bench-v')
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -110,23 +110,17 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install deps
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify packed agent-bench against published dependencies
 run: pnpm run verify:bench:published

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
 NPM_VERSION=$(node -p "require('./bench/package.json').version")
 TAG_VERSION="${GITHUB_REF#refs/tags/agent-bench-v}"
 if [ "$TAG_VERSION" != "$NPM_VERSION" ]; then
 echo "::error::Tag/version mismatch: tag=$TAG_VERSION package=$NPM_VERSION."
Expand All@@ -141,9 +135,6 @@ jobs:
 permissions:
 contents: read
 id-token: write
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -153,7 +144,7 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - run: pnpm install --frozen-lockfile

Expand All@@ -162,10 +153,14 @@ jobs:
 # Requires the npmjs Trusted Publisher on @tangle-network/agent-bench:
 # org tangle-network, repo agent-runtime, workflow publish.yml.
 npm install -g npm@11
 NAME=$(node -p "require('./package.json').name")
 VERSION=$(node -p "require('./package.json').version")
 NAME=$(node -p "require('./bench/package.json').name")
 VERSION=$(node -p "require('./bench/package.json').version")
 if npm view "$NAME@$VERSION" version >/dev/null 2>&1; then
 echo "$NAME@$VERSION already on registry; skipping publish"
 else
 npm publish --provenance --access public
 mkdir -p "$RUNNER_TEMP/agent-bench-package"
 pnpm --dir bench pack --pack-destination "$RUNNER_TEMP/agent-bench-package"
 package="$(find "$RUNNER_TEMP/agent-bench-package" -maxdepth 1 -name '*.tgz' -print -quit)"
 test -n "$package"
 npm publish "$package" --provenance --access public
 fi

20 changes: 8 additions & 12 deletions
bench/package.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -24,23 +24,19 @@
 "test": "node scripts/run-package-tests.mjs",
 "typecheck:public": "tsc -p tsconfig.public.json",
 "verify:package": "node scripts/verify-packed-consumer.mjs",
 "verify:package:local-runtime": "node scripts/verify-packed-consumer.mjs --local-runtime",
 "verify:pier": "tsx scripts/verify-pier-pair.mts"
 },
 "dependencies": {
 "@tangle-network/agent-eval": "^0.117.1",
 "@tangle-network/agent-interface": "^0.25.0",
 "@tangle-network/agent-runtime": "0.94.9",
 "@tangle-network/sandbox": "^0.10.3"
 "@tangle-network/agent-eval": "0.122.1",
 "@tangle-network/agent-interface": "0.30.0",
 "@tangle-network/agent-runtime": "workspace:*",
 "@tangle-network/sandbox": "^0.11.1"
 },
 "devDependencies": {
 "@types/node": "^25.0.0",
 "tsx": "^4.19.0",
 "typescript": "^6.0.3"
 },
 "pnpm": {
 "overrides": {
 "@tangle-network/agent-eval": "0.117.1"
 }
 "@types/node": "^25.9.3",
 "tsx": "^4.22.4",
 "typescript": "^5.7.0"
 },
 "files": [
 "src",
Expand Down

76 changes: 42 additions & 34 deletions
bench/pier_agents/candidate_contract.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -107,6 +107,17 @@ def _object(value: Any, label: str) -> dict[str, Any]:
 return value


def _contract_object(
 value: Any, label: str, *, kind: str | None = None
) -> dict[str, Any]:
 obj = _object(value, label)
 if "schemaVersion" in obj or "version" in obj:
 raise CandidateContractError(f"{label} contains obsolete version fields")
 if kind is not None and obj.get("kind") != kind:
 raise CandidateContractError(f"{label} has the wrong kind")
 return obj


def _string(value: Any, label: str) -> str:
 if not isinstance(value, str) or not value or "\0" in value:
 raise CandidateContractError(f"{label} must be a non-empty string without NUL")
Expand DownExpand Up@@ -236,26 +247,22 @@ def _embedded_bytes(value: Any, label: str) -> bytes:


def _workspace_files(value: Any, label: str) -> tuple[WorkspaceFile, ...]:
 snapshot = _object(value, label)
 if (
 snapshot.get("schemaVersion") != 1
 or snapshot.get("kind") != "agent-candidate-workspace-snapshot"
 ):
 raise CandidateContractError(f"{label} has the wrong snapshot kind")
 material = _object(snapshot.get("material"), f"{label}.material")
 if (
 material.get("schemaVersion") != 1
 or material.get("kind") != "agent-candidate-workspace-manifest"
 ):
 raise CandidateContractError(f"{label}.material has the wrong manifest kind")
 snapshot = _contract_object(
 value, label, kind="agent-candidate-workspace-snapshot"
 )
 material = _contract_object(
 snapshot.get("material"),
 f"{label}.material",
 kind="agent-candidate-workspace-manifest",
 )
 values = material.get("files")
 if not isinstance(values, list):
 raise CandidateContractError(f"{label}.material.files must be an array")
 files: list[WorkspaceFile] = []
 for index, value in enumerate(values):
 obj = _object(value, f"{label}.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"{label}.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"{label} contains unsupported file mode {mode:o}"
 )
Expand DownExpand Up@@ -287,7 +294,7 @@ def _profile_files(value: Any) -> tuple[ProfileFile, ...]:
 for index, value in enumerate(values):
 obj = _object(value, f"profilePlan.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"profilePlan.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"profile plan contains unsupported mode {mode:o}"
 )
Expand DownExpand Up@@ -359,26 +366,23 @@ def load_prepared_candidate_contract(
 raise CandidateContractError(
 "materialization receipt bytes do not match the expected digest"
 )
 if plan.get("schemaVersion") != 1 or plan.get("kind") != _PLAN_KIND:
 raise CandidateContractError("execution plan has the wrong schema or kind")
 if receipt.get("schemaVersion") != 1 or receipt.get("kind") != _RECEIPT_KIND:
 raise CandidateContractError(
 "materialization receipt has the wrong schema or kind"
 )
 plan = _contract_object(plan, "execution plan", kind=_PLAN_KIND)
 receipt = _contract_object(
 receipt, "materialization receipt", kind=_RECEIPT_KIND
 )
 if receipt.get("digestAlgorithm") != "rfc8785-sha256" or "digest" in receipt:
 raise CandidateContractError(
 "materialization receipt must be exact digest-free canonical material"
 )

 execution_evidence = _object(receipt.get("executionPlan"), "receipt.executionPlan")
 execution_evidence = _contract_object(
 receipt.get("executionPlan"),
 "receipt.executionPlan",
 kind=_EXECUTION_EVIDENCE_KIND,
 )
 plan_digest = _digest(
 execution_evidence.get("digest"), "receipt.executionPlan.digest"
 )
 if (
 execution_evidence.get("schemaVersion") != 1
 or execution_evidence.get("kind") != _EXECUTION_EVIDENCE_KIND
 ):
 raise CandidateContractError("receipt execution evidence has the wrong kind")
 if (
 sha256_bytes(plan_raw) != plan_digest
 or execution_evidence.get("material") != plan
Expand All@@ -402,11 +406,16 @@ def load_prepared_candidate_contract(
 execution_id = _string(plan.get("executionId"), "plan.executionId")

 task = _object(plan.get("task"), "plan.task")
 outcome = _object(task.get("outcome"), "plan.task.outcome")
 if outcome.get("kind") != "workspace":
 raise CandidateContractError("Pier requires a workspace task outcome")
 repository = _object(task.get("repository"), "plan.task.repository")
 base_commit = _git_object(
 repository.get("baseCommit"), "plan.task.repository.baseCommit"
 )
 base_tree = _git_object(repository.get("baseTree"), "plan.task.repository.baseTree")
 base_tree = _git_object(
 repository.get("baseTree"), "plan.task.repository.baseTree"
 )
 if len(base_commit) != len(base_tree):
 raise CandidateContractError("task Git objects use different hash formats")
 instruction = _instruction(task.get("instruction"))
Expand DownExpand Up@@ -471,12 +480,11 @@ def load_prepared_candidate_contract(
 if receipt.get("candidateWorkspace") != candidate_snapshot:
 raise CandidateContractError("receipt and plan candidate workspaces differ")

 profile_evidence = _object(receipt.get("profilePlan"), "receipt.profilePlan")
 if (
 profile_evidence.get("schemaVersion") != 1
 or profile_evidence.get("kind") != _PROFILE_PLAN_KIND
 ):
 raise CandidateContractError("receipt profile evidence has the wrong kind")
 profile_evidence = _contract_object(
 receipt.get("profilePlan"),
 "receipt.profilePlan",
 kind=_PROFILE_PLAN_KIND,
 )
 profile_digest = _digest(
 profile_evidence.get("digest"), "receipt.profilePlan.digest"
 )
Expand All@@ -489,7 +497,7 @@ def load_prepared_candidate_contract(
 profile_artifact_material = json.loads(profile_raw)
 except (UnicodeDecodeError, json.JSONDecodeError) as exc:
 raise CandidateContractError("profile artifact is not UTF-8 JSON") from exc
 profile_material = _object(
 profile_material = _contract_object(
 profile_evidence.get("material"), "receipt.profilePlan.material"
 )
 if profile_artifact_material != profile_material:
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
feat(candidate): measure exact candidates before activation by drewstone · Pull Request #547 · tangle-network/agent-runtime · GitHub
Skip to content
feat(candidate): measure exact candidates before activation#547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

drewstone
merged 14 commits into

from

Jul 17, 2026
 Merged


feat(candidate): measure exact candidates before activation
#547
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
00e85d2

feat(candidate): execute exact approved outcomes


drewstone Jul 13, 2026
ddeab85

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
9a2553d

feat(candidate): adopt exact improvement contracts


drewstone Jul 14, 2026
caa1c79

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
2cab99c

refactor(contract): remove candidate schema versions


drewstone Jul 15, 2026
b08bd50

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
cff5376

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
e81b7c1

fix(candidate): materialize approved knowledge


drewstone Jul 15, 2026
52985b5

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
72ebca0

fix(package): verify compatible runtime and bench releases


drewstone Jul 15, 2026
f4aa979

feat(intelligence): bind product metadata to measured comparisons


drewstone Jul 15, 2026
ef4c0b5

fix(intelligence): stabilize measured comparisons


drewstone Jul 15, 2026
908a4c9

Merge remote-tracking branch 'origin/main' into fix/pr547-review


drewstone Jul 17, 2026
0ca7af6

fix(candidate): bind exact experiments before activation


drewstone Jul 17, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


18 changes: 6 additions & 12 deletions
.github/workflows/ci.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,9 +42,6 @@ jobs:

 agent-bench:
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -54,16 +51,13 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install published dependencies
 - name: Install package set
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test
 - name: Build current agent-runtime
 run: pnpm run build

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify agent-bench against current agent-runtime
 run: pnpm run verify:bench

35 changes: 15 additions & 20 deletions
.github/workflows/publish.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -40,6 +40,9 @@ jobs:
 - name: Verify packed package exports
 run: pnpm run verify:package

 - name: Verify agent-bench against this release
 run: pnpm run verify:bench

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
Expand DownExpand Up@@ -98,9 +101,6 @@ jobs:
 # tag/version locking stays intact and the root package is never co-published.
 if: startsWith(github.ref, 'refs/tags/agent-bench-v')
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -110,23 +110,17 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install deps
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify packed agent-bench against published dependencies
 run: pnpm run verify:bench:published

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
 NPM_VERSION=$(node -p "require('./bench/package.json').version")
 TAG_VERSION="${GITHUB_REF#refs/tags/agent-bench-v}"
 if [ "$TAG_VERSION" != "$NPM_VERSION" ]; then
 echo "::error::Tag/version mismatch: tag=$TAG_VERSION package=$NPM_VERSION."
Expand All@@ -141,9 +135,6 @@ jobs:
 permissions:
 contents: read
 id-token: write
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -153,7 +144,7 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - run: pnpm install --frozen-lockfile

Expand All@@ -162,10 +153,14 @@ jobs:
 # Requires the npmjs Trusted Publisher on @tangle-network/agent-bench:
 # org tangle-network, repo agent-runtime, workflow publish.yml.
 npm install -g npm@11
 NAME=$(node -p "require('./package.json').name")
 VERSION=$(node -p "require('./package.json').version")
 NAME=$(node -p "require('./bench/package.json').name")
 VERSION=$(node -p "require('./bench/package.json').version")
 if npm view "$NAME@$VERSION" version >/dev/null 2>&1; then
 echo "$NAME@$VERSION already on registry; skipping publish"
 else
 npm publish --provenance --access public
 mkdir -p "$RUNNER_TEMP/agent-bench-package"
 pnpm --dir bench pack --pack-destination "$RUNNER_TEMP/agent-bench-package"
 package="$(find "$RUNNER_TEMP/agent-bench-package" -maxdepth 1 -name '*.tgz' -print -quit)"
 test -n "$package"
 npm publish "$package" --provenance --access public
 fi

20 changes: 8 additions & 12 deletions
bench/package.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -24,23 +24,19 @@
 "test": "node scripts/run-package-tests.mjs",
 "typecheck:public": "tsc -p tsconfig.public.json",
 "verify:package": "node scripts/verify-packed-consumer.mjs",
 "verify:package:local-runtime": "node scripts/verify-packed-consumer.mjs --local-runtime",
 "verify:pier": "tsx scripts/verify-pier-pair.mts"
 },
 "dependencies": {
 "@tangle-network/agent-eval": "^0.117.1",
 "@tangle-network/agent-interface": "^0.25.0",
 "@tangle-network/agent-runtime": "0.94.9",
 "@tangle-network/sandbox": "^0.10.3"
 "@tangle-network/agent-eval": "0.122.1",
 "@tangle-network/agent-interface": "0.30.0",
 "@tangle-network/agent-runtime": "workspace:*",
 "@tangle-network/sandbox": "^0.11.1"
 },
 "devDependencies": {
 "@types/node": "^25.0.0",
 "tsx": "^4.19.0",
 "typescript": "^6.0.3"
 },
 "pnpm": {
 "overrides": {
 "@tangle-network/agent-eval": "0.117.1"
 }
 "@types/node": "^25.9.3",
 "tsx": "^4.22.4",
 "typescript": "^5.7.0"
 },
 "files": [
 "src",
Expand Down

76 changes: 42 additions & 34 deletions
bench/pier_agents/candidate_contract.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -107,6 +107,17 @@ def _object(value: Any, label: str) -> dict[str, Any]:
 return value


def _contract_object(
 value: Any, label: str, *, kind: str | None = None
) -> dict[str, Any]:
 obj = _object(value, label)
 if "schemaVersion" in obj or "version" in obj:
 raise CandidateContractError(f"{label} contains obsolete version fields")
 if kind is not None and obj.get("kind") != kind:
 raise CandidateContractError(f"{label} has the wrong kind")
 return obj


def _string(value: Any, label: str) -> str:
 if not isinstance(value, str) or not value or "\0" in value:
 raise CandidateContractError(f"{label} must be a non-empty string without NUL")
Expand DownExpand Up@@ -236,26 +247,22 @@ def _embedded_bytes(value: Any, label: str) -> bytes:


def _workspace_files(value: Any, label: str) -> tuple[WorkspaceFile, ...]:
 snapshot = _object(value, label)
 if (
 snapshot.get("schemaVersion") != 1
 or snapshot.get("kind") != "agent-candidate-workspace-snapshot"
 ):
 raise CandidateContractError(f"{label} has the wrong snapshot kind")
 material = _object(snapshot.get("material"), f"{label}.material")
 if (
 material.get("schemaVersion") != 1
 or material.get("kind") != "agent-candidate-workspace-manifest"
 ):
 raise CandidateContractError(f"{label}.material has the wrong manifest kind")
 snapshot = _contract_object(
 value, label, kind="agent-candidate-workspace-snapshot"
 )
 material = _contract_object(
 snapshot.get("material"),
 f"{label}.material",
 kind="agent-candidate-workspace-manifest",
 )
 values = material.get("files")
 if not isinstance(values, list):
 raise CandidateContractError(f"{label}.material.files must be an array")
 files: list[WorkspaceFile] = []
 for index, value in enumerate(values):
 obj = _object(value, f"{label}.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"{label}.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"{label} contains unsupported file mode {mode:o}"
 )
Expand DownExpand Up@@ -287,7 +294,7 @@ def _profile_files(value: Any) -> tuple[ProfileFile, ...]:
 for index, value in enumerate(values):
 obj = _object(value, f"profilePlan.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"profilePlan.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"profile plan contains unsupported mode {mode:o}"
 )
Expand DownExpand Up@@ -359,26 +366,23 @@ def load_prepared_candidate_contract(
 raise CandidateContractError(
 "materialization receipt bytes do not match the expected digest"
 )
 if plan.get("schemaVersion") != 1 or plan.get("kind") != _PLAN_KIND:
 raise CandidateContractError("execution plan has the wrong schema or kind")
 if receipt.get("schemaVersion") != 1 or receipt.get("kind") != _RECEIPT_KIND:
 raise CandidateContractError(
 "materialization receipt has the wrong schema or kind"
 )
 plan = _contract_object(plan, "execution plan", kind=_PLAN_KIND)
 receipt = _contract_object(
 receipt, "materialization receipt", kind=_RECEIPT_KIND
 )
 if receipt.get("digestAlgorithm") != "rfc8785-sha256" or "digest" in receipt:
 raise CandidateContractError(
 "materialization receipt must be exact digest-free canonical material"
 )

 execution_evidence = _object(receipt.get("executionPlan"), "receipt.executionPlan")
 execution_evidence = _contract_object(
 receipt.get("executionPlan"),
 "receipt.executionPlan",
 kind=_EXECUTION_EVIDENCE_KIND,
 )
 plan_digest = _digest(
 execution_evidence.get("digest"), "receipt.executionPlan.digest"
 )
 if (
 execution_evidence.get("schemaVersion") != 1
 or execution_evidence.get("kind") != _EXECUTION_EVIDENCE_KIND
 ):
 raise CandidateContractError("receipt execution evidence has the wrong kind")
 if (
 sha256_bytes(plan_raw) != plan_digest
 or execution_evidence.get("material") != plan
Expand All@@ -402,11 +406,16 @@ def load_prepared_candidate_contract(
 execution_id = _string(plan.get("executionId"), "plan.executionId")

 task = _object(plan.get("task"), "plan.task")
 outcome = _object(task.get("outcome"), "plan.task.outcome")
 if outcome.get("kind") != "workspace":
 raise CandidateContractError("Pier requires a workspace task outcome")
 repository = _object(task.get("repository"), "plan.task.repository")
 base_commit = _git_object(
 repository.get("baseCommit"), "plan.task.repository.baseCommit"
 )
 base_tree = _git_object(repository.get("baseTree"), "plan.task.repository.baseTree")
 base_tree = _git_object(
 repository.get("baseTree"), "plan.task.repository.baseTree"
 )
 if len(base_commit) != len(base_tree):
 raise CandidateContractError("task Git objects use different hash formats")
 instruction = _instruction(task.get("instruction"))
Expand DownExpand Up@@ -471,12 +480,11 @@ def load_prepared_candidate_contract(
 if receipt.get("candidateWorkspace") != candidate_snapshot:
 raise CandidateContractError("receipt and plan candidate workspaces differ")

 profile_evidence = _object(receipt.get("profilePlan"), "receipt.profilePlan")
 if (
 profile_evidence.get("schemaVersion") != 1
 or profile_evidence.get("kind") != _PROFILE_PLAN_KIND
 ):
 raise CandidateContractError("receipt profile evidence has the wrong kind")
 profile_evidence = _contract_object(
 receipt.get("profilePlan"),
 "receipt.profilePlan",
 kind=_PROFILE_PLAN_KIND,
 )
 profile_digest = _digest(
 profile_evidence.get("digest"), "receipt.profilePlan.digest"
 )
Expand All@@ -489,7 +497,7 @@ def load_prepared_candidate_contract(
 profile_artifact_material = json.loads(profile_raw)
 except (UnicodeDecodeError, json.JSONDecodeError) as exc:
 raise CandidateContractError("profile artifact is not UTF-8 JSON") from exc
 profile_material = _object(
 profile_material = _contract_object(
 profile_evidence.get("material"), "receipt.profilePlan.material"
 )
 if profile_artifact_material != profile_material:
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat(candidate): measure exact candidates before activation by drewstone · Pull Request #547 · tangle-network/agent-runtime · GitHub
Skip to content
feat(candidate): measure exact candidates before activation#547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

drewstone
merged 14 commits into

from

Jul 17, 2026
 Merged


feat(candidate): measure exact candidates before activation
#547
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
00e85d2

feat(candidate): execute exact approved outcomes


drewstone Jul 13, 2026
ddeab85

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
9a2553d

feat(candidate): adopt exact improvement contracts


drewstone Jul 14, 2026
caa1c79

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
2cab99c

refactor(contract): remove candidate schema versions


drewstone Jul 15, 2026
b08bd50

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
cff5376

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
e81b7c1

fix(candidate): materialize approved knowledge


drewstone Jul 15, 2026
52985b5

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
72ebca0

fix(package): verify compatible runtime and bench releases


drewstone Jul 15, 2026
f4aa979

feat(intelligence): bind product metadata to measured comparisons


drewstone Jul 15, 2026
ef4c0b5

fix(intelligence): stabilize measured comparisons


drewstone Jul 15, 2026
908a4c9

Merge remote-tracking branch 'origin/main' into fix/pr547-review


drewstone Jul 17, 2026
0ca7af6

fix(candidate): bind exact experiments before activation


drewstone Jul 17, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


18 changes: 6 additions & 12 deletions
.github/workflows/ci.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,9 +42,6 @@ jobs:

 agent-bench:
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -54,16 +51,13 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install published dependencies
 - name: Install package set
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test
 - name: Build current agent-runtime
 run: pnpm run build

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify agent-bench against current agent-runtime
 run: pnpm run verify:bench

35 changes: 15 additions & 20 deletions
.github/workflows/publish.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -40,6 +40,9 @@ jobs:
 - name: Verify packed package exports
 run: pnpm run verify:package

 - name: Verify agent-bench against this release
 run: pnpm run verify:bench

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
Expand DownExpand Up@@ -98,9 +101,6 @@ jobs:
 # tag/version locking stays intact and the root package is never co-published.
 if: startsWith(github.ref, 'refs/tags/agent-bench-v')
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -110,23 +110,17 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install deps
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify packed agent-bench against published dependencies
 run: pnpm run verify:bench:published

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
 NPM_VERSION=$(node -p "require('./bench/package.json').version")
 TAG_VERSION="${GITHUB_REF#refs/tags/agent-bench-v}"
 if [ "$TAG_VERSION" != "$NPM_VERSION" ]; then
 echo "::error::Tag/version mismatch: tag=$TAG_VERSION package=$NPM_VERSION."
Expand All@@ -141,9 +135,6 @@ jobs:
 permissions:
 contents: read
 id-token: write
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -153,7 +144,7 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - run: pnpm install --frozen-lockfile

Expand All@@ -162,10 +153,14 @@ jobs:
 # Requires the npmjs Trusted Publisher on @tangle-network/agent-bench:
 # org tangle-network, repo agent-runtime, workflow publish.yml.
 npm install -g npm@11
 NAME=$(node -p "require('./package.json').name")
 VERSION=$(node -p "require('./package.json').version")
 NAME=$(node -p "require('./bench/package.json').name")
 VERSION=$(node -p "require('./bench/package.json').version")
 if npm view "$NAME@$VERSION" version >/dev/null 2>&1; then
 echo "$NAME@$VERSION already on registry; skipping publish"
 else
 npm publish --provenance --access public
 mkdir -p "$RUNNER_TEMP/agent-bench-package"
 pnpm --dir bench pack --pack-destination "$RUNNER_TEMP/agent-bench-package"
 package="$(find "$RUNNER_TEMP/agent-bench-package" -maxdepth 1 -name '*.tgz' -print -quit)"
 test -n "$package"
 npm publish "$package" --provenance --access public
 fi

20 changes: 8 additions & 12 deletions
bench/package.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -24,23 +24,19 @@
 "test": "node scripts/run-package-tests.mjs",
 "typecheck:public": "tsc -p tsconfig.public.json",
 "verify:package": "node scripts/verify-packed-consumer.mjs",
 "verify:package:local-runtime": "node scripts/verify-packed-consumer.mjs --local-runtime",
 "verify:pier": "tsx scripts/verify-pier-pair.mts"
 },
 "dependencies": {
 "@tangle-network/agent-eval": "^0.117.1",
 "@tangle-network/agent-interface": "^0.25.0",
 "@tangle-network/agent-runtime": "0.94.9",
 "@tangle-network/sandbox": "^0.10.3"
 "@tangle-network/agent-eval": "0.122.1",
 "@tangle-network/agent-interface": "0.30.0",
 "@tangle-network/agent-runtime": "workspace:*",
 "@tangle-network/sandbox": "^0.11.1"
 },
 "devDependencies": {
 "@types/node": "^25.0.0",
 "tsx": "^4.19.0",
 "typescript": "^6.0.3"
 },
 "pnpm": {
 "overrides": {
 "@tangle-network/agent-eval": "0.117.1"
 }
 "@types/node": "^25.9.3",
 "tsx": "^4.22.4",
 "typescript": "^5.7.0"
 },
 "files": [
 "src",
Expand Down

76 changes: 42 additions & 34 deletions
bench/pier_agents/candidate_contract.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -107,6 +107,17 @@ def _object(value: Any, label: str) -> dict[str, Any]:
 return value


def _contract_object(
 value: Any, label: str, *, kind: str | None = None
) -> dict[str, Any]:
 obj = _object(value, label)
 if "schemaVersion" in obj or "version" in obj:
 raise CandidateContractError(f"{label} contains obsolete version fields")
 if kind is not None and obj.get("kind") != kind:
 raise CandidateContractError(f"{label} has the wrong kind")
 return obj


def _string(value: Any, label: str) -> str:
 if not isinstance(value, str) or not value or "\0" in value:
 raise CandidateContractError(f"{label} must be a non-empty string without NUL")
Expand DownExpand Up@@ -236,26 +247,22 @@ def _embedded_bytes(value: Any, label: str) -> bytes:


def _workspace_files(value: Any, label: str) -> tuple[WorkspaceFile, ...]:
 snapshot = _object(value, label)
 if (
 snapshot.get("schemaVersion") != 1
 or snapshot.get("kind") != "agent-candidate-workspace-snapshot"
 ):
 raise CandidateContractError(f"{label} has the wrong snapshot kind")
 material = _object(snapshot.get("material"), f"{label}.material")
 if (
 material.get("schemaVersion") != 1
 or material.get("kind") != "agent-candidate-workspace-manifest"
 ):
 raise CandidateContractError(f"{label}.material has the wrong manifest kind")
 snapshot = _contract_object(
 value, label, kind="agent-candidate-workspace-snapshot"
 )
 material = _contract_object(
 snapshot.get("material"),
 f"{label}.material",
 kind="agent-candidate-workspace-manifest",
 )
 values = material.get("files")
 if not isinstance(values, list):
 raise CandidateContractError(f"{label}.material.files must be an array")
 files: list[WorkspaceFile] = []
 for index, value in enumerate(values):
 obj = _object(value, f"{label}.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"{label}.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"{label} contains unsupported file mode {mode:o}"
 )
Expand DownExpand Up@@ -287,7 +294,7 @@ def _profile_files(value: Any) -> tuple[ProfileFile, ...]:
 for index, value in enumerate(values):
 obj = _object(value, f"profilePlan.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"profilePlan.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"profile plan contains unsupported mode {mode:o}"
 )
Expand DownExpand Up@@ -359,26 +366,23 @@ def load_prepared_candidate_contract(
 raise CandidateContractError(
 "materialization receipt bytes do not match the expected digest"
 )
 if plan.get("schemaVersion") != 1 or plan.get("kind") != _PLAN_KIND:
 raise CandidateContractError("execution plan has the wrong schema or kind")
 if receipt.get("schemaVersion") != 1 or receipt.get("kind") != _RECEIPT_KIND:
 raise CandidateContractError(
 "materialization receipt has the wrong schema or kind"
 )
 plan = _contract_object(plan, "execution plan", kind=_PLAN_KIND)
 receipt = _contract_object(
 receipt, "materialization receipt", kind=_RECEIPT_KIND
 )
 if receipt.get("digestAlgorithm") != "rfc8785-sha256" or "digest" in receipt:
 raise CandidateContractError(
 "materialization receipt must be exact digest-free canonical material"
 )

 execution_evidence = _object(receipt.get("executionPlan"), "receipt.executionPlan")
 execution_evidence = _contract_object(
 receipt.get("executionPlan"),
 "receipt.executionPlan",
 kind=_EXECUTION_EVIDENCE_KIND,
 )
 plan_digest = _digest(
 execution_evidence.get("digest"), "receipt.executionPlan.digest"
 )
 if (
 execution_evidence.get("schemaVersion") != 1
 or execution_evidence.get("kind") != _EXECUTION_EVIDENCE_KIND
 ):
 raise CandidateContractError("receipt execution evidence has the wrong kind")
 if (
 sha256_bytes(plan_raw) != plan_digest
 or execution_evidence.get("material") != plan
Expand All@@ -402,11 +406,16 @@ def load_prepared_candidate_contract(
 execution_id = _string(plan.get("executionId"), "plan.executionId")

 task = _object(plan.get("task"), "plan.task")
 outcome = _object(task.get("outcome"), "plan.task.outcome")
 if outcome.get("kind") != "workspace":
 raise CandidateContractError("Pier requires a workspace task outcome")
 repository = _object(task.get("repository"), "plan.task.repository")
 base_commit = _git_object(
 repository.get("baseCommit"), "plan.task.repository.baseCommit"
 )
 base_tree = _git_object(repository.get("baseTree"), "plan.task.repository.baseTree")
 base_tree = _git_object(
 repository.get("baseTree"), "plan.task.repository.baseTree"
 )
 if len(base_commit) != len(base_tree):
 raise CandidateContractError("task Git objects use different hash formats")
 instruction = _instruction(task.get("instruction"))
Expand DownExpand Up@@ -471,12 +480,11 @@ def load_prepared_candidate_contract(
 if receipt.get("candidateWorkspace") != candidate_snapshot:
 raise CandidateContractError("receipt and plan candidate workspaces differ")

 profile_evidence = _object(receipt.get("profilePlan"), "receipt.profilePlan")
 if (
 profile_evidence.get("schemaVersion") != 1
 or profile_evidence.get("kind") != _PROFILE_PLAN_KIND
 ):
 raise CandidateContractError("receipt profile evidence has the wrong kind")
 profile_evidence = _contract_object(
 receipt.get("profilePlan"),
 "receipt.profilePlan",
 kind=_PROFILE_PLAN_KIND,
 )
 profile_digest = _digest(
 profile_evidence.get("digest"), "receipt.profilePlan.digest"
 )
Expand All@@ -489,7 +497,7 @@ def load_prepared_candidate_contract(
 profile_artifact_material = json.loads(profile_raw)
 except (UnicodeDecodeError, json.JSONDecodeError) as exc:
 raise CandidateContractError("profile artifact is not UTF-8 JSON") from exc
 profile_material = _object(
 profile_material = _contract_object(
 profile_evidence.get("material"), "receipt.profilePlan.material"
 )
 if profile_artifact_material != profile_material:
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat(candidate): measure exact candidates before activation by drewstone · Pull Request #547 · tangle-network/agent-runtime · GitHub
Skip to content
feat(candidate): measure exact candidates before activation#547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

drewstone
merged 14 commits into

from

Jul 17, 2026
 Merged


feat(candidate): measure exact candidates before activation
#547
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
00e85d2

feat(candidate): execute exact approved outcomes


drewstone Jul 13, 2026
ddeab85

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
9a2553d

feat(candidate): adopt exact improvement contracts


drewstone Jul 14, 2026
caa1c79

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
2cab99c

refactor(contract): remove candidate schema versions


drewstone Jul 15, 2026
b08bd50

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
cff5376

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
e81b7c1

fix(candidate): materialize approved knowledge


drewstone Jul 15, 2026
52985b5

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
72ebca0

fix(package): verify compatible runtime and bench releases


drewstone Jul 15, 2026
f4aa979

feat(intelligence): bind product metadata to measured comparisons


drewstone Jul 15, 2026
ef4c0b5

fix(intelligence): stabilize measured comparisons


drewstone Jul 15, 2026
908a4c9

Merge remote-tracking branch 'origin/main' into fix/pr547-review


drewstone Jul 17, 2026
0ca7af6

fix(candidate): bind exact experiments before activation


drewstone Jul 17, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


18 changes: 6 additions & 12 deletions
.github/workflows/ci.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,9 +42,6 @@ jobs:

 agent-bench:
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -54,16 +51,13 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install published dependencies
 - name: Install package set
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test
 - name: Build current agent-runtime
 run: pnpm run build

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify agent-bench against current agent-runtime
 run: pnpm run verify:bench

35 changes: 15 additions & 20 deletions
.github/workflows/publish.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -40,6 +40,9 @@ jobs:
 - name: Verify packed package exports
 run: pnpm run verify:package

 - name: Verify agent-bench against this release
 run: pnpm run verify:bench

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
Expand DownExpand Up@@ -98,9 +101,6 @@ jobs:
 # tag/version locking stays intact and the root package is never co-published.
 if: startsWith(github.ref, 'refs/tags/agent-bench-v')
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -110,23 +110,17 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install deps
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify packed agent-bench against published dependencies
 run: pnpm run verify:bench:published

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
 NPM_VERSION=$(node -p "require('./bench/package.json').version")
 TAG_VERSION="${GITHUB_REF#refs/tags/agent-bench-v}"
 if [ "$TAG_VERSION" != "$NPM_VERSION" ]; then
 echo "::error::Tag/version mismatch: tag=$TAG_VERSION package=$NPM_VERSION."
Expand All@@ -141,9 +135,6 @@ jobs:
 permissions:
 contents: read
 id-token: write
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -153,7 +144,7 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - run: pnpm install --frozen-lockfile

Expand All@@ -162,10 +153,14 @@ jobs:
 # Requires the npmjs Trusted Publisher on @tangle-network/agent-bench:
 # org tangle-network, repo agent-runtime, workflow publish.yml.
 npm install -g npm@11
 NAME=$(node -p "require('./package.json').name")
 VERSION=$(node -p "require('./package.json').version")
 NAME=$(node -p "require('./bench/package.json').name")
 VERSION=$(node -p "require('./bench/package.json').version")
 if npm view "$NAME@$VERSION" version >/dev/null 2>&1; then
 echo "$NAME@$VERSION already on registry; skipping publish"
 else
 npm publish --provenance --access public
 mkdir -p "$RUNNER_TEMP/agent-bench-package"
 pnpm --dir bench pack --pack-destination "$RUNNER_TEMP/agent-bench-package"
 package="$(find "$RUNNER_TEMP/agent-bench-package" -maxdepth 1 -name '*.tgz' -print -quit)"
 test -n "$package"
 npm publish "$package" --provenance --access public
 fi

20 changes: 8 additions & 12 deletions
bench/package.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -24,23 +24,19 @@
 "test": "node scripts/run-package-tests.mjs",
 "typecheck:public": "tsc -p tsconfig.public.json",
 "verify:package": "node scripts/verify-packed-consumer.mjs",
 "verify:package:local-runtime": "node scripts/verify-packed-consumer.mjs --local-runtime",
 "verify:pier": "tsx scripts/verify-pier-pair.mts"
 },
 "dependencies": {
 "@tangle-network/agent-eval": "^0.117.1",
 "@tangle-network/agent-interface": "^0.25.0",
 "@tangle-network/agent-runtime": "0.94.9",
 "@tangle-network/sandbox": "^0.10.3"
 "@tangle-network/agent-eval": "0.122.1",
 "@tangle-network/agent-interface": "0.30.0",
 "@tangle-network/agent-runtime": "workspace:*",
 "@tangle-network/sandbox": "^0.11.1"
 },
 "devDependencies": {
 "@types/node": "^25.0.0",
 "tsx": "^4.19.0",
 "typescript": "^6.0.3"
 },
 "pnpm": {
 "overrides": {
 "@tangle-network/agent-eval": "0.117.1"
 }
 "@types/node": "^25.9.3",
 "tsx": "^4.22.4",
 "typescript": "^5.7.0"
 },
 "files": [
 "src",
Expand Down

76 changes: 42 additions & 34 deletions
bench/pier_agents/candidate_contract.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -107,6 +107,17 @@ def _object(value: Any, label: str) -> dict[str, Any]:
 return value


def _contract_object(
 value: Any, label: str, *, kind: str | None = None
) -> dict[str, Any]:
 obj = _object(value, label)
 if "schemaVersion" in obj or "version" in obj:
 raise CandidateContractError(f"{label} contains obsolete version fields")
 if kind is not None and obj.get("kind") != kind:
 raise CandidateContractError(f"{label} has the wrong kind")
 return obj


def _string(value: Any, label: str) -> str:
 if not isinstance(value, str) or not value or "\0" in value:
 raise CandidateContractError(f"{label} must be a non-empty string without NUL")
Expand DownExpand Up@@ -236,26 +247,22 @@ def _embedded_bytes(value: Any, label: str) -> bytes:


def _workspace_files(value: Any, label: str) -> tuple[WorkspaceFile, ...]:
 snapshot = _object(value, label)
 if (
 snapshot.get("schemaVersion") != 1
 or snapshot.get("kind") != "agent-candidate-workspace-snapshot"
 ):
 raise CandidateContractError(f"{label} has the wrong snapshot kind")
 material = _object(snapshot.get("material"), f"{label}.material")
 if (
 material.get("schemaVersion") != 1
 or material.get("kind") != "agent-candidate-workspace-manifest"
 ):
 raise CandidateContractError(f"{label}.material has the wrong manifest kind")
 snapshot = _contract_object(
 value, label, kind="agent-candidate-workspace-snapshot"
 )
 material = _contract_object(
 snapshot.get("material"),
 f"{label}.material",
 kind="agent-candidate-workspace-manifest",
 )
 values = material.get("files")
 if not isinstance(values, list):
 raise CandidateContractError(f"{label}.material.files must be an array")
 files: list[WorkspaceFile] = []
 for index, value in enumerate(values):
 obj = _object(value, f"{label}.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"{label}.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"{label} contains unsupported file mode {mode:o}"
 )
Expand DownExpand Up@@ -287,7 +294,7 @@ def _profile_files(value: Any) -> tuple[ProfileFile, ...]:
 for index, value in enumerate(values):
 obj = _object(value, f"profilePlan.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"profilePlan.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"profile plan contains unsupported mode {mode:o}"
 )
Expand DownExpand Up@@ -359,26 +366,23 @@ def load_prepared_candidate_contract(
 raise CandidateContractError(
 "materialization receipt bytes do not match the expected digest"
 )
 if plan.get("schemaVersion") != 1 or plan.get("kind") != _PLAN_KIND:
 raise CandidateContractError("execution plan has the wrong schema or kind")
 if receipt.get("schemaVersion") != 1 or receipt.get("kind") != _RECEIPT_KIND:
 raise CandidateContractError(
 "materialization receipt has the wrong schema or kind"
 )
 plan = _contract_object(plan, "execution plan", kind=_PLAN_KIND)
 receipt = _contract_object(
 receipt, "materialization receipt", kind=_RECEIPT_KIND
 )
 if receipt.get("digestAlgorithm") != "rfc8785-sha256" or "digest" in receipt:
 raise CandidateContractError(
 "materialization receipt must be exact digest-free canonical material"
 )

 execution_evidence = _object(receipt.get("executionPlan"), "receipt.executionPlan")
 execution_evidence = _contract_object(
 receipt.get("executionPlan"),
 "receipt.executionPlan",
 kind=_EXECUTION_EVIDENCE_KIND,
 )
 plan_digest = _digest(
 execution_evidence.get("digest"), "receipt.executionPlan.digest"
 )
 if (
 execution_evidence.get("schemaVersion") != 1
 or execution_evidence.get("kind") != _EXECUTION_EVIDENCE_KIND
 ):
 raise CandidateContractError("receipt execution evidence has the wrong kind")
 if (
 sha256_bytes(plan_raw) != plan_digest
 or execution_evidence.get("material") != plan
Expand All@@ -402,11 +406,16 @@ def load_prepared_candidate_contract(
 execution_id = _string(plan.get("executionId"), "plan.executionId")

 task = _object(plan.get("task"), "plan.task")
 outcome = _object(task.get("outcome"), "plan.task.outcome")
 if outcome.get("kind") != "workspace":
 raise CandidateContractError("Pier requires a workspace task outcome")
 repository = _object(task.get("repository"), "plan.task.repository")
 base_commit = _git_object(
 repository.get("baseCommit"), "plan.task.repository.baseCommit"
 )
 base_tree = _git_object(repository.get("baseTree"), "plan.task.repository.baseTree")
 base_tree = _git_object(
 repository.get("baseTree"), "plan.task.repository.baseTree"
 )
 if len(base_commit) != len(base_tree):
 raise CandidateContractError("task Git objects use different hash formats")
 instruction = _instruction(task.get("instruction"))
Expand DownExpand Up@@ -471,12 +480,11 @@ def load_prepared_candidate_contract(
 if receipt.get("candidateWorkspace") != candidate_snapshot:
 raise CandidateContractError("receipt and plan candidate workspaces differ")

 profile_evidence = _object(receipt.get("profilePlan"), "receipt.profilePlan")
 if (
 profile_evidence.get("schemaVersion") != 1
 or profile_evidence.get("kind") != _PROFILE_PLAN_KIND
 ):
 raise CandidateContractError("receipt profile evidence has the wrong kind")
 profile_evidence = _contract_object(
 receipt.get("profilePlan"),
 "receipt.profilePlan",
 kind=_PROFILE_PLAN_KIND,
 )
 profile_digest = _digest(
 profile_evidence.get("digest"), "receipt.profilePlan.digest"
 )
Expand All@@ -489,7 +497,7 @@ def load_prepared_candidate_contract(
 profile_artifact_material = json.loads(profile_raw)
 except (UnicodeDecodeError, json.JSONDecodeError) as exc:
 raise CandidateContractError("profile artifact is not UTF-8 JSON") from exc
 profile_material = _object(
 profile_material = _contract_object(
 profile_evidence.get("material"), "receipt.profilePlan.material"
 )
 if profile_artifact_material != profile_material:
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
feat(candidate): measure exact candidates before activation by drewstone · Pull Request #547 · tangle-network/agent-runtime · GitHub
Skip to content
feat(candidate): measure exact candidates before activation#547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

drewstone
merged 14 commits into

from

Jul 17, 2026
 Merged


feat(candidate): measure exact candidates before activation
#547
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
00e85d2

feat(candidate): execute exact approved outcomes


drewstone Jul 13, 2026
ddeab85

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
9a2553d

feat(candidate): adopt exact improvement contracts


drewstone Jul 14, 2026
caa1c79

Merge remote-tracking branch 'origin/main' into feat/universal-candid…


drewstone Jul 14, 2026
2cab99c

refactor(contract): remove candidate schema versions


drewstone Jul 15, 2026
b08bd50

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
cff5376

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
e81b7c1

fix(candidate): materialize approved knowledge


drewstone Jul 15, 2026
52985b5

Merge remote-tracking branch 'origin/main' into test/unversion-candid…


drewstone Jul 15, 2026
72ebca0

fix(package): verify compatible runtime and bench releases


drewstone Jul 15, 2026
f4aa979

feat(intelligence): bind product metadata to measured comparisons


drewstone Jul 15, 2026
ef4c0b5

fix(intelligence): stabilize measured comparisons


drewstone Jul 15, 2026
908a4c9

Merge remote-tracking branch 'origin/main' into fix/pr547-review


drewstone Jul 17, 2026
0ca7af6

fix(candidate): bind exact experiments before activation


drewstone Jul 17, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


18 changes: 6 additions & 12 deletions
.github/workflows/ci.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,9 +42,6 @@ jobs:

 agent-bench:
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -54,16 +51,13 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install published dependencies
 - name: Install package set
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test
 - name: Build current agent-runtime
 run: pnpm run build

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify agent-bench against current agent-runtime
 run: pnpm run verify:bench

35 changes: 15 additions & 20 deletions
.github/workflows/publish.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -40,6 +40,9 @@ jobs:
 - name: Verify packed package exports
 run: pnpm run verify:package

 - name: Verify agent-bench against this release
 run: pnpm run verify:bench

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
Expand DownExpand Up@@ -98,9 +101,6 @@ jobs:
 # tag/version locking stays intact and the root package is never co-published.
 if: startsWith(github.ref, 'refs/tags/agent-bench-v')
 runs-on: ubuntu-latest
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -110,23 +110,17 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - name: Install deps
 run: pnpm install --frozen-lockfile

 - name: Typecheck public entrypoint against installed packages
 run: pnpm run typecheck:public

 - name: Test deterministic package paths
 run: pnpm test

 - name: Verify packed package in a clean consumer
 run: pnpm run verify:package
 - name: Verify packed agent-bench against published dependencies
 run: pnpm run verify:bench:published

 - name: Verify tag/version lock
 run: |
 NPM_VERSION=$(node -p "require('./package.json').version")
 NPM_VERSION=$(node -p "require('./bench/package.json').version")
 TAG_VERSION="${GITHUB_REF#refs/tags/agent-bench-v}"
 if [ "$TAG_VERSION" != "$NPM_VERSION" ]; then
 echo "::error::Tag/version mismatch: tag=$TAG_VERSION package=$NPM_VERSION."
Expand All@@ -141,9 +135,6 @@ jobs:
 permissions:
 contents: read
 id-token: write
 defaults:
 run:
 working-directory: bench
 steps:
 - uses: actions/checkout@v4

Expand All@@ -153,7 +144,7 @@ jobs:
 with:
 node-version: 22
 cache: pnpm
 cache-dependency-path: bench/pnpm-lock.yaml
 cache-dependency-path: pnpm-lock.yaml

 - run: pnpm install --frozen-lockfile

Expand All@@ -162,10 +153,14 @@ jobs:
 # Requires the npmjs Trusted Publisher on @tangle-network/agent-bench:
 # org tangle-network, repo agent-runtime, workflow publish.yml.
 npm install -g npm@11
 NAME=$(node -p "require('./package.json').name")
 VERSION=$(node -p "require('./package.json').version")
 NAME=$(node -p "require('./bench/package.json').name")
 VERSION=$(node -p "require('./bench/package.json').version")
 if npm view "$NAME@$VERSION" version >/dev/null 2>&1; then
 echo "$NAME@$VERSION already on registry; skipping publish"
 else
 npm publish --provenance --access public
 mkdir -p "$RUNNER_TEMP/agent-bench-package"
 pnpm --dir bench pack --pack-destination "$RUNNER_TEMP/agent-bench-package"
 package="$(find "$RUNNER_TEMP/agent-bench-package" -maxdepth 1 -name '*.tgz' -print -quit)"
 test -n "$package"
 npm publish "$package" --provenance --access public
 fi

20 changes: 8 additions & 12 deletions
bench/package.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -24,23 +24,19 @@
 "test": "node scripts/run-package-tests.mjs",
 "typecheck:public": "tsc -p tsconfig.public.json",
 "verify:package": "node scripts/verify-packed-consumer.mjs",
 "verify:package:local-runtime": "node scripts/verify-packed-consumer.mjs --local-runtime",
 "verify:pier": "tsx scripts/verify-pier-pair.mts"
 },
 "dependencies": {
 "@tangle-network/agent-eval": "^0.117.1",
 "@tangle-network/agent-interface": "^0.25.0",
 "@tangle-network/agent-runtime": "0.94.9",
 "@tangle-network/sandbox": "^0.10.3"
 "@tangle-network/agent-eval": "0.122.1",
 "@tangle-network/agent-interface": "0.30.0",
 "@tangle-network/agent-runtime": "workspace:*",
 "@tangle-network/sandbox": "^0.11.1"
 },
 "devDependencies": {
 "@types/node": "^25.0.0",
 "tsx": "^4.19.0",
 "typescript": "^6.0.3"
 },
 "pnpm": {
 "overrides": {
 "@tangle-network/agent-eval": "0.117.1"
 }
 "@types/node": "^25.9.3",
 "tsx": "^4.22.4",
 "typescript": "^5.7.0"
 },
 "files": [
 "src",
Expand Down

76 changes: 42 additions & 34 deletions
bench/pier_agents/candidate_contract.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -107,6 +107,17 @@ def _object(value: Any, label: str) -> dict[str, Any]:
 return value


def _contract_object(
 value: Any, label: str, *, kind: str | None = None
) -> dict[str, Any]:
 obj = _object(value, label)
 if "schemaVersion" in obj or "version" in obj:
 raise CandidateContractError(f"{label} contains obsolete version fields")
 if kind is not None and obj.get("kind") != kind:
 raise CandidateContractError(f"{label} has the wrong kind")
 return obj


def _string(value: Any, label: str) -> str:
 if not isinstance(value, str) or not value or "\0" in value:
 raise CandidateContractError(f"{label} must be a non-empty string without NUL")
Expand DownExpand Up@@ -236,26 +247,22 @@ def _embedded_bytes(value: Any, label: str) -> bytes:


def _workspace_files(value: Any, label: str) -> tuple[WorkspaceFile, ...]:
 snapshot = _object(value, label)
 if (
 snapshot.get("schemaVersion") != 1
 or snapshot.get("kind") != "agent-candidate-workspace-snapshot"
 ):
 raise CandidateContractError(f"{label} has the wrong snapshot kind")
 material = _object(snapshot.get("material"), f"{label}.material")
 if (
 material.get("schemaVersion") != 1
 or material.get("kind") != "agent-candidate-workspace-manifest"
 ):
 raise CandidateContractError(f"{label}.material has the wrong manifest kind")
 snapshot = _contract_object(
 value, label, kind="agent-candidate-workspace-snapshot"
 )
 material = _contract_object(
 snapshot.get("material"),
 f"{label}.material",
 kind="agent-candidate-workspace-manifest",
 )
 values = material.get("files")
 if not isinstance(values, list):
 raise CandidateContractError(f"{label}.material.files must be an array")
 files: list[WorkspaceFile] = []
 for index, value in enumerate(values):
 obj = _object(value, f"{label}.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"{label}.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"{label} contains unsupported file mode {mode:o}"
 )
Expand DownExpand Up@@ -287,7 +294,7 @@ def _profile_files(value: Any) -> tuple[ProfileFile, ...]:
 for index, value in enumerate(values):
 obj = _object(value, f"profilePlan.material.files[{index}]")
 mode = _integer(obj.get("mode"), f"profilePlan.material.files[{index}].mode")
 if mode not in {0o644, 0o755}:
 if mode > 0o777:
 raise CandidateContractError(
 f"profile plan contains unsupported mode {mode:o}"
 )
Expand DownExpand Up@@ -359,26 +366,23 @@ def load_prepared_candidate_contract(
 raise CandidateContractError(
 "materialization receipt bytes do not match the expected digest"
 )
 if plan.get("schemaVersion") != 1 or plan.get("kind") != _PLAN_KIND:
 raise CandidateContractError("execution plan has the wrong schema or kind")
 if receipt.get("schemaVersion") != 1 or receipt.get("kind") != _RECEIPT_KIND:
 raise CandidateContractError(
 "materialization receipt has the wrong schema or kind"
 )
 plan = _contract_object(plan, "execution plan", kind=_PLAN_KIND)
 receipt = _contract_object(
 receipt, "materialization receipt", kind=_RECEIPT_KIND
 )
 if receipt.get("digestAlgorithm") != "rfc8785-sha256" or "digest" in receipt:
 raise CandidateContractError(
 "materialization receipt must be exact digest-free canonical material"
 )

 execution_evidence = _object(receipt.get("executionPlan"), "receipt.executionPlan")
 execution_evidence = _contract_object(
 receipt.get("executionPlan"),
 "receipt.executionPlan",
 kind=_EXECUTION_EVIDENCE_KIND,
 )
 plan_digest = _digest(
 execution_evidence.get("digest"), "receipt.executionPlan.digest"
 )
 if (
 execution_evidence.get("schemaVersion") != 1
 or execution_evidence.get("kind") != _EXECUTION_EVIDENCE_KIND
 ):
 raise CandidateContractError("receipt execution evidence has the wrong kind")
 if (
 sha256_bytes(plan_raw) != plan_digest
 or execution_evidence.get("material") != plan
Expand All@@ -402,11 +406,16 @@ def load_prepared_candidate_contract(
 execution_id = _string(plan.get("executionId"), "plan.executionId")

 task = _object(plan.get("task"), "plan.task")
 outcome = _object(task.get("outcome"), "plan.task.outcome")
 if outcome.get("kind") != "workspace":
 raise CandidateContractError("Pier requires a workspace task outcome")
 repository = _object(task.get("repository"), "plan.task.repository")
 base_commit = _git_object(
 repository.get("baseCommit"), "plan.task.repository.baseCommit"
 )
 base_tree = _git_object(repository.get("baseTree"), "plan.task.repository.baseTree")
 base_tree = _git_object(
 repository.get("baseTree"), "plan.task.repository.baseTree"
 )
 if len(base_commit) != len(base_tree):
 raise CandidateContractError("task Git objects use different hash formats")
 instruction = _instruction(task.get("instruction"))
Expand DownExpand Up@@ -471,12 +480,11 @@ def load_prepared_candidate_contract(
 if receipt.get("candidateWorkspace") != candidate_snapshot:
 raise CandidateContractError("receipt and plan candidate workspaces differ")

 profile_evidence = _object(receipt.get("profilePlan"), "receipt.profilePlan")
 if (
 profile_evidence.get("schemaVersion") != 1
 or profile_evidence.get("kind") != _PROFILE_PLAN_KIND
 ):
 raise CandidateContractError("receipt profile evidence has the wrong kind")
 profile_evidence = _contract_object(
 receipt.get("profilePlan"),
 "receipt.profilePlan",
 kind=_PROFILE_PLAN_KIND,
 )
 profile_digest = _digest(
 profile_evidence.get("digest"), "receipt.profilePlan.digest"
 )
Expand All@@ -489,7 +497,7 @@ def load_prepared_candidate_contract(
 profile_artifact_material = json.loads(profile_raw)
 except (UnicodeDecodeError, json.JSONDecodeError) as exc:
 raise CandidateContractError("profile artifact is not UTF-8 JSON") from exc
 profile_material = _object(
 profile_material = _contract_object(
 profile_evidence.get("material"), "receipt.profilePlan.material"
 )
 if profile_artifact_material != profile_material:
Expand Down
Loading
Loading