Benchmark Kernel MCP with ClawBench through Harbor

.dockerignore

@@ -0,0 +1,8 @@
+.git
+.github
+.next
+node_modules
+coverage
+.env*
+*.log
+benchmarks/harbor/.image.env

.gitignore

@@ -1,5 +1,7 @@
 # Dependencies
 node_modules/
+__pycache__/
+*.py[cod]
 npm-debug.log*
 yarn-debug.log*
 yarn-error.log*
@@ -107,5 +109,9 @@ Makefile
 # private key
 mcp-key.pem
 
+# Harbor benchmark runtime data
+benchmarks/harbor/.image.env
+benchmarks/harbor/image/source-sha
+
 # TypeScript incremental build cache
 tsconfig.tsbuildinfo

README.md

@@ -261,7 +261,7 @@ Each Kernel feature has a single `manage_*` tool with an `action` parameter, kee
 
 One additional Managed Auth helper (`begin_auth_login`) is marked app-only (`_meta.ui.visibility: ["app"]`); it refuses to execute on hosts that do not declare MCP Apps support. The App forwards the server-issued signed flow checkpoint to the shared `manage_auth_connections` `wait` action, so flow identity and terminal-state decisions stay on the server.
 
-Self-hosted deployments can hide sensitive tool families by setting `KERNEL_MCP_DISABLED_TOOLSETS` to a comma-separated list. For example, `KERNEL_MCP_DISABLED_TOOLSETS=api_keys` prevents `manage_api_keys` from being registered.
+Self-hosted deployments can select tool families with `KERNEL_MCP_ENABLED_TOOLSETS` or hide them with `KERNEL_MCP_DISABLED_TOOLSETS`. Both accept comma- or space-separated toolset names and standalone aliases. For example, `KERNEL_MCP_ENABLED_TOOLSETS="playwright computer"` exposes browser-control tools without browser lifecycle or managed-auth tools, while `KERNEL_MCP_DISABLED_TOOLSETS=api_keys` only removes `manage_api_keys`. `get_connection_context` remains available in either mode.
 
 Call `get_connection_context` before deciding whether to create or select a project. Its canonical `connection_scope` reports whether the connection is organization-wide or fixed to a project. Project-scoped tools advertise an optional `project` (name or ID) and a deprecated `project_id`: organization-wide connections may omit them to preserve organization-wide reads and API default-project behavior, while fixed-project connections may omit them or pass the matching project. Project resources use project-qualified `kernel://orgs/{organizationId}/projects/{projectId}/...` URIs. Authorization remains enforced by the Kernel API; selecting a project never grants access to it.
 

benchmarks/harbor/README.md

@@ -0,0 +1,55 @@
+# Harbor ClawBench benchmark
+
+This directory runs stock Harbor agents (Claude Code, Codex) against a locally built `kernel-mcp-server` on a ClawBench task in a single Hypeman sandbox. The task starts from the Kernel-backed ClawBench Harbor adaptation, replaces Playwright MCP with the local source-pinned Kernel MCP server, and keeps ClawBench attached to the same pre-created browser.
+
+## Requirements
+
+- Harbor 0.21.0 with `harbor-hypeman` 0.1.1 (launched through `uvx`)
+- [uv](https://docs.astral.sh/uv/) and Hypeman CLI credentials
+- A ClawBench checkout containing commit `df6743f` (`kernel/ClawBench` PR #1)
+- `KERNEL_MCP_BENCHMARK_API_KEY` scoped to an isolated evaluation project
+- `KERNEL_MCP_BENCHMARK_PROJECT_ID`
+- `PURELY_MAIL_API_KEY` and `PURELY_MAIL_DOMAIN` for account-task credentials
+- `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN` for Claude Code
+- `OPENAI_API_KEY` for Codex
+
+## Build the image
+
+```bash
+./benchmarks/harbor/build-image.sh
+```
+
+The build uses the current Git SHA, installs dependencies with Bun, runs the production Next.js build, and writes the resulting image reference to the ignored `.image.env` file. Hypeman can report a failed build before the converted image becomes visible; the script performs a bounded 5-minute ready-image check for that case.
+
+## Run the ClawBench Kernel MCP arm
+
+```bash
+export CLAWBENCH_REPO=../ClawBench
+./benchmarks/harbor/clawbench/run-control.sh claude-code \
+ v2-1134-chapter-finder-redcross
+```
+
+Defaults:
+
+| Agent | Version | Model |
+| ----------- | ------: | ----------------- |
+| Claude Code | 2.1.238 | `claude-sonnet-5` |
+| Codex | 0.120.0 | `gpt-5.6-luna` |
+
+Override models with `CLAUDE_BENCHMARK_MODEL` or `CODEX_BENCHMARK_MODEL`. Single-task runs have a 40-minute wall-clock limit; full-suite runs default to 6 hours. Change either with `HARBOR_BENCHMARK_TIMEOUT`.
+
+Pass `all` instead of a task ID to run the complete suite, and set `HARBOR_N_CONCURRENT` to control parallelism:
+
+```bash
+HARBOR_N_CONCURRENT=10 ./benchmarks/harbor/clawbench/run-control.sh codex all
+```
+
+`run-control.sh` adapts the selected ClawBench tasks with `clawbench-harbor-adapt`, converts them with `clawbench/prepare-control.py`, and runs them under Harbor. Each generated task:
+
+- exposes `get_connection_context` and `execute_playwright_code`
+- disables coordinate-based computer actions, browser lifecycle, and managed-auth toolsets
+- instructs the agent to read `./my-info/kernel_browser.json` and use that session ID
+- instructs account tasks to use the supplied PurelyMail credentials instead of managed auth
+- verifies ATIF observations, project scope, exact session reuse, ClawBench interception, replay finalization, and browser deletion
+
+Outputs use the normal Harbor job directory and add `kernel-mcp-control-result.json`, Kernel MCP logs, source manifests, and same-session metrics to the ClawBench verifier artifacts.

benchmarks/harbor/bin/kernel-mcp-local

@@ -0,0 +1,13 @@
+#!/bin/sh
+set -eu
+
+key_file=/run/kernel-mcp-benchmark/api-key
+if [ -z "${KERNEL_API_KEY:-}" ] && [ -r "$key_file" ]; then
+ KERNEL_API_KEY=$(cat "$key_file")
+ export KERNEL_API_KEY
+fi
+: "${KERNEL_API_KEY:?KERNEL_API_KEY is required}"
+
+exec mcp-remote \
+ http://127.0.0.1:3002/mcp \
+ --header "Authorization: Bearer ${KERNEL_API_KEY}"

benchmarks/harbor/bin/start-kernel-mcp-server

@@ -0,0 +1,71 @@
+#!/bin/bash
+set -euo pipefail
+
+: "${KERNEL_API_KEY:?KERNEL_API_KEY is required}"
+
+log_dir=/logs/kernel-mcp
+key_dir=/run/kernel-mcp-benchmark
+mkdir -p "$log_dir" /logs/artifacts "$key_dir"
+chmod 0777 "$log_dir" /logs/artifacts
+chmod 0700 "$key_dir"
+printf '%s' "$KERNEL_API_KEY" >"$key_dir/api-key"
+chmod 0600 "$key_dir/api-key"
+
+redis-server --daemonize yes --bind 127.0.0.1 --port 6379 \
+ --logfile "$log_dir/redis.log" --dir /tmp
+
+export CLERK_SECRET_KEY=${CLERK_SECRET_KEY:-sk_test_kernel_mcp_benchmark_local_only}
+export NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=${NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY:-pk_test_YmVuY2htYXJrLmNsZXJrLmFjY291bnRzLmRldiQ}
+
+cd /opt/kernel-mcp-server
+nohup ./node_modules/.bin/next start -p 3002 \
+ >"$log_dir/server.stdout.log" \
+ 2>"$log_dir/server.stderr.log" &
+echo $! >"$log_dir/server.pid"
+
+for _ in $(seq 1 90); do
+ if curl -fsS -X POST http://127.0.0.1:3002/mcp \
+ -H "Authorization: Bearer ${KERNEL_API_KEY}" \
+ -H "Content-Type: application/json" \
+ -H "Accept: application/json, text/event-stream" \
+ --data '{"jsonrpc":"2.0","id":"benchmark-healthcheck","method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"harbor-healthcheck","version":"1.0.0"}}}' \
+ >"$log_dir/initialize-response.txt"; then
+ break
+ fi
+ sleep 1
+done
+
+if [ ! -s "$log_dir/initialize-response.txt" ]; then
+ echo "Kernel MCP server did not become ready" >&2
+ tail -100 "$log_dir/server.stderr.log" >&2 || true
+ exit 1
+fi
+
+python3 - <<'PY'
+import json
+import os
+import platform
+from datetime import datetime, timezone
+from pathlib import Path
+
+browser_path = Path("/my-info/kernel_browser.json")
+try:
+ browser = json.loads(browser_path.read_text())
+except (OSError, json.JSONDecodeError):
+ browser = {}
+
+manifest = {
+ "kernel_mcp_server_sha": Path("/opt/kernel-mcp-server/SOURCE_SHA").read_text().strip(),
+ "clawbench_source_sha": os.environ.get("CLAWBENCH_SOURCE_SHA", ""),
+ "browser_session_id": browser.get("session_id"),
+ "enabled_toolsets": os.environ.get("KERNEL_MCP_ENABLED_TOOLSETS", ""),
+ "disabled_toolsets": os.environ.get("KERNEL_MCP_DISABLED_TOOLSETS", ""),
+ "image": os.environ.get("KERNEL_MCP_BENCHMARK_IMAGE", ""),
+ "hypeman_instance_name": os.environ.get("HYPEMAN_INSTANCE_NAME", ""),
+ "sandbox_hostname": platform.node(),
+ "started_at": datetime.now(timezone.utc).isoformat(),
+}
+Path("/logs/kernel-mcp/run-manifest.json").write_text(json.dumps(manifest, indent=2))
+PY
+
+printf 'ready\n' >"$log_dir/ready"

benchmarks/harbor/build-image.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+set -euo pipefail
+
+repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+cd "$repo_root"
+
+source_sha=$(git rev-parse HEAD)
+source_sha_file=benchmarks/harbor/image/source-sha
+build_log=$(mktemp)
+trap 'rm -f "$source_sha_file" "$build_log"' EXIT
+
+printf '%s\n' "$source_sha" >"$source_sha_file"
+
+set +e
+hypeman build \
+ --file benchmarks/harbor/image/Dockerfile \
+ --cpus 4 \
+ --memory 8192 \
+ --timeout 1800 \
+ . 2>&1 | tee "$build_log"
+build_status=${PIPESTATUS[0]}
+set -e
+
+build_id=$(sed -n -E 's/^Build (ID|started): //p' "$build_log" | tail -1)
+if [[ -z "$build_id" ]]; then
+ echo "Hypeman did not return a build ID" >&2
+ exit 1
+fi
+
+image_ref="docker.io/builds/$build_id:latest"
+if ((build_status != 0)); then
+ echo "Build record failed; checking for a delayed ready image for up to 5 minutes" >&2
+ image_ready=false
+ for _ in $(seq 1 30); do
+ if hypeman --format json image list | python3 -c '
+import json
+import sys
+
+image_ref = sys.argv[1]
+expected = {image_ref, image_ref.removeprefix("docker.io/")}
+images = json.load(sys.stdin)
+raise SystemExit(
+ 0
+ if any(image.get("name") in expected and image.get("status") == "ready" for image in images)
+ else 1
+)
+' "$image_ref"
+ then
+ image_ready=true
+ break
+ fi
+ sleep 10
+ done
+ if [[ "$image_ready" != true ]]; then
+ exit "$build_status"
+ fi
+fi
+
+cat >benchmarks/harbor/.image.env <<EOF
+KERNEL_MCP_BENCHMARK_IMAGE=$image_ref
+KERNEL_MCP_SOURCE_SHA=$source_sha
+EOF
+chmod 0600 benchmarks/harbor/.image.env
+
+printf 'Wrote benchmarks/harbor/.image.env for %s\n' "$image_ref"