DeepAgent

Manufacture hard, Docker-verifiable SWE tasks as DeepAgent / Harbor pack trees.

Install · Env · CLI · Product N=9 · Hugging Face · Scoreboard · Architecture

---

Overview

DeepAgent builds agent-facing Harbor pack trees from multi-file hard Real-PR tasks. Each certified pack carries a real public repository URL, an immutable base commit, multi-file gold solution, held-out verifier tests, Docker oracle dual-truth (solution reward = 1, null reward = 0), and agent isolation.

This repository is a single installable package (deepagent) at git root. There is no nested second package and no legacy forge product path.

Surface	Path / ref	Role
Current product	datasets/prod_hard_deepswe_med	DeepSWE-median hard Real-PR, N=9 (M27 floors + M28 max 2 packs/repo, unique_repos=7)
HF stable pin	BaseIntelligence/deepagentmain	Current product on Hub (N=9)
HF automation mirror	BaseIntelligence/deepagenttest	CI/dev write target; also N=9
Scoreboard	datasets/panel_prod_hard_deepswe_med_m28	Grok 4.5 + Kimi 2.7-code dual-model matrix
Product detail	datasets/prod_hard_deepswe_med/PRODUCT_README.md	Floors, p50 stats, diversity notes

Primary console entrypoint:deepagent (generate / upload / pull / eval / oracle / version).

Note: An older synthetic forge (swe-forge / Composer path) was removed from this monorepo. The only product story is DeepAgent real_pr Harbor A→Z. A thin internal compat entry swe-factory may still resolve to factory helpers; new work should use deepagent only.

Install

From a clean checkout of this repository root:

cd /path/to/deepagent # git root (this directory)
python3 -m venv .venv
.venv/bin/pip install -U pip
.venv/bin/pip install -e ".[dev]"
cp .env.example .env # set placeholders; never commit .env
.venv/bin/deepagent --help

Python ≥ 3.12. Package name deepagent exposes:

Entry	Module	Role
deepagent	swe_factory.deepagent_cli:app	Primary product CLI
swe-factory	swe_factory.cli:app	Optional internal compat only

Environment (.env)

Copy .env.example. Placeholders only — never commit secrets.

Variable	Purpose
HF_TOKEN	Hugging Face push/pull for BaseIntelligence/deepagent
OPENROUTER_API_KEY	Live panel / eval model calls
OPENROUTER_BASE_URL	Default https://openrouter.ai/api/v1
FACTORY_PANEL_MODELS	Historical default includes k2.6; current median panel uses grok-4.5 + kimi-k2.7-code via explicit --model
FACTORY_BUDGET_USD	Hard cap (default 600)
GITHUB_TOKEN / GH_TOKEN	Live Real-PR mine. Prefer export GITHUB_TOKEN="$(gh auth token)" after gh auth login
OXYLABS_PROXY_URL	SOCKS residential proxy for GitHub REST rate-limit relief
ALL_PROXY / HTTPS_PROXY	Optional proxy chain so git/HTTPS clients share the same egress

GitHub 429 mitigation: authenticated REST (GITHUB_TOKEN / gh auth) plus optional SOCKS via OXYLABS_PROXY_URL (and/or ALL_PROXY / HTTPS_PROXY) is the primary anti-429 path. The Oxylabs realtime Web Scraper API is optional and not required for GitHub REST/Search mining.

Secrets never appear in logs, CLI output, or shipped datasets.

Current product (authoritative)

Field	Value
Product root	datasets/prod_hard_deepswe_med
N	9 certified packs
unique_repos	7
max packs / repo	2 (M28 diversity)
HF stable pin	BaseIntelligence/deepagent revision main (N=9)
HF automation mirror	same repo revision test (also N=9; CI/dev default write target)
Primary CLI	deepagent (generate / upload / pull / eval / oracle)
Scoreboard	datasets/panel_prod_hard_deepswe_med_m28
Default eval models	x-ai/grok-4.5 + moonshotai/kimi-k2.7-code

M27 DeepSWE-median hardness floors

Every keep must pass:

• Multi-file: source files ≥ 4, or hybrid files ≥ 3 + gold added ≥ 500 + hunks ≥ 14
• source hunks ≥ 14
• gold added lines ≥ 400
• F2P nodes ≥ 5
• HarborDocker dual-truth (sol = 1, null = 0) + prompt–verifier alignment
• live-mined source_track=real_pr only (no fixture pad, no hybrid motors)
• M28 diversity: max 2 packs per upstream repo

See datasets/prod_hard_deepswe_med/PRODUCT_README.md, coverage_stats.json, and median_stats.json.

Pack IDs (N=9)

realpr-click-3442 · realpr-itemadapter-101 · realpr-oauthlib-889 · realpr-packaging-1120 · realpr-packaging-1267 · realpr-rich-3930 · realpr-werkzeug-2637 · realpr-werkzeug-3116 · realpr-wtforms-923

Primary CLI (deepagent)

Commands (current product paths)

# Live-mine hard real_pr under M27 floors → prod_hard_deepswe_med
deepagent generate \
--target 10 --min-packs 5 --max-packs 15 \
--out datasets/prod_hard_deepswe_med \
--live-mine --oracle docker --panel offline --pier scripted
# HarborDocker dual-truth on one pack (sol=1 / null=0)
deepagent oracle --pack-dir datasets/prod_hard_deepswe_med/tasks/realpr-click-3442
# Push pack trees + manifest to HF stable pin (main) and/or automation mirror (test)
deepagent upload \
--src datasets/prod_hard_deepswe_med \
--repo-id BaseIntelligence/deepagent \
--revision main
deepagent upload \
--src datasets/prod_hard_deepswe_med \
--repo-id BaseIntelligence/deepagent \
--revision test# Pull packs from HF stable pin (main)
deepagent pull \
--repo-id BaseIntelligence/deepagent \
--revision main \
--out datasets/hf_pull_main
# Pier mini-swe + Harbor dual-model eval (current median models)
deepagent eval \
--product-root datasets/prod_hard_deepswe_med \
--max-packs 9 --k 1 --n-concurrent 5 \
--hard-stop-usd 600 \
--model x-ai/grok-4.5 \
--model moonshotai/kimi-k2.7-code \
--out datasets/panel_prod_hard_deepswe_med_m28
deepagent version
deepagent --help

Command	What it does
generate	Live mine hard real_pr; product out datasets/prod_hard_deepswe_med; Docker oracle only; refuses fixture pad
oracle	HarborDocker dual-truth cert on one pack dir (solution reward 1, null 0; refuse fake)
upload	Validate local pack root, push trees + pack_manifest to HF. Prefer main for the stable pin; test for automation
pull	Download pack trees from HF revision (main or test) into a local out dir
eval	Pier + mini-swe-agent + HarborDocker model eval (n_concurrent 1..5, hard-stop $600, fidelity pier_miniswe_harbor). Current median panel: grok-4.5 + kimi-k2.7-code
version	Package version identity

Operator A→Z loop

1. Install root package (pip install -e ".[dev]") and fill .env.
2. Generate live-mined hard real_pr packs into datasets/prod_hard_deepswe_med.
3. Oracle-cert keep packs (Docker sol=1 / null=0).
4. Upload to HF BaseIntelligence/deepagent revision main (stable) and keep test in sync.
5. Pull (optional) a clean Hub mirror for offline eval.
6. Eval Pier mini-swe + Harbor under the $600 hard-stop with Grok 4.5 + Kimi 2.7-code.

Scoreboard (M28 diversified panel)

Durable dual-model matrix on the current product (observational ranking only; dual-solve rate is the hardness quality gate ≤ 0.30):

Model	pass@1 (k=1)
x-ai/grok-4.5	3/9 ≈ 0.33
moonshotai/kimi-k2.7-code	1/9 ≈ 0.11
dual_solve rate	≈ 0.11 (1/9)

Evidence:

• datasets/panel_prod_hard_deepswe_med_m28/SUMMARY.md
• datasets/panel_prod_hard_deepswe_med_m28/scoreboard.json
• Product notes: datasets/prod_hard_deepswe_med/PRODUCT_README.md

Scoreboard is leaderboard / ranking only. Dual-model success does not auto-drop hardness packs (intrinsic policy).

Hugging Face (BaseIntelligence/deepagent)

Ref	Content
main (stable product pin)	Current N=9 packs from datasets/prod_hard_deepswe_med
test (automation / dev mirror)	Same N=9 product; default write target for CI and live automation

Operator loop: generate → oracle → upload (prefer revision main for the stable public pin; keep test in sync) → pull → eval. Do not embed tokens in CLI flags; use HF_TOKEN env / .env only.

What you get in a pack

Each pack under tasks/<task_id>/:

task.toml # schema 1.1, repository_url, base_commit_hash
instruction.md
pre_artifacts.sh
environment/Dockerfile # agent image @ base SHA; offline runtime
tests/
Dockerfile
test.sh
grader.py
config.json # fail_to_pass / pass_to_pass node ids
test.patch # held-out verifier tests
solution/
solution.patch # multi-file product sources only
solve.sh

Corpus-level artifacts at the pack root:

Artifact	Role
pack_manifest.json	Certified pack index + band metadata
PRODUCT_README.md	Product N, floors, diversity summary
PROVENANCE.md	License, upstream URL, base SHA, language per keep
coverage_stats.json / median_stats.json	Structural p50 / repo diversity
report.md	Language mix, funnel, spend, honesty notes (when present)
oracle_evidence.json / evidence/docker/	Docker sol/null dual-truth index

Architecture

flowchart LR
Allow[Permissive allowlist] --> Mine[Live mine real_pr]
Mine --> Env[Envbuild at base SHA]
Env --> Label[Dual-run F2P / P2P]
Label --> Oracle[Docker oracle sol=1 null=0]
Oracle --> Export[Harbor pack export]
Export --> Product[datasets/prod_hard_deepswe_med N=9]
Product --> HF[HF BaseIntelligence/deepagent @main]
Product --> HFtest[HF @test automation mirror]
HF --> Eval[deepagent eval Pier mini-swe Harbor]
Eval --> Board[panel_prod_hard_deepswe_med_m28]

Loading

Git is the authority for commits and patches. Live GitHub REST prefers GITHUB_TOKEN + optional SOCKS (OXYLABS_PROXY_URL). Realtime Oxylabs page scrape remains optional.

Honesty floors (brief)

• Product N counts live-mined real_pr only. Fixture shortlists never pad current product N.
• Docker oracle only on the certified path (HarborDockerVerifier); fake backends are refused.
• Dual-truth required: solution reward = 1, null reward = 0.
• Multi-file gold from a merged public PR; hard floors follow the DeepSWE-median band (files≥4 OR hybrid 3+500+14; hunks≥14; gold added≥400; F2P≥5) plus M28 max 2 packs/repo.
• Panel / eval spend stops under the hard budget (default $600); never invent panel spend. Current median eval models: grok-4.5 + kimi-k2.7-code.
• Secrets (HF / OpenRouter / GitHub / proxy passwords) stay in env / .env only — never in help examples, logs, or uploaded trees.
• Under-supply and offline modes (panel offline, pier scripted when not live) must be stated honestly in ship reports.

Tests & CI

Local non-integration gate (matches CI quality job):

.venv/bin/ruff format --check src tests
.venv/bin/ruff check src tests
.venv/bin/mypy src
.venv/bin/python -m pytest tests -q -p no:cacheprovider -p no:httpbin -p no:cov \
-m "not integration" -n 2 --maxprocesses=2 --tb=line

Focused suites:

.venv/bin/python -m pytest tests/test_deepagent_cli.py -q -p no:cacheprovider -n 0
.venv/bin/python -m pytest tests/test_hf_packs.py -q -p no:cacheprovider -n 0
.venv/bin/python -m pytest tests/test_ship_deepagent.py -q -p no:cacheprovider -n 0
.venv/bin/python -m pytest tests/test_package_layout.py -q -p no:cacheprovider -n 0

Prefer pytest -n 0 (or ≤ -n 2); never pytest -n auto on shared hosts.

How it works

1. Live-mine multi-file Real-PR candidates (deepagent generate --live-mine).
2. Envbuild agent images pinned at a real base SHA with offline runtime.
3. Label fail-to-pass and pass-to-pass node ids via dual-run suites.
4. Oracle-cert with Docker: solution reward 1, null reward 0; refuse fake.
5. Export Harbor v1.1 tree with held-out tests and isolation-clean agent view.
6. Upload / pull HF (BaseIntelligence/deepagent@main stable pin; @test automation mirror).
7. Eval Pier mini-swe + Harbor under hard-stop budget (fidelity pier_miniswe_harbor) with Grok 4.5 + Kimi 2.7-code for the current median panel.

Documentation

Audience	Guide	Contents
Operators	this README	install, CLI, HF, current product N=9
Product consumers	datasets/prod_hard_deepswe_med/PRODUCT_README.md	floors, diversity, pack list
Scoreboard	datasets/panel_prod_hard_deepswe_med_m28/SUMMARY.md	Grok≈0.33 / Kimi2.7≈0.11 / dual≈0.11
Implementers	docs/architecture.md	pipeline stages and gates
Hardness policy	docs/PRODUCT_HARDNESS.md	DeepSWE-median floors, intrinsic, opt-out
M32 keep/drop	docs/M32_KEEP_DROP.md	monorepo cleanup boundaries

Repository layout

pyproject.toml # package name = deepagent
.env.example
src/swe_factory/
deepagent_cli.py # primary deepagent entrypoint
cli.py # optional swe-factory compat entrypoint
export/hf_packs.py # HF upload / pull
panel/eval_deepagent.py
pipeline/ship_deepagent.py
harbor/ # pack export + docker oracle cert
producers/ # mine / labeling
sources/ # allowlist, git mine, SOCKS proxy
datasets/
prod_hard_deepswe_med/ # CURRENT product: median + diversity N=9
panel_prod_hard_deepswe_med_m28/ # authoritative Grok/Kimi2.7 scoreboard
fixtures/ # unit shortlist only (never product N)
tests/
docs/
architecture.md
PRODUCT_HARDNESS.md
M32_KEEP_DROP.md
assets/banner.png
.github/workflows/ # root package lint + unit gate

Historical bulk trees (test_n10, soft prod_hard_keep, older product archives, live materials workdirs) are not tracked in git after the M32 cleanup. Product truth is only prod_hard_deepswe_med N=9 + the M28 panel (+ HF mirrors).

Limits and non-goals

• Current product N is prod_hard_deepswe_med / HF main (stable pin; test is the automation mirror at the same N=9).
• Not a browser UI; CLI + Docker + Pier only.
• Copyleft / unknown licenses are fail-closed and never appear in PROVENANCE.
• On constrained hosts, prefer serial Docker cert and lower n_concurrent eval.
• No Composer / Cursor / synthetic-forge product path.

Spend

Metric	Value
Cap	$600 (FACTORY_BUDGET_USD / eval hard-stop)
Default eval concurrency	1 (allowlist 1..5)
Current median eval models	grok-4.5 · kimi-k2.7-code
M28 panel spend (durable)	≈ $23.64 under cap
Eval fidelity	pier_miniswe_harbor

Source of truth: panel ledger_summary.json.

License

MIT (see LICENSE and pyproject.toml).