Adversarial Iteration for Underspecified Program Synthesis

LLM-generated code routinely passes the obvious examples and silently fails the hidden ones. IDCS (Iterative Distinguishing of Code and Specs) treats this as a specification problem rather than a model problem: a generator drafts a structured spec from the prompt, a distinguisher critiques it for gaps and ambiguities, a user-proxy answers the questions that need user intent, and a coder implements the final spec. On a held-out hard MBPP+ split, this spec-guided pipeline raises hidden-test pass rate from 73.3% → 96.2%. We also show that the generator and distinguisher prompts themselves can be discovered through adversarial coevolution.

📄 Apart Research submission • 📚 Paper • 🏗 Design notes • 📋 Implementation plan

Setup

git clone https://github.com/aimir/idcs.git
cd idcs
pip install -e ".[dev]"# or: pip install -r requirements.txt

Python 3.11+ required.

Environment variables

Put these in a .env file at the repo root (auto-loaded by python-dotenv):

# Default backend: OpenRouter (any OpenAI-compatible provider works)
OPENROUTER_API_KEY=sk-or-v1-...
IDCS_MODEL=anthropic/claude-sonnet-4.5 # optional, defaults to claude-sonnet-4.5# Or use a local Codex CLI install (no API key needed)
IDCS_BACKEND=codex
IDCS_CODEX_MODEL=gpt-5.4-mini # optional
IDCS_CODEX_TIMEOUT_S=300 # optional, per-call timeout
IDCS_CODEX_SERVICE_TIER=fast # optional
IDCS_CODEX_REASONING_EFFORT=none # optional

Switch backends with IDCS_BACKEND: openrouter (default), openai, openai-compatible, or codex. Use a different mutator model for prompt search with IDCS_MUTATOR_MODEL.

Examples

Smoke: direct vs spec-guided on one MBPP+ task

python scripts/baseline.py --tasks Mbpp/11 --max-turns 1

Output is a side-by-side pass-rate comparison on the hidden plus_input cases.

Batch evaluation on the hard slice

python scripts/batch_baseline.py --dataset hard --workers 4 --max-turns 2

Writes incremental results.jsonl and live-updated summary.json to experiments/runs/. Hard slices available: hard, hard-train, hard-dev, hard-test, hard-extended, plus the curated hardened POC corpus.

Coevolution training

python scripts/train.py \
--benchmark hard \
--epochs 5 --pop-size 8 --elite-size 3 \
--val-fraction 0.2 \
--max-llm-calls 500

Evolves generator and distinguisher prompts via population-based search with anchor-protected elites, task-Pareto elite selection, and an anti-regression reward penalty. Telemetry (per-turn traces, prompt hashes, config snapshot) lands under experiments/runs/<run_id>/.

Reproducing the ceiling result

The diagnostic hand-rules prompts that reach 100% on the hard slice:

IDCS_BACKEND=codex IDCS_CODEX_MODEL=gpt-5.4-mini \
python scripts/batch_baseline.py \
--dataset hard --workers 5 --retries 0 --max-turns 2 \
--generator-prompt-file prompts/hard_mbpp_rules_generator_v0.md \
--distinguisher-prompt-file prompts/hard_mbpp_rules_distinguisher_v0.md

Development

pytest -q # 112 tests
ruff check src tests scripts
mypy src/idcs scripts

Authors

Uri Ariel Chen · Nitzan Pomerantz · Amir Sarid · Amit Saroussi, with Apart Research.