OpenART is an executable benchmark for evaluating the safety of tool-using code agents in long-running, evolving environments. This repository contains its Docker-based runtime, three runnable high-complexity tasks, and the managed tools required by those tasks.
OpenART is the agent version of OpenRT.
[09/02/2026] ⭐ OpenART reached the 100-star milestone on GitHub.
OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution
🥈 #2 Paper of the Day on Hugging Face Papers
Yunhao Chen, Xin Wang, Yixu Wang, Yi Liu, Jie Li, Yan Teng, Xingjun Ma, Xia Hu, and Yu-Gang Jiang
OpenART evaluates the executable environment rather than treating a single prompt as the entire test. The benign task and hidden safety contract remain fixed while target-visible state changes during execution.
| Paper setting | Value |
|---|---|
| Validated stateful scenarios | 10K+ |
| Domains | 50 |
| Capability corpus | 500K+ tools and skills |
| Median task horizon | 97 tool calls |
| Evaluation matrix | 15 agents × 5 models (75 settings) |
| Target-visible attack surfaces | 8 |
| Reference attacker | Evolutionary Markov Hypergraph Attack (EMHA) |
| Pooled strict attack success rate | 85.0% |
The paper reports that environment evolution becomes more effective as workflow complexity grows, and that the agent runtime affects safety beyond the choice of foundation model. Unsafe behavior also tends to appear well after the changed state is first consumed, often through stale assumptions or decisions carried forward from earlier steps.
The public OpenART planner task dataset contains 6,597 verified tasks in 34 zip shards. Its release checks covered checksums, archive integrity, and sample extraction.
We also release the public OpenART tools dataset, which packages 63,697 selected materialized tools from the OpenART capability corpus. This 60K+ subset prioritizes the most usable OpenART tools for task execution, reuse, and reproducible benchmark construction. The release includes metadata, archive manifests, checksums, and a lightweight download helper.
| Path | Contents |
|---|---|
OpenART/ | Runtime, configuration, Dockerfiles, documentation, and tests |
OpenART/examples/tasks/ | Local smoke task and bundled high-complexity tasks |
openart-tools/ | Managed tool subset used by the bundled tasks |
Run the following commands from the repository root:
cd OpenART
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
export PYTHONPATH="$PWD"Docker must be available to the current user. Build the base task image:
docker build -t openart/task-base:latest -f images/Dockerfile.task-base .The local smoke task checks the container and deterministic evaluator paths. It does not require model credentials.
python -m framework.cli run \
--task examples/tasks/local-smoke \
--target-config configs/target-configs/target.local-smoke.yaml \
--eval-strategy deterministic \
--skip-attacker \
--run-id local-smoke \
--output-dir outputs/local-smokeThe default target configuration uses OpenCode. Build its image, copy the
environment template, and add the required model endpoints and credentials to
.env:
docker build -t openart/opencode:latest -f images/Dockerfile.opencode .
cp .env.example .envRun a target without an attacker:
python -m framework.cli run \
--task examples/tasks/high-complexity-kb-integration \
--target-config configs/target-configs/target.yaml \
--tool-store ../openart-tools \
--eval-strategy deterministic \
--skip-attacker \
--run-id kb-integration-target-only \
--output-dir outputs/kb-integration-target-onlyRun the same task with the default OpenCode-compatible attacker:
python -m framework.cli run \
--task examples/tasks/high-complexity-kb-integration \
--target-config configs/target-configs/target.yaml \
--attacker-config configs/attacker-configs/universal/opencode-native-control/config.yaml \
--tool-store ../openart-tools \
--eval-strategy both \
--max-iterations 2 \
--run-id kb-integration-attacked \
--output-dir outputs/kb-integration-attackedThe task graph selects tools from ../openart-tools. Run artifacts are written
under OpenART/outputs/ and ignored by Git.
Build the planner image and generate a task from a checked-in scenario. Planner
model settings are read from .env.
docker build -t openart/safe-world-planner:latest \
-f images/Dockerfile.safe-world-planner .
python -m framework.planner.cli \
--planner-backend opencode \
--scenario-file configs/planner/scenarios/financial-expense-brief.txt \
--tool-store ../openart-tools \
--tool-count 3 \
--complexity-profile stress \
--planner-max-repairs 2 \
--task-id financial-expense-brief \
--output-dir outputs/financial-expense-brief \
--overwritePlanner outputs are also local, ignored artifacts.
OpenART is licensed under the GNU Affero General Public License v3.0.