Toward Precise and Consistent Agent Behaviors across Models Anchored by Validated Social Science Knowledge
🌐 Project Page: cobra.clawder.ai | 📄 Paper: arXiv 2509.13588
If you find CoBRA useful, please star ⭐ this repo to help others discover it!
Demo_Video.mp4
💡 What is Cognitive Bias?
Systematic deviations from rational judgment in human cognition and decision-making. For example, Framing Effect: "90% survival rate" vs. "10% mortality rate" — logically identical, yet people make different choices based on how information is framed.
Reproducibility and controllability are fundamental to scientific research. Yet implicit natural language descriptions — the dominant approach for specifying social agent behaviors in nearly all LLM-based social simulations — often fail to yield consistent behavior across models or capture the nuances of the descriptions.
CoBRA (Cognitive Bias Regulator for Social Agents) is a novel toolkit that lets researchers explicitly specify desired nuances in LLM-based agents and obtain consistent behavior across models.
Through CoBRA, we show how to operationalize validated social science knowledge as reusable "gym" environments for AI — an approach that generalizes to richer social and affective simulations.
The problem and our solution: from inconsistent agent behaviors under implicit specifications to explicit, quantitative control.
At the heart of CoBRA is a novel closed-loop system with two core components:
- Cognitive Bias Index — measures the cognitive bias of a social agent by quantifying its reactions in validated classic social science experiments
- Behavioral Regulation Engine — aligns the agent's behavior to exhibit controlled cognitive bias, via three control methods:
- Prompt Engineering (input space control)
- Representation Engineering (activation space control)
- Fine-tuning (parameter space control)
Example: A researcher specifies a target bias level → CoBRA measures it via classic experiments → iteratively adjusts the agent until it reliably exhibits the desired bias.
# 1. Install dependencies
pip install -r requirements.txt
# 2. Navigate to the unified bias control modulecd examples/unified_bias
# 3. Run a bias experiment
python pipelines.py --bias authority --method repe-linear --model Mistral-7BThat's it. The system will measure and control the agent's Authority Effect bias.
CoBRA/
├── control/ # Core bias control engine
├── examples/
│ ├── unified_bias/ # Main entry point (START HERE)
│ │ ├── pipelines.py # Unified experiment runner
│ │ ├── run_pipelines.py # CLI interface
│ │ ├── ablation/ # Ablation studies
│ │ └── README.md # Full usage guide
│ ├── authority/ # Authority Effect utils
│ ├── bandwagon/ # Bandwagon Effect utils
│ ├── confirmation/ # Confirmation Bias utils
│ └── framing/ # Framing Effect utils
├── generator/ # Data generation utilities
├── data_generated/ # Generated experimental data
├── webdemo/ # Web demonstration interface
└── requirements.txt # Python dependencies
| Component | Description | Documentation |
|---|---|---|
| Cognitive Bias Index | Measures bias strength via classic experiments | data/data_README.md |
| Behavioral Regulation Engine | Three control methods (Prompt/RepE/Finetune) | control/control_README.md |
| Unified Pipeline | Run full experiments with one command | examples/unified_bias/README.md |
| Ablation Studies | Test model/persona/temperature sensitivity | examples/unified_bias/ablation/README.md |
| Data Generator | Create custom bias scenarios and responses | generator/README.md |
| Bias Type | Paradigms | Data Directory | Control Range |
|---|---|---|---|
| Authority Effect | Milgram Obedience, Stanford Prison | data/authority/ | 0-4 scale |
| Bandwagon Effect | Asch's Line, Hotel Towel | data/bandwagon/ | 0-4 scale |
| Confirmation Bias | Wason Selection, Biased Information | data/confirmation/ | 0-4 scale |
| Framing Effect | Asian Disease, Investment/Insurance | data/framing/ | 0-4 scale |
If you use CoBRA in your research, please cite our paper:
@article{liu2025cobra,
title={CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments},
author={Liu, Xuan and Shang, Haoyang and Jin, Haojian},
journal={arXiv preprint arXiv:2509.13588},
year={2025}
}Paper Link:https://arxiv.org/abs/2509.13588
MIT License - see LICENSE for details
For questions, please contact the corresponding author Xuan Liu at xul049@ucsd.edu, or file a GitHub Issue to report bugs and request features.
Need help? Check examples/unified_bias/README.md for detailed walkthroughs. The finetuning code is in the finetuning branch.