QualBench — CI for AI-Generated Code

AI Cost Tracking

• 🤖 LLM usage: $1.1163 (23 commits)
• 👤 Human dev: ~$1136 (11.4h @ $100/h, 30min dedup)

Generated on 2026-07-06 using openrouter/deep/deep-v4-pro

---

Correct code is not the same as mergeable code. eslint + code review, but for AI. Add to your pipeline in 2 minutes.

---

60 seconds to your first score

pip install qualbench
qualbench quickstart

No config, no API keys. QualBench evaluates your current diff and prints a Quality Score.

Add to CI in 2 minutes

# .github/workflows/qualbench.ymlname: QualBenchon: [pull_request]jobs:
quality-check:
runs-on: ubuntu-lateststeps:
- uses: actions/checkout@v4
- uses: semcod/qualbench-action@v1with:
tool: prollamafail_on_score: 70

Every AI-generated PR gets a quality review comment. Set fail_on_score and the pipeline fails if quality is below your threshold.

🧠 QualBench Review
Quality Score: 78/100
❌ Complexity increased (+12%)
⚠ Security: 1 new medium-severity finding
✔ Tests pass, no regressions
Verdict: needs_review

CI/CD Examples

GitHub Action (recommended)

# .github/workflows/qualbench.ymlname: QualBenchon: [pull_request]jobs:
quality-check:
runs-on: ubuntu-lateststeps:
- uses: actions/checkout@v4
- uses: semcod/qualbench-action@v1with:
tool: prollamafail_on_score: 70

GitLab CI

# .gitlab-ci.ymlqualbench:
stage: testimage: python:3.12-slimbefore_script:
- pip install qualbenchscript:
- qualbench run --tool prollama --json --fail-on-score 70only:
- merge_requests

Azure DevOps

# azure-pipelines.ymlsteps:
- task: UsePythonVersion@0inputs:
versionSpec: '3.12'
- script: | pip install qualbench qualbench run --tool prollama --json --fail-on-score 70 displayName: 'QualBench Quality Check'

Jenkins

// Jenkinsfile
stage('Quality Check') {
steps {
sh ''' pip install qualbench qualbench run --tool prollama --fail-on-score 70'''
}
}

CircleCI

# .circleci/config.ymlversion: 2.1jobs:
quality:
docker:
- image: python:3.12-slimsteps:
- checkout
- run: pip install qualbench
- run: qualbench run --tool prollama --fail-on-score 70workflows:
quality-check:
jobs:
- quality

The problem

AI coding tools resolve 70–80% of benchmark tasks. But most AI-generated PRs are not mergeable without human fixes. Every existing benchmark asks "do tests pass?" — nobody asks "would a senior developer approve this PR?"

Six dimensions of production readiness

Dimension	What it measures	Weight
Correctness	All tests pass, no regressions	25%
Mergeability	Would a senior dev merge this? (1–5)	25%
Security	New vulnerabilities introduced	15%
Code quality	Complexity delta, dead code	15%
Iterations	Attempts to reach acceptable output	10%
Cost efficiency	USD per successful patch	10%

Verdicts:ready_to_merge (≥85), needs_review (65–84), not_merge_ready (<65).

CLI

qualbench run --tool prollama # score current diff
qualbench run --tool prollama --json # portable JSON output
qualbench run --mode cheap # lowest-cost models
qualbench quickstart # first score in 60 seconds
qualbench compare my_tool # vs leaderboard
qualbench info # dataset summary
qualbench doctor # check dependencies

One portable format everywhere

CLI, API, GitHub Action — same JSON schema. See docs/schema.md.

Adding your tool

cp runners/template.py runners/my_tool.py
# Implement run() → return portable schema
qualbench run --tool my_tool
# Submit PR with results

License

Licensed under Apache-2.0.

Status

Last updated by taskill at 2026-04-25 13:46 UTC

Metric	Value
HEAD	c199cf4
Coverage	—
Failing tests	—
Commits in last cycle	26

Repository received a mix of fixes, refactors and configuration updates: tests were hardened (mocks added), a JSON test fix was applied, many vallm/style issues and magic numbers were addressed, and release-related features (v0.3.0, Supervisor AI, new runners) were added. PyQual configuration thresholds and gates were adjusted and documentation/TODOs were updated after a PyQual run.