CLLM - Claim LLM

A command-line tool for extracting, evaluating, and comparing scientific claims using Large Language Models.

Overview

CLLM implements a 4-stage workflow for scientific claim verification:

1. Extract Claims: Extract atomic factual claims from a manuscript
2. LLM Evaluation: Group claims into results and evaluate them
3. Peer Review Evaluation: Group claims based on peer review commentary
4. Compare Results: Compare LLM and peer review evaluations

Installation

Prerequisites

• Python >=3.10
• Anthropic API key

Install with uv (recommended)

# From the cllm directorycd cllm
uv pip install -e .

Install with pip

# From the cllm directorycd cllm
pip install -e .

Configuration

Set your Anthropic API key as an environment variable:

export ANTHROPIC_API_KEY=your_api_key_here

Optional configuration:

# Use a different model (default: claude-sonnet-4-5-20250929)export ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
# Set timeout in seconds (default: 600.0)export ANTHROPIC_TIMEOUT=600.0

Usage

Complete Workflow Example

# 1. Extract claims from manuscript
cllm extract manuscript.txt -o claims.json
# 2. Get LLM evaluation
cllm eval manuscript.txt -c claims.json -o eval_llm.json
# 3. Get peer review evaluation
cllm eval manuscript.txt -c claims.json -p peer_reviews.txt -o eval_peers.json
# 4. Compare evaluations
cllm cmp eval_peers.json eval_llm.json -o comparison.json
# Optional: Save metadata for any command
cllm extract manuscript.txt -o claims.json -m metadata.json

Commands

1. Extract Claims

Extract atomic factual claims from a scientific manuscript:

cllm extract manuscript.txt -o claims.json

Options:

• -o, --output: Output JSON file for claims (required)
• -m, --metadata: Optional JSON file for metadata

Output Format:

The output file contains a JSON array of claim objects:

[
{
"claim_id": "C1",
"claim": "The claim text",
"claim_type": "EXPLICIT",
"source_text": "Source quote from manuscript",
"evidence_type": ["DATA", "CITATION"],
"evidence_reasoning": "Explanation of evidence type"
},
{
"claim_id": "C2",
"claim": "Another claim text",
"claim_type": "IMPLICIT",
"source_text": "Source quote",
"evidence_type": ["INFERENCE"],
"evidence_reasoning": "Explanation"
}
]

Metadata Format (if -m is used):

{
"command": "extract",
"manuscript_file": "manuscript.txt",
"manuscript_length": 50000,
"num_claims": 42,
"processing_time_seconds": 15.3
}

2. Evaluate Claims

Evaluate claims and group them into results.

LLM Evaluation:

cllm eval manuscript.txt -c claims.json -o eval_llm.json

Peer Review Evaluation:

cllm eval manuscript.txt -c claims.json -p reviews.txt -o eval_peers.json

Options:

• -c, --claims: Input claims JSON file (required)
• -p, --peer-reviews: Peer review file (optional, changes evaluation mode)
• -o, --output: Output JSON file for evaluations (required)
• -m, --metadata: Optional JSON file for metadata

Output Format:

The output file contains a JSON array of result objects:

[
{
"result_id": "R1",
"claim_ids": ["C1", "C2", "C3"],
"reviewer_id": "LLM",
"reviewer_name": "LLM",
"status": "SUPPORTED",
"status_reasoning": "Explanation of why these claims are grouped and supported"
},
{
"result_id": "R2",
"claim_ids": ["C4"],
"reviewer_id": "0000-0002-1234-5678",
"reviewer_name": "Jane Reviewer",
"status": "UNCERTAIN",
"status_reasoning": "Explanation of uncertainty"
}
]

Metadata Format (if -m is used):

{
"command": "eval",
"manuscript_file": "manuscript.txt",
"claims_file": "claims.json",
"peer_reviews_file": "reviews.txt",
"source": "LLM",
"num_results": 15,
"processing_time_seconds": 22.7
}

3. Compare Results

Compare peer review and LLM evaluations:

cllm cmp eval_peers.json eval_llm.json -o comparison.json

Options:

• -o, --output: Output JSON file for comparison (required)
• -m, --metadata: Optional JSON file for metadata

Output Format:

The output file contains a JSON array of concordance objects:

[
{
"llm_result_id": "R2",
"peer_result_id": "R4",
"llm_status": "SUPPORTED",
"peer_status": "UNSUPPORTED",
"agreement_status": "disagree",
"notes": "Explanation of disagreement"
},
{
"llm_result_id": "R5",
"peer_result_id": null,
"llm_status": "SUPPORTED",
"peer_status": null,
"agreement_status": "disjoint",
"notes": "LLM result with no corresponding peer review"
}
]

Note: For disjoint cases (where a result exists in only one evaluation), the missing side's result_id and status will be null.

Metadata Format (if -m is used):

{
"command": "cmp",
"eval_peers_file": "eval_peers.json",
"eval_llm_file": "eval_llm.json",
"total_comparisons": 20,
"agreements": 15,
"disagreements": 3,
"disjoint": 2,
"agreement_rate": 75.0,
"processing_time_seconds": 8.5
}

Input File Formats

Manuscript Files

Plain text files (.txt) containing the scientific manuscript:

Title: My Scientific Study
Abstract:
This study investigates...
Introduction:
Previous research has shown...

Peer Review Files

Plain text files (.txt) containing peer review comments:

Reviewer 1:
The manuscript presents interesting findings, but...
Reviewer 2:
I have concerns about the methodology...

Features

• 🔍 Atomic Claim Extraction: Identifies discrete, factual statements
• 🤖 LLM Evaluation: Groups and evaluates claims using Claude
• 📝 Peer Review Analysis: Extracts reviewer perspectives on claims
• ⚖️ Concordance Analysis: Compares LLM and peer review assessments
• 💾 JSON Output: All results saved in structured JSON format
• 📊 Metrics: Calculates agreement rates and statistics

Architecture

CLLM is a standalone tool with no external dependencies beyond the Anthropic API:

cllm/
├── cllm/
│ ├── __init__.py
│ ├── cli.py # CLI interface
│ ├── models.py # Pydantic data models
│ ├── verification.py # 4-stage verification logic
│ └── config.py # Configuration management
├── pyproject.toml # Package configuration
└── README.md

Claim Types

• EXPLICIT: Directly stated in the text
• IMPLICIT: Logically follows from what is stated

Evidence Types

• DATA: Based on experimental data or observations
• CITATION: Supported by citation to other work
• KNOWLEDGE: Relies on established scientific knowledge
• INFERENCE: Logical inference from presented information
• SPECULATION: Speculative or hypothetical assertion

Status Values

• SUPPORTED: Claims are well-supported by evidence
• UNSUPPORTED: Claims are not adequately supported
• UNCERTAIN: Insufficient evidence to determine support

Agreement Status

• agree: Same status evaluation from LLM and reviewers
• disagree: Different status evaluations
• disjoint: One evaluator produced a result, but the other did not

Troubleshooting

API Key Not Set

❌ Configuration error: ANTHROPIC_API_KEY environment variable is required.

Solution: Set your API key:

export ANTHROPIC_API_KEY=your_api_key_here

Rate Limiting

If you encounter rate limiting from the Anthropic API, try:

• Waiting a few moments between commands
• Using shorter documents
• Checking your API usage limits

Large Documents

For very large manuscripts (>100,000 characters), processing may take longer. The default timeout is 600 seconds (10 minutes).

Example Workflow

See the example below for a complete end-to-end workflow:

# Set API keyexport ANTHROPIC_API_KEY=sk-ant-...
# Extract claims (with optional metadata)
cllm extract paper.txt -o claims.json -m metadata_extract.json
# Output: 💾 Saved 42 claims to: claims.json# 📊 Saved metadata to: metadata_extract.json# Get LLM evaluation
cllm eval paper.txt -c claims.json -o eval_llm.json -m metadata_eval_llm.json
# Output: 💾 Saved 15 results to: eval_llm.json# 📊 Saved metadata to: metadata_eval_llm.json# Get peer review evaluation
cllm eval paper.txt -c claims.json -p reviews.txt -o eval_peers.json
# Output: 💾 Saved 12 results to: eval_peers.json# Compare evaluations
cllm cmp eval_peers.json eval_llm.json -o comparison.json -m metadata_cmp.json
# Output: 📊 Agreement rate: 75.0%# 💾 Saved 20 concordance rows to: comparison.json# 📊 Saved metadata to: metadata_cmp.json

Note: The -m flag is optional for all commands. If omitted, only the main output file is created.

Development

Running Tests

# Install development dependencies
uv pip install -e ".[dev]"# Run tests (when available)
pytest

Project Structure

• cli.py: Command-line interface using Click
• models.py: Pydantic models for data validation
• verification.py: Core 4-stage verification workflow
• config.py: Configuration management

License

MIT

Support

For issues or questions, please open an issue on GitHub.