DissBench: Evaluating LLM-written Abstracts for Long Scientific Documents

Master Thesis

A Python-based benchmarking tool for evaluating Large Language Models' ability to generate abstracts for scientific dissertations. Uses vLLM for efficient inference and LLM-based evaluation.

It uses a dataset of dissertations and their corresponding abstracts. First, the candidate LLM is asked to write an abstract for the dissertation. After this, the Judge-LLM evaluates the candidate abstract against the original abstract (from the original author of the dissertation). Evaluation is done content-wise and presentation-wise.

Important Note: The DissBench dataset is required. Unfortunately, it can not be published here, as the Deutsche Nationalbibliothek restricts redistribution of its Open Access dissertations.

Features

• Fast inference using vLLM with configurable model parameters
• Support for both local models (via vLLM) and API-based models (via LiteLLM)
• Batched processing with progress tracking and checkpointing
• Configurable generation and evaluation parameters
• Model-specific configurations via JSON files

Installation

Note: Requires Python 3.11 or 3.12

1. Clone this repository
2. Install dependencies:

pip install -r requirements.txt

Usage

Generating Abstracts

Generate abstracts using a model:

python generate.py \
--model "Qwen/Qwen2.5-7B-Instruct" \
--output-dir "results" \
--batch-size 8 \
--temperature 0.7 \
--dataset-path "path/to/dataset.parquet"

For API-based models (OpenAI, Anthropic, etc.), use the --api flag:

python generate.py \
--model "openai/gpt-4o" \
--output-dir "results" \
--api

Generation Parameters

• --model: Model name (prefix with provider name for API models, e.g., 'openai/', 'anthropic/')
• --output-dir: Directory to save results (default: "results")
• --batch-size: Batch size for inference (default: 1)
• --temperature: Generation temperature (optional)
• --max-tokens: Maximum tokens per generation (default: 1000)
• --limit: Limit number of examples to process
• --dataset-path: Path to local parquet dataset file
• --api: Use API-based models via LiteLLM
• --chat-template-variations: Test different chat template formats (choices: none, template, template+prefill, template+reasoning)

Evaluating Results

Evaluate generated abstracts using an LLM judge:

python evaluate.py \
--results-files "results/model_name_1.json""results/model_name_2.json" \
--judge-model "openai/gpt-4o" \
--output-dir "evaluations" \
--batch-size 1

Evaluation Parameters

• --results-files: One or more paths to generation result files
• --judge-model: Model to use for evaluation (default: "openai/gpt-4-turbo")
• --output-dir: Directory to save evaluations (default: "evaluations")
• --batch-size: Batch size for evaluation (default: 1)
• --temperature: Temperature for judge (optional)
• --max-tokens: Max output tokens for judge (default: 1024)
• --api: Use API-based model via LiteLLM
• --limit: Limit the number of items to evaluate
• --reasoning: Enable step-by-step reasoning in evaluation
• --comment: Add a comment to the evaluation results

Output Format

Generation Results (JSON)

{
"generation_metadata": {
"model_name": "model_name",
"temperature": 0.7,
"max_tokens": 1000,
"use_chat_template": false,
"timestamp": "...",
"sample_size": 100
},
"generations": [
{
"idn": "...",
"category": "...",
"total_text_length": 1234,
"total_token_length": 1000,
"abstract_length": 250,
"generated_abstract": "...",
"original_abstract": "...",
"reasoning_content": null,
"url_dnb_archive": "..."
}
]
}

Evaluation Results (JSON)

{
"generation_metadata": { ... },
"evaluation_metadata": {
"judge_model": "gpt-4-turbo",
"temperature": null,
"max_tokens": 1024,
"batch_size": 1,
"timestamp": "..."
},
"evaluations": [
{
"evaluation": {
"scores": {
"Objectives": 1-10,
"Methods": 1-10,
"Results": 1-10,
"Conclusions": 1-10,
"Coherence": 1-10,
"Academic standards": 1-10
},
"comments": {
"Objectives": "...",
"Methods": "...",
"Results": "...",
"Conclusions": "...",
"Coherence": "...",
"Academic standards": "..."
},
"avg_score": 7.5
},
"evaluation_error": null,
"idn": "...",
"category": "...",
"generated_abstract": "...",
"original_abstract": "..."
}
],
"aggregate_statistics": {
"Objectives": {"mean": 7.5, "min": 5, "max": 9, "std": 1.2},
"Methods": { ... },
"Results": { ... },
"Conclusions": { ... },
"Coherence": { ... },
"Academic standards": { ... },
"Average score": { ... }
}
}