BranchesTags

Repository files navigation

HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

HelloBench is an open-source benchmark designed to evaluate the long text generation capabilities of large language models (LLMs). This repository includes the complete test data and evaluation code from the associated paper:

HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

The test data are curated from platforms like Quora and Reddit, providing diverse, real-world challenges to evaluate LLM performance.

Merge to Opencompass

HelloBench is now merged to Opencompass, you can launch the test script automatically in Opencompass, more details can be found in url

Repository Contents

│ LICENSE
│ llm_judge.py
│ README.md
│ regression.py
│ requirements.txt
│ run.py
│
├─Checklists
│ chat_checklist.json
│ heuristic_text_generation_checklist.json
│ open_ended_qa_checklist.json
│ summarization_checklist.json
│ text_completion_checklist.json
│
├─Annotation_Interface
│ main.py
│ stats.jsonl
│
└─HelloBench
│ chat.jsonl
│ heuristic_text_generation.jsonl
│ open_ended_qa.jsonl
│ summarization.jsonl
│ text_completion.jsonl
│
├─length_constrained_experiments_data
heuristic_text_generation_16k.jsonl
heuristic_text_generation_2k.jsonl
heuristic_text_generation_4k.jsonl
heuristic_text_generation_8k.jsonl

Setup Instructions

Language Requirements: Python 3.10 or later

To set up the environment, run the following command in your terminal:

pip install -r requirements.txt

Usage Guidelines

  • Test Data: The core test data can be found in the HelloBench directory.
  • Evaluation Checklists: For predefined evaluation criteria, refer to the Checklists directory.
  • Human Evaluation: To facilitate human evaluation, the required code is located in the Annotation_Interface directory.
  • LLM Response Generation: Use run.py to call an LLM to generate responses for the tasks.
  • LLM Response Judging: To invoke LLMs for evaluating responses, run llm_judge.py.
  • Regression: For Linear Regression, execute regression.py.

For additional details and advanced usage, please refer to the code comments and paper.

Generation Configuration

Here are the generation configurations for different models.

Model Namemax_new_tokenstemperatureversion
GPT-4o-2024-0806163840.8gpt-4o-2024-08-06
Mistral-Large-API163840.8mistral-large-latest
o1-Mini3276810.8o1-mini
Claude-3.5-Sonnet819220.8claude-3-5-sonnet-20240620
Gemini-1.5-Pro819230.8gemini-1.5-pro
Deepseek-API409640.8deepseek-chat
Yi-Large163840.8yi-large
Qwen-Max200050.8qwen-max-0428
GLM-4-API409660.8glm-4-0520
Gemma-2-27B409670.8google/gemma-2-27b-it
LLaMA-3.1-70B163840.8meta-llama/Meta-Llama-3.1-70B-Instruct
Qwen-2-72B163840.8Qwen/Qwen2-72B-Instruct
InternLM-2.5-20B163840.8internlm/internlm2_5-20b-chat
Yi-1.5-34B204880.801-ai/Yi-1.5-34B-Chat
LLaMA-3.1-8B163840.8meta-llama/Meta-Llama-3.1-8B-Instruct
GLM-4-9B163840.8THUDM/glm-4-9b-chat
Qwen-2-7B163840.8Qwen/Qwen2-7B-Instruct
InternLM-2.5-7B163840.8internlm/internlm2_5-7b-chat
Mistral-7B-0.2163840.8mistralai/Mistral-7B-Instruct-v0.2
Phi-3.5-Moe163840.8microsoft/Phi-3.5-MoE-instruct
MAP-Neo204890.8m-a-p/neo_7b_instruct_v0.1
LongWriter-GLM4-9B163840.8THUDM/LongWriter-glm4-9b
Suri-I-ORPO163840.8chtmp223/suri-i-orpo
Yi-1.5-34B-16K8192100.801-ai/Yi-1.5-34B-Chat-16K
InternLM-2.5-7B-1M163840.8internlm/internlm2_5-7b-chat-1m
GLM-4-9B-1M163840.8THUDM/glm-4-9b-chat-1m

Footnotes

  1. For the o1-mini model, the parameter here should be max_completion_tokens instead of max_new_tokens, because it includes reasoning tokens. Therefore, I set it to 32768. You can refer to https://platform.openai.com/docs/guides/reasoning#controlling-costs for more details. ↩

  2. For the reason that claude-3.5-sonnet has max output 8192 tokens. You can refer to https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table for more details. ↩

  3. For the reason that gemini-1.5-pro has max output 8192 tokens. You can refer to https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro for more details. ↩

  4. For the reason that deepseek-chat has max output 4096 tokens. You can refer to https://api-docs.deepseek.com/zh-cn/quick_start/pricing for more details. ↩

  5. For the reason that qwen-max-0428 has max output 2000 tokens. You can refer to https://help.aliyun.com/zh/model-studio/getting-started/models?spm=a2c4g.11186623.0.0.74b04823IseC0N#9f8890ce29g5u for more details. ↩

  6. For the reason that glm-4-0520 has max output 4096 tokens. You can refer to https://bigmodel.cn/dev/howuse/model for more details. ↩

  7. Gemma-2-27B has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=4096. ↩

  8. Yi-1.5-34B-Chat has max_position_embeddings with 4096 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩

  9. neo_7b_instruct_v0.1 has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩

  10. Yi-1.5-34B-Chat-16K has max_position_embeddings with 16384 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=8192. ↩

About

HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

Resources

Readme
MIT license
Activity

Stars

60 stars

Watchers

1 watching

Forks

3 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
HelloBench is an open-source benchmark designed to evaluate the long text generation capabilities of large language models (LLMs). This repository includes the complete test data and evaluation code from the associated paper:
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
The test data are curated from platforms like Quora and Reddit, providing diverse, real-world challenges to evaluate LLM performance.
Merge to Opencompass
HelloBench is now merged to Opencompass, you can launch the test script automatically in Opencompass, more details can be found in url
Repository Contents
│ LICENSE
│ llm_judge.py
│ README.md
│ regression.py
│ requirements.txt
│ run.py
│
├─Checklists
│ chat_checklist.json
│ heuristic_text_generation_checklist.json
│ open_ended_qa_checklist.json
│ summarization_checklist.json
│ text_completion_checklist.json
│
├─Annotation_Interface
│ main.py
│ stats.jsonl
│
└─HelloBench
│ chat.jsonl
│ heuristic_text_generation.jsonl
│ open_ended_qa.jsonl
│ summarization.jsonl
│ text_completion.jsonl
│
├─length_constrained_experiments_data
heuristic_text_generation_16k.jsonl
heuristic_text_generation_2k.jsonl
heuristic_text_generation_4k.jsonl
heuristic_text_generation_8k.jsonl

Setup Instructions
Language Requirements: Python 3.10 or later
To set up the environment, run the following command in your terminal:
pip install -r requirements.txt
Usage Guidelines
  • Test Data: The core test data can be found in the HelloBench directory.
  • Evaluation Checklists: For predefined evaluation criteria, refer to the Checklists directory.
  • Human Evaluation: To facilitate human evaluation, the required code is located in the Annotation_Interface directory.
  • LLM Response Generation: Use run.py to call an LLM to generate responses for the tasks.
  • LLM Response Judging: To invoke LLMs for evaluating responses, run llm_judge.py.
  • Regression: For Linear Regression, execute regression.py.
For additional details and advanced usage, please refer to the code comments and paper.
Generation Configuration
Here are the generation configurations for different models.
Model Namemax_new_tokenstemperatureversion
GPT-4o-2024-0806163840.8gpt-4o-2024-08-06
Mistral-Large-API163840.8mistral-large-latest
o1-Mini3276810.8o1-mini
Claude-3.5-Sonnet819220.8claude-3-5-sonnet-20240620
Gemini-1.5-Pro819230.8gemini-1.5-pro
Deepseek-API409640.8deepseek-chat
Yi-Large163840.8yi-large
Qwen-Max200050.8qwen-max-0428
GLM-4-API409660.8glm-4-0520
Gemma-2-27B409670.8google/gemma-2-27b-it
LLaMA-3.1-70B163840.8meta-llama/Meta-Llama-3.1-70B-Instruct
Qwen-2-72B163840.8Qwen/Qwen2-72B-Instruct
InternLM-2.5-20B163840.8internlm/internlm2_5-20b-chat
Yi-1.5-34B204880.801-ai/Yi-1.5-34B-Chat
LLaMA-3.1-8B163840.8meta-llama/Meta-Llama-3.1-8B-Instruct
GLM-4-9B163840.8THUDM/glm-4-9b-chat
Qwen-2-7B163840.8Qwen/Qwen2-7B-Instruct
InternLM-2.5-7B163840.8internlm/internlm2_5-7b-chat
Mistral-7B-0.2163840.8mistralai/Mistral-7B-Instruct-v0.2
Phi-3.5-Moe163840.8microsoft/Phi-3.5-MoE-instruct
MAP-Neo204890.8m-a-p/neo_7b_instruct_v0.1
LongWriter-GLM4-9B163840.8THUDM/LongWriter-glm4-9b
Suri-I-ORPO163840.8chtmp223/suri-i-orpo
Yi-1.5-34B-16K8192100.801-ai/Yi-1.5-34B-Chat-16K
InternLM-2.5-7B-1M163840.8internlm/internlm2_5-7b-chat-1m
GLM-4-9B-1M163840.8THUDM/glm-4-9b-chat-1m
Footnotes
  1. For the o1-mini model, the parameter here should be max_completion_tokens instead of max_new_tokens, because it includes reasoning tokens. Therefore, I set it to 32768. You can refer to https://platform.openai.com/docs/guides/reasoning#controlling-costs for more details. ↩
  2. For the reason that claude-3.5-sonnet has max output 8192 tokens. You can refer to https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table for more details. ↩
  3. For the reason that gemini-1.5-pro has max output 8192 tokens. You can refer to https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro for more details. ↩
  4. For the reason that deepseek-chat has max output 4096 tokens. You can refer to https://api-docs.deepseek.com/zh-cn/quick_start/pricing for more details. ↩
  5. For the reason that qwen-max-0428 has max output 2000 tokens. You can refer to https://help.aliyun.com/zh/model-studio/getting-started/models?spm=a2c4g.11186623.0.0.74b04823IseC0N#9f8890ce29g5u for more details. ↩
  6. For the reason that glm-4-0520 has max output 4096 tokens. You can refer to https://bigmodel.cn/dev/howuse/model for more details. ↩
  7. Gemma-2-27B has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=4096. ↩
  8. Yi-1.5-34B-Chat has max_position_embeddings with 4096 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  9. neo_7b_instruct_v0.1 has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  10. Yi-1.5-34B-Chat-16K has max_position_embeddings with 16384 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=8192. ↩
About
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Resources
Readme
MIT license
Activity
Stars
60 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
HelloBench is an open-source benchmark designed to evaluate the long text generation capabilities of large language models (LLMs). This repository includes the complete test data and evaluation code from the associated paper:
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
The test data are curated from platforms like Quora and Reddit, providing diverse, real-world challenges to evaluate LLM performance.
Merge to Opencompass
HelloBench is now merged to Opencompass, you can launch the test script automatically in Opencompass, more details can be found in url
Repository Contents
│ LICENSE
│ llm_judge.py
│ README.md
│ regression.py
│ requirements.txt
│ run.py
│
├─Checklists
│ chat_checklist.json
│ heuristic_text_generation_checklist.json
│ open_ended_qa_checklist.json
│ summarization_checklist.json
│ text_completion_checklist.json
│
├─Annotation_Interface
│ main.py
│ stats.jsonl
│
└─HelloBench
│ chat.jsonl
│ heuristic_text_generation.jsonl
│ open_ended_qa.jsonl
│ summarization.jsonl
│ text_completion.jsonl
│
├─length_constrained_experiments_data
heuristic_text_generation_16k.jsonl
heuristic_text_generation_2k.jsonl
heuristic_text_generation_4k.jsonl
heuristic_text_generation_8k.jsonl

Setup Instructions
Language Requirements: Python 3.10 or later
To set up the environment, run the following command in your terminal:
pip install -r requirements.txt
Usage Guidelines
  • Test Data: The core test data can be found in the HelloBench directory.
  • Evaluation Checklists: For predefined evaluation criteria, refer to the Checklists directory.
  • Human Evaluation: To facilitate human evaluation, the required code is located in the Annotation_Interface directory.
  • LLM Response Generation: Use run.py to call an LLM to generate responses for the tasks.
  • LLM Response Judging: To invoke LLMs for evaluating responses, run llm_judge.py.
  • Regression: For Linear Regression, execute regression.py.
For additional details and advanced usage, please refer to the code comments and paper.
Generation Configuration
Here are the generation configurations for different models.
Model Namemax_new_tokenstemperatureversion
GPT-4o-2024-0806163840.8gpt-4o-2024-08-06
Mistral-Large-API163840.8mistral-large-latest
o1-Mini3276810.8o1-mini
Claude-3.5-Sonnet819220.8claude-3-5-sonnet-20240620
Gemini-1.5-Pro819230.8gemini-1.5-pro
Deepseek-API409640.8deepseek-chat
Yi-Large163840.8yi-large
Qwen-Max200050.8qwen-max-0428
GLM-4-API409660.8glm-4-0520
Gemma-2-27B409670.8google/gemma-2-27b-it
LLaMA-3.1-70B163840.8meta-llama/Meta-Llama-3.1-70B-Instruct
Qwen-2-72B163840.8Qwen/Qwen2-72B-Instruct
InternLM-2.5-20B163840.8internlm/internlm2_5-20b-chat
Yi-1.5-34B204880.801-ai/Yi-1.5-34B-Chat
LLaMA-3.1-8B163840.8meta-llama/Meta-Llama-3.1-8B-Instruct
GLM-4-9B163840.8THUDM/glm-4-9b-chat
Qwen-2-7B163840.8Qwen/Qwen2-7B-Instruct
InternLM-2.5-7B163840.8internlm/internlm2_5-7b-chat
Mistral-7B-0.2163840.8mistralai/Mistral-7B-Instruct-v0.2
Phi-3.5-Moe163840.8microsoft/Phi-3.5-MoE-instruct
MAP-Neo204890.8m-a-p/neo_7b_instruct_v0.1
LongWriter-GLM4-9B163840.8THUDM/LongWriter-glm4-9b
Suri-I-ORPO163840.8chtmp223/suri-i-orpo
Yi-1.5-34B-16K8192100.801-ai/Yi-1.5-34B-Chat-16K
InternLM-2.5-7B-1M163840.8internlm/internlm2_5-7b-chat-1m
GLM-4-9B-1M163840.8THUDM/glm-4-9b-chat-1m
Footnotes
  1. For the o1-mini model, the parameter here should be max_completion_tokens instead of max_new_tokens, because it includes reasoning tokens. Therefore, I set it to 32768. You can refer to https://platform.openai.com/docs/guides/reasoning#controlling-costs for more details. ↩
  2. For the reason that claude-3.5-sonnet has max output 8192 tokens. You can refer to https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table for more details. ↩
  3. For the reason that gemini-1.5-pro has max output 8192 tokens. You can refer to https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro for more details. ↩
  4. For the reason that deepseek-chat has max output 4096 tokens. You can refer to https://api-docs.deepseek.com/zh-cn/quick_start/pricing for more details. ↩
  5. For the reason that qwen-max-0428 has max output 2000 tokens. You can refer to https://help.aliyun.com/zh/model-studio/getting-started/models?spm=a2c4g.11186623.0.0.74b04823IseC0N#9f8890ce29g5u for more details. ↩
  6. For the reason that glm-4-0520 has max output 4096 tokens. You can refer to https://bigmodel.cn/dev/howuse/model for more details. ↩
  7. Gemma-2-27B has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=4096. ↩
  8. Yi-1.5-34B-Chat has max_position_embeddings with 4096 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  9. neo_7b_instruct_v0.1 has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  10. Yi-1.5-34B-Chat-16K has max_position_embeddings with 16384 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=8192. ↩
About
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Resources
Readme
MIT license
Activity
Stars
60 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
HelloBench is an open-source benchmark designed to evaluate the long text generation capabilities of large language models (LLMs). This repository includes the complete test data and evaluation code from the associated paper:
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
The test data are curated from platforms like Quora and Reddit, providing diverse, real-world challenges to evaluate LLM performance.
Merge to Opencompass
HelloBench is now merged to Opencompass, you can launch the test script automatically in Opencompass, more details can be found in url
Repository Contents
│ LICENSE
│ llm_judge.py
│ README.md
│ regression.py
│ requirements.txt
│ run.py
│
├─Checklists
│ chat_checklist.json
│ heuristic_text_generation_checklist.json
│ open_ended_qa_checklist.json
│ summarization_checklist.json
│ text_completion_checklist.json
│
├─Annotation_Interface
│ main.py
│ stats.jsonl
│
└─HelloBench
│ chat.jsonl
│ heuristic_text_generation.jsonl
│ open_ended_qa.jsonl
│ summarization.jsonl
│ text_completion.jsonl
│
├─length_constrained_experiments_data
heuristic_text_generation_16k.jsonl
heuristic_text_generation_2k.jsonl
heuristic_text_generation_4k.jsonl
heuristic_text_generation_8k.jsonl

Setup Instructions
Language Requirements: Python 3.10 or later
To set up the environment, run the following command in your terminal:
pip install -r requirements.txt
Usage Guidelines
  • Test Data: The core test data can be found in the HelloBench directory.
  • Evaluation Checklists: For predefined evaluation criteria, refer to the Checklists directory.
  • Human Evaluation: To facilitate human evaluation, the required code is located in the Annotation_Interface directory.
  • LLM Response Generation: Use run.py to call an LLM to generate responses for the tasks.
  • LLM Response Judging: To invoke LLMs for evaluating responses, run llm_judge.py.
  • Regression: For Linear Regression, execute regression.py.
For additional details and advanced usage, please refer to the code comments and paper.
Generation Configuration
Here are the generation configurations for different models.
Model Namemax_new_tokenstemperatureversion
GPT-4o-2024-0806163840.8gpt-4o-2024-08-06
Mistral-Large-API163840.8mistral-large-latest
o1-Mini3276810.8o1-mini
Claude-3.5-Sonnet819220.8claude-3-5-sonnet-20240620
Gemini-1.5-Pro819230.8gemini-1.5-pro
Deepseek-API409640.8deepseek-chat
Yi-Large163840.8yi-large
Qwen-Max200050.8qwen-max-0428
GLM-4-API409660.8glm-4-0520
Gemma-2-27B409670.8google/gemma-2-27b-it
LLaMA-3.1-70B163840.8meta-llama/Meta-Llama-3.1-70B-Instruct
Qwen-2-72B163840.8Qwen/Qwen2-72B-Instruct
InternLM-2.5-20B163840.8internlm/internlm2_5-20b-chat
Yi-1.5-34B204880.801-ai/Yi-1.5-34B-Chat
LLaMA-3.1-8B163840.8meta-llama/Meta-Llama-3.1-8B-Instruct
GLM-4-9B163840.8THUDM/glm-4-9b-chat
Qwen-2-7B163840.8Qwen/Qwen2-7B-Instruct
InternLM-2.5-7B163840.8internlm/internlm2_5-7b-chat
Mistral-7B-0.2163840.8mistralai/Mistral-7B-Instruct-v0.2
Phi-3.5-Moe163840.8microsoft/Phi-3.5-MoE-instruct
MAP-Neo204890.8m-a-p/neo_7b_instruct_v0.1
LongWriter-GLM4-9B163840.8THUDM/LongWriter-glm4-9b
Suri-I-ORPO163840.8chtmp223/suri-i-orpo
Yi-1.5-34B-16K8192100.801-ai/Yi-1.5-34B-Chat-16K
InternLM-2.5-7B-1M163840.8internlm/internlm2_5-7b-chat-1m
GLM-4-9B-1M163840.8THUDM/glm-4-9b-chat-1m
Footnotes
  1. For the o1-mini model, the parameter here should be max_completion_tokens instead of max_new_tokens, because it includes reasoning tokens. Therefore, I set it to 32768. You can refer to https://platform.openai.com/docs/guides/reasoning#controlling-costs for more details. ↩
  2. For the reason that claude-3.5-sonnet has max output 8192 tokens. You can refer to https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table for more details. ↩
  3. For the reason that gemini-1.5-pro has max output 8192 tokens. You can refer to https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro for more details. ↩
  4. For the reason that deepseek-chat has max output 4096 tokens. You can refer to https://api-docs.deepseek.com/zh-cn/quick_start/pricing for more details. ↩
  5. For the reason that qwen-max-0428 has max output 2000 tokens. You can refer to https://help.aliyun.com/zh/model-studio/getting-started/models?spm=a2c4g.11186623.0.0.74b04823IseC0N#9f8890ce29g5u for more details. ↩
  6. For the reason that glm-4-0520 has max output 4096 tokens. You can refer to https://bigmodel.cn/dev/howuse/model for more details. ↩
  7. Gemma-2-27B has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=4096. ↩
  8. Yi-1.5-34B-Chat has max_position_embeddings with 4096 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  9. neo_7b_instruct_v0.1 has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  10. Yi-1.5-34B-Chat-16K has max_position_embeddings with 16384 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=8192. ↩
About
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Resources
Readme
MIT license
Activity
Stars
60 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
HelloBench is an open-source benchmark designed to evaluate the long text generation capabilities of large language models (LLMs). This repository includes the complete test data and evaluation code from the associated paper:
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
The test data are curated from platforms like Quora and Reddit, providing diverse, real-world challenges to evaluate LLM performance.
Merge to Opencompass
HelloBench is now merged to Opencompass, you can launch the test script automatically in Opencompass, more details can be found in url
Repository Contents
│ LICENSE
│ llm_judge.py
│ README.md
│ regression.py
│ requirements.txt
│ run.py
│
├─Checklists
│ chat_checklist.json
│ heuristic_text_generation_checklist.json
│ open_ended_qa_checklist.json
│ summarization_checklist.json
│ text_completion_checklist.json
│
├─Annotation_Interface
│ main.py
│ stats.jsonl
│
└─HelloBench
│ chat.jsonl
│ heuristic_text_generation.jsonl
│ open_ended_qa.jsonl
│ summarization.jsonl
│ text_completion.jsonl
│
├─length_constrained_experiments_data
heuristic_text_generation_16k.jsonl
heuristic_text_generation_2k.jsonl
heuristic_text_generation_4k.jsonl
heuristic_text_generation_8k.jsonl

Setup Instructions
Language Requirements: Python 3.10 or later
To set up the environment, run the following command in your terminal:
pip install -r requirements.txt
Usage Guidelines
  • Test Data: The core test data can be found in the HelloBench directory.
  • Evaluation Checklists: For predefined evaluation criteria, refer to the Checklists directory.
  • Human Evaluation: To facilitate human evaluation, the required code is located in the Annotation_Interface directory.
  • LLM Response Generation: Use run.py to call an LLM to generate responses for the tasks.
  • LLM Response Judging: To invoke LLMs for evaluating responses, run llm_judge.py.
  • Regression: For Linear Regression, execute regression.py.
For additional details and advanced usage, please refer to the code comments and paper.
Generation Configuration
Here are the generation configurations for different models.
Model Namemax_new_tokenstemperatureversion
GPT-4o-2024-0806163840.8gpt-4o-2024-08-06
Mistral-Large-API163840.8mistral-large-latest
o1-Mini3276810.8o1-mini
Claude-3.5-Sonnet819220.8claude-3-5-sonnet-20240620
Gemini-1.5-Pro819230.8gemini-1.5-pro
Deepseek-API409640.8deepseek-chat
Yi-Large163840.8yi-large
Qwen-Max200050.8qwen-max-0428
GLM-4-API409660.8glm-4-0520
Gemma-2-27B409670.8google/gemma-2-27b-it
LLaMA-3.1-70B163840.8meta-llama/Meta-Llama-3.1-70B-Instruct
Qwen-2-72B163840.8Qwen/Qwen2-72B-Instruct
InternLM-2.5-20B163840.8internlm/internlm2_5-20b-chat
Yi-1.5-34B204880.801-ai/Yi-1.5-34B-Chat
LLaMA-3.1-8B163840.8meta-llama/Meta-Llama-3.1-8B-Instruct
GLM-4-9B163840.8THUDM/glm-4-9b-chat
Qwen-2-7B163840.8Qwen/Qwen2-7B-Instruct
InternLM-2.5-7B163840.8internlm/internlm2_5-7b-chat
Mistral-7B-0.2163840.8mistralai/Mistral-7B-Instruct-v0.2
Phi-3.5-Moe163840.8microsoft/Phi-3.5-MoE-instruct
MAP-Neo204890.8m-a-p/neo_7b_instruct_v0.1
LongWriter-GLM4-9B163840.8THUDM/LongWriter-glm4-9b
Suri-I-ORPO163840.8chtmp223/suri-i-orpo
Yi-1.5-34B-16K8192100.801-ai/Yi-1.5-34B-Chat-16K
InternLM-2.5-7B-1M163840.8internlm/internlm2_5-7b-chat-1m
GLM-4-9B-1M163840.8THUDM/glm-4-9b-chat-1m
Footnotes
  1. For the o1-mini model, the parameter here should be max_completion_tokens instead of max_new_tokens, because it includes reasoning tokens. Therefore, I set it to 32768. You can refer to https://platform.openai.com/docs/guides/reasoning#controlling-costs for more details. ↩
  2. For the reason that claude-3.5-sonnet has max output 8192 tokens. You can refer to https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table for more details. ↩
  3. For the reason that gemini-1.5-pro has max output 8192 tokens. You can refer to https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro for more details. ↩
  4. For the reason that deepseek-chat has max output 4096 tokens. You can refer to https://api-docs.deepseek.com/zh-cn/quick_start/pricing for more details. ↩
  5. For the reason that qwen-max-0428 has max output 2000 tokens. You can refer to https://help.aliyun.com/zh/model-studio/getting-started/models?spm=a2c4g.11186623.0.0.74b04823IseC0N#9f8890ce29g5u for more details. ↩
  6. For the reason that glm-4-0520 has max output 4096 tokens. You can refer to https://bigmodel.cn/dev/howuse/model for more details. ↩
  7. Gemma-2-27B has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=4096. ↩
  8. Yi-1.5-34B-Chat has max_position_embeddings with 4096 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  9. neo_7b_instruct_v0.1 has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  10. Yi-1.5-34B-Chat-16K has max_position_embeddings with 16384 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=8192. ↩
About
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Resources
Readme
MIT license
Activity
Stars
60 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
HelloBench is an open-source benchmark designed to evaluate the long text generation capabilities of large language models (LLMs). This repository includes the complete test data and evaluation code from the associated paper:
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
The test data are curated from platforms like Quora and Reddit, providing diverse, real-world challenges to evaluate LLM performance.
Merge to Opencompass
HelloBench is now merged to Opencompass, you can launch the test script automatically in Opencompass, more details can be found in url
Repository Contents
│ LICENSE
│ llm_judge.py
│ README.md
│ regression.py
│ requirements.txt
│ run.py
│
├─Checklists
│ chat_checklist.json
│ heuristic_text_generation_checklist.json
│ open_ended_qa_checklist.json
│ summarization_checklist.json
│ text_completion_checklist.json
│
├─Annotation_Interface
│ main.py
│ stats.jsonl
│
└─HelloBench
│ chat.jsonl
│ heuristic_text_generation.jsonl
│ open_ended_qa.jsonl
│ summarization.jsonl
│ text_completion.jsonl
│
├─length_constrained_experiments_data
heuristic_text_generation_16k.jsonl
heuristic_text_generation_2k.jsonl
heuristic_text_generation_4k.jsonl
heuristic_text_generation_8k.jsonl

Setup Instructions
Language Requirements: Python 3.10 or later
To set up the environment, run the following command in your terminal:
pip install -r requirements.txt
Usage Guidelines
  • Test Data: The core test data can be found in the HelloBench directory.
  • Evaluation Checklists: For predefined evaluation criteria, refer to the Checklists directory.
  • Human Evaluation: To facilitate human evaluation, the required code is located in the Annotation_Interface directory.
  • LLM Response Generation: Use run.py to call an LLM to generate responses for the tasks.
  • LLM Response Judging: To invoke LLMs for evaluating responses, run llm_judge.py.
  • Regression: For Linear Regression, execute regression.py.
For additional details and advanced usage, please refer to the code comments and paper.
Generation Configuration
Here are the generation configurations for different models.
Model Namemax_new_tokenstemperatureversion
GPT-4o-2024-0806163840.8gpt-4o-2024-08-06
Mistral-Large-API163840.8mistral-large-latest
o1-Mini3276810.8o1-mini
Claude-3.5-Sonnet819220.8claude-3-5-sonnet-20240620
Gemini-1.5-Pro819230.8gemini-1.5-pro
Deepseek-API409640.8deepseek-chat
Yi-Large163840.8yi-large
Qwen-Max200050.8qwen-max-0428
GLM-4-API409660.8glm-4-0520
Gemma-2-27B409670.8google/gemma-2-27b-it
LLaMA-3.1-70B163840.8meta-llama/Meta-Llama-3.1-70B-Instruct
Qwen-2-72B163840.8Qwen/Qwen2-72B-Instruct
InternLM-2.5-20B163840.8internlm/internlm2_5-20b-chat
Yi-1.5-34B204880.801-ai/Yi-1.5-34B-Chat
LLaMA-3.1-8B163840.8meta-llama/Meta-Llama-3.1-8B-Instruct
GLM-4-9B163840.8THUDM/glm-4-9b-chat
Qwen-2-7B163840.8Qwen/Qwen2-7B-Instruct
InternLM-2.5-7B163840.8internlm/internlm2_5-7b-chat
Mistral-7B-0.2163840.8mistralai/Mistral-7B-Instruct-v0.2
Phi-3.5-Moe163840.8microsoft/Phi-3.5-MoE-instruct
MAP-Neo204890.8m-a-p/neo_7b_instruct_v0.1
LongWriter-GLM4-9B163840.8THUDM/LongWriter-glm4-9b
Suri-I-ORPO163840.8chtmp223/suri-i-orpo
Yi-1.5-34B-16K8192100.801-ai/Yi-1.5-34B-Chat-16K
InternLM-2.5-7B-1M163840.8internlm/internlm2_5-7b-chat-1m
GLM-4-9B-1M163840.8THUDM/glm-4-9b-chat-1m
Footnotes
  1. For the o1-mini model, the parameter here should be max_completion_tokens instead of max_new_tokens, because it includes reasoning tokens. Therefore, I set it to 32768. You can refer to https://platform.openai.com/docs/guides/reasoning#controlling-costs for more details. ↩
  2. For the reason that claude-3.5-sonnet has max output 8192 tokens. You can refer to https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table for more details. ↩
  3. For the reason that gemini-1.5-pro has max output 8192 tokens. You can refer to https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro for more details. ↩
  4. For the reason that deepseek-chat has max output 4096 tokens. You can refer to https://api-docs.deepseek.com/zh-cn/quick_start/pricing for more details. ↩
  5. For the reason that qwen-max-0428 has max output 2000 tokens. You can refer to https://help.aliyun.com/zh/model-studio/getting-started/models?spm=a2c4g.11186623.0.0.74b04823IseC0N#9f8890ce29g5u for more details. ↩
  6. For the reason that glm-4-0520 has max output 4096 tokens. You can refer to https://bigmodel.cn/dev/howuse/model for more details. ↩
  7. Gemma-2-27B has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=4096. ↩
  8. Yi-1.5-34B-Chat has max_position_embeddings with 4096 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  9. neo_7b_instruct_v0.1 has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  10. Yi-1.5-34B-Chat-16K has max_position_embeddings with 16384 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=8192. ↩
About
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Resources
Readme
MIT license
Activity
Stars
60 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
HelloBench is an open-source benchmark designed to evaluate the long text generation capabilities of large language models (LLMs). This repository includes the complete test data and evaluation code from the associated paper:
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
The test data are curated from platforms like Quora and Reddit, providing diverse, real-world challenges to evaluate LLM performance.
Merge to Opencompass
HelloBench is now merged to Opencompass, you can launch the test script automatically in Opencompass, more details can be found in url
Repository Contents
│ LICENSE
│ llm_judge.py
│ README.md
│ regression.py
│ requirements.txt
│ run.py
│
├─Checklists
│ chat_checklist.json
│ heuristic_text_generation_checklist.json
│ open_ended_qa_checklist.json
│ summarization_checklist.json
│ text_completion_checklist.json
│
├─Annotation_Interface
│ main.py
│ stats.jsonl
│
└─HelloBench
│ chat.jsonl
│ heuristic_text_generation.jsonl
│ open_ended_qa.jsonl
│ summarization.jsonl
│ text_completion.jsonl
│
├─length_constrained_experiments_data
heuristic_text_generation_16k.jsonl
heuristic_text_generation_2k.jsonl
heuristic_text_generation_4k.jsonl
heuristic_text_generation_8k.jsonl

Setup Instructions
Language Requirements: Python 3.10 or later
To set up the environment, run the following command in your terminal:
pip install -r requirements.txt
Usage Guidelines
  • Test Data: The core test data can be found in the HelloBench directory.
  • Evaluation Checklists: For predefined evaluation criteria, refer to the Checklists directory.
  • Human Evaluation: To facilitate human evaluation, the required code is located in the Annotation_Interface directory.
  • LLM Response Generation: Use run.py to call an LLM to generate responses for the tasks.
  • LLM Response Judging: To invoke LLMs for evaluating responses, run llm_judge.py.
  • Regression: For Linear Regression, execute regression.py.
For additional details and advanced usage, please refer to the code comments and paper.
Generation Configuration
Here are the generation configurations for different models.
Model Namemax_new_tokenstemperatureversion
GPT-4o-2024-0806163840.8gpt-4o-2024-08-06
Mistral-Large-API163840.8mistral-large-latest
o1-Mini3276810.8o1-mini
Claude-3.5-Sonnet819220.8claude-3-5-sonnet-20240620
Gemini-1.5-Pro819230.8gemini-1.5-pro
Deepseek-API409640.8deepseek-chat
Yi-Large163840.8yi-large
Qwen-Max200050.8qwen-max-0428
GLM-4-API409660.8glm-4-0520
Gemma-2-27B409670.8google/gemma-2-27b-it
LLaMA-3.1-70B163840.8meta-llama/Meta-Llama-3.1-70B-Instruct
Qwen-2-72B163840.8Qwen/Qwen2-72B-Instruct
InternLM-2.5-20B163840.8internlm/internlm2_5-20b-chat
Yi-1.5-34B204880.801-ai/Yi-1.5-34B-Chat
LLaMA-3.1-8B163840.8meta-llama/Meta-Llama-3.1-8B-Instruct
GLM-4-9B163840.8THUDM/glm-4-9b-chat
Qwen-2-7B163840.8Qwen/Qwen2-7B-Instruct
InternLM-2.5-7B163840.8internlm/internlm2_5-7b-chat
Mistral-7B-0.2163840.8mistralai/Mistral-7B-Instruct-v0.2
Phi-3.5-Moe163840.8microsoft/Phi-3.5-MoE-instruct
MAP-Neo204890.8m-a-p/neo_7b_instruct_v0.1
LongWriter-GLM4-9B163840.8THUDM/LongWriter-glm4-9b
Suri-I-ORPO163840.8chtmp223/suri-i-orpo
Yi-1.5-34B-16K8192100.801-ai/Yi-1.5-34B-Chat-16K
InternLM-2.5-7B-1M163840.8internlm/internlm2_5-7b-chat-1m
GLM-4-9B-1M163840.8THUDM/glm-4-9b-chat-1m
Footnotes
  1. For the o1-mini model, the parameter here should be max_completion_tokens instead of max_new_tokens, because it includes reasoning tokens. Therefore, I set it to 32768. You can refer to https://platform.openai.com/docs/guides/reasoning#controlling-costs for more details. ↩
  2. For the reason that claude-3.5-sonnet has max output 8192 tokens. You can refer to https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table for more details. ↩
  3. For the reason that gemini-1.5-pro has max output 8192 tokens. You can refer to https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro for more details. ↩
  4. For the reason that deepseek-chat has max output 4096 tokens. You can refer to https://api-docs.deepseek.com/zh-cn/quick_start/pricing for more details. ↩
  5. For the reason that qwen-max-0428 has max output 2000 tokens. You can refer to https://help.aliyun.com/zh/model-studio/getting-started/models?spm=a2c4g.11186623.0.0.74b04823IseC0N#9f8890ce29g5u for more details. ↩
  6. For the reason that glm-4-0520 has max output 4096 tokens. You can refer to https://bigmodel.cn/dev/howuse/model for more details. ↩
  7. Gemma-2-27B has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=4096. ↩
  8. Yi-1.5-34B-Chat has max_position_embeddings with 4096 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  9. neo_7b_instruct_v0.1 has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  10. Yi-1.5-34B-Chat-16K has max_position_embeddings with 16384 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=8192. ↩
About
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Resources
Readme
MIT license
Activity
Stars
60 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
HelloBench is an open-source benchmark designed to evaluate the long text generation capabilities of large language models (LLMs). This repository includes the complete test data and evaluation code from the associated paper:
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
The test data are curated from platforms like Quora and Reddit, providing diverse, real-world challenges to evaluate LLM performance.
Merge to Opencompass
HelloBench is now merged to Opencompass, you can launch the test script automatically in Opencompass, more details can be found in url
Repository Contents
│ LICENSE
│ llm_judge.py
│ README.md
│ regression.py
│ requirements.txt
│ run.py
│
├─Checklists
│ chat_checklist.json
│ heuristic_text_generation_checklist.json
│ open_ended_qa_checklist.json
│ summarization_checklist.json
│ text_completion_checklist.json
│
├─Annotation_Interface
│ main.py
│ stats.jsonl
│
└─HelloBench
│ chat.jsonl
│ heuristic_text_generation.jsonl
│ open_ended_qa.jsonl
│ summarization.jsonl
│ text_completion.jsonl
│
├─length_constrained_experiments_data
heuristic_text_generation_16k.jsonl
heuristic_text_generation_2k.jsonl
heuristic_text_generation_4k.jsonl
heuristic_text_generation_8k.jsonl

Setup Instructions
Language Requirements: Python 3.10 or later
To set up the environment, run the following command in your terminal:
pip install -r requirements.txt
Usage Guidelines
  • Test Data: The core test data can be found in the HelloBench directory.
  • Evaluation Checklists: For predefined evaluation criteria, refer to the Checklists directory.
  • Human Evaluation: To facilitate human evaluation, the required code is located in the Annotation_Interface directory.
  • LLM Response Generation: Use run.py to call an LLM to generate responses for the tasks.
  • LLM Response Judging: To invoke LLMs for evaluating responses, run llm_judge.py.
  • Regression: For Linear Regression, execute regression.py.
For additional details and advanced usage, please refer to the code comments and paper.
Generation Configuration
Here are the generation configurations for different models.
Model Namemax_new_tokenstemperatureversion
GPT-4o-2024-0806163840.8gpt-4o-2024-08-06
Mistral-Large-API163840.8mistral-large-latest
o1-Mini3276810.8o1-mini
Claude-3.5-Sonnet819220.8claude-3-5-sonnet-20240620
Gemini-1.5-Pro819230.8gemini-1.5-pro
Deepseek-API409640.8deepseek-chat
Yi-Large163840.8yi-large
Qwen-Max200050.8qwen-max-0428
GLM-4-API409660.8glm-4-0520
Gemma-2-27B409670.8google/gemma-2-27b-it
LLaMA-3.1-70B163840.8meta-llama/Meta-Llama-3.1-70B-Instruct
Qwen-2-72B163840.8Qwen/Qwen2-72B-Instruct
InternLM-2.5-20B163840.8internlm/internlm2_5-20b-chat
Yi-1.5-34B204880.801-ai/Yi-1.5-34B-Chat
LLaMA-3.1-8B163840.8meta-llama/Meta-Llama-3.1-8B-Instruct
GLM-4-9B163840.8THUDM/glm-4-9b-chat
Qwen-2-7B163840.8Qwen/Qwen2-7B-Instruct
InternLM-2.5-7B163840.8internlm/internlm2_5-7b-chat
Mistral-7B-0.2163840.8mistralai/Mistral-7B-Instruct-v0.2
Phi-3.5-Moe163840.8microsoft/Phi-3.5-MoE-instruct
MAP-Neo204890.8m-a-p/neo_7b_instruct_v0.1
LongWriter-GLM4-9B163840.8THUDM/LongWriter-glm4-9b
Suri-I-ORPO163840.8chtmp223/suri-i-orpo
Yi-1.5-34B-16K8192100.801-ai/Yi-1.5-34B-Chat-16K
InternLM-2.5-7B-1M163840.8internlm/internlm2_5-7b-chat-1m
GLM-4-9B-1M163840.8THUDM/glm-4-9b-chat-1m
Footnotes
  1. For the o1-mini model, the parameter here should be max_completion_tokens instead of max_new_tokens, because it includes reasoning tokens. Therefore, I set it to 32768. You can refer to https://platform.openai.com/docs/guides/reasoning#controlling-costs for more details. ↩
  2. For the reason that claude-3.5-sonnet has max output 8192 tokens. You can refer to https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table for more details. ↩
  3. For the reason that gemini-1.5-pro has max output 8192 tokens. You can refer to https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro for more details. ↩
  4. For the reason that deepseek-chat has max output 4096 tokens. You can refer to https://api-docs.deepseek.com/zh-cn/quick_start/pricing for more details. ↩
  5. For the reason that qwen-max-0428 has max output 2000 tokens. You can refer to https://help.aliyun.com/zh/model-studio/getting-started/models?spm=a2c4g.11186623.0.0.74b04823IseC0N#9f8890ce29g5u for more details. ↩
  6. For the reason that glm-4-0520 has max output 4096 tokens. You can refer to https://bigmodel.cn/dev/howuse/model for more details. ↩
  7. Gemma-2-27B has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=4096. ↩
  8. Yi-1.5-34B-Chat has max_position_embeddings with 4096 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  9. neo_7b_instruct_v0.1 has max_position_embeddings with 8192 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=2048. ↩
  10. Yi-1.5-34B-Chat-16K has max_position_embeddings with 16384 tokens, and you need to balance the input and output, thus result the selection of max_new_tokens=8192. ↩
About
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Resources
Readme
MIT license
Activity
Stars
60 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages