All issues

Search results

OpenAI-compat: --model openai-chat-completions with full chat URL, not curator or vLLM

Status: Open.

#161In mlfoundations/evalchemy;

· cursor opened on Aug 28, 2026

[bug] GPQA-Diamond ground truth answers are all D

Status: Open.

#155In mlfoundations/evalchemy;

· xiefiona opened on Feb 24, 2026

vLLM backend crashes with "please provide at least one prompt" when max_gen_toks exceeds model context window

Status: Open.

#152In mlfoundations/evalchemy;

· Ali-Elganzory opened on Jan 3, 2026

[bug] LiveCodeBenchv5

Status: Open.

#149In mlfoundations/evalchemy;

· cyr0930 opened on Dec 23, 2025

[Bug] google/codegemma-7b-it Evaluation Results All Zero

Status: Open.

#147In mlfoundations/evalchemy;

· juyongjiang opened on Dec 16, 2025

Only return loose result in IFEval?

Status: Open.

#145In mlfoundations/evalchemy;

· luyy12 opened on Oct 28, 2025

final answer extraction in reasoning model's output

Status: Open.

#144In mlfoundations/evalchemy;

· maziye-azad opened on Oct 2, 2025

LM Eval Tasks not found

Status: Open.

#143In mlfoundations/evalchemy;

· AahilA opened on Sep 18, 2025

Difference between self.max_tokens self.max_new_tokens in eval_instruct.py

Status: Open.

#142In mlfoundations/evalchemy;

· richardbaihe opened on Sep 3, 2025

Add logical reasoning task tests? (e.g., Enigmata data)

Status: Open.

#141In mlfoundations/evalchemy;

· Siki-cloud opened on Aug 29, 2025

Support for SWEBench

Status: Open.

#140In mlfoundations/evalchemy;

· sravan500 opened on Aug 12, 2025

Request to add Debug mode

Status: Open.

#139In mlfoundations/evalchemy;

· richardzhuang0412 opened on Jul 23, 2025