Updates and announcements from the LLMEval team.
Our paper on robust and fair LLM evaluation has been accepted to ACL 2026. With 220K generative questions across 13 disciplines and anti-cheating mechanisms, LLMEval-Fair sets a new standard for trustworthy model benchmarking.
LLMEval-Med introduces a benchmark of 2,996 clinical questions built from real electronic health records, with physician-validated evaluation covering five core medical dimensions.
Using the freshly released 2024 Chinese Gaokao math papers — highly original and confidential — we evaluate leading LLMs with both LaTeX and escape-character prompts.
Our foundational paper at AAAI 2024 systematically studies the 'how to evaluate' question — comparing evaluation criteria, annotator types, scoring methods, and ranking systems across 20 LLMs with 2,186 participants.