A benchmark of expert-level academic questions to assess AI capabilities
Document Type
Article
Publication Date
1-29-2026
Abstract
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity’s Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.
Keywords
Artificial Intelligence; Benchmarking; Educational Measurement; Humans; Language
Publication Title
Nature
ISSN
0028-0836
DOI
10.1038/s41586-025-09962-4
Recommended Citation
Hendrycks, Dan; Mazeika, Mantas; Zhang, Oliver; Hausenloy, Jason; Ren, Richard; Kim, Ryan; Khoja, Adam; Li, Nathaniel; Gatti, Alice; Phan, Long; Wang, Alexandr; Yue, Summer; Telluri, Anwith; Wu, Aidan; Wang, Kaixin; Nagumalli, Laasya; Nguyen, Leon; Zhang, Alex; and Universiti Malaya, "A benchmark of expert-level academic questions to assess AI capabilities" (2026). Research Publications (2026 to 2030). 1429.
https://knova.um.edu.my/research_publications_2026_2030/1429
Volume
649
Issue
8099
First Page
1139
Last Page
1146
Publisher
Nature Research