Humanity's Last Exam is a benchmark designed to test the capabilities of artificial intelligence systems at the frontier of knowledge. It was created by a coalition of researchers, including Dan Hendrycks of the Center for AI Safety and others, as a successor to earlier tests like the Massive Multitask Language Understanding (MMLU) benchmark. The exam consists of hundreds of extremely difficult questions across mathematics, physics, biology, law, and other specialized domains, intended to measure whether AI can solve problems that require deep reasoning and expert-level knowledge. The questions were crowdsourced from experts worldwide and vetted to ensure they are not easily answerable by current models through memorization or pattern matching. The prediction market question asks whether any language model will achieve a specific accuracy threshold, set at 90% or higher, on this exam before December 31, 2026. This benchmark is distinct from earlier tests because it was specifically designed to be harder than any existing AI evaluation, with questions that often require multi-step reasoning, novel problem-solving, and integration of knowledge from multiple fields. The interest in this market reflects broader debates about the pace of AI progress, the reliability of benchmarks, and whether current scaling laws will continue to produce rapid gains in capability. Some researchers argue that achieving 90% on Humanity's Last Exam would represent a significant milestone, potentially indicating that AI systems have reached a level of competence that approaches or exceeds human experts in many domains. Others caution that benchmark scores can be misleading due to data contamination, overfitting, or the narrowness of the test itself. The market's resolution depends on publicly verifiable results from the official benchmark, which is maintained by the Center for AI Safety and associated researchers. As of early 2025, no model has publicly reported scores above 50% on this exam, though some frontier labs like OpenAI, Google DeepMind, and Anthropic are likely testing their systems internally.

Historical Context

The development of AI benchmarks has evolved significantly over the past decade. Early benchmarks like the Stanford Question Answering Dataset (SQuAD) and the General Language Understanding Evaluation (GLUE) focused on narrow tasks such as reading comprehension and sentiment analysis. These were superseded by more comprehensive tests like MMLU, introduced in 2020 by Dan Hendrycks and colleagues, which covered 57 subjects from elementary mathematics to professional law. MMLU quickly became a standard for evaluating large language models, with GPT-4 achieving approximately 86% accuracy in 2023, a result that surprised many researchers. That same year, the concept of 'Humanity's Last Exam' was proposed as a response to concerns that MMLU and other benchmarks were becoming saturated. The idea was to create a test that would remain challenging for several years, similar to how the ImageNet benchmark for computer vision was eventually surpassed. The exam was officially released in early 2024, with questions curated from over 500 experts across 100 fields. Each question required at least 30 minutes for a human expert to solve, and the authors explicitly designed them to resist memorization by using novel problem formulations. The benchmark's name reflects its intended role as the final evaluation needed before AI systems could be considered broadly superhuman. In 2024, OpenAI's GPT-4o scored approximately 44% on the exam, while Google's Gemini Ultra scored around 42%, far below the 90% threshold. These results set the baseline for the prediction market. The historical pattern of AI progress on benchmarks suggests that scores often improve rapidly after initial releases, but the difficulty of Humanity's Last Exam may slow this trend. For example, MMLU saw a 20 percentage point improvement in its first three years, but the harder questions on the new exam may require breakthroughs in reasoning rather than simple scaling.

Frequently Asked Questions