AI benchmarks

Extracted Attributes

• Importance

  Crucial for fair comparison, identifying areas for improvement, ensuring transparency

• Key Function

  Comparing different AI designs, how they learn, and how they work in real-world settings

• Primary Purpose

  Evaluating capabilities and performance of artificial intelligence models

• Metrics Measured

  Inference speed, accuracy, resource utilization, complex reasoning, problem-solving, reading comprehension, long-context comprehension, agentic reasoning, adaptation to new contexts, ability to solve novel machine learning tasks

• Example Benchmark

  MATH 500

• HLE Subject Scope

  Broad range of subjects

• HLE Question Count

  2,500 questions

• Challenges Addressed

  Overfitting, architectural differences between models (e.g., Convolutional Neural Networks vs Large Language Models)

Humanity's Last Exam

Humanity's Last Exam (HLE) is a language model benchmark consisting of 2,500 questions across a broad range of subjects. It was created jointly by the Center for AI Safety and Scale AI.

Web Search Results

• Test scores of AI systems on various capabilities relative to human ...

  This dataset captures the progression of AI evaluation benchmarks, reflecting their adaptation to the rapid advancements in AI technology. The benchmarks cover a wide range of tasks, from language understanding to image processing, and are designed to test AI models' capabilities in various domains. The dataset includes performance metrics for each benchmark, providing insights into AI models' proficiency in different areas of machine learning research. [...] BBH (BIG-Bench Hard): This benchmark serves as a rigorous evaluation framework for advanced language models, targeting their capacity for complex reasoning and problem-solving. It identifies tasks where AI models traditionally underperform compared to human benchmarks, emphasizing the enhancement of AI reasoning through innovative prompting methods like Chain-of-Thought. [...] SQuAD 1.1 and 2.0 (Stanford Question Answering Dataset): These benchmarks evaluate the reading comprehension abilities of AI models, requiring them to extract or infer answers from textual passages. SQuAD 2.0 further introduces the challenge of discerning unanswerable questions, adding a layer of complexity in judgment and inference.

• AI Model Performance - International Test and Evaluation Association

  An AI benchmark tool aims to fix this by providing a single interface to evaluate diverse AI models on many tasks, data sets, and methods to evaluate their performance against hardware and computational constraints. By enforcing a common inferencing methodology, this tool can compare different AI designs, how they learn, and how they work in real-world settings. In this paper, we propose a comprehensive and generalized AI benchmark harness that is compatible with convolutional neural networks [...] The AIIA DNN Benchmark Overview1 provides a comprehensive benchmark to evaluate the performance of deep neural networks (DNN) in various hardware and software configurations. It focuses on measuring inference speed, accuracy, and resource utilization for different DNN models. The benchmark suite enables comparisons between frameworks like TensorFlow, PyTorch, and ONNX. This tool is particularly useful for developers and researchers in optimizing AI workloads. By standardizing evaluation [...] 2. Related Work --------------- The development of a comprehensive benchmark harness is very important to evaluate the performance of a model before its deployment onto its target platform. Designing a comprehensive benchmarking tool that can scale to benchmark a variety of models that include both CNNs and LLMs is challenging due to architectural differences. This literature review examines recent efforts towards the development of AI model benchmark tools.

• AI Benchmarking Dashboard - Epoch AI

  The benchmark is meant to measure how well models can handle long contexts, while being more challenging than traditional “needle in a haystack” evaluations which resemble simple recognition or retrieval. For example, a model which could identify a specific word in the text but can’t understand enough about the story to answer a question about a character’s state of mind would thus get a score on Fiction.liveBench that better represents its long-context comprehension abilities. [...] The AI Benchmarking Hub is supported by a grant from the UK AI Security Institute. This funding enables us to conduct rigorous, independent evaluations of leading AI models on challenging benchmarks and make the results freely available to researchers and the public. ### Who can I contact with questions or comments about the data? Image 54 Feedback can be directed to tom@epoch.ai. Methodology ----------- ### Epoch-evaluated benchmarks [...] The WeirdML benchmark evaluates models’ ability to solve novel machine learning tasks that require careful thinking and understanding rather than applying standard recipes. It tests models across six diverse tasks: shape recognition (easy and hard variants), image patch reconstruction (easy and hard variants), chess game outcome prediction, and semi-supervised digit classification.

• LLM Leaderboard 2025 - Vellum AI

  AverageGRIND Independently run Vellum benchmark that tests how well models adapt to new contexts instead of relying on pre-learned patterns.AIME 2024 Data from the AIME 2024, a competitive high school math benchmark.GPQA Data from the GPQA Diamond, a very complex benchmark that evaluates quality and reliability across biology, physics, and chemistry.SWE Bench Data from the SWE Bechmark that evaluates if LLMs can resolve GitHub Issues. It measures agentic reasoning.MATH 500 Data from the MATH [...] Data from the Humanity's Last Exam, which is the most challenging benchmark across multiple domains. Score (Percentage) 50 40 30 20 10 0 Image 28 Grok 4 25.4 Image 29 Gemini 2.5 Pro 21.6 Image 30 OpenAI o3 20.32 Image 31 OpenAI o4-mini 14.28 Image 32 OpenAI o3-mini 14 Fastest and most affordable models Fastest Models Tokens per second. Higher is better. Tokens/seconds 2500 2000 1500 1000 500 0 Image 33 Llama 4 Scout 2600 Image 34 Llama 3.3 70b 2500 [...] 60% 50% Image 8 Grok 3 [Beta] 93.3 Image 9 OpenAI o4-mini 92.7 Image 10 Grok 4 91.7 Image 11 OpenAI o3 88.9 Image 12 Gemini 2.5 Pro 88 Best in Agentic Coding (SWE Bench) Data from the SWE Bechmark that evaluates if LLMs can resolve GitHub Issues. It measures agentic reasoning. Score (Percentage) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Image 13 Grok 4 75 Image 14 Claude 4 Sonnet 72.7 Image 15 Claude 4 Opus 72.5 Image 16 Claude 3.7 Sonnet [R] 70.3

• LiveBench

  | Qwen2.5 Max | Alibaba | 51.93 | 38.53 | 66.79 | 3.33 | 56.87 | 64.27 | 58.37 | 75.35 | | Claude 3.5 Sonnet | Anthropic | 51.80 | 43.22 | 73.90 | 15.00 | 50.54 | 56.19 | 54.48 | 69.30 | | GPT-4.1 Mini | OpenAI | 51.57 | 53.78 | 72.11 | 6.67 | 58.78 | 61.34 | 38.00 | 70.31 | | Mistral Medium 3 | Mistral AI | 50.65 | 41.97 | 61.48 | 15.00 | 59.74 | 60.20 | 44.74 | 71.40 | | Phi-4 Reasoning Plus | Microsoft | 49.27 | 57.83 | 60.59 | 5.00 | 62.83 | 54.74 | 30.69 | 73.17 | [...] | Gemini 2.5 Flash | Google | 64.42 | 78.53 | 63.53 | 18.33 | 84.10 | 69.85 | 57.04 | 79.56 | | Qwen 3 32B | Alibaba | 63.71 | 83.08 | 64.24 | 10.00 | 80.05 | 68.29 | 55.15 | 85.17 | | Claude 4 Sonnet | Anthropic | 63.37 | 54.86 | 78.25 | 25.00 | 76.39 | 64.68 | 67.18 | 77.25 | | Kimi K2 Instruct | Moonshot AI | 62.70 | 62.97 | 71.78 | 20.00 | 74.41 | 63.41 | 63.85 | 82.47 | | Grok 3 Mini Beta (High) | xAI | 62.36 | 87.61 | 54.52 | 15.00 | 77.00 | 64.58 | 59.09 | 78.70 | [...] | DeepSeek R1 Distill Llama 70B | DeepSeek | 48.53 | 59.81 | 46.65 | 6.67 | 58.80 | 60.81 | 37.05 | 69.94 | | Llama 4 Maverick 17B 128E Instruct | Meta | 47.78 | 43.83 | 54.19 | 3.33 | 60.58 | 47.11 | 49.65 | 75.75 | | GPT-4o | OpenAI | 47.43 | 39.75 | 69.29 | 8.33 | 41.48 | 63.53 | 44.68 | 64.94 | | Gemini 2.0 Flash Lite | Google | 46.78 | 32.25 | 59.31 | 5.00 | 54.97 | 65.39 | 33.94 | 76.63 | | Command A | Cohere | 44.17 | 36.33 | 54.26 | 5.00 | 45.54 | 48.46 | 36.70 | 82.90 |