| Benchmark | Description | Setup | Claude Sonnet 3 | Claude Opus 3 | Claude Haiku 3 | GPT-4 | GPT-3.5 | Gemini 1.0 Ultra | Gemini 1.5 Pro | Gemini 1.0 Pro |
|---|---|---|---|---|---|---|---|---|---|---|
| MMLU | General reasoning | 5-shot / 5-shot CoT | 79.0% 81.5% |
86.8% 88.2% |
75.2% 76.7% |
86.4% | 70.0% | 83.7% | 81.9% | 71.8% |
| MATH1 | Mathematical problem solving | 4-shot / 0-shot / Maj@32, 4-shot | 40.5% 43.1% 55.1% |
61% 60.1% 73.7% |
40.9% 38.9% 50.3% |
52.9% 42.5% |
34.1% | 53.2% | 58.5% | 32.6% |
| GSM8K2 | Grade school math | Elicitation varies by model | 92.3% | 95.0% | 88.9% | 92.0% | 57.1% | 94.4% | 91.7% | 86.5% |
| HumanEval | Python coding tasks | 0-shot | 73.0% | 84.9% | 75.9% | 67.0% | 48.1% | 74.4% | 71.9% | 67.7% |
| GPQA (Diamond)3 | Graduate-level Q&A | 0-shot CoT / Maj@32, 5-shot CoT | 40.4% 46.3% |
50.4% 59.5% |
33.3% 40.1% |
35.7% | 28.1% | — | — | — |
| MGSM | Multilingual math | 8-shot / 0-shot | 83.7% 83.5% |
90.5% 90.7% |
76.5% 75.1% |
74.5% | — | 79.0% | 88.7% | 63.5% |
| DROP | Reading comprehension, arithmetic | 3-shot, F1 | 78.9 | 83.1 | 78.4 | 80.9 | 64.1 | 82.4 | 78.9 | 74.1 |
| BIG-Bench-Hard | Mixed evaluations | 3-shot CoT | 82.9% | 86.8% | 73.7% | 83.1% | 66.6% | 83.6% | 84.0% | 75.0% |
| ARC-Challenge | Common-sense reasoning | 25-shot | 93.2% | 96.4% | 89.2% | 96.3% | 85.2% | — | — | — |
| HellaSwag | Common-sense reasoning | 10-shot | 89.0% | 95.4% | 85.9% | 95.3% | 85.5% | 87.8% | 92.5% | 84.7% |
| PubMedQA | Biomedical questions | 5-shot / 0-shot | 78.3% 79.7% |
75.8% 74.9% |
76.0% 78.5% |
74.4% 75.2% |
60.2% 71.6% |
— | — | — |
| WinoGrande | Common-sense reasoning | 5-shot | 75.1% | 88.5% | 74.2% | 87.5% | — | — | — | — |
| RACE-H | Reading comprehension | 5-shot | 88.8% | 92.9% | 87.0% | — | — | — | — | — |
| APPS | Python coding tasks | 0-shot | 55.9% | 70.2% | 54.8% | — | — | — | — | — |
| MBPP | Code generation | Pass@1 | 79.4% | 86.4% | 80.4% | — | — | — | — | — |
| LSAT | Law School Admission Test | 5-shot CoT, scaled score | 158.3 | 161 | 156.3 | 163 | 149 | — | — | — |
| MBE | Multistate Bar Exam | 0-shot CoT | 71% | 85% | 64% | 75.7% | 45.1% | — | — | — |
| AMC 20234 | High school math contests | AMC 8 / 10 / 12, of 150 | 54/150 24/150 27/150 |
84/150 72/150 63/150 |
36/150 54/150 48/150 |
— 36/150 60/150 |
— 36/150 30/150 |
— | — | — |
| Multilingual MMLU5 | Multilingual reasoning | 5-shot | 69.0% | 79.1% | 65.2% | — | — | — | — | — |
| QuALITY5 | Long-input reading comprehension | 1-shot / 0-shot | 85.9% 84.9% |
90.5% 89.2% |
80.2% 79.4% |
— | — | — | — | — |
| Needle In A Haystack5 | Long context retrieval | All context lengths / 200k tokens | 95.4% 91.4% |
99.4% 98.3% |
95.9% 91.9% |
— | — | — | — | — |