Long ContextActive
Trust Score:
80
OpenAI-MRCR
MRCR v2 8-needle long-context benchmark where a model must recover the requested instance from repeated similar requests in a synthetic conversation.
Launched: —Refresh: static
Status Assessment (active):
Actively tracked as MRCR v2 (8-needle) on LLM Stats, whose 2026-07-22 snapshot lists 21 self-reported model results with a wide 13.5%-91.5% displayed score spread, corresponding to raw mean_similarity values of 0.135-0.915.
Performance Timeline
Longitudinal progression of model scores against human baselines.Performance & Historical Trajectory
Empirical score progression across model release dates and evaluation rounds.
Independent Vendor-Reported Human Baseline (85.0%)
| Model | Score | Date | Source Type | Provenance |
|---|---|---|---|---|
| GPT-5.4 mini (LLM Stats MRCR v2 8-needle) | 33.6% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemma 3 27B (LLM Stats MRCR v2 8-needle) | 13.5% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemini 2.5 Pro Preview 06-05 (LLM Stats MRCR v2 8-needle) | 16.4% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemma 4 E2B (LLM Stats MRCR v2 8-needle) | 19.1% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemini 3.5 Flash-Lite (LLM Stats MRCR v2 8-needle) | 21.3% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemini 3 Flash (LLM Stats MRCR v2 8-needle) | 22.1% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemma 4 E4B (LLM Stats MRCR v2 8-needle) | 25.4% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemini 3 Pro (LLM Stats MRCR v2 8-needle) | 26.3% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemini 3.1 Pro (LLM Stats MRCR v2 8-needle) | 26.3% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemini 3.5 Flash (LLM Stats MRCR v2 8-needle) | 26.6% | 2024-06-01 | vendor-reported | Source ↗ |
| GPT-5.4 nano (LLM Stats MRCR v2 8-needle) | 33.1% | 2024-06-01 | vendor-reported | Source ↗ |
| GPT-5.6 Sol (LLM Stats MRCR v2 8-needle) | 91.5% | 2024-06-01 | vendor-reported | Source ↗ |
| GPT-5.6 Luna (LLM Stats MRCR v2 8-needle) | 41.3% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemma 4 12B (LLM Stats MRCR v2 8-needle) | 43.4% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemma 4 26B-A4B (LLM Stats MRCR v2 8-needle) | 44.1% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemini 3.6 Flash (LLM Stats MRCR v2 8-needle) | 54% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemini 3.1 Flash-Lite (LLM Stats MRCR v2 8-needle) | 60.1% | 2024-06-01 | vendor-reported | Source ↗ |
| Gemma 4 31B (LLM Stats MRCR v2 8-needle) | 66.4% | 2024-06-01 | vendor-reported | Source ↗ |
| GPT-5.5 (LLM Stats MRCR v2 8-needle) | 74% | 2024-06-01 | vendor-reported | Source ↗ |
| Claude Opus 4.6 (LLM Stats MRCR v2 8-needle) | 76% | 2024-06-01 | vendor-reported | Source ↗ |
| GPT-5.6 Terra (LLM Stats MRCR v2 8-needle) | 89.6% | 2024-06-01 | vendor-reported | Source ↗ |
Human Baseline & Difficulty Horizon
Calibrated human reference points, specialist benchmarks, and ceiling thresholds.Measured Human Score85.0%Domain Expert Baseline
Baseline Protocol & Interpretation
Human performance on multi-hop reading comprehension across long-context books.
Metric & Scoring Methodology
Verification protocols, aggregation formulas, and specialized metric variants.Primary Metric:
MRCR v2 8-needle mean_similarityScoring Engine:composite
Dataset & Compute Cost
Evaluation volume, public availability, API pricing, and local hardware requirements.Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:
$5 – $20 USD for full benchmark evaluation run on frontier APIs.
Recommended Local GPU Setup:
1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang
Official Dataset & Benchmark Files:Download / View Dataset Repository ↗
How to Run & Reproduce
Standardized evaluation protocols, CLI commands, and reproducible runner templates.Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifiercomposite
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks openai-mrcr --batch_size autoOption BOpenCompass Evaluation Framework
opencompass --datasets openai-mrcr --models <model_config>Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
)Standardized Reporting Requirement:
When publishing results for OpenAI-MRCR, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.
Contamination & Memorization Analysis
Audit of pretraining exposure risks, memorization vectors, and refresh policies.Overall Contamination Risk:HIGH
Refresh Cadence:
Static fixed snapshot
Test Set Exposure:
Public on web / HuggingFace