ReasoningDeprecated
Trust Score:
10

BIG-Bench

204-task community-built mega-suite spanning reasoning, knowledge, math, code and social bias — the sprawling proving ground BBH was distilled from.

Launched: —Refresh: static
Status Assessment (deprecated):

No longer maintained as an evaluation target. Its 'canary' GUID — meant to keep tasks out of training data — was widely ignored, so the suite leaked into web-scale corpora; task quality is famously uneven; and the field moved to distillates like BBH. Deprecated rather than saturated: models don't uniformly max it, it simply stopped being used.

Performance Timeline

Longitudinal progression of model scores against human baselines.
No performance score history recorded yet.

Human Baseline & Difficulty Horizon

Calibrated human reference points, specialist benchmarks, and ceiling thresholds.
Measured Human Score84.3%Domain Expert Baseline
Baseline Protocol & Interpretation

Crowdworker and domain expert aggregate average across the 200+ collaborative BIG-bench tasks.

Metric & Scoring Methodology

Verification protocols, aggregation formulas, and specialized metric variants.
Primary Metric:normalized preferred-metric aggregate (0–100)
Scoring Engine:composite

Dataset & Compute Cost

Evaluation volume, public availability, API pricing, and local hardware requirements.
Total Dataset Size—Annotated evaluation items
Public Test SetPublicOpenly mirrored on repositories
Access GatingOpen AccessUnrestricted download
Evaluation LicenseOpen AccessDataset usage and redistribution terms
Frontier API Compute Cost:

$5 – $20 USD for full benchmark evaluation run on frontier APIs.

Recommended Local GPU Setup:

1x NVIDIA RTX 4090 (24GB) or A100 (40GB/80GB) via vLLM / SGLang

Official Dataset & Benchmark Files:Download / View Dataset Repository ↗

How to Run & Reproduce

Standardized evaluation protocols, CLI commands, and reproducible runner templates.
Prompt Regimezero-shot
Reasoning Modedirect
Sampling Temp0
Pass@k Budgetk = 1
Tools & SandboxPure Text
Scoring Verifiercomposite
Option AEleutherAI LM-Evaluation-Harness (Open-Weight Models)
lm_eval --model hf --model_args pretrained=<model_path> --tasks big-bench --batch_size auto
Option BOpenCompass Evaluation Framework
opencompass --datasets big-bench --models <model_config>
Python APIDeterministic Inference Loop Snippet
# Standard API Evaluation Loop
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
Standardized Reporting Requirement:

When publishing results for BIG-Bench, always report the exact prompt template, few-shot exemplar ordering, sampling temperature (temperature=0), maximum reasoning budget tokens, and the precise timestamped model snapshot ID.

Contamination & Memorization Analysis

Audit of pretraining exposure risks, memorization vectors, and refresh policies.
Overall Contamination Risk:HIGH
Refresh Cadence:

Static fixed snapshot

Test Set Exposure:

Public on web / HuggingFace