Liang 2022 - HELM - Overview
Summary
HELM (Liang, Bommasani, Lee, Tsipras, Soylu et al., TMLR 2023, arXiv 2211.09110) is a benchmark design for language models built top-down: a taxonomy of scenarios (task, domain, language) and metrics (desiderata), from which it selects 16 core scenarios, each measured on 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency; 98 of 112 pairs, 87.5%), plus 7 targeted evaluations (26 scenarios). It evaluates 30 models under standardized 5-shot prompting, raising average scenario coverage per model from 17.9% to 96.0%, and reports 25 top-level findings. Key lessons: benchmarks encode values; multi-metric reporting exposes tradeoffs; standardization matters because results swing wildly with prompt and adaptation choices.
Overview
Research question. Language models are general-purpose text interfaces with a vast surface of capabilities and risks that is “poorly understood”. What does it mean to benchmark them holistically, and what do the models look like when measured that way?
Three elements of holistic evaluation (Sec. 1).
- Broad coverage and recognition of incompleteness: a top-down taxonomy that also makes explicit what is missing.
- Multi-metric measurement: every desideratum measured for each scenario so metrics beyond accuracy do not become second-class.
- Standardization: the object evaluated is the language model, not a scenario-specific system, so the adaptation strategy must be controlled and all models run on the same scenarios.
Headline numbers.
| Quantity | Value |
|---|---|
| Core scenarios | 16 (6 user-facing tasks; English) |
| Metric categories | 7 (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) |
| (core scenario, metric) pairs measured | 98 of 112 (87.5%) |
| Targeted evaluations | 7, through 26 additional scenarios |
| Scenarios new to mainstream LM evaluation | 21 |
| Models evaluated | 30 (16 families, 12 organizations); 10 open, 17 limited-access, 3 closed |
| Mean share of core scenarios previously evaluated per model | 17.9% before; 96.0% after |
| Runs | 4,939 (model, scenario) runs; 12,169,227,491 tokens; 17,431,479 queries; $38,001 commercial API cost; about 19,500 GPU hours for open models |
| Adaptation | 5-shot prompting, 3 random in-context-example seeds |
| Top-level findings | 25 |
Contributions (Sec. 1.3). (1) Taxonomy of scenarios and metrics. (2) Broad coverage (16 + 26 scenarios). (3) Evaluation of 30 models under standardized conditions. (4) Empirical findings. (5) Interactive results website, released raw prompts and completions, and a modular codebase.
Main Content
Source Map
(Page numbers are PDF page indices, located by text search; boundaries are approximate to within a page.)
Examples
Score vector versus score matrix
ImageNet assigns each model one number (top-1 accuracy). GLUE assigns a vector (nine task accuracies, averaged to one headline score). HELM assigns a matrix: for each of 16 scenarios, scores on up to 7 metrics. For one model that is up to numbers, of which are defined. See Deng 2009 - ImageNet - Overview and Wang 2018 - GLUE - Overview for the two predecessors the paper compares against.
Connections
- Lineage. The paper explicitly follows “the precedent set by GLUE and SuperGLUE” (Wang 2018 - GLUE - Overview) and cites ImageNet as the benchmark that shaped computer vision (Deng 2009 - ImageNet - Overview).
- Goodhart. The authors warn that benchmark validity is critical because of Strathern’s Law (“When a measure becomes a target, it ceases to be a good measure”); see Manheim Garrabrant 2018 - Variants of Goodhart’s Law - Overview.
- LLM evaluation background: Evaluating LLM Systems - Benchmarks, Hallucination and Human Preference, In-Context Learning and Few-Shot Prompting, Neural Scaling Laws, RLHF and Instruction Tuning.
Relevance to marketing measurement. HELM’s design recipe applies to evaluating MMM or incrementality methods: enumerate the use cases (scenarios) and the desiderata (accuracy of ROI, interval calibration, robustness to data perturbations, fairness across segments or geographies, compute cost), measure all of them for every scenario, standardize the “adaptation” (priors, tuning budget, data preprocessing), and publish what the benchmark does not cover. See Experimental Benchmarks for Observational Ad Measurement for an experiment-based ground truth.