Gebru 2021 - Questions I - Motivation, Composition and Collection Process
Summary
The first three of seven lifecycle sections of the datasheet: Motivation (why the dataset exists and who paid), Composition (what an instance is, sampling, labels, splits, noise, confidentiality, subpopulations) and Collection Process (how data was acquired, by whom, when, with what ethical review and consent). Composition and Collection should be read before collecting and answered after collection is complete.
Overview
Section 3 organises the questions to “roughly match the key stages of the dataset lifecycle: motivation, composition, collection process, preprocessing/cleaning/labeling, uses, distribution, and maintenance.” Questions that do not apply “should be skipped”; creators are urged to “answer as many questions as possible rather than skipping the datasheet creation process entirely.” Each section ends with Any other comments?. Questions that apply only to datasets relating to people come last in Composition and Collection; take “a broad interpretation” (any dataset containing text written by people relates to people).
Datasheet workflow (when to answer)
Section Read Answer Motivation at design time after creation Composition before any data collection once collection is complete Collection Process before any collection, to flag issues once collection is complete Preprocessing/cleaning/labeling before any such step once done Uses during reflection before release Distribution before distribution (internal or external) prior to distribution Maintenance before distribution prior to distribution
Main Content
3.1 Motivation (3 prompts + comments)
Intent: make creators “clearly articulate their reasons for creating the dataset and … promote transparency about funding interests”, relevant for research datasets.
Motivation questions
- For what purpose was the dataset created? Was there a specific task in mind? A specific gap to fill?
- Who created the dataset (team, research group) and on behalf of which entity (company, institution, organization)?
- Who funded the creation? If a grant exists: grantor, grant name and number.
- Any other comments?
3.2 Composition (12 core prompts + 3 people-only)
Mostly informs consumers’ fitness-for-task decisions; some questions elicit GDPR-type compliance information.
Composition questions
- What do the instances represent (documents, photos, people, countries)? Multiple instance types (movies, users, ratings; nodes and edges)?
- How many instances in total (per type)?
- All possible instances or a sample (not necessarily random)? If a sample: what is the larger set; is the sample representative (e.g., geographic coverage); how was representativeness validated; if not representative, why not (e.g., diversity, withheld instances)?
- What data does each instance consist of: raw (unprocessed text/images) or features?
- Is there a label or target per instance?
- Is information missing from individual instances, and why (not counting intentional removals; redacted text may count)?
- Are relationships between instances explicit (ratings, social links)?
- Recommended data splits (train/dev/test), with rationale?
- Errors, sources of noise, redundancies?
- Self-contained or reliant on external resources (websites, tweets, other datasets)? If external: (a) guarantees they exist and remain constant; (b) official archival versions including the external resources as of creation; (c) restrictions (licenses, fees).
- Confidential data (legal privilege, doctor-patient confidentiality, non-public communications)?
- Offensive, insulting, threatening or anxiety-inducing content if viewed directly?
If the dataset does not relate to people, skip the rest.
- Does it identify subpopulations (age, gender)? How identified; what distributions?
- Can individuals be identified, directly or in combination with other data?
- Sensitive data (race/ethnic origin, sexual orientation, religion, political opinion, union membership, location, financial or health data, biometric/genetic data, government IDs such as SSNs, criminal history)?
- Any other comments?
(The paper uses bullets, not numbers; numbering here is for reference only. Items 3 and 10 are the multi-part questions most relevant to benchmark validity.)
3.3 Collection Process (6 core prompts + 5 people-only)
Dual goals: tell consumers about quality and provenance, and “help researchers and practitioners to create alternative datasets with similar characteristics.”
Collection-process questions
- How was data for each instance acquired? Directly observable (raw text, ratings), reported by subjects (surveys), or inferred/derived (part-of-speech tags, model-based guesses of age or language)? If reported or derived, was it validated, and how?
- What mechanisms or procedures (sensors, manual curation, software, APIs)? How were they validated?
- Sampling strategy if a sample (deterministic; probabilistic with stated probabilities)?
- Who was involved (students, crowdworkers, contractors) and how were they compensated (e.g., crowdworker pay)?
- Over what timeframe? Does it match the creation timeframe of the underlying data (e.g., a recent crawl of old news)?
- Ethical review (e.g., IRB)? Outcomes and supporting documentation.
If the dataset does not relate to people, skip the rest.
- Collected directly from individuals or via third parties (websites)?
- Notification: were individuals notified? How (screenshots, exact language)?
- Consent: did individuals consent to collection and use? How requested; exact language.
- Revocation: a mechanism to revoke consent later or for certain uses?
- Impact analysis (e.g., a data protection impact analysis) on data subjects: outcomes and documentation.
- Any other comments?
Examples
Applying "Is the sample representative?" to a benchmark
Setup. A benchmark claims to measure general language understanding but consists of 8 previously published datasets. Question 3 (Composition). The larger set is “all language-understanding tasks practitioners care about.” The sample is a convenience selection of prior shared tasks. A truthful answer says it is not a random sample, was chosen for availability and format uniformity, and has not been validated against a deployment distribution. Interpretation. That honest answer is the seed of the task selection bias analysed in Dehghani 2021 - Task Selection Bias Case Studies: the datasheet makes the sampling frame explicit, so consumers can ask whether their task lies inside it.
Connections
- Composition Q3 and Q8 (recommended splits) are the datasheet counterpart of Dehghani et al.’s demand for multiple fixed splits in benchmark design (Dehghani 2021 - Guidelines, Statistical Testing and Living Benchmarks).
- Collection Q3 (sampling strategy) corresponds to selection into experiments in Gordon et al 2019 - Facebook RCTs vs Observational Ad Measurement - Overview, where who is randomized determines what generalises.
- Composition Q6 (missingness) and Q9 (noise) parallel the label-error evidence in Dehghani 2021 - Community Bias and the GLUE Example.