Fairness slices

• slice metrics
• bootstrap
• shipped feature

What is fairness slicing?

Fairness slicing splits a dataset into groups by a metadata column, such as accent or gender, and compares model performance across those groups. Speech models often do worse for non-native speakers or under-represented groups, and that only shows up when results are measured per group. A raw gap is not proof of bias on its own, so this panel adds statistics to tell a real gap from noise.

Grouping

You pick a dataset, a grouping column from its metadata, and a model. You can slice by up to 2 grouping keys at once (e.g. accent and gender together, not just one at a time). You can also filter rows by other columns, but a column can't be both a filter and a grouping key.

Every group is compared against a reference group — by default the largest one — not against every other group. Set reference_group to pick a different baseline.

Metrics

Nine metrics are available. Transcription models (Whisper) use the error-rate family; classification models (Wav2Vec2 emotion) use the classification family; representation and grounding metrics come from the model's embeddings and saliency maps.

Transcription

Text is normalised with Whisper's own normaliser before scoring, so punctuation and casing don't count as errors and numbers stay comparable with published Whisper results. Each group also reports its substitution, deletion, and insertion rates separately. A high insertion ratio with few deletions is the usual signature of hallucination.

Classification

Classification groups also report macro-recall, balanced accuracy, mean confidence, and a per-group confusion matrix.

Representation

These use the model's embedding vectors and are computed when include_representation is on.

High silhouette or leakage means the model's internal representation has learned to tell groups apart as a side effect — a warning sign for downstream disparate treatment, even before you look at output accuracy. Combined with the disparity result, the panel labels each case as representation-linked disparity, representation only, disparity not representation-linked, or no structure, no disparity.

Grounding

These use saliency maps on a speaker-balanced sample of items (12 per group by default) and are computed when include_explanations is on.

If one group consistently gets lower grounding lift, the model is "listening to the wrong part" of the audio more often for that group — a different kind of unfairness than raw error rate.

Is a gap real? The statistics

For every group × metric, the panel compares the group against the reference and reports:

• Gap — mean(group) − mean(reference), plus the ratio (e.g. 1.4× the reference WER) and an effect size (the gap in units of natural variation).
• 95% confidence interval — from a bootstrap: the data is resampled 2,000 times (n_bootstrap, 200–20,000) and the gap recomputed each time. The interval is the 2.5th–97.5th percentile of those gaps. If it doesn't cross zero, the gap is unlikely to be pure chance. This is the percentile bootstrap, not BCa.
• p-value and p_adjusted — the raw p-value is corrected with Holm-Bonferroni so that running many groups × metrics at once doesn't produce spurious "significant" results. Always read p_adjusted, not the raw value.

How the resampling is done depends on the data:

• Paired by content — if groups read the same sentences and at least 5 are shared, the same sentence is compared across groups and those pairs are resampled. Most powerful and most trustworthy.
• By speaker — otherwise whole speakers are resampled, so one voice can't dominate the result.
• By utterance — used only when no speaker IDs exist.

Design: matched, partially matched, unmatched

Before trusting any comparison, the panel checks whether the groups spoke the same content:

• matched — every group covers ≥ 95% of the same content. Any gap is down to how it was said, not what.
• partially_matched — some overlap; pairing is done on the shared subset only.
• unmatched — different groups said different things. A gap may reflect accent or may just reflect harder sentences, so every verdict carries a content-confounded caveat and is never called causal.

Minimum detectable effect

The MDE is the smallest true gap this specific sample could have detected at 95% confidence and 80% power. It stops "no significant difference" from being misread as "we checked and it's fine" when the data was simply too small to see anything.

Verdicts

Each comparison gets one verdict, checked in this order:

Thresholds are configurable per run: absolute_error_gap (0.05), disparity_ratio (1.25), grounding_gap (0.15), and alpha (0.05).

Alongside per-group verdicts, each metric gets corpus-level summary stats: max gap, disparity ratio (worst ÷ best group), Gini, and weighted standard deviation, with the best and worst group named.

Minimums

A slice needs enough data before its statistics mean anything, so two minimums are enforced:

Groups below either minimum are excluded from the results rather than reported with unreliable statistics.

Dataset size

Fairness analysis is capped at ≤ 20,000 items total per run.

How long it takes

Behind the scenes the job runs in four stages: it prepares (cache check, grouping, dropping small groups, picking the reference, classifying the design), infers in parallel shards of about 8 items, optionally explains a sampled subset with saliency on the GPU, then aggregates all statistics. Identical requests are cached and return instantly. A corrupt file is recorded as a per-item failure without sinking the whole job, and you can cancel a running job at any stage.

Reading the results

• Disparity radar — one axis per metric, one line per group, so you can see at a glance which groups the model treats differently and on which metric.
• Group error bars — per-group metric values with bootstrap confidence intervals, so you can tell a real gap from noise.
• Verdict cards — the plain-English verdict for each group and metric, with the CI, p_adjusted, estimator, and block count spelled out.
• Phone confusion matrix — where phone-level annotations exist (for example, L2-ARCTIC), a confusion matrix of which phones get mistaken for which, broken out by group.

Dataset-specific analyses

Speech Accent Archive (SAA)

150 speakers all read the same paragraph, so the design is naturally matched and content can't be a confound. That unlocks two extra analyses:

• Word-position error heatmap — because every transcript aligns to the same reference, you get an error rate per word position, turning "WER is higher" into "this group mishandles these specific words".
• Age-of-English-onset regression — WER ≈ β0 + β1·age_english_onset + β2·age, plus a Spearman correlation, testing whether learning English later predicts worse transcription independent of current age.

Both are descriptive, not significance-tested; read them alongside the reported n.

L2-ARCTIC

Non-native English speech where a trained linguist marked the exact time window, expected phoneme, and produced phoneme of every pronunciation error (891 annotated errors in the built-in subset). It adds:

• Phone-error grounding — does saliency land on the moments a linguist flagged? Scored with AUROC (0.5 = random, 1.0 = perfect), average precision, and AUROC within speech, which only counts speech frames and so proves the model is honing in on the error rather than just preferring speech over silence.
• Accentedness regression — WER ≈ intercept + β·accentedness + group terms, where accentedness is human-verified error count per phone. If WER is already explained by how mispronounced an utterance was, a group's extra term shows whether the model penalises that accent beyond what the pronunciation itself predicts.

Common misreadings

• disparity_detected on an unmatched design is real but not proof of accent-specific bias — hence the content-confounded flag.
• speaker_confounded means "get more speakers", not "no bias".
• inconclusive — underpowered is not "fine". It means the sample was too small to detect a gap of practical size.
• Custom single-file uploads aren't eligible: fairness needs a dataset with a grouping column and enough items per group.

Glossary