Data selection methods are almost always evaluated against a random subset of the same size. That comparison tests the ranking criterion. It does not answer the question a practitioner faces: should I select a subset, or just train on everything I already have?
This repository runs both comparisons side by side — 5,280 fine-tuning runs over five text-classification tasks, two corruption mechanisms, five selection methods and three encoders — and contains everything needed to reproduce them.
Five conditions, evaluated on the same clean test set. all data is the whole pool after corruption, random 20% a uniformly random fifth of it, selected 20% the fifth a method picks from it. The two additions are clean 20%, a random fifth of the pool before corruption, and clean 100%.
After four fifths of the pool is replaced with off-domain text, selected subsets beat matched random subsets by +0.147 — and beat full-data training in 1 of 15 dataset–model cells. Judged one way it is a large win; judged the other it is not a win at all.
The flat clean 20% line is what a selector is really chasing: the data your budget would have bought had the corruption never happened. Any controlled study already runs it, as its clean-condition random control. With it on the page the comparison becomes an identity:
(selected 20%) − (all data) = D − B − S
D damage clean 100% − all data what the corruption cost a full-data learner
B budget cost clean 100% − clean 20% what a 20% budget costs on clean data
S shortfall clean 20% − selected 20% how far the selector falls short of an
uncorrupted subset of the same size
Selection pays exactly when D > B + S. Two of the three terms have nothing
to do with the selection method — which is why an evaluation reporting only the
random-subset comparison cannot tell the contamination case from the noise case,
though the right advice in the two is opposite.
Cells where the selected 20% beats training on all the data, out of 15 per corner
(5 datasets × 3 encoders). w (s): w cells with a positive mean paired
difference, s of them significant at α = .05.
| method | clean | off-domain | label noise | both | total |
|---|---|---|---|---|---|
| RHO-Loss | 0/15 (0) | 6/15 (1) | 11/15 (11) | 15/15 (15) | 32/60 (27) |
| GradMatch | 2/15 (0) | 0/15 (0) | 6/15 (5) | 14/15 (12) | 22/60 (17) |
| InfoBatch | 4/15 (0) | 5/15 (0) | 3/15 (0) | 5/15 (0) | 17/60 (0) |
| GREATS | 0/15 (0) | 1/15 (0) | 5/15 (5) | 3/15 (3) | 9/60 (8) |
| MATES | 0/15 (0) | 1/15 (0) | 4/15 (3) | 3/15 (3) | 8/60 (6) |
Training on everything is the better option in the large majority of cells, and
not one of the 75 clean-pool cells is a significant win for selection. A selector
can only recover damage that was actually done, so D is a ceiling on what any
method can buy at any budget.
The leftover term S separates the methods, and turns results the usual
comparison leaves as brute facts into a diagnosis:
| method | shortfall S (60 cells) | on clean pools | dS/dD | R² of unfitted D − B |
|---|---|---|---|---|
| RHO-Loss | 0.034 ±.007 | 0.003 ±.012 | 0.16 ±.02 | 0.88 |
| GradMatch | 0.098 ±.024 | 0.007 ±.004 | 0.38 ±.05 | 0.21 |
| InfoBatch | 0.159 ±.026 | −0.001 ±.001 | 1.00 ±.03 | −36.09 |
| GREATS | 0.201 ±.058 | 0.095 ±.037 | 0.36 ±.12 | −0.55 |
| MATES | 0.212 ±.023 | 0.132 ±.025 | 0.35 ±.07 | −1.18 |
D − B fits nothing, so it can and does go negative; those values are printed
rather than suppressed. A slope dS/dD of 1 means the method removes none of the
corruption — InfoBatch sits exactly there, which is correct for a method whose
stated objective is lossless speed-up. GREATS and MATES lose more than a tenth of
an accuracy point before any corruption is introduced. GradMatch fails on
Emotion by selecting badly, not by lacking opportunity.
For RHO-Loss the account also predicts out of sample. D − B explains 88% of the
variance in advantage over 60 cells with nothing fitted, against 23% for the
corrupted fraction and 46% for intrinsic quality scores of the pool. It forecasts
the noise level at which selection starts to win:
A separate 330-run sweep, sharing no endpoint with the grid above. Every input to the forecast comes from runs that never touched a noisy pool; it lands in the right interval in 3 of 3 cells for RHO-Loss and 1 of 3 for GradMatch. The account is method-dependent — it holds for RHO-Loss, weakens for GradMatch, and fails for the other three — and this repository reports where it fails as well as where it holds.
Component 1 perturb → score axes perturbations/ scoring/
Component 2 select → fine-tune → eval selection/
Analysis tables, plots, statistics selection/analyze.py axis_trends.py axis_stats.py
Two ways to degrade a pool.eta flips a fraction of training labels to a
different class. rho replaces a fraction of examples with text from an unrelated
corpus, labelled by a deterministic function of its own source class — so the
substituted subtask stays internally consistent and perfectly learnable, and only
its relevance is destroyed. That is what keeps the second dial from being a
confounded version of the first.
| run | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
eta | 0.0 | 0.6 | 0.0 | 0.6 | 0.2 | 0.4 |
rho | 0.8 | 0.0 | 0.0 | 0.8 | 0.0 | 0.0 |
| corner | off-domain | label noise | clean | both | sweep | sweep |
The five conditions, in the vocabulary of the code. There is no separate "clean" condition — it is the same condition run on the unperturbed corner:
| in the figure above | run | condition |
|---|---|---|
| all data | corrupted corner | random |
| random 20% | corrupted corner | random_ratio@<method> |
| selected 20% | corrupted corner | <method> |
| clean 20% | run3 | random_ratio@<method> |
| clean 100% | run3 | random |
D, B and S are differences between those cells. The pipeline produces all
five per (dataset, corner, method, seed) in
results/component2/<dataset>.json; the decomposition itself is arithmetic on
that file and is not computed by the analysis modules, which report the
advantage over each method's matched-random control and its correlation with the
intrinsic axes.
Selection methods. Each is one module in selection/ exposing
method(ds, model) -> model.
| Condition | Method | Reference |
|---|---|---|
rho_loss | reducible holdout loss | Mindermann et al., ICML 2022 |
gradmatch / gradmatchpb | OMP gradient matching (per-class / per-batch) | Killamsetty et al., ICML 2021 |
greats | greedy second-order ghost-gradient selection | Wang et al., NeurIPS 2024 |
mates | data-influence model with Gumbel-Top-k sampling | Yu et al., NeurIPS 2024 |
infobatch | unbiased dynamic pruning of well-learned samples | Qin et al., ICLR 2024 |
random / random_ratio | full-pool ERM / random at another method's budget | — |
clean_only | trains only on the trusted guide slice (equal-use control) | — |
Intrinsic diagnostics. Component 1 also scores each corrupted pool on two
training-free axes — Learnable (pointwise V-information, Ethayarajh et al.,
ICML 2022) and Worth-Learning (DSIR importance, Xie et al., NeurIPS 2023). They
confirm the two dials move different things, and they are what D − B is measured
against: they describe the data, whereas D and B describe what the data and
the budget do to a particular learner.
Evaluation protocol. Every reported number is measured on the clean,
unperturbed test set. Under the default *_source: clean_eval, every guidance /
holdout / reference set a method needs is carved from that same clean split —
disjoint from the scored remainder, fixed across seeds, so no condition sees its
own test data. Three of the five methods consume a 1,024-example trusted slice
that full-data training never sees; clean_only isolates what that slice is worth
on its own.
python3 -m venv .venv &&source .venv/bin/activate # Python 3.11
pip install -r requirements.txt # on GPU, install torch per pytorch.org firstDatasets, perturbation runs and encoder are chosen once in config.yml; no stage
takes its own dataset flags.
bash scripts/dry-run.sh # tiny end-to-end check, run this first
bash scripts/run.sh # full pipeline (Component 1 then 2)
python -m selection.axis_trends # cross-dataset trend table + figurescripts/run.sh drives both components through the scheduler, packing as many
jobs onto each GPU as VRAM allows. To run a stage on its own:
python -m perturbations.make_runs # 1. build + score the perturbed runs
python -m selection.run # 2. train every method vs. random
python -m selection.analyze --base ag_news # 3. per-dataset summary tableSee scripts/README.md for the helper scripts and
perturbations/README.md for make_runs flags and
caching.
scripts/run.sh on its defaults is exactly one encoder's grid: 5 datasets × 4
corners × 11 conditions × 8 seeds = 1,760 runs. The 11 conditions are the five
selection methods, their five budget-matched random controls, and full-pool ERM.
The full study is that grid run three times, with model_name swapped in
config.yml and a disjoint seed family each time — google/bert_uncased_L-8_H-512_A-8
(BERT-medium), google/bert_uncased_L-4_H-256_A-4 (BERT-mini) and
distilbert-base-uncased.
bash scripts/run.sh # one encoder's grid
bash scripts/run-noise-sweep.sh # the 330-run label-noise sweep
python scripts/analyze_noise_sweep.py # its tables + figures
python -m selection.axis_stats # pooled statistics over the gridconfig.yml is the single source of truth. The knobs that change the scope of a
run:
datasets— base datasets to run end-to-end. Any name registered inscoring.data_loader._DATASET_REGISTRYwithtextandlabelcolumns works.runs— which(eta, rho)corners to build, score and train on.model_name— one Hugging Face id, fanned out to every component, every auxiliary model and every checkpoint directory name.
Runtime is dominated by the row caps (component1.max_*_examples,
component2.max_*_examples; 0 = whole split) and by compute.max_parallel_jobs.
The caps are part of the checkpoint cache key, so changing one resolves the right
checkpoint automatically.
Adding a new base dataset needs one entry in OFF_TARGET
(perturbations/make_runs.py), naming the off-domain text source for the rho dial.
results/
perturbation_runs/<dataset>.json axis scores per run
perturbation_runs/learnable_vs_worth_learning.png
component2/<dataset>.json per-method accuracy per run + axis scores
component2/plots/ per-dataset figures (and seed<N>/ per seed)
component2/plots/axis_trends.png pooled advantage-vs-axis trends
Inputs and intermediates live under data/ (perturbed CSVs, HF cache), models/
(f_V / g_V checkpoints) and outputs/worth_learning/ (DSIR); all are derived
from component1.data_dir / models_dir and are git-ignored.
pytest tests -qtests/ checks each ported selection method against its reference implementation
— GREATS ghost-gradient scores against brute-force per-sample gradients,
GradMatch's OMP against CORDS, the MATES oracle and Gumbel-Top-k kernel, RHO-LOSS
irreducible-loss selection, InfoBatch's prune/rescale — plus end-to-end fine-tunes
under each condition.
| Path | Contents |
|---|---|
perturbations/ | the eta/rho dials and the run materialiser (Component 1) |
scoring/ | the intrinsic axis scorers: learnable/ (PVI), worth_learning/ (DSIR) |
selection/ | selection methods, training driver, scheduler, plots and statistics (Component 2) |
profiling/ | opt-in latency / GPU-utilisation profiling (--profile) |
scripts/ | end-to-end run, dry run and analysis helpers |
tests/ | reference-parity and end-to-end tests |
MIT — see LICENSE.