v. 2.2.0

[ddgunizar]

Interpolation and Boundary robustness are now independent scores

• Section A/B used to show the same combined score on both the Interpolation and
  Boundary robustness (formerly "Extrapolation" - renamed since the metric evaluates
  robustness at the edges of the observed y-range via repeated CV, not genuine
  out-of-domain extrapolation) columns, with the right-hand column mostly duplicating or
  leaving blank its sub-metrics. Both columns now have their own independent 0-10 score,
  their own score icon, and five dedicated sub-metrics each (0-2 points each), scored and
  explained separately

Boundary robustness now uses the same repeated-CV methodology as Interpolation

• The Low/High (bottom/top 20% of y, formerly labeled "Q1"/"Q5" - renamed since they are
  sorted-CV folds, not statistical quartiles) predictions and the Applicability Domain
  analysis used to come from a single train-once/test-once fit. They are now computed with
  the same 10x repeated 5-fold CV that Interpolation's "10x 5-fold CV"/"test" plot already
  used, scoped to each extreme's own 80/20 partition of the sorted dataset. This makes
  Boundary robustness's RMSE directly comparable to Interpolation's RMSE (both are averages
  over the same kind of repeated procedure, instead of one being averaged and the other a
  single noisy draw), and it fixed a real bug where the "Scaled RMSE (Low/High, sorted CV)"
  text shown in the report was still silently reading VERIFY's old single-pass values even
  after the plots themselves had already switched to the new repeated-CV predictions

Boundary robustness sub-metrics redesigned

• "Sorted CV, top 20% (High)" / "bottom 20% (Low)": scored on scaled RMSE (unchanged
  thresholds), now penalized by how many of the extreme predictions actually cross beyond
  the training range ("real range extension" / crossing rate) instead of by Spearman rank
• New "Spearman rank" item: Spearman rank correlation within the Low/High folds is now its
  own dedicated, independently-scored item instead of only acting as an internal penalty
  inside the RMSE items above
• "Degradation ratio" item: compares each extreme fold's (Low/High) scaled RMSE against the
  RMSE of its own "remaining 80%" baseline from the same sorted-CV run. Low and High are
  now scored independently (+1 each, <=1.5x tiering) instead of taking the worse (max) of
  the two sides, so a badly-degraded side is no longer invisible just because the other
  side happens to be fine
• Applicability domain (leverage): scoring and Williams-plot methodology unchanged, now
  also fed by the repeated-CV predictions instead of a single fit
• New unscored "Additional diagnostics" item: shows the global Spearman rank over the
  whole sorted dataset as context (the local, per-extreme Spearman is already scored in
  item 3), without double-counting rank that other items already score
• Every Boundary robustness sub-score is now a plain integer (no more .5 values feeding
  into the column total)

New Interpolation item: train vs. validation gap

• New item 4, "Train vs validation gap": compares each fold's out-of-fold validation RMSE
  against that same fold's own in-fold training RMSE (scaled RMSE (test) <= 1.25x train:
  +2, <= 1.5x: +1), to catch a model that fits its own training folds much better than it
  generalizes to their held-out validation fold - a form of overfitting that a validation-only
  metric can miss. The existing "CV vs test" comparison (now item 5) was retitled "CV vs
  test consistency" to disambiguate it from this new item, since both compare two error
  values but on different axes (fold-internal train/validation vs. overall CV/test)

Interpolation item 6 redesigned into three stability facets

• "Avg. standard deviation (SD)" is renamed "Prediction stability" and now averages three
  facets (each scored 0-2, final score is their average, rounded): (a) SD of the
  repeated-CV test-set predictions (unchanged), (b) SD of the out-of-fold
  train+validation predictions (new), (c) coefficient of variation of the aggregate RMSE
  across the 10 CV repeats (new) - a dataset-wide view of the same "how much does this
  depend on the random split" question that (a)/(b) ask per-point
• New plot: out-of-fold CV predictions +- SD, mirroring the existing test-set +- SD plot

Report PDF rendering fixes

• Fixed inconsistently cropped titles on the small report thumbnails (VERIFY tests, Low/High,
  applicability domain, Williams plot, SD plots): some showed no title, others a
  half-cut one, depending on each image's exact aspect ratio. The saved PNG files in
  PREDICT/VERIFY keep their full title as before; the report's image containers now use a
  height measured precisely from each image type so the title is cropped out cleanly and
  consistently instead of only in some plots
• Fixed a large blank gap in the Pearson correlation heatmap (worst on databases with very
  few descriptors, e.g. only 1 of 4 cells visible with 2 descriptors): masking the upper
  triangle left the always-fully-masked first row and last column as still-valid (if
  invisible) axes content, which bbox_inches='tight' does not crop away. The heatmap
  now trims that empty row/column from the plot only; the function's returned correlation
  matrix is unchanged (kept full-size, since PREDICT's correlation-pair detection indexes
  into it positionally and would silently miscount pairs if it were trimmed too)

VERIFY: new flawed-model test - cluster mean baseline

• Added a fourth flawed-model comparison, alongside y_mean/y_shuffle/onehot: a
  baseline that clusters the descriptors into two groups with KMeans and predicts each
  point using its own cluster's training-y mean, evaluated out-of-fold with the same
  repeated 10x 5-fold CV used for the real model. This specifically targets bimodal/
  structured datasets, where a trivial "which cluster am I in" rule can otherwise reach
  deceptively good RMSE and go undetected by the existing three tests

Bayesian Optimization: integer hyperparameters no longer reported with decimals

• Initial BO points are generated with Latin Hypercube Sampling and probed directly (instead
  of queued lazily), and every suggestion from the optimizer now has its integer-type
  hyperparameters (e.g. n_estimators, max_depth) rounded before being registered.
  The model itself was always fit with the correctly rounded value, but the reported/saved
  "best params" could previously show misleading decimals for these hyperparameters

Removed obsolete LOOCV documentation

• The --kfold help text described a "for databases with less than 50 points, do LOOCV"
  auto-behavior that was never implemented (kfold has always simply defaulted to 5,
  regardless of dataset size). The help text now matches the actual behavior

VERIFY module: fixed a systematic bias in the one-hot test

• The one-hot flawed-model test binarized every descriptor with "value == 0 -> 0, else ->
  1". For any strictly-positive continuous descriptor with no literal zeros (e.g. a
  molecular weight), this silently collapsed it into a useless constant column instead of
  a genuine two-way split, which could make the test fail regardless of whether the model
  was actually flawed. Continuous descriptors are now binarized with an unsupervised
  per-descriptor median split (value >= median -> 1, else -> 0), which does not require
  literal zeros and does not look at the target value (keeping the test an unbiased
  sanity check)
• Descriptors that already have 2 or fewer unique values are left untouched instead of
  being median-split: applying the median split to an imbalanced already-binary
  descriptor (its minority class under 50%) reproduces the same collapse-to-constant
  failure, just triggered by class imbalance instead of by missing literal zeros. This is
  handled per descriptor, so datasets mixing binary and continuous descriptors are scored
  correctly instead of only checking the dataset as a whole
• A discrete descriptor with many ties at its median can still collapse to a constant
  after the split; that single descriptor is now dropped from the one-hot matrix instead
  of letting it invalidate the whole test
• The one-hot test is skipped entirely (reported as N/A, not scored) when every descriptor
  already has 2 or fewer unique values, or when every descriptor collapses to a constant
  after binarization, since there is no continuous variation left to destroy and the test
  would otherwise always report a spurious failure

Bayesian Optimization: hyperparameter search space revised

• Bounds are now tuned for ROBERT's typical dataset sizes (mostly 20-100 datapoints, up to
  ~10,000 at most) rather than scikit-learn's generic defaults, which assume much larger
  samples. RF/GB's max_depth are no longer shared: GB (boosting) now searches a
  shallower 2-10 range instead of RF's 5-20, matching scikit-learn's own default of 3 for
  gradient boosting. NN's alpha lower bound widened down to scikit-learn's own default
  (0.0001), which the previous 0.01 floor excluded from the search entirely
• Removed three hyperparameters that were being searched without any real effect on the
  fitted model: GB's validation_fraction (only used if n_iter_no_change is set,
  which ROBERT never does, so scikit-learn silently ignores it), RF/GB's
  min_weight_fraction_leaf (constrains leaf size by weighted sample fraction, but
  ROBERT never passes sample_weight to fit(), so with implicit uniform weights this
  is a redundant, weaker echo of min_samples_leaf/min_samples_split, already
  searched), and NN's tol (affects when the solver stops, not the quality of the fit)
• Integer-type hyperparameters can now define a step size (e.g. n_estimators in steps
  of 10, NN's max_iter in steps of 25) instead of every single integer, so each BO
  evaluation explores meaningfully different configurations instead of two iterations
  landing on near-identical values (e.g. 71 vs. 72 trees)

New --all_models option: full VERIFY/PREDICT/REPORT for every screened model

• GENERATE already fits all 4 models (RF/GB/NN/MVL) during hyperparameter screening, but
  VERIFY/PREDICT/REPORT only ever ran on the single best one. With --all_models True,
  every model GENERATE screened gets its own full VERIFY -> PREDICT -> REPORT pass and its
  own PDF (e.g. ROBERT_report_RF_No_PFI.pdf, ROBERT_report_NN_PFI.pdf, ...), so the
  user can compare and pick a model manually instead of only ever seeing the
  auto-selected best one. Only this VERIFY/PREDICT/REPORT tail runs once per model - the
  expensive BO search in GENERATE is unaffected, since it already computes all 4 models
  regardless of this option
• Section F (Model Screening) shows a different heatmap when this option is active: instead
  of GENERATE's raw combined-RMSE-per-model heatmap (only reflects the BO search
  criterion), it shows each model's final Interpolation/Boundary robustness score (0-10), which
  is a more informative basis for comparing models once VERIFY/PREDICT have finished

PREDICT: faster SHAP analysis for RF and GB

• SHAP was always run as shap.Explainer(model.predict, X), i.e. through the generic
  predict-function wrapper. This hides the tree structure from SHAP, so it fell back to the
  Exact/Permutation explainer, which scales poorly with dataset size and descriptor count.
  RF and GB now use shap.TreeExplainer(model) directly, which computes the same exact
  Shapley values through the polynomial-time TreeSHAP algorithm instead of the slow
  generic fallback - same results, much faster on large datasets. NN and MVL are unaffected
  (TreeSHAP doesn't apply to them)
• Fixed a crash introduced by the change above for RF/GB classification models:
  TreeExplainer returns SHAP values per class (an extra 3rd array dimension) instead of
  the plain per-descriptor 2D array the generic explainer returned, which broke both the
  summary plot and the printed min/max SHAP values per descriptor. The last class (the
  positive class in binary problems) is now selected right after computing SHAP values, so
  plotting and the printed summary both consistently work on a 2D array again

REPORT: classification Interpolation score now reaches 10, same as regression

• Classification's Interpolation score used to max out at 8 (CV predictions, max 3 + test
  predictions, max 3 + the flawed-models penalty, always ≤ 0 + the MCC-difference item, max 2),
  missing the two points regression gets from item 6 (prediction stability) because that item's
  SD-based, %-of-y-range definition doesn't apply to a discrete label. Item 6 is now defined for
  classification too, adapting the same three facets: (a)/(b) the disagreement rate between
  individual CV repeats and the already majority-voted class (test set and out-of-fold
  train+validation, respectively), and (c) the coefficient of variation of the per-repeat MCC
  instead of the per-repeat RMSE - each scored 0-2 with the same ≤15%/≤25% thresholds used by
  regression's facet (c), then averaged and rounded, same as regression. This was a stale
  denominator bug in passing (the percentage-of-max used to look up the score tier was dividing
  by 9 - a leftover from an assumption that the flawed-models item could contribute +1, which it
  never does - before this fix corrected it to 8, now superseded by the real max of 10)

GENERATE/REPORT: --test_set 0 now works on its own, and the score is hidden when the
test set isn't the standard split

• --test_set 0 used to still get silently raised to 0.2 by the --auto_test safety
  net (needing --auto_test False on top to actually take effect) even though an
  explicit 0 is an unambiguous, deliberate "train on 100% of the data" choice, not a value
  that needs the safety net - only small-but-nonzero values (a likely accident) still get
  raised
• The ROBERT score's thresholds were calibrated assuming the standard ~20% test_set split
  (see the note in score.rst). With no internal test set at all (--test_set 0, or any
  other non-standard --test_set), the "test set predictions" score component silently
  computed as 0 (NaN <= threshold comparisons are always False in Python, so this
  failed silently, no crash and no warning), which could tank an otherwise good model's
  score for a reason invisible in the PDF. Section A now shows a short explanatory notice
  instead of a score in this case, the "Overall assessment" line shows "Not available"
  instead of a verdict based on that score, and Section B (entirely a breakdown of the
  score) is skipped - the rest of the report (SHAP, PFI, outliers, reproducibility, etc.)
  is unaffected

GENERATE: fixed the STRATIFIED split for classification, and warn on split methods
that don't apply to classification

• --split stratified used to cap the min/max target value and bin the remaining values
  into quantiles via pd.qcut() before stratifying - a regression-only approach (binning
  a continuous target). For classification this ignored the actual class labels entirely,
  relying on qcut accidentally degenerating to ~2 bins for a binary target to produce a
  passably-balanced split; for 3+ classes it was blocked outright (fell back to RND). Now
  classification stratifies directly on the class labels via StratifiedShuffleSplit,
  which works for any number of classes
• While fixing this, found and fixed a real pre-existing bug affecting the regression
  STRATIFIED split too: StratifiedShuffleSplit.split() yields (train_idx, test_idx)
  in that order, but the code unpacked it as for test_idx, _ in ..., taking the train
  indices as if they were the test indices. This had been silently compensated by a second,
  opposite bug (test_size=(100 - size) / 100 instead of size / 100, i.e. requesting
  the complement), so the two bugs cancelled out and regression's STRATIFIED split ended up
  correct anyway - fixed to compute both correctly on their own, without relying on that
  accidental cancellation
• EVEN/EXTRA_Q1/EXTRA_Q5 rely on a continuous, ordered target value (e.g. "the lowest 20% of
  y") that has no defined meaning for a discrete class label, regardless of class count -
  choosing one of these for a classification run now logs a warning and falls back to
  STRATIFIED (previously this fallback only existed for KN/STRATIFIED with 3+ classes, and
  fell back to RND)
• Classification's default split (--split auto, or no --split at all) changed from
  RND to STRATIFIED - preserving class proportions in both train and test is standard
  practice for classification, and a plain random split can, by chance, leave a class
  under/over-represented, especially on small datasets

[Copilot]

🟡 Changes recommended

One or more issues must be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

v2.2.0 expands scoring, repeated-CV boundary analysis, classification handling, reporting, and model evaluation workflows.

Changes:

• Independent Interpolation and Boundary robustness scoring.
• New VERIFY diagnostics, stability metrics, SHAP optimizations, and split handling.
• Added --all_models reporting and EVALUATE smoke coverage.

File summaries

File	Description
tests/test_easyrob.py	Updated as part of this pull request.
tests/test_6evaluate.py	Updated as part of this pull request.
tests/test_6evaluate.future	Updated as part of this pull request.
tests/test_5aqme_n_full.py	Updated as part of this pull request.
tests/test_4predict.py	Updated as part of this pull request.
tests/test_3verify.py	Updated as part of this pull request.
tests/test_2generate.py	Updated as part of this pull request.
tests/test_1curate.py	Updated as part of this pull request.
setup.py	Updated as part of this pull request.
robert/verify.py	Updated as part of this pull request.
robert/robert.py	Updated as part of this pull request.
robert/report_utils.py	Updated as part of this pull request.
robert/predict.py	Updated as part of this pull request.
robert/predict_utils.py	Updated as part of this pull request.
robert/gui_easyrob/utils/predictions_utils.py	Updated as part of this pull request.
robert/gui_easyrob/tutorials/overview.md	Updated as part of this pull request.
robert/gui_easyrob/tabs/predictions.py	Updated as part of this pull request.
robert/gui_easyrob/tabs/advanced_options.py	Updated as part of this pull request.
robert/gui_easyrob/main/window.py	Updated as part of this pull request.
robert/generate.py	Updated as part of this pull request.
robert/generate_utils.py	Updated as part of this pull request.
robert/evaluate.py	Updated as part of this pull request.
robert/curate.py	Updated as part of this pull request.
robert/argument_parser.py	Updated as part of this pull request.
robert/aqme.py	Updated as part of this pull request.
robert/api.py	Updated as part of this pull request.
README.md	Updated as part of this pull request.
docs/Tutorials/overview.rst	Updated as part of this pull request.
docs/Report/score.rst	Updated as part of this pull request.
docs/Report/images/spearman_rank_diagram.svg	Updated as part of this pull request.
docs/Report/images/sorted_low_high_diagram.svg	Updated as part of this pull request.
docs/Report/images/leverage_diagram.svg	Updated as part of this pull request.
docs/Report/images/degradation_ratio_diagram.svg	Updated as part of this pull request.
docs/Report/images/crossing_rate_diagram.svg	Updated as part of this pull request.
docs/README.rst	Updated as part of this pull request.
docs/Misc/versions.rst	Updated as part of this pull request.
.circleci/config.yml	Updated as part of this pull request.

Review details

Suppressed comments (9)

robert/gui_easyrob/utils/predictions_utils.py:155

• The new report puts Interpolation on the left and Boundary robustness on the right inside both suffix-specific PDFs, so model_key is not a valid left/right selector here. As written, the No_PFI extraction selects the Interpolation column while the PFI extraction selects the Boundary column, causing the dashboard score/fragment to be wrong even if the correct suffix PDF is supplied; select the region by metric instead.
  robert/predict_utils.py:271
• The classification stability facet is defined per CV repeat, but y_pred_test_all contains one prediction for every fold of every repeat and this code compares all raw fold predictions directly with the majority vote. That measures fold-level disagreement rather than repeat-level disagreement and can change the score; aggregate each repeat's fold predictions before computing the disagreement rate.
  robert/predict_utils.py:254
• These stability diagnostics are computed using model_data['repeat_kfolds'], but the output label is hardcoded to “10 repeats” (the same label is used for the train agreement and MCC lines below). A run with --repeat_kfolds 5, for example, reports the wrong methodology; use the configured repeat count in every stability label and in the corresponding parser.
  robert/predict_utils.py:249
• y_pred_train_all contains one prediction per point for every fold of every repeat, so its columns are repeat_kfolds * kfold, not one column per repeat. Computing RMSE/MCC across those columns treats individual fold predictions as separate repeats and does not measure the stated per-repeat stability; group each set of folds into one prediction per repeat before calculating the coefficient of variation.
  robert/predict_utils.py:275
• The train+validation classification facet has the same repeat-boundary problem: y_pred_train_all contains fold-level predictions, but this computes one agreement rate over all folds against a single vote across all repeats. That is not the specified repeat-to-repeat agreement; aggregate each repeat's out-of-fold predictions before comparing with the overall majority vote.
  robert/report_utils.py:1264
• The new item is described as comparing each fold's validation RMSE with that same fold's training RMSE, but this loop finds one aggregate Train fit (in-fold) line and divides the aggregate CV RMSE by the aggregate in-fold RMSE. Per-fold gaps are never retained, so a badly overfit fold can be hidden by averaging; compute and score the fold-level ratios.
  robert/utils.py:1642
• StratifiedShuffleSplit only preserves at least one sample per class in the train/test split; it does not ensure that the resulting training set has at least kfold samples per class for the downstream StratifiedKFold. A small balanced classification dataset (for example, 5 samples per class with the default 4-point test set) leaves fewer than five members per class, causing warnings and folds without a class during every CV/BO run. The split or kfold value needs validation/adjustment before proceeding.
  robert/utils.py:3288
• For multiclass classifiers, TreeExplainer returns one explanation per class, but this unconditionally keeps only the last class. Since this release now supports classification splits with three or more classes, the report's unlabeled SHAP plot and min/max summary can describe an arbitrary class rather than the model's predictions; handle multiclass outputs explicitly (aggregate or label/render per-class values) instead of treating the last class as universally positive.
  robert/utils.py:3262
• For RF/GB classifiers, the previous generic explainer wrapped loaded_model.predict (hard class labels), while TreeExplainer(loaded_model) explains the estimator's continuous tree output/probability and the code selects the positive-class channel. Thus the classification SHAP values and plots change semantically, despite the PR describing this as the same result; explicitly choose and document a matching output (for example predict_proba if probabilities are intended) or preserve the old predictor target.

• Files reviewed: 34/59 changed files
• Comments generated: 14
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 40.08778% with 819 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.77%. Comparing base (ebedbe1) to head (9b07173).

Files with missing lines	Patch %	Lines
robert/report.py	7.90%	338 Missing ⚠️
robert/report_utils.py	5.08%	317 Missing ⚠️
robert/utils.py	79.61%	84 Missing ⚠️
robert/verify.py	78.72%	20 Missing ⚠️
robert/predict.py	54.05%	17 Missing ⚠️
robert/generate_utils.py	27.77%	13 Missing ⚠️
robert/aqme.py	0.00%	7 Missing ⚠️
robert/gui_easyrob/utils/predictions_utils.py	76.92%	6 Missing ⚠️
robert/robert.py	0.00%	6 Missing ⚠️
robert/evaluate.py	0.00%	3 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #85      +/-   ##
==========================================
- Coverage   57.23%   55.77%   -1.47%     
==========================================
  Files          32       32              
  Lines        8804     9492     +688     
==========================================
+ Hits         5039     5294     +255     
- Misses       3765     4198     +433     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.