xgbwwdata

A small, extensible dataset scanning + loading library designed for large-scale XGBoost2WW / WeightWatcher experiments.

What it does

• Scan one or more dataset sources (default: all supported sources)
• Apply consistent structural guards
  • min_rows, max_rows (subsample)
  • max_features (after preprocessing)
  • max_dense_elements (n_rows * n_features)
  • optional preprocessing (median/mode impute + one-hot)
• Run a 1-round XGBoost smoke test to confirm trainability
• Return a pandas.DataFrame registry of passing datasets with stable dataset_uids
• Load a dataset later by dataset_uid with the same preprocessing pipeline

Supported sources

• openml → openml:<did>
• pmlb → pmlb:<name>
• keel → keel:<name> (optional; requires keel-ds + XGBWW_ENABLE_KEEL=1)
• libsvm → libsvm:<filename> (from LIBSVM dataset pages)
• amlb → amlb_openml:<did> (OpenML suites 269/270/271 by default)

Install

Local editable install:

pip install -e .

Install all runtime dependencies from requirements.txt:

pip install -r requirements.txt

KEEL is now optional and disabled by default. To enable it:

export XGBWW_ENABLE_KEEL=1
pip install keel-ds

Colab / notebook fresh-clone install

Use the included helper so install + module refresh happens in one step (no manual kernel restart dance):

# Fresh clone
!rm-rf/content/repo_xgbwwdata
!gitclonehttps://github.com/CalculatedContent/xgbwwdata.git/content/repo_xgbwwdata# Installs requirements + editable package, clears stale xgbwwdata modules,# removes common repo-shadowing paths, and verifies import.%run/content/repo_xgbwwdata/scripts/colab_install.py--repo/content/repo_xgbwwdatafromxgbwwdataimportFilters, scan_datasets, load_datasetprint("import OK")

If you prefer fully manual steps, the helper script is equivalent to:

• upgrading pip/setuptools/wheel
• installing requirements.txt
• installing -e with --no-deps
• clearing stale xgbwwdata* modules and cache paths in the current kernel.

Quickstart

Notebook examples

• XGBWW_Dataset_Catalog_Checkpoint.ipynb — builds a dataset catalog with Drive checkpoint/resume.
• XGBWW_Catalog_Random5_XGBoost_Accuracy.ipynb — trains one model for random per-source samples.
• XGBWW_Multisource_Experiment.ipynb — multi-source XGBoost2WW experiment workflow.
• XGBWW_Checkpoint_PerSource_Validation.ipynb — validates checkpoint reuse per source.
• XGBWW_OpenML_1049_W7W8_Alpha_Checkpoint.ipynb — OpenML dataset 1049 run, tracking W7/W8 alpha by round with Drive checkpoints.

Scan datasets

fromxgbwwdataimportFilters, enable_logging, scan_datasetsenable_logging() # show scan progress in stdoutfilters=Filters(
min_rows=200,
max_rows=60000,
max_features=50000,
max_dense_elements=int(2e8),
preprocess=True,
)
df=scan_datasets(
sources=["openml", "pmlb", "keel", "libsvm", "amlb"], # explicit list (keel optional)limit=200, # stop after N passing datasets (None = all)filters=filters,
smoke_train=True, # 1-round XGBoost checkrandom_state=0,
log_every=10, # emit aggregate progress every N datasets
)
print(df.head())
print(df["source"].value_counts())

If sources is omitted, defaults are ["openml", "pmlb", "libsvm", "amlb"]. KEEL is included only when XGBWW_ENABLE_KEEL=1.

enable_logging() attaches a stream handler for the xgbwwdata logger so you can see per-dataset and periodic progress updates while scanning.

Load a dataset by id

fromxgbwwdataimportload_datasetrow=df.iloc[0]
X, y, meta=load_dataset(row["dataset_uid"], filters=filters)
# X is ready for xgboost.DMatrix / xgb.train