Skip to content

fix(preprocessor): raise a clear error on duplicate column names #37

Description

@ChrisW09

Summary

Fitting on a DataFrame with duplicate column labels fails with an opaque
AttributeError: 'DataFrame' object has no attribute 'dtype' from deep inside
_detect_column_types, with nothing pointing at the real cause.

Reproduction

importnumpyasnp, pandasaspdfrompretabimportPreprocessorrng=np.random.default_rng(0)
dup=pd.DataFrame(np.column_stack([rng.normal(size=50)] *2), columns=["a", "a"])
Preprocessor(numerical_method="minmax").fit(dup, rng.normal(size=50))
AttributeError: 'DataFrame' object has no attribute 'dtype'
... pretab/preprocessor.py, line 307, in _detect_column_types

Expected

A PretabDataError naming the duplicated labels and explaining that column names must be
unique — the ColumnTransformer this builds keys its transformers by column name, so
duplicates cannot be routed unambiguously even if detection were fixed.

Actual

AttributeError from numpy/pandas internals.

Root cause

pretab/preprocessor.py:288-312. The loop does X[col], which returns a DataFrame
rather than a Series when col is duplicated, so X[col].dtype does not exist. (nunique()
on the preceding line happens to work on a frame, which is why the failure surfaces on the
dtype access rather than earlier.)

Suggested fix

Check up front in fit / _detect_column_types, after the dict/ndarray coercion:

duplicated=X.columns[X.columns.duplicated()].unique().tolist()
ifduplicated:
raisePretabDataError(
f"Duplicate column names are not supported: {duplicated}.\n""Fix: rename the columns so every name is unique."
)

Cheap to check and turns a confusing internal failure into an actionable one.

Environment

  • pretab 0.1.0 (main @ 51c3043)
  • Python 3.11.15, numpy 2.4.6, pandas 2.3.3, scikit-learn 1.9.0, scipy 1.17.1
  • macOS (darwin 25.5.0)

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions