Skip to content

Remove nan-likes from category header - #1037

Merged
PGijsbers merged 2 commits into
developfrom
fix_1036
Mar 12, 2021
Merged

Remove nan-likes from category header#1037
PGijsbers merged 2 commits into
developfrom
fix_1036

Conversation

@PGijsbers

Copy link
Copy Markdown
Collaborator

Pandas does not accept None/nan as a category (note: of course it does allow nan-values in the data itself). However outside source (i.e. ARFF files) do allow nan as a category, so we must filter these.

Penguins has the column: @ATTRIBUTE sex {?,FEMALE,MALE,_}

Running

importopenmlpenguins=openml.datasets.get_dataset(42585)
data, *_=penguins.get_data()
print(data.head())

Before:

Traceback (mostrecentcalllast):
File"E:/repositories/openml-python/mwe.py", line4, in<module>data, *_=penguins.get_data()
File"E:\repositories\openml-python\openml\datasets\dataset.py", line693, inget_datadata, categorical, attribute_names=self._load_data()
File"E:\repositories\openml-python\openml\datasets\dataset.py", line531, in_load_datareturnself._cache_compressed_file_from_file(file_to_load)
File"E:\repositories\openml-python\openml\datasets\dataset.py", line488, in_cache_compressed_file_from_filedata, categorical, attribute_names=self._parse_data_from_arff(data_file)
File"E:\repositories\openml-python\openml\datasets\dataset.py", line445, in_parse_data_from_arffself._unpack_categories(X[column_name], categories_names[column_name])
File"E:\repositories\openml-python\openml\datasets\dataset.py", line650, in_unpack_categoriesraw_cat=pd.Categorical(col, ordered=True, categories=categories)
File"E:\repositories\openml-python\venv\lib\site-packages\pandas\core\arrays\categorical.py", line316, in__init__dtype=CategoricalDtype._from_values_or_dtype(
File"E:\repositories\openml-python\venv\lib\site-packages\pandas\core\dtypes\dtypes.py", line330, in_from_values_or_dtypedtype=CategoricalDtype(categories, ordered)
File"E:\repositories\openml-python\venv\lib\site-packages\pandas\core\dtypes\dtypes.py", line222, in__init__self._finalize(categories, ordered, fastpath=False)
File"E:\repositories\openml-python\venv\lib\site-packages\pandas\core\dtypes\dtypes.py", line369, in_finalizecategories=self.validate_categories(categories, fastpath=fastpath)
File"E:\repositories\openml-python\venv\lib\site-packages\pandas\core\dtypes\dtypes.py", line543, invalidate_categoriesraiseValueError("Categorial categories cannot be null")
ValueError: CategorialcategoriescannotbenullProcessfinishedwithexitcode1

After:

 species island culmen_length_mm ... flipper_length_mm body_mass_g sex
0 Adelie Torgersen 39.1 ... 181.0 3750.0 MALE
1 Adelie Torgersen 39.5 ... 186.0 3800.0 FEMALE
2 Adelie Torgersen 40.3 ... 195.0 3250.0 FEMALE
3 Adelie Torgersen NaN ... NaN NaN NaN
4 Adelie Torgersen 36.7 ... 193.0 3450.0 FEMALE
[5 rows x 7 columns]

Pandas does not accept None/nan as a category (note: of course
it does allow nan-values in the data itself). However outside source
(i.e. ARFF files) do allow nan as a category, so we must filter these.
@PGijsbers
PGijsbers requested a review from mfeurerMarch 12, 2021 09:59
@PGijsbers
PGijsbers merged commit 4aec00a into developMar 12, 2021
@PGijsbers
PGijsbers deleted the fix_1036 branch March 12, 2021 13:09
PGijsbers added a commit to Mirkazemi/openml-python that referenced this pull request Feb 23, 2023
* Remove nan-likes from category header
Pandas does not accept None/nan as a category (note: of course
it does allow nan-values in the data itself). However outside source
(i.e. ARFF files) do allow nan as a category, so we must filter these.
* Test output of _unpack_categories
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@PGijsbers@mfeurer