To handle non-actionable steps in sklearn

[Neeratyoy]

Reference Issue

Addresses #825 and #480.

What does this PR implement/fix? Explain your changes.

The serialization expects each step to be a sklearn module and eventually an OpenMLFlow or a None. For ColumnTransform operations, a drop or passthrough can be specified to include a step in a pipeline but not execute it. The changes in this PR incorporates this additional string handling of the keywords drop and passthrough in a ColumnTransformer call.

How should this PR be tested?

importopenmlimportnumpyasnpimportpandasaspdfromsklearn.pipelineimportmake_pipelinefromsklearn.composeimportColumnTransformerfromsklearn.preprocessingimportOneHotEncoder, StandardScalerfromsklearn.imputeimportSimpleImputerfromsklearn.treeimportDecisionTreeClassifierdataset_id=61dataset=openml.datasets.get_dataset(dataset_id)
task=openml.tasks.get_task(59)
X, y, categorical_ind, feature_names=dataset.get_data(target=dataset.default_target_attribute, dataset_format='array')
categorical_ind=np.array(categorical_ind)
cat_idx, =np.where(categorical_ind)
cont_idx, =np.where(~categorical_ind)
clf=make_pipeline(ColumnTransformer([('cat', make_pipeline(SimpleImputer(strategy='most_frequent'), OneHotEncoder()), cat_idx.tolist()),
('cont', make_pipeline(SimpleImputer(strategy='median'), StandardScaler()), cont_idx.tolist())]) , DecisionTreeClassifier(max_depth=1))
ifnotcategorical_ind.any():
clf[0].set_params(cat='drop')
ifnot (~categorical_ind).any():
clf[0].set_params(cont='drop')
hotencoded=clf[:-1].fit_transform(X)
bla=openml.runs.run_model_on_task(model=clf, task=task, return_flow=True)

Any other comments?

This PR handles the specific case of a ColumnTransform. I'm not sure if there are other such keywords which should be handled equivalently. We can then maintain a keyword listing, based on the sklearn version under extension.py and handle them with a similar logic.
If there are more diverse cases that may arise, some examples would help in designing a generalized solution in that case.
Would need to accordingly design relevant unit tests too.

[codecov-io]

Codecov Report

Merging #866 into develop will increase coverage by 2.23%.
The diff coverage is 83.33%.

@@ Coverage Diff @@## develop #866 +/- ##
===========================================
+ Coverage 88.38% 90.62% +2.23% 
===========================================
Files 37 37 Lines 4298 6163 +1865 ===========================================
+ Hits 3799 5585 +1786 - Misses 499 578 +79

Impacted Files	Coverage Δ
openml/extensions/sklearn/extension.py	93.43% <83.33%> (+2.11%)	⬆️
openml/evaluations/evaluation.py	63.88% <0%> (-2.78%)	⬇️
openml/extensions/functions.py	95.65% <0%> (-1.13%)	⬇️
openml/utils.py	90.66% <0%> (-0.41%)	⬇️
openml/exceptions.py	93.47% <0%> (-0.08%)	⬇️
openml/extensions/sklearn/__init__.py	100% <0%> (ø)	⬆️
openml/setups/__init__.py	100% <0%> (ø)	⬆️
openml/tasks/__init__.py	100% <0%> (ø)	⬆️
openml/__init__.py	100% <0%> (ø)	⬆️
openml/extensions/__init__.py	100% <0%> (ø)	⬆️
... and 28 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e489f41...3b03d15. Read the comment docs.

[mfeurer]

I think your changes all make sense, but require some explanation in the code (and tests).

[mfeurer]

Could you please also add some error handling for earlier scikit-learn versions? Right now the tests fail due to a hard-to-interpret error message.

[Neeratyoy]

@mfeurer one of the old unit tests that I had to change test_serialize_feature_union fails for versions 0.18.2 and 0.19.2.
It appears that the manner in which we are serializing, the behaviour of one of the intermediate stages have changed since these two versions. To be more specific, the parameter metadata extraction.

Should I then try to change the code to see how this can be handled or skip that particular unit test?

[mfeurer]

Should I then try to change the code to see how this can be handled or skip that particular unit test?

You can skip that unit test.