not JSON serializable error when running flow on a task

[dbehrouz]

Description

I encounter the following error when running a flow (for example 7103) on a task (for example 146607).
<openml.flows.flow.OpenMLFlow object at 0x11cc43110> is not JSON serializable

Steps/Code to Reproduce

Example:

import openml
flow = openml.flows.get_flow(7103)
task = openml.tasks.get_task(146607)
run = runs.run_flow_on_task(task, flow)

Expected Results

OpenMLRun object

Actual Results

---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-102-f7fb112d9304> in <module>()
----> 1 run = runs.run_flow_on_task(task, flow)

/usr/local/lib/python2.7/site-packages/openml-0.7.0-py2.7.egg/openml/runs/functions.pyc in run_flow_on_task(task, flow, avoid_duplicate_runs, flow_tags, seed)
 85 if avoid_duplicate_runs and flow_id:
 86 flow_from_server = get_flow(flow_id)
---> 87 setup_id = setup_exists(flow_from_server, flow.model)
 88 ids = _run_exists(task.task_id, setup_id)
 89 if ids:

/usr/local/lib/python2.7/site-packages/openml-0.7.0-py2.7.egg/openml/setups/functions.pyc in setup_exists(flow, model)
 44 raise ValueError('This should not happen!')
 45 
---> 46 openml_param_settings = openml.runs.OpenMLRun._parse_parameters(flow, model)
 47 description = xmltodict.unparse(_to_dict(flow.flow_id,
 48 openml_param_settings),

/usr/local/lib/python2.7/site-packages/openml-0.7.0-py2.7.egg/openml/runs/run.pyc in _parse_parameters(flow, model)
 360 flow_dict = get_flow_dict(flow)
 361 parameters = extract_parameters(flow, flow_dict, model,
--> 362 True, flow.flow_id)
 363 
 364 return parameters

/usr/local/lib/python2.7/site-packages/openml-0.7.0-py2.7.egg/openml/runs/run.pyc in extract_parameters(_flow, _flow_dict, component_model, _main_call, main_id)
 340 raise e
 341 elif not isinstance(step, openml.flows.OpenMLFlow):
--> 342 raise e
 343 continue
 344 else:

TypeError: <openml.flows.flow.OpenMLFlow object at 0x11cc43110> is not JSON serializable

Versions

Darwin-17.5.0-x86_64-i386-64bit
('Python', '2.7.15 (default, May 28 2018, 14:20:45) \n[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)]')
('NumPy', '1.13.3')
('SciPy', '1.0.0')
('Scikit-Learn', '0.18.1')
('OpenML', '0.7.0')

[janvanrijn]

This code doesn't reproduce the same error on my side. Which branch are you running?

/home/janvanrijn/anaconda3/envs/openml-python2/bin/python2.7 /home/janvanrijn/projects/openml-python/test_jan.py
Traceback (most recent call last):
 File "/home/janvanrijn/projects/openml-python/test_jan.py", line 5, in <module>
 run = openml.runs.run_flow_on_task(task, flow)
 File "/home/janvanrijn/projects/openml-python/openml/runs/functions.py", line 80, in run_flow_on_task
 flow.model = _get_seeded_model(flow.model, seed=seed)
 File "/home/janvanrijn/projects/openml-python/openml/runs/functions.py", line 292, in _get_seeded_model
 model_params = model.get_params()
AttributeError: 'NoneType' object has no attribute 'get_params'

(Which is something we should fix though)

[dbehrouz]

I'm getting the error on the master branch.

I will dig a bit deeper later (I think I got a similar error to yours when I was trying to run an "mlr" flow on the specified task).

[dbehrouz]

@janvanrijn Running the same code on the develop shows a different error from the master branch and the one you got

/usr/local/lib/python2.7/site-packages/openml-0.8.0.dev0-py2.7.egg/openml/runs/functions.pyc in run_flow_on_task(task, flow, avoid_duplicate_runs, flow_tags, seed, add_local_measures)
 105 
 106 # execute the run
--> 107 res = _run_task_get_arffcontent(flow.model, task, add_local_measures=add_local_measures)
 108 
 109 # in case the flow not exists, we will get a "False" back (which can be

/usr/local/lib/python2.7/site-packages/openml-0.8.0.dev0-py2.7.egg/openml/runs/functions.pyc in _run_task_get_arffcontent(model, task, add_local_measures)
 411 res = _run_model_on_fold(model_fold, task, rep_no, fold_no, sample_no,
 412 can_measure_runtime=can_measure_runtime,
--> 413 add_local_measures=add_local_measures)
 414 arff_datacontent_fold, arff_tracecontent_fold, user_defined_measures_fold, model_fold = res
 415 

/usr/local/lib/python2.7/site-packages/openml-0.8.0.dev0-py2.7.egg/openml/runs/functions.pyc in _run_model_on_fold(model, task, rep_no, fold_no, sample_no, can_measure_runtime, add_local_measures)
 508 if can_measure_runtime:
 509 modelfit_starttime = time.process_time()
--> 510 model.fit(trainX, trainY)
 511 
 512 if can_measure_runtime:

/usr/local/lib/python2.7/site-packages/sklearn/pipeline.pyc in fit(self, X, y, **fit_params)
 266 This estimator
 267 """
--> 268 Xt, fit_params = self._fit(X, y, **fit_params)
 269 if self._final_estimator is not None:
 270 self._final_estimator.fit(Xt, y, **fit_params)

/usr/local/lib/python2.7/site-packages/sklearn/pipeline.pyc in _fit(self, X, y, **fit_params)
 232 pass
 233 elif hasattr(transform, "fit_transform"):
--> 234 Xt = transform.fit_transform(Xt, y, **fit_params_steps[name])
 235 else:
 236 Xt = transform.fit(Xt, y, **fit_params_steps[name]) \

/usr/local/lib/python2.7/site-packages/sklearn/preprocessing/data.pyc in fit_transform(self, X, y)
 1900 """
 1901 return _transform_selected(X, self._fit_transform,
-> 1902 self.categorical_features, copy=True)
 1903 
 1904 def _transform(self, X):

/usr/local/lib/python2.7/site-packages/sklearn/preprocessing/data.pyc in _transform_selected(X, transform, selected, copy)
 1700 return transform(X)
 1701 
-> 1702 if len(selected) == 0:
 1703 return X
 1704 

TypeError: object of type 'NoneType' has no len()

[mfeurer]

@d-behi and @janvanrijn I can reproduce this and will no try to figure out what's going wrong here.

[mfeurer]

@d-behi I can reproduce your error on the development branch, too, using the following code:

import openml

import sklearn.pipeline
import sklearn.preprocessing
import sklearn.feature_selection
import sklearn.svm

flow = sklearn.pipeline.Pipeline(
 [
 ('imputation', sklearn.preprocessing.Imputer()),
 ('onehot', sklearn.preprocessing.OneHotEncoder(handle_unknown='ignore',
 categorical_features=None)),
 ('variancethreshold', sklearn.feature_selection.VarianceThreshold()),
 ('classifier', sklearn.svm.SVC())
 ]
)

task = openml.tasks.get_task(146607)
run = openml.runs.run_model_on_task(task, flow)

[mfeurer]

Replacing the None by a list solves this issue:

import openml

import sklearn.pipeline
import sklearn.preprocessing
import sklearn.feature_selection
import sklearn.svm

#flow = openml.flows.get_flow(7103)

flow = sklearn.pipeline.Pipeline(
 [
 ('imputation', sklearn.preprocessing.Imputer()),
 ('onehot', sklearn.preprocessing.OneHotEncoder(handle_unknown='ignore',
 categorical_features=[])),
 ('variancethreshold', sklearn.feature_selection.VarianceThreshold()),
 ('classifier', sklearn.svm.SVC())
 ]
)

task = openml.tasks.get_task(3)
#run = openml.runs.run_flow_on_task(task, flow)
run = openml.runs.run_model_on_task(task, flow)

[mfeurer]

Okay, I found the issue. When downloading the flow and not having the study14 package available, the flow will still deserialize and will actually have an empty step (for the imputation). However, the parameter extraction for the run upload cannot cope with this empty step. I will create a hotfix which gives a better error message.