First run of sklearn classifier crashes constantly

[janvanrijn]

see attached error.

My guess: flow exists is called, and it returns null. Flow can be uploaded, but flow id is not fixed.

should be tested with a sentinel

Traceback (most recent call last):
 File "/home/janvanrijn/projects/sklearn-bot/examples/run_on_study.py", line 53, in <module>
 run()
 File "/home/janvanrijn/projects/sklearn-bot/examples/run_on_study.py", line 45, in run
 args.upload_result)
 File "/home/janvanrijn/projects/sklearn-bot/sklearnbot/bot/run.py", line 69, in run_bot_on_task
 run = openml.runs.run_model_on_task(task, classifier)
 File "/home/janvanrijn/projects/openml-python/openml/runs/functions.py", line 48, in run_model_on_task
 add_local_measures=add_local_measures)
 File "/home/janvanrijn/projects/openml-python/openml/runs/functions.py", line 146, in run_flow_on_task
 "same: '%s' vs '%s'" % (str(flow.flow_id), str(flow_id))
ValueError: Result from API call flow_exists and flow.flow_id are not same: '108950' vs 'False'

[mfeurer]

Could you please attach a minimal example to reproduce this?

[janvanrijn]

It is kind of hard to create a MWE, because it's hard to unit test. (while writing this, I found a nifty solution)

The current unit tests use the run_flow_on_task fn, which automatically uploads the flow, whereas the run_model_on_task potentially uploads the flow internally, triggering this problem.

import hashlib
import openml
import sklearn.pipeline
import sklearn.tree
import time

openml.config.server = 'https://test.openml.org/api/v1'

md5 = hashlib.md5()
md5.update(str(time.time()).encode('utf-8'))
sentinel = md5.hexdigest()[:10]
sentinel = 'TEST%s' % sentinel

task = openml.tasks.get_task(115)
clf = sklearn.pipeline.Pipeline(steps=[(sentinel,
 sklearn.tree.DecisionTreeClassifier())])
openml.runs.run_model_on_task(clf, task)

[PGijsbers]

The MWE does not produce an error for me. I think this got fixed in #516.