Cache runs for later upload

[joaquinvanschoren]

On some computing clusters, worker nodes don't have internet access. Hence, we need to store openml runs to disk and upload them later by a separate process.

This basically means separating run_task and publish_run, but we do lack code to dump runs to disk and to retrieve them later.

[mfeurer]

I had this issue as well. In my opinion it's easier to tunnel the connection to the server via SSH instead of writing lots of python code to accomplish this. What do you think?

[rhiever]

In my specific case, the worker nodes don't have access to the internet but the head node (which connects to the same hard drive) does. If there were separate run_task and publish_run functions, then that would be perfect---as long as the run_task function provided me an ID to pass to publish_run.

[janvanrijn]

Seems easy and straightforward. Run task and run publish are two separate functions right now. However, run task should also save all it's outputs to a folder on the file system (in case of a cached run). A node with internet access can later upload these runs.

Only complication is the flows. There must be some caching directory that contains all concerning flows beforehand (the algorithm, but not the hyperparameter settings). This seems doable in the auto-sklearn world; what about TPOT?

[rhiever]

Only complication is the flows. There must be some caching directory that contains all concerning flows beforehand (the algorithm, but not the hyperparameter settings). This seems doable in the auto-sklearn world; what about TPOT?

I'm not sure I understand. Why would there need to be a directory containing all of the flows beforehand? Aren't the sklearn pipeline --> flows conversions done locally?

[joaquinvanschoren]

Yes, but I guess you want to store the flows/pipelines after you created them when working on a cluster without internet access? They will have to be uploaded before the runs are uploaded.

On Tue, Mar 28, 2017 at 9:59 PM Randy Olson ***@***.***> wrote: Only complication is the flows. There must be some caching directory that contains all concerning flows beforehand (the algorithm, but not the hyperparameter settings). This seems doable in the auto-sklearn world; what about TPOT? I'm not sure I understand. Why would there need to be a directory containing all of the flows beforehand? Aren't the sklearn pipeline --> flows conversions done locally? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#219 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABpQV6wO7XKaJsqO6C6kmrMNer2wznlVks5rqWaggaJpZM4Mq1Kp> .

-- Thank you, Joaquin

[rhiever]

Aren't they uploaded together? Or can they?

[janvanrijn]

Come to think of it, shouldn't be a problem either. In technical terms: in order to match the hyperparameter settings of a run to a flow, the flow actually needs to be already on the server.. however, this can also be carried out at upload time. Short answer: indeed no problem

…

On 28 Mar 2017 11:23 p.m., "Randy Olson" ***@***.***> wrote: Aren't they uploaded together? Or can they? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#219 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACL7-gzu-7UkY84RImgA_m13Eps6mcP_ks5rqXpkgaJpZM4Mq1Kp> .

[joaquinvanschoren]

Ok, so what needs to be done? Just write all the run objects to disk, and then read them and publish them in a separate process?

On Wed, 29 Mar 2017 at 00:17, janvanrijn ***@***.***> wrote: Come to think of it, shouldn't be a problem either. In technical terms: in order to match the hyperparameter settings of a run to a flow, the flow actually needs to be already on the server.. however, this can also be carried out at upload time. Short answer: indeed no problem On 28 Mar 2017 11:23 p.m., "Randy Olson" ***@***.***> wrote: > Aren't they uploaded together? Or can they? > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #219 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/ACL7-gzu-7UkY84RImgA_m13Eps6mcP_ks5rqXpkgaJpZM4Mq1Kp > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#219 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABpQV3ytQrWLtPA6JxcARcZA2iYEyF0kks5rqYbogaJpZM4Mq1Kp> .

-- Thank you, Joaquin

[mfeurer]

The merge of PR #253 should allow for running the task on a downloaded flow object. @joaquinvanschoren could you please check for this?

[rhiever]

Awesome. Has the API changed at all to accommodate caches? If not, I will test this on our HPC soon.

[mfeurer]

Could you please be more specific in what you mean with:

Has the API changed at all to accommodate caches?

Disclaimer: this feature is a by-product of fixing a different issue. I have not tested whether it works for the usecase at hand, but it should.

[rhiever]

Sure! What I mean is: If I want to run a task on a HPC that has nodes with no internet connection, currently I run a script like:

task_id = 123
print('Running task {}...'.format(task_id))
task = tasks.get_task(task_id)
clf = pipeline.make_pipeline(decomposition.PCA(), ensemble.RandomForestClassifier(n_estimators=500))
run = runs.run_task(task, clf)
run.publish()
print('Successfully submitted task {}'.format(task_id))

But of course that script fails because the HPC nodes do not have an internet connection. How does that code change now that OpenML supports caching? I assume there will be two different scripts now: one to run the task and cache it, and one to publish all the cached tasks.

[mfeurer]

I just wanted to write a few short snippets on how this works, but found out that there is no way to store a flow offline. Once this part is there, we should be able to provide you with an example.

[rhiever]

Thank you @mfeurer, it's much appreciated. I will keep an eye on this issue in the meantime.