Standardize use of n_jobs and reporting of computation time by Neeratyoy · Pull Request #1038 · openml/openml-python

Neeratyoy · 2021-03-12T17:46:28Z

Reference Issue

Addresses #895.

mfeurer

Hey, I renamed the file so I could render it on my local machine. Also, I left yet another few comments :)

mfeurer · 2021-04-06T08:34:28Z

Hey, I think the example is really getting along well. Based on the current status I'm wondering three things:

Should we have a summary section which summarizes the behavior of OpenML-Python, scikit-learn and joblib?
Should we have a section which shows that the measured numbers will differ based on the backend?
We currently don't have case 2 from Why is computation time not reported if n_jobs != 1 or != None? #895 (comment)

Neeratyoy · 2021-04-06T15:01:19Z

Should we have a summary section which summarizes the behavior of OpenML-Python, scikit-learn and joblib?

Do you mean their interaction and behaviour? It would be useful I guess, but the only concern or question is what (exactly) to summarize I guess.

Should we have a section which shows that the measured numbers will differ based on the backend?

Though we could, from the OpenML API standpoint, we can also ignore it. If the duty of parallelization is delegated to scikit-learn or OpenML (as we show in the example), I feel it is likely that the user may not set the backend with a parallel_backend context.
Having said that, it also might be fair to show a 4th case in the example, giving an example of changing the backend.

On second thought, this might necessitate the mention of how the backends change in the entire stack of function calls going through OpenML to scikit-learn pipelines. This is obviously quite complicated.

We currently don't have case 2 from #895 (comment)

In the latest SGDClassifier documentation, the parallelization happens through joblib which I thought we are already covering.
While the HistGradientBoosting, still appears to be experimental, so didn't quite think about including it..

mfeurer · 2021-04-06T18:10:08Z

Do you mean their interaction and behaviour? It would be useful I guess, but the only concern or question is what (exactly) to summarize I guess.

I think the main caveats to pay attention to, like the gist of #895

Though we could, from the OpenML API standpoint, we can also ignore it.

Probably I mixed up something. But an example on how this can be changed might be a good idea!

In the latest SGDClassifier documentation, the parallelization happens through joblib which I thought we are already covering.
While the HistGradientBoosting, still appears to be experimental, so didn't quite think about including it..

Yes, but that's for fitting multiple classifiers for a "one vs all" scheme. I was more thinking about something like the neural network where the number of used cores can't be set via the API.

mfeurer · 2021-04-08T07:47:01Z

+################################################################################
+# Summmary
+# *********
+# OpenML records model runtimes for the CPU-clock and the wall-clock times. The above


Hm, should we explicitly say "the scikit-learn extension"? Everything we're writing here is exclusive about the scikit-learn extension, so it could be confusing otherwise.

mfeurer · 2021-04-09T08:45:36Z

I still can't annotate all my comments... anyway, linear SVM also releases the GIL: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/svm/_liblinear.pyx#L61

Maybe naive bayes doesn't release the GIL?

* Minor reshuffling * Update examples/30_extended/fetch_runtimes_tutorial.py Co-authored-by: Neeratyoy Mallik <neeratyoy@gmail.com> Co-authored-by: Neeratyoy Mallik <neeratyoy@gmail.com>

…ion time (#1038)

@mfeurer

) * Unit test to test existence of refit time * Measuring runtime always * Removing redundant check in unit test * Updating docs with runtimes * Adding more utilities to new example * Removing refit_time + fetching trace runtime in example * rename example * Reiterating with changes to example from @mfeurer suggestions * Including refit time and other minor formatting * Adding more cases + a concluding summary * Cosmetic changes * Adding 5th case with no release of GIL * Removing debug code * Runtime measurement example updates (openml#1052) * Minor reshuffling * Update examples/30_extended/fetch_runtimes_tutorial.py Co-authored-by: Neeratyoy Mallik <neeratyoy@gmail.com> Co-authored-by: Neeratyoy Mallik <neeratyoy@gmail.com> Co-authored-by: Matthias Feurer <feurerm@informatik.uni-freiburg.de>

Neeratyoy added 4 commits March 12, 2021 18:43

Unit test to test existence of refit time
82a5ff6

Measuring runtime always
0566220

Removing redundant check in unit test
e527bea

Updating docs with runtimes
c2d2757

Neeratyoy requested a review from mfeurer March 24, 2021 13:55

mfeurer reviewed Mar 25, 2021
View reviewed changes

Adding more utilities to new example
7559c74

Neeratyoy requested a review from mfeurer March 29, 2021 22:39

Removing refit_time + fetching trace runtime in example
be73637

Neeratyoy marked this pull request as ready for review March 30, 2021 14:10

Neeratyoy force-pushed the fix_895 branch from 4bbf738 to be73637Compare March 30, 2021 14:12

Merge branch 'develop' into fix_895
b7e0ae5

mfeurer reviewed Mar 31, 2021
View reviewed changes

mfeurerand others added 2 commits March 31, 2021 22:32

rename example
dcaf34b

Reiterating with changes to example from@mfeurersuggestions
ba19ab7