Abnormal task retrieval time

[gabartas]

Hello everyone
I am using pyhon_develop on my windows machine, and when I try to use get_task(taskid) for my experiments, some tasks take an awfully long time to be retrieved, independantly of their size (as an example, the task 3 is taking ages, though it has a reasonable size). I've done some tests, and it appears that it is the same tasks that take long to be retrieved each time. I attached a sample of my code for you. I used the print in order to see if the get_task was working, as I saw that the get_tasks would take what seems to ba an abnormally long time.
dataset_collection.txt

[mfeurer]

Hi @gabartas thanks for reporting this. I do not have a windows machine with python available so I can only check this on Ubuntu. I reduced your example to:

import openml as ml
import time

ml.config.set_cache_directory('/tmp/openml')

tasksid = [3889, 3902, 219, 9914, 9952, 3917, 9957, 9971, 3918, 3913, 9970,
 14965, 145976, 145854, 145804, 7295, 145855, 3, 125923, 145972, 37,
 31, 146066, 3493, 145862, 145836, 3903, 145878, 9967, 9946, 3899,
 146085, 145834, 6566, 3954, 3494, 145853, 34536, 146065, 145833,
 3891, 146064, 9889, 146012, 10101, 14951, 3896, 3492, 3485,
 145872, 145857, 9976, 9980, 43, 145979]


for task_id in tasksid:
 start_time = time.time()
 ml.tasks.get_task(task_id)
 print(task_id, '%2f' % (time.time() - start_time))

and got the following results:

3889 5.907928
3902 1.048350
219 4.880913
9914 1.191978
9952 1.211666
3917 1.267980
9957 1.586261
9971 0.756712
3918 0.870034
3913 5.784485
9970 1.046599
14965 5.356120
145976 0.764453
145854 8.813734
145804 0.740089
7295 4.490297
145855 1.212775
3 1.188197
125923 5.786461
145972 5.841808
37 0.294157
31 0.328659
146066 0.722201
3493 0.768212
145862 0.339064
145836 0.749331
3903 0.926486
145878 0.792823
9967 6.030899
9946 0.297444
3899 7.111604
146085 7.806968
145834 1.056016
6566 3.692524
3954 2.590942
3494 0.273833
145853 2.704920
34536 3.669005
146065 0.286731
145833 3.982687
3891 5.168065
146064 0.755054
9889 0.623262
146012 3.754874
10101 0.286485
14951 7.275544
3896 1.347626
3492 0.280251
3485 2.112590
145872 0.476039
145857 0.688469
9976 1.278439
9980 0.845591
43 1.448483
145979 0.748834

so no abnormal retrieval times for me. Could you please post the output of the script here?

[gabartas]

Hello Matthias, I would gladly send you the output but the problem is that for some tasks (as the task id = 3), the script does not budge at all (I left it running for more than 30 minutes, which was long enough, I think, to conclude of a abnormal retrieval time). Here is the (incomplete) output of your script which I left running for all a night : 3889 6.306526 3902 0.189595 219 0.864736 9914 0.118189 9952 0.137386 3917 0.136635 9957 0.126037 9971 0.081447 3918 0.092994 3913 0.086359 9970 0.236280 14965 1.279656 145976 0.092956 145854 5.264925 145804 0.098583 7295 0.841696 145855 0.317240 As you can see, the task number 3 seems to take an infinite time to be loaded, and it is not the only one in this case.

…

________________________________ De : Matthias Feurer <notifications@github.com> Envoyé : jeudi 7 décembre 2017 17:51 À : openml/openml-python Cc : gabartas; Mention Objet : Re: [openml/openml-python] Abnormal task retrieval time (#385) Hi @gabartas<https://github.com/gabartas> thanks for reporting this. I do not have a windows machine with python available so I can only check this on Ubuntu. I reduced your example to: import openml as ml import time ml.config.set_cache_directory('/tmp/openml') tasksid = [3889, 3902, 219, 9914, 9952, 3917, 9957, 9971, 3918, 3913, 9970, 14965, 145976, 145854, 145804, 7295, 145855, 3, 125923, 145972, 37, 31, 146066, 3493, 145862, 145836, 3903, 145878, 9967, 9946, 3899, 146085, 145834, 6566, 3954, 3494, 145853, 34536, 146065, 145833, 3891, 146064, 9889, 146012, 10101, 14951, 3896, 3492, 3485, 145872, 145857, 9976, 9980, 43, 145979] for task_id in tasksid: start_time = time.time() ml.tasks.get_task(task_id) print(task_id, '%2f' % (time.time() - start_time)) and got the following results: 3889 5.907928 3902 1.048350 219 4.880913 9914 1.191978 9952 1.211666 3917 1.267980 9957 1.586261 9971 0.756712 3918 0.870034 3913 5.784485 9970 1.046599 14965 5.356120 145976 0.764453 145854 8.813734 145804 0.740089 7295 4.490297 145855 1.212775 3 1.188197 125923 5.786461 145972 5.841808 37 0.294157 31 0.328659 146066 0.722201 3493 0.768212 145862 0.339064 145836 0.749331 3903 0.926486 145878 0.792823 9967 6.030899 9946 0.297444 3899 7.111604 146085 7.806968 145834 1.056016 6566 3.692524 3954 2.590942 3494 0.273833 145853 2.704920 34536 3.669005 146065 0.286731 145833 3.982687 3891 5.168065 146064 0.755054 9889 0.623262 146012 3.754874 10101 0.286485 14951 7.275544 3896 1.347626 3492 0.280251 3485 2.112590 145872 0.476039 145857 0.688469 9976 1.278439 9980 0.845591 43 1.448483 145979 0.748834 so no abnormal retrieval times for me. Could you please post the output of the script here? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#385 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AJu-wr1jBH9Rtfz5Bo1gYO1vCedXrlyfks5s-BeJgaJpZM4Q2cHd>.

[mfeurer]

Could you please check if there is a directory 3 in your task cache directory and remove it if it's there? Also, please check if you have a directory 3 in your dataset cache directory and remove it. Then, try again. I'm not really sure where the cache directory ends up on Windows, but I assume it is in your user directory in a folder called .openml.

[gabartas]

I did it for both my general tmp file and the one created by your script and nothing changed.

…

________________________________ De : Matthias Feurer <notifications@github.com> Envoyé : lundi 11 décembre 2017 13:31 À : openml/openml-python Cc : gabartas; Mention Objet : Re: [openml/openml-python] Abnormal task retrieval time (#385) Could you please check if there is a directory 3 in your task cache directory and remove it if it's there? Also, please check if you have a directory 3 in your dataset cache directory and remove it. Then, try again. I'm not really sure where the cache directory ends up on Windows, but I assume it is in your user directory in a folder called .openml. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#385 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AJu-wl9H9QHlXFnYD8SDeVeVvSuXBGDfks5s_SCEgaJpZM4Q2cHd>.

[mfeurer]

Is it possible for you to profile the code?

[gabartas]

Hello, sorry I never saw your answer, here is my complete code (I've edited out my keys), I'm not sure if it's what you mean by "profile the code". The code keeps going wrong when getting the task for the task 3 (the 'task acquired' message never pops up) :

import openml as ml
from sklearn import preprocessing, ensemble
from pprint import pprint
import numpy as np
import pandas as pd

#a functions which find and get the best run on a task
from annexe import get_best_run

#the tasks I intend to collect
tasksid = [3889, 3902, 219, 9914, 9952, 3917, 9957, 9971, 3918, 3913, 9970,
14965, 145976, 145854, 145804, 7295, 145855, 3, 125923, 145972, 37,
31, 146066, 3493, 145862, 145836, 3903, 145878, 9967, 9946, 3899,
146085, 145834, 6566, 3954, 3494, 145853, 34536, 146065, 145833,
3891, 146064, 9889, 146012, 10101, 14951, 3896, 3492, 3485,
145872, 145857, 9976, 9980, 43, 145979]

#the flows concerning the experiment
flowsid = [5636, 5527, 6794, 6766, 6767]

datasets = []
labels = []

for t in tasksid:
print(t)
task = ml.tasks.get_task(t)
print("task acquired")
datasets.append(task.get_dataset().get_data())
print("dataset acquired")
bestscore = 0
bestid = 0
score = 0
tempid = 0
for f in flowsid:
myrun = ml.runs.list_runs(flow = [f], task = [t], size = 1)
tempid = list(myrun)[0]
score = ml.runs.get_run(tempid).evaluations['area_under_roc_curve']
if score>bestscore:
bestid = f
bestscore = score
labels.append(bestid)

np.save("datasets.npy",datasets)
np.save("labels.npy",labels)

[mfeurer]

No, I meat that it would be great if you could run your code using cProfile to see what part exactly is slow. You can also check this answer on stackoverflow.

[PGijsbers]

Task 3 was one of the tasks affected by a bug which prevented users from downloading said task through the Python API (all tasks which shared task and dataset id had this). It has been fixed in #429.