fit something with parallel_backend is unreasonably slow

[GuoHao150]

I set up a Dask cluster and tries to reproduce the distributed Machine Learning tutorial from https://github.com/dask/dask-tutorial/blob/master/08_machine_learning.ipynb when I run the command

%%time
with joblib.parallel_backend("dask", scatter=[X, y]):
    grid_search.fit(X, y)

it's not complete even after an hour and compare to the fitting without Dask as a backend I thought it's unreasonably slow. Since it does not give me an error message I wonder if anybody here could give me some advice. Thanks

[GuoHao150]

Just realize this is a silly question and in case someone encounters the same problem make sure that all your nodes installed the dask-ml

[H4dr1en]

Shouldn't be implemented a warning for such case?

[jakirkham]

Shouldn't be implemented a warning for such case?

Please see PR ( dask/distributed#2627 ).

[donniedarko-zen]

I have exactly the same problem. dask-ml is installed. However, whenever I run the parallel_backend, it is extremely slow. Do you think what is the problem?

here is the code that I want to test it

from dask.distributed import Client
from sklearn.externals.joblib import parallel_backend
from sklearn.cluster import DBSCAN
from sklearn.datasets import make_blobs
import datetime

X, y = make_blobs(n_samples = 1500, n_features = 2, centers = 3, cluster_std = 2.1)
client = Client()
now = datetime.datetime.now()
model = DBSCAN(eps = 0.5, min_samples = 30)
with parallel_backend('dask'):
    model.fit(X)
print(datetime.datetime.now() - now)

[TomAugspurger]

@iremozen-edremit do you have environment variables set up to connect to a distributed cluster? If not, creating a Client will just make a local cluster, which will generally be slower that joblib's other backends.

[donniedarko-zen]

@TomAugspurger, to be honest, I'm not sure what you say. Here is the output of the command "$ set"
(base) irem@irem:~$ set BASH=/bin/bash BASHOPTS=checkwinsize:cmdhist:complete_fullquote:expand_aliases:extglob:extquote:force_fignore:histappend:interactive_comments:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_COMPLETION_VERSINFO=([0]="2" [1]="8") BASH_LINENO=() BASH_SOURCE=() BASH_VERSINFO=([0]="4" [1]="4" [2]="19" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu") BASH_VERSION='4.4.19(1)-release' CLUTTER_IM_MODULE=xim COLORTERM=truecolor COLUMNS=80 CONDA_DEFAULT_ENV=base CONDA_EXE=/home/irem/ENTER/bin/conda CONDA_PREFIX=/home/irem/ENTER CONDA_PROMPT_MODIFIER='(base) ' CONDA_PYTHON_EXE=/home/irem/ENTER/bin/python CONDA_SHLVL=1 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path DESKTOP_SESSION=ubuntu DIRSTACK=() DISPLAY=:0 EUID=1000 GDMSESSION=ubuntu GJS_DEBUG_OUTPUT=stderr GJS_DEBUG_TOPICS='JS ERROR;JS LOG' GNOME_DESKTOP_SESSION_ID=this-is-deprecated GNOME_SHELL_SESSION_MODE=ubuntu GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/94e1831f_87e3_43f0_bbf6_4cf7c971a57e GNOME_TERMINAL_SERVICE=:1.78 GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 GROUPS=() GTK_IM_MODULE=ibus GTK_MODULES=gail:atk-bridge HISTCONTROL=ignoreboth HISTFILE=/home/irem/.bash_history HISTFILESIZE=2000 HISTSIZE=1000 HOME=/home/irem HOSTNAME=irem HOSTTYPE=x86_64 IFS=$' \t\n' IM_CONFIG_PHASE=2 LANG=en_GB.UTF-8 LANGUAGE=en_GB:en LESSCLOSE='/usr/bin/lesspipe %s %s' LESSOPEN='| /usr/bin/lesspipe %s' LINES=24 LOGNAME=irem
How can I check environment varaibles? or would you send me a link to follow up the set up it, pls?

[TomAugspurger]

https://docs.dask.org/en/latest/setup.html

The dask Joblib backend is helpful in distributed mode, not local.

[H4dr1en]

Then it is contradictory to this statement:

[TomAugspurger]

Your comparing dasks schedulers. I’m talking about joblib’s backends.

…

On May 10, 2019, at 07:25, H4dr1en ***@***.***> wrote: Then it is contradictory to this statement: — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[TomAugspurger]

As noted in https://joblib.readthedocs.io/en/latest/auto_examples/parallel/distributed_backend_simple.html#sphx-glr-auto-examples-parallel-distributed-backend-simple-py,

This example shows the simplest usage of the dask distributed backend, on the local computer.

This is useful for prototyping a solution, to later be run on a truly distributed cluster, as the only change to be made is the address of the scheduler.

So you've created a dask.distributed "cluster" locally on your machine. But to make it useful you need to connect to a real cluster (setup described in https://docs.dask.org/en/latest/setup.html)