Deepo is a series of Docker images that
- allows you to quickly set up your deep learning research environment
- supports almost all commonly used deep learning frameworks
- supports GPU acceleration (CUDA and cuDNN included), also works in CPU-only mode
- works on Linux (CPU version/GPU version), Windows (CPU version) and OS X (CPU version)
and their Dockerfile generator that
- allows you to customize your own environment with Lego-like modules
- automatically resolves the dependencies for you
Step 1. Install Docker and nvidia-docker.
Step 2. Obtain the all-in-one image from Docker Hub
docker pull ufoym/deepoFor users in China who may suffer from slow speeds when pulling the image from the public Docker registry, you can pull deepo images from the China registry mirror by specifying the full path, including the registry, in your docker pull command, for example:
docker pull registry.docker-cn.com/ufoym/deepoNow you can try this command:
docker run --runtime=nvidia --rm ufoym/deepo nvidia-smiThis should work and enables Deepo to use the GPU from inside a docker container. If this does not work, search the issues section on the nvidia-docker GitHub -- many solutions are already documented. To get an interactive shell to a container that will not be automatically deleted after you exit do
docker run --runtime=nvidia -it ufoym/deepo bashIf you want to share your data and configurations between the host (your machine or VM) and the container in which you are using Deepo, use the -v option, e.g.
docker run --runtime=nvidia -it -v /host/data:/data -v /host/config:/config ufoym/deepo bashThis will make /host/data from the host visible as /data in the container, and /host/config as /config. Such isolation reduces the chances of your containerized experiments overwriting or using wrong data.
Please note that some frameworks (e.g. PyTorch) use shared memory to share data between processes, so if multiprocessing is used the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host or --shm-size command line options to docker run.
docker run --runtime=nvidia -it --ipc=host ufoym/deepo bashStep 1. Install Docker.
Step 2. Obtain the all-in-one image from Docker Hub
docker pull ufoym/deepo:cpuNow you can try this command:
docker run -it ufoym/deepo:cpu bashIf you want to share your data and configurations between the host (your machine or VM) and the container in which you are using Deepo, use the -v option, e.g.
docker run -it -v /host/data:/data -v /host/config:/config ufoym/deepo:cpu bashThis will make /host/data from the host visible as /data in the container, and /host/config as /config. Such isolation reduces the chances of your containerized experiments overwriting or using wrong data.
Please note that some frameworks (e.g. PyTorch) use shared memory to share data between processes, so if multiprocessing is used the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host or --shm-size command line options to docker run.
docker run -it --ipc=host ufoym/deepo:cpu bashYou are now ready to begin your journey.
$ python
>>>importtensorflow>>>importsonnet>>>importtorch>>>importkeras>>>importmxnet>>>importcntk>>>importchainer>>>importtheano>>>importlasagne>>>importcaffe>>>importcaffe2$ caffe --version
caffe version 1.0.0
$ darknet
usage: darknet <function>
$ th
│ ______ __ | Torch7
│ /_ __/__ ________/ / | Scientific computing for Lua.
│ / / / _ \/ __/ __/ _ \ | Type ? for help
│ /_/ \___/_/ \__/_//_/ | https://github.com/torch
│ | http://torch.ch
│
│th>
Note that docker pull ufoym/deepo mentioned in Quick Start will give you a standard image containing all available deep learning frameworks. You can customize your own environment as well.
If you prefer a specific framework rather than an all-in-one image, just append a tag with the name of the framework. Take tensorflow for example:
docker pull ufoym/deepo:tensorflowdocker pull ufoym/deepo:all-jupyterdocker run --runtime=nvidia -it -p 8888:8888 --ipc=host ufoym/deepo:all-jupyter jupyter notebook --no-browser --ip=0.0.0.0 --allow-root --NotebookApp.token= --notebook-dir='/root'git clone https://github.com/ufoym/deepo.git
cd deepo/generatorFor example, if you like pytorch and lasagne, then
python generate.py Dockerfile pytorch lasagneThis should generate a Dockerfile that contains everything for building pytorch and lasagne. Note that the generator can handle automatic dependency processing and topologically sort the lists. So you don't need to worry about missing dependencies and the list order.
You can also specify the version of Python:
python generate.py Dockerfile pytorch lasagne python==3.6docker build -t my/deepo .This may take several minutes as it compiles a few libraries from scratch.
| . | modern-deep-learning | dl-docker | jupyter-deeplearning | Deepo |
|---|---|---|---|---|
| ubuntu | 16.04 | 14.04 | 14.04 | 18.04 |
| cuda | X | 8.0 | 6.5-8.0 | 8.0-10.0/None |
| cudnn | X | v5 | v2-5 | v7 |
| onnx | X | X | X | O |
| theano | X | O | O | O |
| tensorflow | O | O | O | O |
| sonnet | X | X | X | O |
| pytorch | X | X | X | O |
| keras | O | O | O | O |
| lasagne | X | O | O | O |
| mxnet | X | X | X | O |
| cntk | X | X | X | O |
| chainer | X | X | X | O |
| caffe | O | O | O | O |
| caffe2 | X | X | X | O |
| torch | X | O | O | O |
| darknet | X | X | X | O |
| . | CUDA 10.0 / Python 3.6 | CPU-only / Python 3.6 |
|---|---|---|
| all-in-one | latestallall-py36py36-cu100all-py36-cu100 | all-py36-cpuall-cpupy36-cpucpu |
| all-in-one with jupyter | all-jupyter-py36-cu100all-jupyter-py36all-jupyter | all-py36-jupyter-cpupy36-jupyter-cpu |
| Theano | theano-py36-cu100theano-py36theano | theano-py36-cputheano-cpu |
| TensorFlow | tensorflow-py36-cu100tensorflow-py36tensorflow | tensorflow-py36-cputensorflow-cpu |
| Sonnet | sonnet-py36-cu100sonnet-py36sonnet | sonnet-py36-cpusonnet-cpu |
| PyTorch / Caffe2 | pytorch-py36-cu100pytorch-py36pytorch | pytorch-py36-cpupytorch-cpu |
| Keras | keras-py36-cu100keras-py36keras | keras-py36-cpukeras-cpu |
| Lasagne | lasagne-py36-cu100lasagne-py36lasagne | lasagne-py36-cpulasagne-cpu |
| MXNet | mxnet-py36-cu100mxnet-py36mxnet | mxnet-py36-cpumxnet-cpu |
| CNTK | cntk-py36-cu100cntk-py36cntk | cntk-py36-cpucntk-cpu |
| Chainer | chainer-py36-cu100chainer-py36chainer | chainer-py36-cpuchainer-cpu |
| Caffe | caffe-py36-cu100caffe-py36caffe | caffe-py36-cpucaffe-cpu |
| Torch | torch-cu100torch | torch-cpu |
| Darknet | darknet-cu100darknet | darknet-cpu |
| . | CUDA 9.0 / Python 3.6 | CUDA 9.0 / Python 2.7 | CPU-only / Python 3.6 | CPU-only / Python 2.7 |
|---|---|---|---|---|
| all-in-one | py36-cu90all-py36-cu90 | all-py27-cu90all-py27py27-cu90 | all-py27-cpupy27-cpu | |
| all-in-one with jupyter | all-jupyter-py36-cu90 | all-py27-jupyterpy27-jupyter | all-py27-jupyter-cpupy27-jupyter-cpu | |
| Theano | theano-py36-cu90 | theano-py27-cu90theano-py27 | theano-py27-cpu | |
| TensorFlow | tensorflow-py36-cu90 | tensorflow-py27-cu90tensorflow-py27 | tensorflow-py27-cpu | |
| Sonnet | sonnet-py36-cu90 | sonnet-py27-cu90sonnet-py27 | sonnet-py27-cpu | |
| PyTorch | pytorch-py36-cu90 | pytorch-py27-cu90pytorch-py27 | pytorch-py27-cpu | |
| Keras | keras-py36-cu90 | keras-py27-cu90keras-py27 | keras-py27-cpu | |
| Lasagne | lasagne-py36-cu90 | lasagne-py27-cu90lasagne-py27 | lasagne-py27-cpu | |
| MXNet | mxnet-py36-cu90 | mxnet-py27-cu90mxnet-py27 | mxnet-py27-cpu | |
| CNTK | cntk-py36-cu90 | cntk-py27-cu90cntk-py27 | cntk-py27-cpu | |
| Chainer | chainer-py36-cu90 | chainer-py27-cu90chainer-py27 | chainer-py27-cpu | |
| Caffe | caffe-py36-cu90 | caffe-py27-cu90caffe-py27 | caffe-py27-cpu | |
| Caffe2 | caffe2-py36-cu90caffe2-py36caffe2 | caffe2-py27-cu90caffe2-py27 | caffe2-py36-cpucaffe2-cpu | caffe2-py27-cpu |
| Torch | torch-cu90 | torch-cu90torch | torch-cpu | |
| Darknet | darknet-cu90 | darknet-cu90darknet | darknet-cpu |
@misc{ming2017deepo,
author = {Ming Yang},
title = {Deepo: set up deep learning environment in a single command line.},
year = {2017},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/ufoym/deepo}}
}
We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us.
Deepo is MIT licensed.