Cannot stop container; status gets out of sync with LXC

[stefan-pdx]

I came across an interesting behavior where a Docker container's status got out of sync with an lxc container.

1. docker ps showed my container (ae23c705afdb) with a status of running. However, whenever I try running docker kill or docker stop, the command hangs indefinitely. docker rm says that it cannot remove a container that is running.
2. docker inspect ae23c705afdb confirmed the running state and showed a PID of 11780.
3. ps aux | grep 11780 showed a process that was running (lxc-start -n ae23c705afdbbcfcd723c7bb17fbdbc7c8632da41e5e8c38bbf714a701b5b536 -f /var/lib/docker/containers/...), but it's state is shown as D, or "uninterruptible sleep". This process thus does not respond to any interrupts.
4. lxc-list does not show the corresponding lxc container running, so lxc-kill xxx does not work.

I ended up just having to do a reboot. Any thoughts on this?

[nekto0n]

I think it would be useful to attach to lcx-start process and take a look where it blocks. gdb -p <PID> and then issue bt command to show stacktrace. Output of strace -fp <PID can be helpful too.
BTW, google took me to launchpad bug which looks similar.

[pwaller]

Sounds like it might be related to #1300?

[mahendra]

@nekto0n, @pwaller I faced the same problem. However, I am seeing it with a simple sleep command.

• lxc-ls and lxc-kill dont work on the docker thingy even when it is running.
• lxc-info shows that the container is STOPPED whereas docker ps shows it as UP.

For eg: I run

$ docker run -i -t ubuntu sleep 600

On another terminal

$ docker ps
ID IMAGE COMMAND CREATED STATUS PORTS
1c49f1d5ccd4 ubuntu:12.04 sleep 600 33 seconds ago Up 32 seconds 
$
$ ps -eaf | grep lxc-start
root 6352 1698 0 16:23 pts/16 00:00:00 lxc-start -n 1c49f1d5ccd41a7436596f6fdbc53158986f51522caefc166bc88cb248997e30 -f /var/lib/docker/containers/1c49f1d5ccd41a7436596f6fdbc53158986f51522caefc166bc88cb248997e30/config.lxc -- /.dockerinit -g 172.17.42.1 -e TERM=xterm -e HOME=/ -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -e container=lxc -e HOSTNAME=1c49f1d5ccd4 -- sleep 600
$
$ sudo lxc-info -n 1c49f1d5ccd41a7436596f6fdbc53158986f51522caefc166bc88cb248997e30
state: STOPPED
mahendra@kautilya:~/affirm/salt/srv/salt$ sudo docker ps
ID IMAGE COMMAND CREATED STATUS PORTS
1c49f1d5ccd4 ubuntu:12.04 sleep 600 About a minute ago Up About a minute 
mahendra@kautilya:~/affirm/salt/srv/salt$ 
$
$ sudo lxc-ls
$ sudo lxc-ls
$ sudo lxc-kill -n 1c49f1d5ccd41a7436596f6fdbc53158986f51522caefc166bc88cb248997e30 15
lxc-kill: failed to get the init pid
$ sudo docker ps
ID IMAGE COMMAND CREATED STATUS PORTS
1c49f1d5ccd4 ubuntu:12.04 sleep 600 2 minutes ago Up 2 minutes 

My system information

$ docker version
Client version: 0.6.4
Go version (client): go1.1.2
Git commit (client): 2f74b1c
Server version: 0.6.4
Git commit (server): 2f74b1c
Go version (server): go1.1.2
Last stable version: 0.6.4
$ uname -a
Linux kautilya 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
$ 

[SamSaffron]

I just got a similar one, docker stop would not stop it, it simply hung.

when I did inspect it had a pid loggged, when I looked up the pid it was not there.

eventually I restarted the docker service and stop started working, then I was able to rm the image but got:

Unable to remove filesystem for ac772babe9ba8ed8dc1369fb59ea07ac0e82c48002c3feb31635aaff4a414679: remove /var/lib/docker/containers/ac772babe9ba8ed8dc1369fb59ea07ac0e82c48002c3feb31635aaff4a414679/rw: device or resource busy

[shykes]

Tentatively scheduling for 0.8.

I think this will be affected by the execution drivers work, so @crosbymichael and @creack I'll assign it to one of you.

[gesellix]

We had similar problems with non stoppable containers/hanging processes and subsequently locked files on the following configuration (RedHat EL 6.4) and a Docker container with a CMD issuing the Tomcat start command catalina.sh run:

Linux solv213 3.8.13-13.el6uek.x86_64 #1 SMP Wed Aug 21 14:28:36 PDT 2013 x86_64 x86_64 x86_64 GNU/Linux

Docker version:

Client version: 0.8.0
Go version (client): go1.2
Git commit (client): cc3a8c8/0.8.0
Server version: 0.8.0
Git commit (server): cc3a8c8/0.8.0
Go version (server): go1.2

Changing the CMD to /bin/bash -c "startup.sh; while [ true ]; do sleep 1; done;" made the stopping/redeployments work.

Is there any advice on what the CMD instructions should start?

[kuno]

Similar issue on a ubntu 13.04 box, with we have few container running

Linux box 3.8.0-19-generic #30-Ubuntu SMP Wed May 1 16:35:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Which left about 300 MB free memory. The docker is latest one, v0.8.1

Client version: 0.8.1
Go version (client): go1.2
Git commit (client): a1598d1
Server version: 0.8.1
Git commit (server): a1598d1
Go version (server): go1.2
Last stable version: 0.8.1

[gesellix]

Some problems described at http://phusion.github.io/baseimage-docker/ might be relevant. Whether the proposed solution of their baseimage is a good one should everyone decide for themselves :)

In order to minimize suprises or to avoid too many issues I would propose to add some details to the Docker documentation.