There seems to be an issue in docker stop that causes it to intermittently deadlock and hang the execution context performing the stop. It seems to be a race condition as it's very sporadic.
I've tracked the issue down to a block on container.Wait() here: https://github.com/dotcloud/docker/blob/master/container.go#L886 The prior step is that docker tries to directly kill the process which doesn't error but also doesn't actually succeed in killing it either.
In this scenario the lxc container is already gone but the process is still running and docker thinks the container still exists. When it's in this state docker stop on the container just hangs because it waits to acquire the lock. Killing the process manually will cause docker to think the container exited and will free the lock. It requires a kill -9 to kill the process but I'm not sure why Process.Kill isn't able to do it directly.
Here's the log of a hung session (the last line is a debug I added right before L886 of container.go):
2013/07/25 21:41:10 POST /v1.3/containers/2b1213eb763b/stop?t=1
2013/07/25 21:41:10 lxc-kill: '2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd' is not running
lxc-kill: failed to get the init pid
2013/07/25 21:41:10 Failed to send SIGTERM to the process, force killing
2013/07/25 21:41:10 error killing container 2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd (lxc-kill: '2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd' is not running
lxc-kill: failed to get the init pid
, exit status 255)
2013/07/25 21:41:20 Container 2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd failed to exit within 10 seconds of lxc SIGKILL - trying direct SIGKILL
2013/07/25 21:41:20 Waiting on Container 2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd to exit
There seems to be an issue in docker stop that causes it to intermittently deadlock and hang the execution context performing the stop. It seems to be a race condition as it's very sporadic.
I've tracked the issue down to a block on container.Wait() here: https://github.com/dotcloud/docker/blob/master/container.go#L886 The prior step is that docker tries to directly kill the process which doesn't error but also doesn't actually succeed in killing it either.
In this scenario the lxc container is already gone but the process is still running and docker thinks the container still exists. When it's in this state docker stop on the container just hangs because it waits to acquire the lock. Killing the process manually will cause docker to think the container exited and will free the lock. It requires a kill -9 to kill the process but I'm not sure why Process.Kill isn't able to do it directly.
Here's the log of a hung session (the last line is a debug I added right before L886 of container.go):