Volumes lost if container fails to start

[buckett]

If you have a container using volumes persisted between restarts (eg a MySQL container and /var/lib/mysql) if the container fails to restart the volumes get wiped out when the container does start up.

To reproduce this, create a fig.yml containing:

app:
 image: ubuntu
 ports:
 - "6543:6543"
 volumes:
 - /opt
 command: ["/bin/bash", "-c", "/bin/cat </opt/last ; /bin/date > /opt/last"]

When this container runs it attempts to read the date when container was last run and and the write todays date back to the file. I have a mapped port as it's an easy way to stop the container from starting. The first time this is run you see:

$ fig up
Creating bug_app_1...
Attaching to bug_app_1
app_1 | /bin/bash: /opt/last: No such file or directory
bug_app_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

which is expected, if you then run it again you see the date when it was last run (as the /opt volume is persisted:

$ fig up
Recreating bug_app_1...
Attaching to bug_app_1
app_1 | Sat Nov 8 07:20:03 UTC 2014
bug_app_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

If you now start another container with also wants port 6543 and leave it running in another terminal:

docker run -p 6543:6543 --rm -it ubuntu

and attempt to startup the fig managed container again it fails:

$ fig up
Recreating bug_app_1...
Cannot start container fabfd220db3d728213ad6bc410f737ccccbf8414284aad5e27ebdc8cc97f91fe: Bind for 0.0.0.0:6543 failed: port is already allocated

If you now stop the docker container (to release the port) and restart the fig managed container the /opt volume is empty:

$ fig up
Recreating bug_app_1...
Attaching to bug_app_1
app_1 | /bin/bash: /opt/last: No such file or directory
bug_app_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

I would have expected it to still have the /opt volume from the previous runs.

[aanand]

Yep, this is a side-effect of our somewhat-flaky volumes_from/intermediate_container recreate strategy.

I proposed a solution for doing away with VolumesFrom and intermediate containers when recreating on the thread for #517, but it got lost in the discussion as it was a bit of a tangent. Might be worth exploring properly.

[dnephin]

@aanand I thought I saw that somewhere. There are a bunch of related issues, I restated a similar solution in #447 (comment)

[aanand]

Thinking it through, that approach should fix this bug, but there's still a risk of losing the reference to the volume if the new container fails at the create step, not the start step.

1. Inspect the old container's volumes config and hold onto it in memory.
2. Destroy the old container.
3. Create a new container with the same name and copied volumes config.
4. Start the new container.

If we fail at 4, we've still got a container sitting around with the volumes config preserved. But if we fail at 3, the volumes config only exists in memory, so we'll lose it if we exit.

[anentropic]

this seems to apply to links as well as volumes

I have following fig.yml

postgres:
 image: kartoza/postgis
 volumes_from:
 - postgresdata
 expose:
 - 5432

postgresdata:
 image: dockerfile/ubuntu

myapp:
 image: myapp
 links:
 - postgres
 expose:
 - 3031
 command: --socket :3031

if I change the config of myapp and try to restart it, it ends up terminating my whole fig cluster, which is not very desirable

$ fig restart myapp
Restarting myproject_myapp_1...
Cannot restart container 2a8f2b63a268349e47434198d34099aa2ae1399052f80cc83c750bcbd8437c9a:
Container 44b5d8e6c9512ba495f223267ae6c577c189aa33e86b5659d5f7eb500c3d5342 not found.
Impossible to mount its volumes

in my other terminal I see:

myproject_myapp_1 exited with code -1
Gracefully stopping... (press Ctrl+C again to force)
Stopping myproject_nginx_1...
Stopping myproject_postgres_1...

I'm not really sure what happened here. There are no volumes mounted on the myproject_myapp_1 either by its Dockerfile or by my fig.yml

I can only assume it has followed links to the myproject_postgres_1 container, which uses volumes_from a third container

more odd, if I do docker ps --all I can see that 2a8f2b63a268 is the myproject_myapp_1 container... but I don't see 44b5d8e6c951 at all

any idea what fig is looking for here?

[anentropic]

hmm, fig just calls out to docker, this is actually a problem with docker, I get same symptoms if I do docker restart de7431a5e02b for my app container

[SamMorrowDrums]

I had a datavolume for postgress and mongo, and after this failure:

Starting BrainsApp_DATAVOLUME_1...
Cannot start container 5c721dbb615b5f52afc6f99bbbd5ef2d9c019269f3ce9d385f8df2c2417eeee1: fork/exec /var/lib/docker/init/dockerinit-1.0.1: cannot allocate memory

When I restarted the machine fig start wouldn't work, so I ran fig build and lost everything. Tried re-attaching old containers, but nothing seemed to work.

It strikes me that whatever failure handling is needed, this situation is incredibly high risk, and made me question using Fig with Docker at all for persistent data.

Failure recovery, where data has not actually been corrupted, should not lose, or allow you to destroy your data without warning or prompt. That's too dangerous for production apps to accept as a risk. I know backup is necessary, but this is different, this should not be an emergency run the backup recovery script situation.

Let me know if anyone knows how to handle this please.

[aanand]

@SamMorrowDrums What do you mean you "lost everything"? fig build doesn't touch containers, it just builds images.

[SamMorrowDrums]

Sorry, I ran fig up -d after fig build.

After that I was unable to find the orphaned container. I couldn't find an existing data volume that contained the database files.

[aanand]

We definitely need a container recreate process that's resilient to machine failure at every step. The weakness right now is that we don't keep track of the intermediate container except in memory, so if we fail between deleting the old container and creating the new one, we'll lose data.

We'll need to keep this in mind alongside plans to switch to using renaming (#874) or labels (moby/moby#9882).

[Starefossen]

Yes, please. I too have had volumes disappear on me when Docker fails to build.