Right now, if the Limbo process crashes, ECS is configured to restart it. If the crash is deterministic -- for example, if there are bad credentials or bad code was pushed -- ECS will restart it indefinitely, which causes a ton of log streams to be created in CloudWatch logs (among other problems). I've poked around the ECS documentation and don't see any configuration setting that tells it to stop restarting after too many tries. We could probably write a Lambda function that reads the event stream coming off of ECS and looks for this (although would need state for doing this).
Right now, if the Limbo process crashes, ECS is configured to restart it. If the crash is deterministic -- for example, if there are bad credentials or bad code was pushed -- ECS will restart it indefinitely, which causes a ton of log streams to be created in CloudWatch logs (among other problems). I've poked around the ECS documentation and don't see any configuration setting that tells it to stop restarting after too many tries. We could probably write a Lambda function that reads the event stream coming off of ECS and looks for this (although would need state for doing this).