devops-sandbox

A self-service platform for spinning up isolated temporary environments, deploying apps, simulating outages, monitoring health, and auto-destroying everything. Think miniature internal Heroku.

Every environment is short-lived by design.

---

Architecture

 ┌─────────────────────────────────────────────────┐
│ Linux VM / Host │
│ │
User / CI │ ┌─────────────────────────────────────────────┐ │
│ │ │ Docker Engine │ │
│ make / curl │ │ │ │
▼ │ │ ┌──────────┐ ┌──────────┐ ┌─────────┐ │ │
┌───────────┐ │ │ │ │ │ │ │ │ │ │
│ Makefile │──────────────▶│ │ │ nginx │ │ API │ │ daemon │ │ │
│ (make up) │ │ │ │ :8080 │ │ :7000 │ │(cleanup)│ │ │
└───────────┘ │ │ │ │ │ │ │ │ │ │
│ │ └────┬─────┘ └────┬─────┘ └────┬────┘ │ │
┌───────────┐ │ │ │ │ │ │ │
│ REST API │──────────────▶│ │ │ sandbox-nginx-net │ │ │
│ /envs │ │ │ │───────────────────── │ │ │
└───────────┘ │ │ │ │ │
│ │ ┌─────────────────────────────────┐ │ │ │
│ │ │ Sandbox Environments │ │ │ │
│ │ │ │ │ │ │
│ │ │ ┌───────────┐ ┌───────────┐ │ │ │ │
│ │ │ │env-abc123 │ │env-def456 │ │◀┘ │ │
│ │ │ │ app:5000 │ │ app:5000 │ │ │ │
│ │ │ │ net: own │ │ net: own │ │ │ │
│ │ │ └───────────┘ └───────────┘ │ │ │
│ │ └─────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌──────────┐ logs/ ──────────────────────▶│ │
│ │ │ monitor │ envs/ (state files) │ │
│ │ │(health) │ nginx/conf.d/ (per-env) │ │
│ │ └──────────┘ │ │
│ └─────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘
Request flow:
Browser → Nginx (:8080) → upstream sandbox-app-<env_id>:5000
Nginx routes by Host header: env-abc123.sandbox.local → env-abc123 container
Data flow:
create_env.sh → Docker network + container + Nginx conf + state file
cleanup_daemon.sh → reads envs/*.json → calls destroy_env.sh when TTL expired
health_monitor.py → polls localhost:<port>/health every 30s → writes health.log

---

Prerequisites

• Docker ≥ 24.x and Docker Compose ≥ 2.20
• Python 3.11+ (on the host, for the health monitor)
• Bash 4+, GNU Make
• A Linux VM (tested on Ubuntu 22.04/24.04)

---

Quick Start

Zero to first running environment in 5 commands:

git clone https://github.com/YOUR_USERNAME/devops-sandbox.git
cd devops-sandbox
cp .env.example .env # review defaults; edit ports if needed
make up # starts Nginx, API, cleanup daemon, monitor
make create # prompts for name + TTL, creates env

After make create you'll see:

╔══════════════════════════════════════════╗
║ Environment Ready! ║
║ ID: env-1716000000-a1b2c3 ║
║ URL: http://localhost:8412 ║
║ TTL: 1800s (expires in 30 min) ║
╚══════════════════════════════════════════╝

---

Full Demo Walkthrough

1 — Create an environment

make create
# name: myapp# ttl: 300 (5 minutes for demo)

Or via the API:

curl -s -X POST http://localhost:7000/envs \
-H 'Content-Type: application/json' \
-d '{"name":"myapp","ttl":300}'| python3 -m json.tool

2 — Confirm it's running

make status
# or
curl -s http://localhost:7000/envs | python3 -m json.tool

Hit the app directly (port shown at creation time):

curl http://localhost:<PORT>/health
# {"status": "ok", "env_id": "env-...", ...}

3 — Check health

make health
# or via API (last 10 results):
curl -s http://localhost:7000/envs/<ENV_ID>/health | python3 -m json.tool

4 — Simulate an outage

Crash the container:

make simulate ENV=env-1716000000-a1b2c3 MODE=crash

Pause it (freeze, not kill):

make simulate ENV=env-1716000000-a1b2c3 MODE=pause

Network isolation:

make simulate ENV=env-1716000000-a1b2c3 MODE=network

Or via the API:

curl -s -X POST http://localhost:7000/envs/<ENV_ID>/outage \
-H 'Content-Type: application/json' \
-d '{"mode":"crash"}'

5 — Observe degradation

Within 90 seconds the health monitor will detect failures. After 3 consecutive failures, status becomes degraded:

make health
# status=degraded
curl -s http://localhost:7000/envs/<ENV_ID>/health

Watch the health log live:

tail -f logs/<ENV_ID>/health.log

6 — Recover

make simulate ENV=env-1716000000-a1b2c3 MODE=recover
# or
curl -s -X POST http://localhost:7000/envs/<ENV_ID>/outage \
-H 'Content-Type: application/json' \
-d '{"mode":"recover"}'

Health monitor detects recovery and resets status to running.

7 — View logs

make logs ENV=env-1716000000-a1b2c3
# tails logs/<ENV_ID>/app.log live# or via API (last 100 lines):
curl -s http://localhost:7000/envs/<ENV_ID>/logs

8 — Manual destroy

make destroy ENV=env-1716000000-a1b2c3

9 — Auto-destroy (TTL)

If you set a short TTL (e.g. 60s), the cleanup daemon destroys it automatically. Watch it happen:

tail -f logs/cleanup.log

---

API Reference

Method	Endpoint	Description
POST	/envs	Create env — body: {name, ttl}
GET	/envs	List all active envs + TTL remaining
GET	/envs/:id	Get single env details
DELETE	/envs/:id	Destroy env
GET	/envs/:id/logs	Last 100 lines of app.log
GET	/envs/:id/health	Last 10 health check results
POST	/envs/:id/outage	Trigger simulation — body: {mode}

---

Make Targets

Target	Description
make up	Start Nginx, daemon, API, monitor
make down	Stop everything, destroy all envs
make create	Interactive: create new env
make destroy ENV=<id>	Destroy specific env
make logs ENV=<id>	Tail env app.log (live)
make health	Show all env health statuses
make simulate ENV=<id> MODE=<mode>	Run outage simulation
make status	List envs via API (JSON)
make clean	Wipe all state, logs, archives

Outage modes

Mode	Effect	Recovery
crash	docker kill — hard stop	MODE=recover
pause	docker pause — freeze process	MODE=recover
network	Disconnect from Docker networks	MODE=recover
recover	Unpause / restart / reconnect as needed	—
stress	CPU spike via stress-ng or Python burner (60s)	Self-resolving

---

Nginx Routing

Nginx is the front door for all environments. Each create_env.sh call writes a config to nginx/conf.d/<ENV_ID>.conf and runs nginx -s reload. On destroy, the file is removed and Nginx is reloaded again.

Routing strategy: Host-header based. Each env gets a virtual server name <ENV_ID>.sandbox.local. For local testing, hit by port directly (each env gets a random host port). For proper hostname routing, add entries to /etc/hosts or use a wildcard DNS entry.

Network: Nginx runs in sandbox-nginx-net. App containers are also joined to this network at creation time, so Nginx can upstream to sandbox-app-<ENV_ID>:5000 by container name.

---

Log Shipping

Approach A (implemented): At container creation, docker logs -f <container> >> logs/<ENV_ID>/app.log & is run and the PID saved to logs/<ENV_ID>/log_shipper.pid. On destroy, this PID is killed before container removal to prevent zombie processes.

Logs are archived to logs/archived/<ENV_ID>/ on destroy and remain queryable.

---

Monitoring (optional — Netdata)

Netdata is an optional add-on that gives you a live dashboard of every container's CPU, memory, network I/O, and disk — with zero configuration. It auto-discovers all sandbox containers via the Docker socket the moment they start.

Start:

make monitoring-up
# Dashboard: http://localhost:19999

Stop:

make monitoring-down

What you get instantly, with no setup:

• Per-container CPU and memory graphs — including each sandbox-app-<id> as it's created
• Host-level system metrics (load, disk, network)
• Built-in alerts for memory pressure and high CPU
• Live log of containers appearing and disappearing as you create/destroy envs

Metrics are retained in a Docker volume (netdata-lib, netdata-cache) so they survive restarts. Configuration is in monitor/netdata/netdata.conf — the defaults are fine for local use.

---

File Structure

devops-sandbox/
├── platform/
│ ├── create_env.sh # Spin up environment
│ ├── destroy_env.sh # Tear down environment
│ ├── cleanup_daemon.sh # TTL auto-expire loop
│ ├── simulate_outage.sh # Chaos injection
│ ├── api.py # Flask REST API
│ └── lib/
│ └── common.sh # Shared functions (state, docker, nginx helpers)
├── apps/
│ └── demo/
│ ├── app.py # Demo HTTP server (/ /health /info)
│ └── Dockerfile
├── nginx/
│ ├── nginx.conf # Main config (includes conf.d/)
│ └── conf.d/ # Auto-generated per-env configs (gitignored)
├── monitor/
│ ├── health_monitor.py # 30s health poller → health.log
│ └── netdata/
│ └── netdata.conf # Netdata config (update_every, retention, plugins)
├── scripts/
│ ├── inspect.sh # Pretty-print env runtime state
│ ├── list_envs.sh # Formatted table of all active envs
│ ├── build_demo_app.sh # Build sandbox-demo-app:latest
│ ├── export_logs.sh # Tarball logs for any env
│ ├── prune_archives.sh # Remove archived logs older than N days
│ └── reset_platform.sh # Nuclear wipe with confirmation
├── tests/
│ ├── test_api.sh # 12 API integration assertions
│ ├── test_lifecycle.sh # Full create→crash→recover→destroy cycle
│ ├── test_outage.sh # All outage modes tested end-to-end
│ └── test_cleanup_daemon.sh # TTL auto-expiry test (~2 min)
├── logs/ # gitignored
│ ├── cleanup.log
│ ├── <env_id>/
│ │ ├── app.log
│ │ └── health.log
│ └── archived/
├── envs/ # gitignored — runtime state JSONs
├── .env.example
├── .gitignore
├── docker-compose.yml # Core platform (nginx, api, daemon, monitor)
├── docker-compose.monitoring.yml # Optional Netdata (make monitoring-up)
├── Dockerfile.api
├── Makefile
├── requirements.txt
├── CONTRIBUTING.md
└── README.md

---

Known Limitations

• Single VM only. No distributed scheduling — everything runs on one host. This is by design.
• Port allocation. Env ports are random (8100–9000). High concurrency could exhaust this range. For >900 envs, widen the range.
• No auth on the API. The API is unauthenticated. Do not expose port 7000 publicly without adding auth middleware.
• Demo app is ephemeral. The bundled Python HTTP server is not production-grade — it's a placeholder to satisfy /health. Swap it for your own image via create_env.sh.
• Nginx config reload is not atomic. Between rm and nginx -s reload, Nginx may briefly serve a 502 for that env. For production, use nginx -t validation before reloading.
• Log shipper depends on docker logs. On high-throughput containers this can lag. For production use Approach B (Loki/Fluentd via Docker socket).
• Cleanup daemon requires bash + python3 in the daemon container. The docker:24-cli image installs these at startup, which adds ~5s cold start.
• No TLS. All traffic is plain HTTP. Add Certbot + nginx SSL termination for production use.