feat: add upgrade-prod.sh for release-to-prod deployments - #284
feat: add upgrade-prod.sh for release-to-prod deployments#284sheepdestroyer wants to merge 5 commits into
Conversation
- Clones a release tag to a temp dir, rsyncs pod.yaml, start-stack.sh, litellm/, router/, scripts/ into ~/prod/LLM-Routing - Self-copy guard: re-execs from /tmp so the script can safely update itself - Per-directory rsync (no data/ contamination from --delete) - Pre-flight env var validation: checks OPENROUTER_API_KEY, OLLAMA_API_KEY, LLAMA_CLASSIFIER_URL, PUBLIC_BASE_URL before stopping the pod - Non-interactive auto-proceed when stdin is not a TTY - .env and data/ are never touched
- Remove manual pod stop before rsync (let start-stack.sh --pull handle graceful shutdown, which includes pre-deploy database backup) - Guard cleanup rm commands with -n checks to prevent noisy errors on early exit when TEMP_DIR/UPGRADE_PROD_SELF_PATH are unset - Use git clone -q instead of piping to tail -1 (preserves full error output from stderr on clone failure) - Delete duplicate scripts/upgrade-prod.sh (root-level is canonical)
Root is for primary orchestration (start-stack.sh); supporting tools live under scripts/ (backup.sh, upgrade-prod.sh, etc.). README already documents it at this path.
- Replace ${!var:-} (syntax error under set -u) with [[ -v var ]] + ${!var}
- Fix leaked loop variable in error example: use ${missing_vars[0]} instead of $var
Both flagged by Gemini Code Assist on PR #283.There was a problem hiding this comment.
Sorry @sheepdestroyer, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Warning Review limit reached
Next review available in:27 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request updates the production upgrade script scripts/upgrade-prod.sh by improving the cleanup logic, silencing the git clone command, and introducing pre-flight validation for required environment variables sourced from .env. It also removes the manual pod shutdown step, delegating graceful shutdown to start-stack.sh --pull. Feedback suggests temporarily disabling set -u when sourcing .env to prevent potential crashes from unbound variables.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| echo "🛑 Stopping $POD_NAME (SIGTERM, 30s)..." | ||
| podman pod stop -t 30 "$POD_NAME" 2>/dev/null || true | ||
| # Source .env to validate required vars are present | ||
| set -a; source "$PROD_DIR/.env"; set +a |
There was a problem hiding this comment.
Since set -u is active in this script (set -euo pipefail on line 15), sourcing a .env file that contains any unbound variables (e.g., FOO=$BAR where BAR is unset, or conditional expansions without defaults) will cause the script to crash immediately. Since .env files are user-configured and may not be strictly compliant with set -u, it is safer to temporarily disable set -u while sourcing .env and re-enable it afterward.
| set-a;source"$PROD_DIR/.env";set +a | |
| set+u | |
| set -a;source"$PROD_DIR/.env";set +a | |
| set -u |
…vars Gemini Code Assist flagged that sourcing .env with set -u active (set -euo pipefail) crashes when .env contains unbound variable refs like FOO=$BAR. Temporarily disable nounset, source, then re-enable. Verified: old behavior crashes on unbound vars, new behavior sources successfully and validates required vars.
sheepdestroyer
commented
Jul 12, 2026
Closing in favor of a fresh PR with Gemini Code Assist review fix applied (see #285). |
Summary
Adds
scripts/upgrade-prod.sh— syncs runtime files from a GitHub release tag into~/prod/LLM-Routingand redeploys.What it does
pod.yaml,start-stack.sh,litellm/,router/,scripts/into prodDATA_ROOTare used at runtime;start-stack.sh --pullhandles graceful shutdown with pre-deploy backup--delete(clean stale files within each dir;.envanddata/never touched).envbefore touching anything.envanddata/are NEVER touchedLocation
Under
scripts/per repo convention — root is for primary orchestration (start-stack.sh),scripts/for supporting tools. README already documents it at this path.Why
Prod deployments previously required manual file syncing. This standardizes the workflow and prevents the single-rsync corruption seen during the v0.1.5 deployment.
Review fixes applied (from PR #283 Gemini Code Assist)
${!var:-}(syntax error underset -u) with[[ -v "$var" ]] || [[ -z "${!var}" ]]— correctly checks both unset and empty vars without crashing${missing_vars[0]}instead of$var(which always held the last iterated value, not the actual missing one)Verification
start-dev.sh --full-rebuild): all 24 canonical endpoints pass (verify_canonical_endpoints.py --dev)bash -n) cleanNotes