Skip to content

feat: add upgrade-prod.sh for release-to-prod deployments#283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sheepdestroyer wants to merge 4 commits into from
Closed

feat: add upgrade-prod.sh for release-to-prod deployments #283

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1233f05
feat: add upgrade-prod.sh for syncing runtime files from releases
Jul 12, 2026
c9892a2
fix: address Gemini Code Assist review on upgrade-prod.sh
Jul 12, 2026
1000c62
fix: move upgrade-prod.sh to scripts/ per repo convention
Jul 12, 2026
a349e97
fix: bash syntax error in env var validation (Gemini review)
Jul 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 26 additions & 9 deletions scripts/upgrade-prod.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -24,8 +24,8 @@ fi

# ── centralized cleanup ──
cleanup() {
rm -rf "${TEMP_DIR:-}"
rm -f "${UPGRADE_PROD_SELF_PATH:-}"
[[ -n "${TEMP_DIR:-}" ]] && rm -rf "$TEMP_DIR"
[[ -n "${UPGRADE_PROD_SELF_PATH:-}" ]] && rm -f "$UPGRADE_PROD_SELF_PATH"
}
trap cleanup EXIT

Expand DownExpand Up@@ -64,7 +64,7 @@ echo "🏷 Target release: $TAG"
TEMP_DIR=$(mktemp -d /tmp/llm-routing-upgrade.XXXXXX)

echo "📥 Cloning $REPO @ $TAG..."
git clone --depth 1 --branch "$TAG" "https://github.com/$REPO.git" "$TEMP_DIR" 2>&1 | tail -1
git clone -q --depth 1 --branch "$TAG" "https://github.com/$REPO.git" "$TEMP_DIR"

# ── verify the tag has the files we need ──
for f in pod.yaml start-stack.sh litellm/ router/ scripts/; do
Expand DownExpand Up@@ -104,20 +104,37 @@ else
echo "Non-interactive shell detected, proceeding with upgrade..."
fi

# ── pre-flight: validate PROD_DIR ──
# ── pre-flight: validate PROD_DIR and required env vars ──
if [ ! -f "$PROD_DIR/.env" ]; then
echo "❌ $PROD_DIR/.env not found. Is PROD_DIR correct?"
exit 1
fi

# ── stop the pod gracefully before touching files ──
POD_NAME="${POD_NAME:-agent-router-pod}"
if podman pod exists "$POD_NAME" 2>/dev/null; then
echo "🛑 Stopping $POD_NAME (SIGTERM, 30s)..."
podman pod stop -t 30 "$POD_NAME" 2>/dev/null || true
# Source .env to validate required vars are present
set -a; source "$PROD_DIR/.env"; set +a

missing_vars=()
for var in OPENROUTER_API_KEY OLLAMA_API_KEY LLAMA_CLASSIFIER_URL PUBLIC_BASE_URL; do
if [[ ! -v "$var" ]] || [[ -z "${!var}" ]]; then
missing_vars+=("$var")
fi
done
Comment on lines +117 to +121

@gemini-code-assistgemini-code-assistBotJul 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

In Bash, combining indirect expansion (${!var}) with parameter expansion modifiers like :- (i.e., ${!var:-}) is a syntax error (bad substitution) and will cause the script to crash immediately when executed.

Additionally, because set -u is active (via set -euo pipefail on line 15), referencing an unset variable indirectly via ${!var} will trigger an unbound variable error and terminate the script.

To safely check if a dynamically named variable is set and non-empty under set -u, you can use the [[ -v var_name ]] built-in to check if it is defined, and only evaluate ${!var} if it is set.

Suggested change
forvarin OPENROUTER_API_KEY OLLAMA_API_KEY LLAMA_CLASSIFIER_URL PUBLIC_BASE_URL;do
if [ -z"${!var:-}" ];then
missing_vars+=("$var")
fi
done
if! [[ -v $var ]] || [ -z"${!var}" ];then
missing_vars+=("$var")
fi


if [ ${#missing_vars[@]} -gt 0 ]; then
echo "❌ Missing required env vars in $PROD_DIR/.env:"
for var in "${missing_vars[@]}"; do
echo " - $var"
done
echo ""
echo "Add them before upgrading. Example:"
echo " echo '${missing_vars[0]}=your_value' >> $PROD_DIR/.env"
exit 1
fi

# ── rsync runtime files ──
# The pod stays running during rsync — it only uses rendered configs from
# DATA_ROOT, not the source files being synced. start-stack.sh --pull
# handles graceful shutdown (including pre-deploy backup) before redeploy.
echo "📋 Syncing runtime files..."
# Sync directories with --delete (clean stale files within each dir)
rsync -a --delete "$TEMP_DIR/litellm/" "$PROD_DIR/litellm/"
Expand Down