fix(e2e): add retry-with-backoff for DigitalOcean 422 droplet limit errors

[la14-1]

Why: E2E suite fails permanently on DigitalOcean 422 "droplet limit exceeded" errors instead of retrying — makes the suite fragile to transient capacity issues during parallel batch provisioning.

Changes

• Add retry-with-backoff (30s delay, up to 3 retries) for 422 errors in provision_agent() (sh/e2e/lib/provision.sh)
• Detects droplet limit / quota errors in CLI stderr output and retries the full provision cycle
• Non-retryable errors (auth failures, config issues) fail immediately without retry
• macOS bash 3.x compatible (uses $((...)) not ((var++)))

Fixes#2516

-- refactor/code-health

[louisgv]

Security Review

Verdict: APPROVED
Commit: c8b3c93

Findings

No security issues found. The retry-with-backoff implementation is safe and follows project conventions:

• ✅ Loop bounds properly validated (max 3 retries)
• ✅ Fixed sleep duration (30s) - no user-controlled timing
• ✅ Grep pattern is fixed string literal - no regex injection
• ✅ File existence checks before grep operations
• ✅ Integer arithmetic uses bash 3.2-compatible syntax
• ✅ Error messages use script-controlled counters only
• ✅ All variables properly quoted and locally scoped with _ prefix
• ✅ State tracking via integer flag prevents logic errors
• ✅ Clean failure paths with proper error reporting

Tests

• bash -n: PASS (no syntax errors)
• bun test: N/A (shell script only)
• curl|bash: OK (no network calls in changed code)
• macOS compat: OK (no bash 3.x incompatibilities introduced)

Implementation Quality

The retry logic is conservative and defensive:

• Only retries on specific droplet limit / quota error patterns
• Validates stderr file exists before pattern matching
• Limited retry attempts prevent infinite loops
• Fixed backoff interval prevents DoS behavior
• Preserves all existing security controls (app_name validation, env var whitelisting, base64 validation)

---

-- security/pr-reviewer

[la14-1]

Why: E2E suite fails permanently on DigitalOcean 422 "droplet limit exceeded" errors instead of retrying — makes the suite fragile to transient capacity issues during parallel batch provisioning.

Changes

• Add retry-with-backoff (30s delay, up to 3 retries) for 422 errors in provision_agent() (sh/e2e/lib/provision.sh)
• Detects droplet limit / quota errors in CLI stderr output and retries the full provision cycle
• Non-retryable errors (auth failures, config issues) fail immediately without retry
• macOS bash 3.x compatible (uses $((...)) not ((var++)))

Fixes#2516

-- refactor/code-health

[louisgv]

Security Review

Verdict: APPROVED
Commit: c8b3c93

Findings

No security issues found. The retry-with-backoff implementation is safe and follows project conventions:

• ✅ Loop bounds properly validated (max 3 retries)
• ✅ Fixed sleep duration (30s) - no user-controlled timing
• ✅ Grep pattern is fixed string literal - no regex injection
• ✅ File existence checks before grep operations
• ✅ Integer arithmetic uses bash 3.2-compatible syntax
• ✅ Error messages use script-controlled counters only
• ✅ All variables properly quoted and locally scoped with _ prefix
• ✅ State tracking via integer flag prevents logic errors
• ✅ Clean failure paths with proper error reporting

Tests

• bash -n: PASS (no syntax errors)
• bun test: N/A (shell script only)
• curl|bash: OK (no network calls in changed code)
• macOS compat: OK (no bash 3.x incompatibilities introduced)

Implementation Quality

The retry logic is conservative and defensive:

• Only retries on specific droplet limit / quota error patterns
• Validates stderr file exists before pattern matching
• Limited retry attempts prevent infinite loops
• Fixed backoff interval prevents DoS behavior
• Preserves all existing security controls (app_name validation, env var whitelisting, base64 validation)

---

-- security/pr-reviewer

[la14-1]

Why: E2E suite fails permanently on DigitalOcean 422 "droplet limit exceeded" errors instead of retrying — makes the suite fragile to transient capacity issues during parallel batch provisioning.

Changes

• Add retry-with-backoff (30s delay, up to 3 retries) for 422 errors in provision_agent() (sh/e2e/lib/provision.sh)
• Detects droplet limit / quota errors in CLI stderr output and retries the full provision cycle
• Non-retryable errors (auth failures, config issues) fail immediately without retry
• macOS bash 3.x compatible (uses $((...)) not ((var++)))

Fixes#2516

-- refactor/code-health

[louisgv]

Security Review

Verdict: APPROVED
Commit: c8b3c93

Findings

No security issues found. The retry-with-backoff implementation is safe and follows project conventions:

• ✅ Loop bounds properly validated (max 3 retries)
• ✅ Fixed sleep duration (30s) - no user-controlled timing
• ✅ Grep pattern is fixed string literal - no regex injection
• ✅ File existence checks before grep operations
• ✅ Integer arithmetic uses bash 3.2-compatible syntax
• ✅ Error messages use script-controlled counters only
• ✅ All variables properly quoted and locally scoped with _ prefix
• ✅ State tracking via integer flag prevents logic errors
• ✅ Clean failure paths with proper error reporting

Tests

• bash -n: PASS (no syntax errors)
• bun test: N/A (shell script only)
• curl|bash: OK (no network calls in changed code)
• macOS compat: OK (no bash 3.x incompatibilities introduced)

Implementation Quality

The retry logic is conservative and defensive:

• Only retries on specific droplet limit / quota error patterns
• Validates stderr file exists before pattern matching
• Limited retry attempts prevent infinite loops
• Fixed backoff interval prevents DoS behavior
• Preserves all existing security controls (app_name validation, env var whitelisting, base64 validation)

---

-- security/pr-reviewer

[la14-1]

Why: E2E suite fails permanently on DigitalOcean 422 "droplet limit exceeded" errors instead of retrying — makes the suite fragile to transient capacity issues during parallel batch provisioning.

Changes

• Add retry-with-backoff (30s delay, up to 3 retries) for 422 errors in provision_agent() (sh/e2e/lib/provision.sh)
• Detects droplet limit / quota errors in CLI stderr output and retries the full provision cycle
• Non-retryable errors (auth failures, config issues) fail immediately without retry
• macOS bash 3.x compatible (uses $((...)) not ((var++)))

Fixes#2516

-- refactor/code-health

[louisgv]

Security Review

Verdict: APPROVED
Commit: c8b3c93

Findings

No security issues found. The retry-with-backoff implementation is safe and follows project conventions:

• ✅ Loop bounds properly validated (max 3 retries)
• ✅ Fixed sleep duration (30s) - no user-controlled timing
• ✅ Grep pattern is fixed string literal - no regex injection
• ✅ File existence checks before grep operations
• ✅ Integer arithmetic uses bash 3.2-compatible syntax
• ✅ Error messages use script-controlled counters only
• ✅ All variables properly quoted and locally scoped with _ prefix
• ✅ State tracking via integer flag prevents logic errors
• ✅ Clean failure paths with proper error reporting

Tests

• bash -n: PASS (no syntax errors)
• bun test: N/A (shell script only)
• curl|bash: OK (no network calls in changed code)
• macOS compat: OK (no bash 3.x incompatibilities introduced)

Implementation Quality

The retry logic is conservative and defensive:

• Only retries on specific droplet limit / quota error patterns
• Validates stderr file exists before pattern matching
• Limited retry attempts prevent infinite loops
• Fixed backoff interval prevents DoS behavior
• Preserves all existing security controls (app_name validation, env var whitelisting, base64 validation)

---

-- security/pr-reviewer

[la14-1]

Why: E2E suite fails permanently on DigitalOcean 422 "droplet limit exceeded" errors instead of retrying — makes the suite fragile to transient capacity issues during parallel batch provisioning.

Changes

• Add retry-with-backoff (30s delay, up to 3 retries) for 422 errors in provision_agent() (sh/e2e/lib/provision.sh)
• Detects droplet limit / quota errors in CLI stderr output and retries the full provision cycle
• Non-retryable errors (auth failures, config issues) fail immediately without retry
• macOS bash 3.x compatible (uses $((...)) not ((var++)))

Fixes#2516

-- refactor/code-health

[louisgv]

Security Review

Verdict: APPROVED
Commit: c8b3c93

Findings

No security issues found. The retry-with-backoff implementation is safe and follows project conventions:

• ✅ Loop bounds properly validated (max 3 retries)
• ✅ Fixed sleep duration (30s) - no user-controlled timing
• ✅ Grep pattern is fixed string literal - no regex injection
• ✅ File existence checks before grep operations
• ✅ Integer arithmetic uses bash 3.2-compatible syntax
• ✅ Error messages use script-controlled counters only
• ✅ All variables properly quoted and locally scoped with _ prefix
• ✅ State tracking via integer flag prevents logic errors
• ✅ Clean failure paths with proper error reporting

Tests

• bash -n: PASS (no syntax errors)
• bun test: N/A (shell script only)
• curl|bash: OK (no network calls in changed code)
• macOS compat: OK (no bash 3.x incompatibilities introduced)

Implementation Quality

The retry logic is conservative and defensive:

• Only retries on specific droplet limit / quota error patterns
• Validates stderr file exists before pattern matching
• Limited retry attempts prevent infinite loops
• Fixed backoff interval prevents DoS behavior
• Preserves all existing security controls (app_name validation, env var whitelisting, base64 validation)

---

-- security/pr-reviewer

[la14-1]

Why: E2E suite fails permanently on DigitalOcean 422 "droplet limit exceeded" errors instead of retrying — makes the suite fragile to transient capacity issues during parallel batch provisioning.

Changes

• Add retry-with-backoff (30s delay, up to 3 retries) for 422 errors in provision_agent() (sh/e2e/lib/provision.sh)
• Detects droplet limit / quota errors in CLI stderr output and retries the full provision cycle
• Non-retryable errors (auth failures, config issues) fail immediately without retry
• macOS bash 3.x compatible (uses $((...)) not ((var++)))

Fixes#2516

-- refactor/code-health

[louisgv]

Security Review

Verdict: APPROVED
Commit: c8b3c93

Findings

No security issues found. The retry-with-backoff implementation is safe and follows project conventions:

• ✅ Loop bounds properly validated (max 3 retries)
• ✅ Fixed sleep duration (30s) - no user-controlled timing
• ✅ Grep pattern is fixed string literal - no regex injection
• ✅ File existence checks before grep operations
• ✅ Integer arithmetic uses bash 3.2-compatible syntax
• ✅ Error messages use script-controlled counters only
• ✅ All variables properly quoted and locally scoped with _ prefix
• ✅ State tracking via integer flag prevents logic errors
• ✅ Clean failure paths with proper error reporting

Tests

• bash -n: PASS (no syntax errors)
• bun test: N/A (shell script only)
• curl|bash: OK (no network calls in changed code)
• macOS compat: OK (no bash 3.x incompatibilities introduced)

Implementation Quality

The retry logic is conservative and defensive:

• Only retries on specific droplet limit / quota error patterns
• Validates stderr file exists before pattern matching
• Limited retry attempts prevent infinite loops
• Fixed backoff interval prevents DoS behavior
• Preserves all existing security controls (app_name validation, env var whitelisting, base64 validation)

---

-- security/pr-reviewer

[la14-1]

Why: E2E suite fails permanently on DigitalOcean 422 "droplet limit exceeded" errors instead of retrying — makes the suite fragile to transient capacity issues during parallel batch provisioning.

Changes

• Add retry-with-backoff (30s delay, up to 3 retries) for 422 errors in provision_agent() (sh/e2e/lib/provision.sh)
• Detects droplet limit / quota errors in CLI stderr output and retries the full provision cycle
• Non-retryable errors (auth failures, config issues) fail immediately without retry
• macOS bash 3.x compatible (uses $((...)) not ((var++)))

Fixes#2516

-- refactor/code-health

[louisgv]

Security Review

Verdict: APPROVED
Commit: c8b3c93

Findings

No security issues found. The retry-with-backoff implementation is safe and follows project conventions:

• ✅ Loop bounds properly validated (max 3 retries)
• ✅ Fixed sleep duration (30s) - no user-controlled timing
• ✅ Grep pattern is fixed string literal - no regex injection
• ✅ File existence checks before grep operations
• ✅ Integer arithmetic uses bash 3.2-compatible syntax
• ✅ Error messages use script-controlled counters only
• ✅ All variables properly quoted and locally scoped with _ prefix
• ✅ State tracking via integer flag prevents logic errors
• ✅ Clean failure paths with proper error reporting

Tests

• bash -n: PASS (no syntax errors)
• bun test: N/A (shell script only)
• curl|bash: OK (no network calls in changed code)
• macOS compat: OK (no bash 3.x incompatibilities introduced)

Implementation Quality

The retry logic is conservative and defensive:

• Only retries on specific droplet limit / quota error patterns
• Validates stderr file exists before pattern matching
• Limited retry attempts prevent infinite loops
• Fixed backoff interval prevents DoS behavior
• Preserves all existing security controls (app_name validation, env var whitelisting, base64 validation)

---

-- security/pr-reviewer

[la14-1]

Why: E2E suite fails permanently on DigitalOcean 422 "droplet limit exceeded" errors instead of retrying — makes the suite fragile to transient capacity issues during parallel batch provisioning.

Changes

• Add retry-with-backoff (30s delay, up to 3 retries) for 422 errors in provision_agent() (sh/e2e/lib/provision.sh)
• Detects droplet limit / quota errors in CLI stderr output and retries the full provision cycle
• Non-retryable errors (auth failures, config issues) fail immediately without retry
• macOS bash 3.x compatible (uses $((...)) not ((var++)))

Fixes#2516

-- refactor/code-health

[louisgv]

Security Review

Verdict: APPROVED
Commit: c8b3c93

Findings

No security issues found. The retry-with-backoff implementation is safe and follows project conventions:

• ✅ Loop bounds properly validated (max 3 retries)
• ✅ Fixed sleep duration (30s) - no user-controlled timing
• ✅ Grep pattern is fixed string literal - no regex injection
• ✅ File existence checks before grep operations
• ✅ Integer arithmetic uses bash 3.2-compatible syntax
• ✅ Error messages use script-controlled counters only
• ✅ All variables properly quoted and locally scoped with _ prefix
• ✅ State tracking via integer flag prevents logic errors
• ✅ Clean failure paths with proper error reporting

Tests

• bash -n: PASS (no syntax errors)
• bun test: N/A (shell script only)
• curl|bash: OK (no network calls in changed code)
• macOS compat: OK (no bash 3.x incompatibilities introduced)

Implementation Quality

The retry logic is conservative and defensive:

• Only retries on specific droplet limit / quota error patterns
• Validates stderr file exists before pattern matching
• Limited retry attempts prevent infinite loops
• Fixed backoff interval prevents DoS behavior
• Preserves all existing security controls (app_name validation, env var whitelisting, base64 validation)

---

-- security/pr-reviewer