fix(preview): point preview lambdas at the branch RDS, not DBInstances[0]

[nourshoreibah]

The bug

Auth failed in every preview environment while prod was fine. Multiple devs hit it. Symptom from the UI: login appears to work, then hangs ~11s and errors out.

.github/workflows/preview-env.yml resolved the preview lambdas' DB_HOST like this:

DB_HOST=$(aws rds describe-db-instances --query "DBInstances[0].Endpoint.Address" --output text)

This account hosts several unrelated C4C databases and the branch instance is not first:

#	instance
0	bhchp-postgres ← what previews got
1	c4c-hats
2	fcc-postgres
3	terraform-2025…001 ← the actual branch DB (prod uses this)

So all 12 preview lambdas across PRs #302 and #310 pointed at another project's database, whose security group blackholes our traffic. Every query hit the 5s connectionTimeoutMillis in db.ts and threw.

Why it looked like an auth bug

authenticateRequest wrapped both the JWT verification and the branch.users lookup in one try/catch that returned isAuthenticated: false. A dead database therefore surfaced as 401 Authentication required.

That is also actively harmful beyond the bad diagnostics: the frontend treats 401 as an expired session, so authedFetch refreshes, retries, then calls endSession() — a database blip logs the user out.

Login and refresh kept working throughout because they only talk to Cognito and never touch the DB, which is exactly why this looked like "auth is broken" rather than "the DB is unreachable."

Evidence

Against the PR-310 preview API, with a valid token:

probe	before	after
/auth/me, real token	401 in 5.06s / 5.12s / 5.58s	200 in 1.46s, correct user
/auth/me, forged signature (real kid)	401 in 165ms	unchanged
nc to bhchp-postgres:5432	hangs (blackholed)	n/a
nc to branch RDS:5432	0.27s	n/a

The forged-token probe is what isolated it: JWKS fetch and JWT verification were healthy and failed fast, so the 5s had to come from the line after verify(). aws-jwt-verify's own timeouts (1500ms socket / 3000ms response) never matched 5.06s; connectionTimeoutMillis: 5000 did, exactly.

The fix

1. preview-env.yml — stop re-deriving DB_HOST. The prod lambda config the workflow already reads carries the right value, so inherit it like every other DB credential. Added a guard that fails the job if DB_HOST / DB creds / Cognito ids are missing, instead of silently shipping a preview that 401s on every call.
2. shared/lambda-auth/src/authenticate.ts — only catch token verification. DB errors and a missing COGNITO_USER_POOL_ID now propagate into the handlers' existing 500 mapping. No handler changes needed; they all already have an outer catch → json(500, …).
3. Tests flipped to assert the new behavior. Two of them previously documented this footgun as intended — including the comment "This is why a missing env var manifests as blanket silent 401s across all six lambdas rather than a loud 500."

Already applied out of band

The workflow only resolves lambda env on label-add, so the 12 already-created preview lambdas kept the bad DB_HOST regardless of this change. I repatched all of them to the branch RDS, which is what infrastructure/preview/variables.tf already documents as intended ("Preview envs deliberately reuse the shared RDS + Cognito pool"). PR-310's preview is working now — verified with the 200 above.

Verification

• shared/lambda-auth: npx jest → 42/42 pass; npx tsc --noEmit clean.
• apps/backend/lambdas/auth: 67 unit tests pass. 3 auth.e2e.test.ts failures are pre-existing — they need a lambda on localhost:3000 and fail identically on clean main.
• Live: /auth/me on the PR-310 preview returns 200 with the correct user record.

Follow-ups not in scope here

• Preview envs share the prod database and Cognito pool. That's a deliberate, documented tradeoff, but it means a preview can write to prod data.
• apps/backend/lambdas/*/db.ts duplicates the same Pool config six times, so the 5s timeout and TLS rules have to be kept in sync by hand.

🤖 Generated with Claude Code

[github-actions]

🌿 ⏳ Creating preview environment… (logs)

[github-actions]

🌿 Preview environment — ready ✅

Open:https://d3nmtjoh6ir9ym.cloudfront.net/pr-316/
API:https://dr79itqqoj.execute-api.us-east-2.amazonaws.com/prod

Shared RDS + Cognito (prod data); DB migrations are not applied here — if this PR adds a migration, endpoints using the new columns will fail until it merges. New commits update this environment in place — a note is posted here on each update. Remove the test-environment label or close the PR to tear it down.

[github-actions]

🌿 Preview environment torn down 🧹 — the stack for this PR has been destroyed.

[nourshoreibah]

The bug

Auth failed in every preview environment while prod was fine. Multiple devs hit it. Symptom from the UI: login appears to work, then hangs ~11s and errors out.

.github/workflows/preview-env.yml resolved the preview lambdas' DB_HOST like this:

DB_HOST=$(aws rds describe-db-instances --query "DBInstances[0].Endpoint.Address" --output text)

This account hosts several unrelated C4C databases and the branch instance is not first:

#	instance
0	bhchp-postgres ← what previews got
1	c4c-hats
2	fcc-postgres
3	terraform-2025…001 ← the actual branch DB (prod uses this)

So all 12 preview lambdas across PRs #302 and #310 pointed at another project's database, whose security group blackholes our traffic. Every query hit the 5s connectionTimeoutMillis in db.ts and threw.

Why it looked like an auth bug

authenticateRequest wrapped both the JWT verification and the branch.users lookup in one try/catch that returned isAuthenticated: false. A dead database therefore surfaced as 401 Authentication required.

That is also actively harmful beyond the bad diagnostics: the frontend treats 401 as an expired session, so authedFetch refreshes, retries, then calls endSession() — a database blip logs the user out.

Login and refresh kept working throughout because they only talk to Cognito and never touch the DB, which is exactly why this looked like "auth is broken" rather than "the DB is unreachable."

Evidence

Against the PR-310 preview API, with a valid token:

probe	before	after
/auth/me, real token	401 in 5.06s / 5.12s / 5.58s	200 in 1.46s, correct user
/auth/me, forged signature (real kid)	401 in 165ms	unchanged
nc to bhchp-postgres:5432	hangs (blackholed)	n/a
nc to branch RDS:5432	0.27s	n/a

The forged-token probe is what isolated it: JWKS fetch and JWT verification were healthy and failed fast, so the 5s had to come from the line after verify(). aws-jwt-verify's own timeouts (1500ms socket / 3000ms response) never matched 5.06s; connectionTimeoutMillis: 5000 did, exactly.

The fix

1. preview-env.yml — stop re-deriving DB_HOST. The prod lambda config the workflow already reads carries the right value, so inherit it like every other DB credential. Added a guard that fails the job if DB_HOST / DB creds / Cognito ids are missing, instead of silently shipping a preview that 401s on every call.
2. shared/lambda-auth/src/authenticate.ts — only catch token verification. DB errors and a missing COGNITO_USER_POOL_ID now propagate into the handlers' existing 500 mapping. No handler changes needed; they all already have an outer catch → json(500, …).
3. Tests flipped to assert the new behavior. Two of them previously documented this footgun as intended — including the comment "This is why a missing env var manifests as blanket silent 401s across all six lambdas rather than a loud 500."

Already applied out of band

The workflow only resolves lambda env on label-add, so the 12 already-created preview lambdas kept the bad DB_HOST regardless of this change. I repatched all of them to the branch RDS, which is what infrastructure/preview/variables.tf already documents as intended ("Preview envs deliberately reuse the shared RDS + Cognito pool"). PR-310's preview is working now — verified with the 200 above.

Verification

• shared/lambda-auth: npx jest → 42/42 pass; npx tsc --noEmit clean.
• apps/backend/lambdas/auth: 67 unit tests pass. 3 auth.e2e.test.ts failures are pre-existing — they need a lambda on localhost:3000 and fail identically on clean main.
• Live: /auth/me on the PR-310 preview returns 200 with the correct user record.

Follow-ups not in scope here

• Preview envs share the prod database and Cognito pool. That's a deliberate, documented tradeoff, but it means a preview can write to prod data.
• apps/backend/lambdas/*/db.ts duplicates the same Pool config six times, so the 5s timeout and TLS rules have to be kept in sync by hand.

🤖 Generated with Claude Code

[github-actions]

🌿 ⏳ Creating preview environment… (logs)

[github-actions]

🌿 Preview environment — ready ✅

Open:https://d3nmtjoh6ir9ym.cloudfront.net/pr-316/
API:https://dr79itqqoj.execute-api.us-east-2.amazonaws.com/prod

Shared RDS + Cognito (prod data); DB migrations are not applied here — if this PR adds a migration, endpoints using the new columns will fail until it merges. New commits update this environment in place — a note is posted here on each update. Remove the test-environment label or close the PR to tear it down.

[github-actions]

🌿 Preview environment torn down 🧹 — the stack for this PR has been destroyed.

[nourshoreibah]

The bug

Auth failed in every preview environment while prod was fine. Multiple devs hit it. Symptom from the UI: login appears to work, then hangs ~11s and errors out.

.github/workflows/preview-env.yml resolved the preview lambdas' DB_HOST like this:

DB_HOST=$(aws rds describe-db-instances --query "DBInstances[0].Endpoint.Address" --output text)

This account hosts several unrelated C4C databases and the branch instance is not first:

#	instance
0	bhchp-postgres ← what previews got
1	c4c-hats
2	fcc-postgres
3	terraform-2025…001 ← the actual branch DB (prod uses this)

So all 12 preview lambdas across PRs #302 and #310 pointed at another project's database, whose security group blackholes our traffic. Every query hit the 5s connectionTimeoutMillis in db.ts and threw.

Why it looked like an auth bug

authenticateRequest wrapped both the JWT verification and the branch.users lookup in one try/catch that returned isAuthenticated: false. A dead database therefore surfaced as 401 Authentication required.

That is also actively harmful beyond the bad diagnostics: the frontend treats 401 as an expired session, so authedFetch refreshes, retries, then calls endSession() — a database blip logs the user out.

Login and refresh kept working throughout because they only talk to Cognito and never touch the DB, which is exactly why this looked like "auth is broken" rather than "the DB is unreachable."

Evidence

Against the PR-310 preview API, with a valid token:

probe	before	after
/auth/me, real token	401 in 5.06s / 5.12s / 5.58s	200 in 1.46s, correct user
/auth/me, forged signature (real kid)	401 in 165ms	unchanged
nc to bhchp-postgres:5432	hangs (blackholed)	n/a
nc to branch RDS:5432	0.27s	n/a

The forged-token probe is what isolated it: JWKS fetch and JWT verification were healthy and failed fast, so the 5s had to come from the line after verify(). aws-jwt-verify's own timeouts (1500ms socket / 3000ms response) never matched 5.06s; connectionTimeoutMillis: 5000 did, exactly.

The fix

1. preview-env.yml — stop re-deriving DB_HOST. The prod lambda config the workflow already reads carries the right value, so inherit it like every other DB credential. Added a guard that fails the job if DB_HOST / DB creds / Cognito ids are missing, instead of silently shipping a preview that 401s on every call.
2. shared/lambda-auth/src/authenticate.ts — only catch token verification. DB errors and a missing COGNITO_USER_POOL_ID now propagate into the handlers' existing 500 mapping. No handler changes needed; they all already have an outer catch → json(500, …).
3. Tests flipped to assert the new behavior. Two of them previously documented this footgun as intended — including the comment "This is why a missing env var manifests as blanket silent 401s across all six lambdas rather than a loud 500."

Already applied out of band

The workflow only resolves lambda env on label-add, so the 12 already-created preview lambdas kept the bad DB_HOST regardless of this change. I repatched all of them to the branch RDS, which is what infrastructure/preview/variables.tf already documents as intended ("Preview envs deliberately reuse the shared RDS + Cognito pool"). PR-310's preview is working now — verified with the 200 above.

Verification

• shared/lambda-auth: npx jest → 42/42 pass; npx tsc --noEmit clean.
• apps/backend/lambdas/auth: 67 unit tests pass. 3 auth.e2e.test.ts failures are pre-existing — they need a lambda on localhost:3000 and fail identically on clean main.
• Live: /auth/me on the PR-310 preview returns 200 with the correct user record.

Follow-ups not in scope here

• Preview envs share the prod database and Cognito pool. That's a deliberate, documented tradeoff, but it means a preview can write to prod data.
• apps/backend/lambdas/*/db.ts duplicates the same Pool config six times, so the 5s timeout and TLS rules have to be kept in sync by hand.

🤖 Generated with Claude Code

[github-actions]

🌿 ⏳ Creating preview environment… (logs)

[github-actions]

🌿 Preview environment — ready ✅

Open:https://d3nmtjoh6ir9ym.cloudfront.net/pr-316/
API:https://dr79itqqoj.execute-api.us-east-2.amazonaws.com/prod

Shared RDS + Cognito (prod data); DB migrations are not applied here — if this PR adds a migration, endpoints using the new columns will fail until it merges. New commits update this environment in place — a note is posted here on each update. Remove the test-environment label or close the PR to tear it down.

[github-actions]

🌿 Preview environment torn down 🧹 — the stack for this PR has been destroyed.

[nourshoreibah]

The bug

Auth failed in every preview environment while prod was fine. Multiple devs hit it. Symptom from the UI: login appears to work, then hangs ~11s and errors out.

.github/workflows/preview-env.yml resolved the preview lambdas' DB_HOST like this:

DB_HOST=$(aws rds describe-db-instances --query "DBInstances[0].Endpoint.Address" --output text)

This account hosts several unrelated C4C databases and the branch instance is not first:

#	instance
0	bhchp-postgres ← what previews got
1	c4c-hats
2	fcc-postgres
3	terraform-2025…001 ← the actual branch DB (prod uses this)

So all 12 preview lambdas across PRs #302 and #310 pointed at another project's database, whose security group blackholes our traffic. Every query hit the 5s connectionTimeoutMillis in db.ts and threw.

Why it looked like an auth bug

authenticateRequest wrapped both the JWT verification and the branch.users lookup in one try/catch that returned isAuthenticated: false. A dead database therefore surfaced as 401 Authentication required.

That is also actively harmful beyond the bad diagnostics: the frontend treats 401 as an expired session, so authedFetch refreshes, retries, then calls endSession() — a database blip logs the user out.

Login and refresh kept working throughout because they only talk to Cognito and never touch the DB, which is exactly why this looked like "auth is broken" rather than "the DB is unreachable."

Evidence

Against the PR-310 preview API, with a valid token:

probe	before	after
/auth/me, real token	401 in 5.06s / 5.12s / 5.58s	200 in 1.46s, correct user
/auth/me, forged signature (real kid)	401 in 165ms	unchanged
nc to bhchp-postgres:5432	hangs (blackholed)	n/a
nc to branch RDS:5432	0.27s	n/a

The forged-token probe is what isolated it: JWKS fetch and JWT verification were healthy and failed fast, so the 5s had to come from the line after verify(). aws-jwt-verify's own timeouts (1500ms socket / 3000ms response) never matched 5.06s; connectionTimeoutMillis: 5000 did, exactly.

The fix

1. preview-env.yml — stop re-deriving DB_HOST. The prod lambda config the workflow already reads carries the right value, so inherit it like every other DB credential. Added a guard that fails the job if DB_HOST / DB creds / Cognito ids are missing, instead of silently shipping a preview that 401s on every call.
2. shared/lambda-auth/src/authenticate.ts — only catch token verification. DB errors and a missing COGNITO_USER_POOL_ID now propagate into the handlers' existing 500 mapping. No handler changes needed; they all already have an outer catch → json(500, …).
3. Tests flipped to assert the new behavior. Two of them previously documented this footgun as intended — including the comment "This is why a missing env var manifests as blanket silent 401s across all six lambdas rather than a loud 500."

Already applied out of band

The workflow only resolves lambda env on label-add, so the 12 already-created preview lambdas kept the bad DB_HOST regardless of this change. I repatched all of them to the branch RDS, which is what infrastructure/preview/variables.tf already documents as intended ("Preview envs deliberately reuse the shared RDS + Cognito pool"). PR-310's preview is working now — verified with the 200 above.

Verification

• shared/lambda-auth: npx jest → 42/42 pass; npx tsc --noEmit clean.
• apps/backend/lambdas/auth: 67 unit tests pass. 3 auth.e2e.test.ts failures are pre-existing — they need a lambda on localhost:3000 and fail identically on clean main.
• Live: /auth/me on the PR-310 preview returns 200 with the correct user record.

Follow-ups not in scope here

• Preview envs share the prod database and Cognito pool. That's a deliberate, documented tradeoff, but it means a preview can write to prod data.
• apps/backend/lambdas/*/db.ts duplicates the same Pool config six times, so the 5s timeout and TLS rules have to be kept in sync by hand.

🤖 Generated with Claude Code

[github-actions]

🌿 ⏳ Creating preview environment… (logs)

[github-actions]

🌿 Preview environment — ready ✅

Open:https://d3nmtjoh6ir9ym.cloudfront.net/pr-316/
API:https://dr79itqqoj.execute-api.us-east-2.amazonaws.com/prod

Shared RDS + Cognito (prod data); DB migrations are not applied here — if this PR adds a migration, endpoints using the new columns will fail until it merges. New commits update this environment in place — a note is posted here on each update. Remove the test-environment label or close the PR to tear it down.

[github-actions]

🌿 Preview environment torn down 🧹 — the stack for this PR has been destroyed.

[nourshoreibah]

The bug

Auth failed in every preview environment while prod was fine. Multiple devs hit it. Symptom from the UI: login appears to work, then hangs ~11s and errors out.

.github/workflows/preview-env.yml resolved the preview lambdas' DB_HOST like this:

DB_HOST=$(aws rds describe-db-instances --query "DBInstances[0].Endpoint.Address" --output text)

This account hosts several unrelated C4C databases and the branch instance is not first:

#	instance
0	bhchp-postgres ← what previews got
1	c4c-hats
2	fcc-postgres
3	terraform-2025…001 ← the actual branch DB (prod uses this)

So all 12 preview lambdas across PRs #302 and #310 pointed at another project's database, whose security group blackholes our traffic. Every query hit the 5s connectionTimeoutMillis in db.ts and threw.

Why it looked like an auth bug

authenticateRequest wrapped both the JWT verification and the branch.users lookup in one try/catch that returned isAuthenticated: false. A dead database therefore surfaced as 401 Authentication required.

That is also actively harmful beyond the bad diagnostics: the frontend treats 401 as an expired session, so authedFetch refreshes, retries, then calls endSession() — a database blip logs the user out.

Login and refresh kept working throughout because they only talk to Cognito and never touch the DB, which is exactly why this looked like "auth is broken" rather than "the DB is unreachable."

Evidence

Against the PR-310 preview API, with a valid token:

probe	before	after
/auth/me, real token	401 in 5.06s / 5.12s / 5.58s	200 in 1.46s, correct user
/auth/me, forged signature (real kid)	401 in 165ms	unchanged
nc to bhchp-postgres:5432	hangs (blackholed)	n/a
nc to branch RDS:5432	0.27s	n/a

The forged-token probe is what isolated it: JWKS fetch and JWT verification were healthy and failed fast, so the 5s had to come from the line after verify(). aws-jwt-verify's own timeouts (1500ms socket / 3000ms response) never matched 5.06s; connectionTimeoutMillis: 5000 did, exactly.

The fix

1. preview-env.yml — stop re-deriving DB_HOST. The prod lambda config the workflow already reads carries the right value, so inherit it like every other DB credential. Added a guard that fails the job if DB_HOST / DB creds / Cognito ids are missing, instead of silently shipping a preview that 401s on every call.
2. shared/lambda-auth/src/authenticate.ts — only catch token verification. DB errors and a missing COGNITO_USER_POOL_ID now propagate into the handlers' existing 500 mapping. No handler changes needed; they all already have an outer catch → json(500, …).
3. Tests flipped to assert the new behavior. Two of them previously documented this footgun as intended — including the comment "This is why a missing env var manifests as blanket silent 401s across all six lambdas rather than a loud 500."

Already applied out of band

The workflow only resolves lambda env on label-add, so the 12 already-created preview lambdas kept the bad DB_HOST regardless of this change. I repatched all of them to the branch RDS, which is what infrastructure/preview/variables.tf already documents as intended ("Preview envs deliberately reuse the shared RDS + Cognito pool"). PR-310's preview is working now — verified with the 200 above.

Verification

• shared/lambda-auth: npx jest → 42/42 pass; npx tsc --noEmit clean.
• apps/backend/lambdas/auth: 67 unit tests pass. 3 auth.e2e.test.ts failures are pre-existing — they need a lambda on localhost:3000 and fail identically on clean main.
• Live: /auth/me on the PR-310 preview returns 200 with the correct user record.

Follow-ups not in scope here

• Preview envs share the prod database and Cognito pool. That's a deliberate, documented tradeoff, but it means a preview can write to prod data.
• apps/backend/lambdas/*/db.ts duplicates the same Pool config six times, so the 5s timeout and TLS rules have to be kept in sync by hand.

🤖 Generated with Claude Code

[github-actions]

🌿 ⏳ Creating preview environment… (logs)

[github-actions]

🌿 Preview environment — ready ✅

Open:https://d3nmtjoh6ir9ym.cloudfront.net/pr-316/
API:https://dr79itqqoj.execute-api.us-east-2.amazonaws.com/prod

Shared RDS + Cognito (prod data); DB migrations are not applied here — if this PR adds a migration, endpoints using the new columns will fail until it merges. New commits update this environment in place — a note is posted here on each update. Remove the test-environment label or close the PR to tear it down.

[github-actions]

🌿 Preview environment torn down 🧹 — the stack for this PR has been destroyed.

[nourshoreibah]

The bug

Auth failed in every preview environment while prod was fine. Multiple devs hit it. Symptom from the UI: login appears to work, then hangs ~11s and errors out.

.github/workflows/preview-env.yml resolved the preview lambdas' DB_HOST like this:

DB_HOST=$(aws rds describe-db-instances --query "DBInstances[0].Endpoint.Address" --output text)

This account hosts several unrelated C4C databases and the branch instance is not first:

#	instance
0	bhchp-postgres ← what previews got
1	c4c-hats
2	fcc-postgres
3	terraform-2025…001 ← the actual branch DB (prod uses this)

So all 12 preview lambdas across PRs #302 and #310 pointed at another project's database, whose security group blackholes our traffic. Every query hit the 5s connectionTimeoutMillis in db.ts and threw.

Why it looked like an auth bug

authenticateRequest wrapped both the JWT verification and the branch.users lookup in one try/catch that returned isAuthenticated: false. A dead database therefore surfaced as 401 Authentication required.

That is also actively harmful beyond the bad diagnostics: the frontend treats 401 as an expired session, so authedFetch refreshes, retries, then calls endSession() — a database blip logs the user out.

Login and refresh kept working throughout because they only talk to Cognito and never touch the DB, which is exactly why this looked like "auth is broken" rather than "the DB is unreachable."

Evidence

Against the PR-310 preview API, with a valid token:

probe	before	after
/auth/me, real token	401 in 5.06s / 5.12s / 5.58s	200 in 1.46s, correct user
/auth/me, forged signature (real kid)	401 in 165ms	unchanged
nc to bhchp-postgres:5432	hangs (blackholed)	n/a
nc to branch RDS:5432	0.27s	n/a

The forged-token probe is what isolated it: JWKS fetch and JWT verification were healthy and failed fast, so the 5s had to come from the line after verify(). aws-jwt-verify's own timeouts (1500ms socket / 3000ms response) never matched 5.06s; connectionTimeoutMillis: 5000 did, exactly.

The fix

1. preview-env.yml — stop re-deriving DB_HOST. The prod lambda config the workflow already reads carries the right value, so inherit it like every other DB credential. Added a guard that fails the job if DB_HOST / DB creds / Cognito ids are missing, instead of silently shipping a preview that 401s on every call.
2. shared/lambda-auth/src/authenticate.ts — only catch token verification. DB errors and a missing COGNITO_USER_POOL_ID now propagate into the handlers' existing 500 mapping. No handler changes needed; they all already have an outer catch → json(500, …).
3. Tests flipped to assert the new behavior. Two of them previously documented this footgun as intended — including the comment "This is why a missing env var manifests as blanket silent 401s across all six lambdas rather than a loud 500."

Already applied out of band

The workflow only resolves lambda env on label-add, so the 12 already-created preview lambdas kept the bad DB_HOST regardless of this change. I repatched all of them to the branch RDS, which is what infrastructure/preview/variables.tf already documents as intended ("Preview envs deliberately reuse the shared RDS + Cognito pool"). PR-310's preview is working now — verified with the 200 above.

Verification

• shared/lambda-auth: npx jest → 42/42 pass; npx tsc --noEmit clean.
• apps/backend/lambdas/auth: 67 unit tests pass. 3 auth.e2e.test.ts failures are pre-existing — they need a lambda on localhost:3000 and fail identically on clean main.
• Live: /auth/me on the PR-310 preview returns 200 with the correct user record.

Follow-ups not in scope here

• Preview envs share the prod database and Cognito pool. That's a deliberate, documented tradeoff, but it means a preview can write to prod data.
• apps/backend/lambdas/*/db.ts duplicates the same Pool config six times, so the 5s timeout and TLS rules have to be kept in sync by hand.

🤖 Generated with Claude Code

[github-actions]

🌿 ⏳ Creating preview environment… (logs)

[github-actions]

🌿 Preview environment — ready ✅

Open:https://d3nmtjoh6ir9ym.cloudfront.net/pr-316/
API:https://dr79itqqoj.execute-api.us-east-2.amazonaws.com/prod

Shared RDS + Cognito (prod data); DB migrations are not applied here — if this PR adds a migration, endpoints using the new columns will fail until it merges. New commits update this environment in place — a note is posted here on each update. Remove the test-environment label or close the PR to tear it down.

[github-actions]

🌿 Preview environment torn down 🧹 — the stack for this PR has been destroyed.

[nourshoreibah]

The bug

Auth failed in every preview environment while prod was fine. Multiple devs hit it. Symptom from the UI: login appears to work, then hangs ~11s and errors out.

.github/workflows/preview-env.yml resolved the preview lambdas' DB_HOST like this:

DB_HOST=$(aws rds describe-db-instances --query "DBInstances[0].Endpoint.Address" --output text)

This account hosts several unrelated C4C databases and the branch instance is not first:

#	instance
0	bhchp-postgres ← what previews got
1	c4c-hats
2	fcc-postgres
3	terraform-2025…001 ← the actual branch DB (prod uses this)

So all 12 preview lambdas across PRs #302 and #310 pointed at another project's database, whose security group blackholes our traffic. Every query hit the 5s connectionTimeoutMillis in db.ts and threw.

Why it looked like an auth bug

authenticateRequest wrapped both the JWT verification and the branch.users lookup in one try/catch that returned isAuthenticated: false. A dead database therefore surfaced as 401 Authentication required.

That is also actively harmful beyond the bad diagnostics: the frontend treats 401 as an expired session, so authedFetch refreshes, retries, then calls endSession() — a database blip logs the user out.

Login and refresh kept working throughout because they only talk to Cognito and never touch the DB, which is exactly why this looked like "auth is broken" rather than "the DB is unreachable."

Evidence

Against the PR-310 preview API, with a valid token:

probe	before	after
/auth/me, real token	401 in 5.06s / 5.12s / 5.58s	200 in 1.46s, correct user
/auth/me, forged signature (real kid)	401 in 165ms	unchanged
nc to bhchp-postgres:5432	hangs (blackholed)	n/a
nc to branch RDS:5432	0.27s	n/a

The forged-token probe is what isolated it: JWKS fetch and JWT verification were healthy and failed fast, so the 5s had to come from the line after verify(). aws-jwt-verify's own timeouts (1500ms socket / 3000ms response) never matched 5.06s; connectionTimeoutMillis: 5000 did, exactly.

The fix

1. preview-env.yml — stop re-deriving DB_HOST. The prod lambda config the workflow already reads carries the right value, so inherit it like every other DB credential. Added a guard that fails the job if DB_HOST / DB creds / Cognito ids are missing, instead of silently shipping a preview that 401s on every call.
2. shared/lambda-auth/src/authenticate.ts — only catch token verification. DB errors and a missing COGNITO_USER_POOL_ID now propagate into the handlers' existing 500 mapping. No handler changes needed; they all already have an outer catch → json(500, …).
3. Tests flipped to assert the new behavior. Two of them previously documented this footgun as intended — including the comment "This is why a missing env var manifests as blanket silent 401s across all six lambdas rather than a loud 500."

Already applied out of band

The workflow only resolves lambda env on label-add, so the 12 already-created preview lambdas kept the bad DB_HOST regardless of this change. I repatched all of them to the branch RDS, which is what infrastructure/preview/variables.tf already documents as intended ("Preview envs deliberately reuse the shared RDS + Cognito pool"). PR-310's preview is working now — verified with the 200 above.

Verification

• shared/lambda-auth: npx jest → 42/42 pass; npx tsc --noEmit clean.
• apps/backend/lambdas/auth: 67 unit tests pass. 3 auth.e2e.test.ts failures are pre-existing — they need a lambda on localhost:3000 and fail identically on clean main.
• Live: /auth/me on the PR-310 preview returns 200 with the correct user record.

Follow-ups not in scope here

• Preview envs share the prod database and Cognito pool. That's a deliberate, documented tradeoff, but it means a preview can write to prod data.
• apps/backend/lambdas/*/db.ts duplicates the same Pool config six times, so the 5s timeout and TLS rules have to be kept in sync by hand.

🤖 Generated with Claude Code

[github-actions]

🌿 ⏳ Creating preview environment… (logs)

[github-actions]

🌿 Preview environment — ready ✅

Open:https://d3nmtjoh6ir9ym.cloudfront.net/pr-316/
API:https://dr79itqqoj.execute-api.us-east-2.amazonaws.com/prod

Shared RDS + Cognito (prod data); DB migrations are not applied here — if this PR adds a migration, endpoints using the new columns will fail until it merges. New commits update this environment in place — a note is posted here on each update. Remove the test-environment label or close the PR to tear it down.

[github-actions]

🌿 Preview environment torn down 🧹 — the stack for this PR has been destroyed.

[nourshoreibah]

The bug

Auth failed in every preview environment while prod was fine. Multiple devs hit it. Symptom from the UI: login appears to work, then hangs ~11s and errors out.

.github/workflows/preview-env.yml resolved the preview lambdas' DB_HOST like this:

DB_HOST=$(aws rds describe-db-instances --query "DBInstances[0].Endpoint.Address" --output text)

This account hosts several unrelated C4C databases and the branch instance is not first:

#	instance
0	bhchp-postgres ← what previews got
1	c4c-hats
2	fcc-postgres
3	terraform-2025…001 ← the actual branch DB (prod uses this)

So all 12 preview lambdas across PRs #302 and #310 pointed at another project's database, whose security group blackholes our traffic. Every query hit the 5s connectionTimeoutMillis in db.ts and threw.

Why it looked like an auth bug

authenticateRequest wrapped both the JWT verification and the branch.users lookup in one try/catch that returned isAuthenticated: false. A dead database therefore surfaced as 401 Authentication required.

That is also actively harmful beyond the bad diagnostics: the frontend treats 401 as an expired session, so authedFetch refreshes, retries, then calls endSession() — a database blip logs the user out.

Login and refresh kept working throughout because they only talk to Cognito and never touch the DB, which is exactly why this looked like "auth is broken" rather than "the DB is unreachable."

Evidence

Against the PR-310 preview API, with a valid token:

probe	before	after
/auth/me, real token	401 in 5.06s / 5.12s / 5.58s	200 in 1.46s, correct user
/auth/me, forged signature (real kid)	401 in 165ms	unchanged
nc to bhchp-postgres:5432	hangs (blackholed)	n/a
nc to branch RDS:5432	0.27s	n/a

The forged-token probe is what isolated it: JWKS fetch and JWT verification were healthy and failed fast, so the 5s had to come from the line after verify(). aws-jwt-verify's own timeouts (1500ms socket / 3000ms response) never matched 5.06s; connectionTimeoutMillis: 5000 did, exactly.

The fix

1. preview-env.yml — stop re-deriving DB_HOST. The prod lambda config the workflow already reads carries the right value, so inherit it like every other DB credential. Added a guard that fails the job if DB_HOST / DB creds / Cognito ids are missing, instead of silently shipping a preview that 401s on every call.
2. shared/lambda-auth/src/authenticate.ts — only catch token verification. DB errors and a missing COGNITO_USER_POOL_ID now propagate into the handlers' existing 500 mapping. No handler changes needed; they all already have an outer catch → json(500, …).
3. Tests flipped to assert the new behavior. Two of them previously documented this footgun as intended — including the comment "This is why a missing env var manifests as blanket silent 401s across all six lambdas rather than a loud 500."

Already applied out of band

The workflow only resolves lambda env on label-add, so the 12 already-created preview lambdas kept the bad DB_HOST regardless of this change. I repatched all of them to the branch RDS, which is what infrastructure/preview/variables.tf already documents as intended ("Preview envs deliberately reuse the shared RDS + Cognito pool"). PR-310's preview is working now — verified with the 200 above.

Verification

• shared/lambda-auth: npx jest → 42/42 pass; npx tsc --noEmit clean.
• apps/backend/lambdas/auth: 67 unit tests pass. 3 auth.e2e.test.ts failures are pre-existing — they need a lambda on localhost:3000 and fail identically on clean main.
• Live: /auth/me on the PR-310 preview returns 200 with the correct user record.

Follow-ups not in scope here

• Preview envs share the prod database and Cognito pool. That's a deliberate, documented tradeoff, but it means a preview can write to prod data.
• apps/backend/lambdas/*/db.ts duplicates the same Pool config six times, so the 5s timeout and TLS rules have to be kept in sync by hand.

🤖 Generated with Claude Code

[github-actions]

🌿 ⏳ Creating preview environment… (logs)

[github-actions]

🌿 Preview environment — ready ✅

Open:https://d3nmtjoh6ir9ym.cloudfront.net/pr-316/
API:https://dr79itqqoj.execute-api.us-east-2.amazonaws.com/prod

Shared RDS + Cognito (prod data); DB migrations are not applied here — if this PR adds a migration, endpoints using the new columns will fail until it merges. New commits update this environment in place — a note is posted here on each update. Remove the test-environment label or close the PR to tear it down.

[github-actions]

🌿 Preview environment torn down 🧹 — the stack for this PR has been destroyed.