fix(gmail): retry + defer write-backs instead of dropping them on quota

[KrisBraun]

Problem

Gmail star/read write-backs (onThreadToDo/onThreadRead → GmailApi.modifyThread, one Gmail call per thread state change) had no rate-limit handling. A single per-user-per-minute quota 403 rateLimitExceeded threw straight through GmailApi.call; the API worker's updates queue consumer (processTwistBatch) caught it, captured it to error tracking, and continued — so the user's to-do/read change silently never reached Gmail and was never retried.

This is the gmail_quota half of PostHog issue 019ed581 — the sibling of the incremental-sync OOM fixed in #241. Both fired together from one composite "Gmail & Calendar" connector during a 2026-06-29 burst (46 write-back quota 403s in ~90s for a single user). #241 bounds the sync read storm; it does not touch the write-back path.

Fix (retry + defer — no suppression)

Two layers so a write-back actually completes instead of being dropped:

1. GmailApi.call bounded retry (gmail-api.ts): new isGmailRateLimitError() (429, or 403 carrying rateLimitExceeded / userRateLimitExceeded / Quota exceeded). call() now retries rate-limit / 5xx / dropped-connection with short backoff [500, 1500]ms, honoring Retry-After up to a 3s cap — a longer server-requested wait throws so the caller defers rather than blocking the isolate. Benefits every Gmail call (complements fix(gmail): bound incremental sync per pass to stop isolate OOM #241's read path). Permission 403s and 404s are not retried.

2. Deferred write-back drain (sync.ts): on sustained quota, onThreadToDo / onThreadRead append the write-back to a persisted writeback_pending queue (dedup by kind+threadId, last-write-wins, capped at 500) and schedule a drain — instead of dropping it. Non-rate-limit errors still throw (stay visible). processWriteBackRetryFn drains ≤20/pass, bumps+retries rate-limited items up to MAX_WRITEBACK_ATTEMPTS = 5, drops on lost-auth / permanent error with a log (the state still lives in Plot), and re-queues only while work remains (self-terminating).

   The drain is scheduled with a keyed, delayedscheduleTask("gmail-writeback-retry", cb, { runAt: +60s }) so a burst of enqueues collapses to a single task that fires after the per-minute quota window clears — it never hot-loops the way an immediate runTask self-chain would.

Wired through both the standalone Gmail connector and the composite Gmail & Calendar (Google) connector via the shared GmailSyncHost scheduler boundary.

Tests

• New gmail-api-retry.test.ts (13): retry-then-succeed for 429 / 403-rateLimit / 5xx; no-retry for permission-403 / 404; Retry-After-too-long defers; retry exhaustion; isGmailRateLimitError classifier.
• New gmail-writeback-defer.test.ts (11): defer-on-rate-limit + rethrow-on-other for both callbacks; drain success/clear, rate-limited keep+bump+re-queue, give-up after max attempts, batch bound + overflow re-queue, lost-token drop; mergePendingWriteBack dedup.
• gmail 87 tests, google 64, tsc + plot lint clean. Connector-only → no changeset.

Stacked on #241 (now merged).

🤖 Generated with Claude Code

[KrisBraun]

Problem

Gmail star/read write-backs (onThreadToDo/onThreadRead → GmailApi.modifyThread, one Gmail call per thread state change) had no rate-limit handling. A single per-user-per-minute quota 403 rateLimitExceeded threw straight through GmailApi.call; the API worker's updates queue consumer (processTwistBatch) caught it, captured it to error tracking, and continued — so the user's to-do/read change silently never reached Gmail and was never retried.

This is the gmail_quota half of PostHog issue 019ed581 — the sibling of the incremental-sync OOM fixed in #241. Both fired together from one composite "Gmail & Calendar" connector during a 2026-06-29 burst (46 write-back quota 403s in ~90s for a single user). #241 bounds the sync read storm; it does not touch the write-back path.

Fix (retry + defer — no suppression)

Two layers so a write-back actually completes instead of being dropped:

1. GmailApi.call bounded retry (gmail-api.ts): new isGmailRateLimitError() (429, or 403 carrying rateLimitExceeded / userRateLimitExceeded / Quota exceeded). call() now retries rate-limit / 5xx / dropped-connection with short backoff [500, 1500]ms, honoring Retry-After up to a 3s cap — a longer server-requested wait throws so the caller defers rather than blocking the isolate. Benefits every Gmail call (complements fix(gmail): bound incremental sync per pass to stop isolate OOM #241's read path). Permission 403s and 404s are not retried.

2. Deferred write-back drain (sync.ts): on sustained quota, onThreadToDo / onThreadRead append the write-back to a persisted writeback_pending queue (dedup by kind+threadId, last-write-wins, capped at 500) and schedule a drain — instead of dropping it. Non-rate-limit errors still throw (stay visible). processWriteBackRetryFn drains ≤20/pass, bumps+retries rate-limited items up to MAX_WRITEBACK_ATTEMPTS = 5, drops on lost-auth / permanent error with a log (the state still lives in Plot), and re-queues only while work remains (self-terminating).

   The drain is scheduled with a keyed, delayedscheduleTask("gmail-writeback-retry", cb, { runAt: +60s }) so a burst of enqueues collapses to a single task that fires after the per-minute quota window clears — it never hot-loops the way an immediate runTask self-chain would.

Wired through both the standalone Gmail connector and the composite Gmail & Calendar (Google) connector via the shared GmailSyncHost scheduler boundary.

Tests

• New gmail-api-retry.test.ts (13): retry-then-succeed for 429 / 403-rateLimit / 5xx; no-retry for permission-403 / 404; Retry-After-too-long defers; retry exhaustion; isGmailRateLimitError classifier.
• New gmail-writeback-defer.test.ts (11): defer-on-rate-limit + rethrow-on-other for both callbacks; drain success/clear, rate-limited keep+bump+re-queue, give-up after max attempts, batch bound + overflow re-queue, lost-token drop; mergePendingWriteBack dedup.
• gmail 87 tests, google 64, tsc + plot lint clean. Connector-only → no changeset.

Stacked on #241 (now merged).

🤖 Generated with Claude Code

[KrisBraun]

Problem

Gmail star/read write-backs (onThreadToDo/onThreadRead → GmailApi.modifyThread, one Gmail call per thread state change) had no rate-limit handling. A single per-user-per-minute quota 403 rateLimitExceeded threw straight through GmailApi.call; the API worker's updates queue consumer (processTwistBatch) caught it, captured it to error tracking, and continued — so the user's to-do/read change silently never reached Gmail and was never retried.

This is the gmail_quota half of PostHog issue 019ed581 — the sibling of the incremental-sync OOM fixed in #241. Both fired together from one composite "Gmail & Calendar" connector during a 2026-06-29 burst (46 write-back quota 403s in ~90s for a single user). #241 bounds the sync read storm; it does not touch the write-back path.

Fix (retry + defer — no suppression)

Two layers so a write-back actually completes instead of being dropped:

1. GmailApi.call bounded retry (gmail-api.ts): new isGmailRateLimitError() (429, or 403 carrying rateLimitExceeded / userRateLimitExceeded / Quota exceeded). call() now retries rate-limit / 5xx / dropped-connection with short backoff [500, 1500]ms, honoring Retry-After up to a 3s cap — a longer server-requested wait throws so the caller defers rather than blocking the isolate. Benefits every Gmail call (complements fix(gmail): bound incremental sync per pass to stop isolate OOM #241's read path). Permission 403s and 404s are not retried.

2. Deferred write-back drain (sync.ts): on sustained quota, onThreadToDo / onThreadRead append the write-back to a persisted writeback_pending queue (dedup by kind+threadId, last-write-wins, capped at 500) and schedule a drain — instead of dropping it. Non-rate-limit errors still throw (stay visible). processWriteBackRetryFn drains ≤20/pass, bumps+retries rate-limited items up to MAX_WRITEBACK_ATTEMPTS = 5, drops on lost-auth / permanent error with a log (the state still lives in Plot), and re-queues only while work remains (self-terminating).

   The drain is scheduled with a keyed, delayedscheduleTask("gmail-writeback-retry", cb, { runAt: +60s }) so a burst of enqueues collapses to a single task that fires after the per-minute quota window clears — it never hot-loops the way an immediate runTask self-chain would.

Wired through both the standalone Gmail connector and the composite Gmail & Calendar (Google) connector via the shared GmailSyncHost scheduler boundary.

Tests

• New gmail-api-retry.test.ts (13): retry-then-succeed for 429 / 403-rateLimit / 5xx; no-retry for permission-403 / 404; Retry-After-too-long defers; retry exhaustion; isGmailRateLimitError classifier.
• New gmail-writeback-defer.test.ts (11): defer-on-rate-limit + rethrow-on-other for both callbacks; drain success/clear, rate-limited keep+bump+re-queue, give-up after max attempts, batch bound + overflow re-queue, lost-token drop; mergePendingWriteBack dedup.
• gmail 87 tests, google 64, tsc + plot lint clean. Connector-only → no changeset.

Stacked on #241 (now merged).

🤖 Generated with Claude Code

[KrisBraun]

Problem

Gmail star/read write-backs (onThreadToDo/onThreadRead → GmailApi.modifyThread, one Gmail call per thread state change) had no rate-limit handling. A single per-user-per-minute quota 403 rateLimitExceeded threw straight through GmailApi.call; the API worker's updates queue consumer (processTwistBatch) caught it, captured it to error tracking, and continued — so the user's to-do/read change silently never reached Gmail and was never retried.

This is the gmail_quota half of PostHog issue 019ed581 — the sibling of the incremental-sync OOM fixed in #241. Both fired together from one composite "Gmail & Calendar" connector during a 2026-06-29 burst (46 write-back quota 403s in ~90s for a single user). #241 bounds the sync read storm; it does not touch the write-back path.

Fix (retry + defer — no suppression)

Two layers so a write-back actually completes instead of being dropped:

1. GmailApi.call bounded retry (gmail-api.ts): new isGmailRateLimitError() (429, or 403 carrying rateLimitExceeded / userRateLimitExceeded / Quota exceeded). call() now retries rate-limit / 5xx / dropped-connection with short backoff [500, 1500]ms, honoring Retry-After up to a 3s cap — a longer server-requested wait throws so the caller defers rather than blocking the isolate. Benefits every Gmail call (complements fix(gmail): bound incremental sync per pass to stop isolate OOM #241's read path). Permission 403s and 404s are not retried.

2. Deferred write-back drain (sync.ts): on sustained quota, onThreadToDo / onThreadRead append the write-back to a persisted writeback_pending queue (dedup by kind+threadId, last-write-wins, capped at 500) and schedule a drain — instead of dropping it. Non-rate-limit errors still throw (stay visible). processWriteBackRetryFn drains ≤20/pass, bumps+retries rate-limited items up to MAX_WRITEBACK_ATTEMPTS = 5, drops on lost-auth / permanent error with a log (the state still lives in Plot), and re-queues only while work remains (self-terminating).

   The drain is scheduled with a keyed, delayedscheduleTask("gmail-writeback-retry", cb, { runAt: +60s }) so a burst of enqueues collapses to a single task that fires after the per-minute quota window clears — it never hot-loops the way an immediate runTask self-chain would.

Wired through both the standalone Gmail connector and the composite Gmail & Calendar (Google) connector via the shared GmailSyncHost scheduler boundary.

Tests

• New gmail-api-retry.test.ts (13): retry-then-succeed for 429 / 403-rateLimit / 5xx; no-retry for permission-403 / 404; Retry-After-too-long defers; retry exhaustion; isGmailRateLimitError classifier.
• New gmail-writeback-defer.test.ts (11): defer-on-rate-limit + rethrow-on-other for both callbacks; drain success/clear, rate-limited keep+bump+re-queue, give-up after max attempts, batch bound + overflow re-queue, lost-token drop; mergePendingWriteBack dedup.
• gmail 87 tests, google 64, tsc + plot lint clean. Connector-only → no changeset.

Stacked on #241 (now merged).

🤖 Generated with Claude Code

[KrisBraun]

Problem

Gmail star/read write-backs (onThreadToDo/onThreadRead → GmailApi.modifyThread, one Gmail call per thread state change) had no rate-limit handling. A single per-user-per-minute quota 403 rateLimitExceeded threw straight through GmailApi.call; the API worker's updates queue consumer (processTwistBatch) caught it, captured it to error tracking, and continued — so the user's to-do/read change silently never reached Gmail and was never retried.

This is the gmail_quota half of PostHog issue 019ed581 — the sibling of the incremental-sync OOM fixed in #241. Both fired together from one composite "Gmail & Calendar" connector during a 2026-06-29 burst (46 write-back quota 403s in ~90s for a single user). #241 bounds the sync read storm; it does not touch the write-back path.

Fix (retry + defer — no suppression)

Two layers so a write-back actually completes instead of being dropped:

1. GmailApi.call bounded retry (gmail-api.ts): new isGmailRateLimitError() (429, or 403 carrying rateLimitExceeded / userRateLimitExceeded / Quota exceeded). call() now retries rate-limit / 5xx / dropped-connection with short backoff [500, 1500]ms, honoring Retry-After up to a 3s cap — a longer server-requested wait throws so the caller defers rather than blocking the isolate. Benefits every Gmail call (complements fix(gmail): bound incremental sync per pass to stop isolate OOM #241's read path). Permission 403s and 404s are not retried.

2. Deferred write-back drain (sync.ts): on sustained quota, onThreadToDo / onThreadRead append the write-back to a persisted writeback_pending queue (dedup by kind+threadId, last-write-wins, capped at 500) and schedule a drain — instead of dropping it. Non-rate-limit errors still throw (stay visible). processWriteBackRetryFn drains ≤20/pass, bumps+retries rate-limited items up to MAX_WRITEBACK_ATTEMPTS = 5, drops on lost-auth / permanent error with a log (the state still lives in Plot), and re-queues only while work remains (self-terminating).

   The drain is scheduled with a keyed, delayedscheduleTask("gmail-writeback-retry", cb, { runAt: +60s }) so a burst of enqueues collapses to a single task that fires after the per-minute quota window clears — it never hot-loops the way an immediate runTask self-chain would.

Wired through both the standalone Gmail connector and the composite Gmail & Calendar (Google) connector via the shared GmailSyncHost scheduler boundary.

Tests

• New gmail-api-retry.test.ts (13): retry-then-succeed for 429 / 403-rateLimit / 5xx; no-retry for permission-403 / 404; Retry-After-too-long defers; retry exhaustion; isGmailRateLimitError classifier.
• New gmail-writeback-defer.test.ts (11): defer-on-rate-limit + rethrow-on-other for both callbacks; drain success/clear, rate-limited keep+bump+re-queue, give-up after max attempts, batch bound + overflow re-queue, lost-token drop; mergePendingWriteBack dedup.
• gmail 87 tests, google 64, tsc + plot lint clean. Connector-only → no changeset.

Stacked on #241 (now merged).

🤖 Generated with Claude Code

[KrisBraun]

Problem

Gmail star/read write-backs (onThreadToDo/onThreadRead → GmailApi.modifyThread, one Gmail call per thread state change) had no rate-limit handling. A single per-user-per-minute quota 403 rateLimitExceeded threw straight through GmailApi.call; the API worker's updates queue consumer (processTwistBatch) caught it, captured it to error tracking, and continued — so the user's to-do/read change silently never reached Gmail and was never retried.

This is the gmail_quota half of PostHog issue 019ed581 — the sibling of the incremental-sync OOM fixed in #241. Both fired together from one composite "Gmail & Calendar" connector during a 2026-06-29 burst (46 write-back quota 403s in ~90s for a single user). #241 bounds the sync read storm; it does not touch the write-back path.

Fix (retry + defer — no suppression)

Two layers so a write-back actually completes instead of being dropped:

1. GmailApi.call bounded retry (gmail-api.ts): new isGmailRateLimitError() (429, or 403 carrying rateLimitExceeded / userRateLimitExceeded / Quota exceeded). call() now retries rate-limit / 5xx / dropped-connection with short backoff [500, 1500]ms, honoring Retry-After up to a 3s cap — a longer server-requested wait throws so the caller defers rather than blocking the isolate. Benefits every Gmail call (complements fix(gmail): bound incremental sync per pass to stop isolate OOM #241's read path). Permission 403s and 404s are not retried.

2. Deferred write-back drain (sync.ts): on sustained quota, onThreadToDo / onThreadRead append the write-back to a persisted writeback_pending queue (dedup by kind+threadId, last-write-wins, capped at 500) and schedule a drain — instead of dropping it. Non-rate-limit errors still throw (stay visible). processWriteBackRetryFn drains ≤20/pass, bumps+retries rate-limited items up to MAX_WRITEBACK_ATTEMPTS = 5, drops on lost-auth / permanent error with a log (the state still lives in Plot), and re-queues only while work remains (self-terminating).

   The drain is scheduled with a keyed, delayedscheduleTask("gmail-writeback-retry", cb, { runAt: +60s }) so a burst of enqueues collapses to a single task that fires after the per-minute quota window clears — it never hot-loops the way an immediate runTask self-chain would.

Wired through both the standalone Gmail connector and the composite Gmail & Calendar (Google) connector via the shared GmailSyncHost scheduler boundary.

Tests

• New gmail-api-retry.test.ts (13): retry-then-succeed for 429 / 403-rateLimit / 5xx; no-retry for permission-403 / 404; Retry-After-too-long defers; retry exhaustion; isGmailRateLimitError classifier.
• New gmail-writeback-defer.test.ts (11): defer-on-rate-limit + rethrow-on-other for both callbacks; drain success/clear, rate-limited keep+bump+re-queue, give-up after max attempts, batch bound + overflow re-queue, lost-token drop; mergePendingWriteBack dedup.
• gmail 87 tests, google 64, tsc + plot lint clean. Connector-only → no changeset.

Stacked on #241 (now merged).

🤖 Generated with Claude Code

[KrisBraun]

Problem

Gmail star/read write-backs (onThreadToDo/onThreadRead → GmailApi.modifyThread, one Gmail call per thread state change) had no rate-limit handling. A single per-user-per-minute quota 403 rateLimitExceeded threw straight through GmailApi.call; the API worker's updates queue consumer (processTwistBatch) caught it, captured it to error tracking, and continued — so the user's to-do/read change silently never reached Gmail and was never retried.

This is the gmail_quota half of PostHog issue 019ed581 — the sibling of the incremental-sync OOM fixed in #241. Both fired together from one composite "Gmail & Calendar" connector during a 2026-06-29 burst (46 write-back quota 403s in ~90s for a single user). #241 bounds the sync read storm; it does not touch the write-back path.

Fix (retry + defer — no suppression)

Two layers so a write-back actually completes instead of being dropped:

1. GmailApi.call bounded retry (gmail-api.ts): new isGmailRateLimitError() (429, or 403 carrying rateLimitExceeded / userRateLimitExceeded / Quota exceeded). call() now retries rate-limit / 5xx / dropped-connection with short backoff [500, 1500]ms, honoring Retry-After up to a 3s cap — a longer server-requested wait throws so the caller defers rather than blocking the isolate. Benefits every Gmail call (complements fix(gmail): bound incremental sync per pass to stop isolate OOM #241's read path). Permission 403s and 404s are not retried.

2. Deferred write-back drain (sync.ts): on sustained quota, onThreadToDo / onThreadRead append the write-back to a persisted writeback_pending queue (dedup by kind+threadId, last-write-wins, capped at 500) and schedule a drain — instead of dropping it. Non-rate-limit errors still throw (stay visible). processWriteBackRetryFn drains ≤20/pass, bumps+retries rate-limited items up to MAX_WRITEBACK_ATTEMPTS = 5, drops on lost-auth / permanent error with a log (the state still lives in Plot), and re-queues only while work remains (self-terminating).

   The drain is scheduled with a keyed, delayedscheduleTask("gmail-writeback-retry", cb, { runAt: +60s }) so a burst of enqueues collapses to a single task that fires after the per-minute quota window clears — it never hot-loops the way an immediate runTask self-chain would.

Wired through both the standalone Gmail connector and the composite Gmail & Calendar (Google) connector via the shared GmailSyncHost scheduler boundary.

Tests

• New gmail-api-retry.test.ts (13): retry-then-succeed for 429 / 403-rateLimit / 5xx; no-retry for permission-403 / 404; Retry-After-too-long defers; retry exhaustion; isGmailRateLimitError classifier.
• New gmail-writeback-defer.test.ts (11): defer-on-rate-limit + rethrow-on-other for both callbacks; drain success/clear, rate-limited keep+bump+re-queue, give-up after max attempts, batch bound + overflow re-queue, lost-token drop; mergePendingWriteBack dedup.
• gmail 87 tests, google 64, tsc + plot lint clean. Connector-only → no changeset.

Stacked on #241 (now merged).

🤖 Generated with Claude Code

[KrisBraun]

Problem

Gmail star/read write-backs (onThreadToDo/onThreadRead → GmailApi.modifyThread, one Gmail call per thread state change) had no rate-limit handling. A single per-user-per-minute quota 403 rateLimitExceeded threw straight through GmailApi.call; the API worker's updates queue consumer (processTwistBatch) caught it, captured it to error tracking, and continued — so the user's to-do/read change silently never reached Gmail and was never retried.

This is the gmail_quota half of PostHog issue 019ed581 — the sibling of the incremental-sync OOM fixed in #241. Both fired together from one composite "Gmail & Calendar" connector during a 2026-06-29 burst (46 write-back quota 403s in ~90s for a single user). #241 bounds the sync read storm; it does not touch the write-back path.

Fix (retry + defer — no suppression)

Two layers so a write-back actually completes instead of being dropped:

1. GmailApi.call bounded retry (gmail-api.ts): new isGmailRateLimitError() (429, or 403 carrying rateLimitExceeded / userRateLimitExceeded / Quota exceeded). call() now retries rate-limit / 5xx / dropped-connection with short backoff [500, 1500]ms, honoring Retry-After up to a 3s cap — a longer server-requested wait throws so the caller defers rather than blocking the isolate. Benefits every Gmail call (complements fix(gmail): bound incremental sync per pass to stop isolate OOM #241's read path). Permission 403s and 404s are not retried.

2. Deferred write-back drain (sync.ts): on sustained quota, onThreadToDo / onThreadRead append the write-back to a persisted writeback_pending queue (dedup by kind+threadId, last-write-wins, capped at 500) and schedule a drain — instead of dropping it. Non-rate-limit errors still throw (stay visible). processWriteBackRetryFn drains ≤20/pass, bumps+retries rate-limited items up to MAX_WRITEBACK_ATTEMPTS = 5, drops on lost-auth / permanent error with a log (the state still lives in Plot), and re-queues only while work remains (self-terminating).

   The drain is scheduled with a keyed, delayedscheduleTask("gmail-writeback-retry", cb, { runAt: +60s }) so a burst of enqueues collapses to a single task that fires after the per-minute quota window clears — it never hot-loops the way an immediate runTask self-chain would.

Wired through both the standalone Gmail connector and the composite Gmail & Calendar (Google) connector via the shared GmailSyncHost scheduler boundary.

Tests

• New gmail-api-retry.test.ts (13): retry-then-succeed for 429 / 403-rateLimit / 5xx; no-retry for permission-403 / 404; Retry-After-too-long defers; retry exhaustion; isGmailRateLimitError classifier.
• New gmail-writeback-defer.test.ts (11): defer-on-rate-limit + rethrow-on-other for both callbacks; drain success/clear, rate-limited keep+bump+re-queue, give-up after max attempts, batch bound + overflow re-queue, lost-token drop; mergePendingWriteBack dedup.
• gmail 87 tests, google 64, tsc + plot lint clean. Connector-only → no changeset.

Stacked on #241 (now merged).

🤖 Generated with Claude Code