fix(gmail,outlook-mail): durable recovery for stranded mailbox watches on upgrade

[KrisBraun]

Problem

A mail connection (Gmail or Outlook) could be left permanently and silently dead with no self-recovery on deploy — the failure mode the durable-recurring-tasks work (#215) set out to eliminate, but with a gap.

upgrade() only re-asserted the recurring self-heal/renewal tasks when the watch/subscription sentinel (mailbox_webhook / mailbox_subscription) was already stored, and the cron maintenance sweep is gated on a recurring_meta.ever marker. Two stranded states slip through both recovery paths:

1. The sentinel never persisted — a prior setupWatch() / setupMailboxSubscription() threw before its set() (e.g. Gmail's 400 "Only one user push notification client allowed per developer"). There was nothing to re-assert, and because scheduleRecurring never ran, the sweep's ever marker was never set either.
2. The watch/subscription expired while the self-heal/renewal chain was dead.

Either way the connection stayed dead until the user manually re-enabled a channel.

Real-world hit

A Gmail "Personal" connection stopped ingesting after a June reconnect and produced zero mail for ~18 days across multiple deploys, while the same user's other Gmail connection (continuously healthy) was unaffected. The recovery paths shipped in #215 couldn't reach it because its watch state had never persisted / had expired with a dead chain.

Fix

upgrade() now always runs a recoverMailboxDelivery() backstop for any instance with enabled channels:

• Healthy watch (sentinel present and unexpired) → only re-assert the recurring tasks (unchanged behavior).
• Missing or expired watch → re-establish it and re-walk every enabled label/folder, so mail that accumulated while delivery was dead is backfilled. The backfill upserts by source (no duplicates) and uses initial-sync semantics (unread:false, archived:false), so it never spams notifications.

Gmail's legacy per-channel→mailbox migration is extracted into migrateLegacyPerChannelState(); the watch is then (re-)established by the shared recovery path (removing the old if (migratedAny) setupMailboxWebhook() tail).

Both connectors get the same shape (recoverMailboxDelivery + requeueInitialSync), since Outlook-mail had the identical gating gap (if (existing?.subscriptionId) re-assert, no else).

Why this fixes it for all affected users

It runs on every deploy for every non-archived instance (the deploy "Upgrading active twists" phase). Any Gmail/Outlook connection currently stranded with a missing/expired watch will re-establish live delivery and backfill its gap on the next deploy — no manual re-enable required.

Testing

• Gmail: 47 tests pass (4 new recoverMailboxDelivery cases: stranded, expired, healthy, no-channels)
• Outlook-mail: 35 tests pass (4 new equivalent cases)
• Both pnpm build (tsc) + plot lint clean
• Connector-only change → no changeset (per public/AGENTS.md)

Deploy / verify notes

• This is a public-submodule connector change. To reach production it needs the superproject (plotday/core) submodule pointer bumped to the merge commit, then a deploy (the deploy-connectors (gmail/outlook-mail) jobs).
• A connection that already self-healed its live watch (sentinel now fresh) takes the healthy path and is not re-backfilled; recovering its historical gap still needs a one-time channel re-enable (which triggers onChannelEnabled's full backfill). The auto-backfill here targets connections still stranded at deploy time.

🤖 Generated with Claude Code

[KrisBraun]

Problem

A mail connection (Gmail or Outlook) could be left permanently and silently dead with no self-recovery on deploy — the failure mode the durable-recurring-tasks work (#215) set out to eliminate, but with a gap.

upgrade() only re-asserted the recurring self-heal/renewal tasks when the watch/subscription sentinel (mailbox_webhook / mailbox_subscription) was already stored, and the cron maintenance sweep is gated on a recurring_meta.ever marker. Two stranded states slip through both recovery paths:

1. The sentinel never persisted — a prior setupWatch() / setupMailboxSubscription() threw before its set() (e.g. Gmail's 400 "Only one user push notification client allowed per developer"). There was nothing to re-assert, and because scheduleRecurring never ran, the sweep's ever marker was never set either.
2. The watch/subscription expired while the self-heal/renewal chain was dead.

Either way the connection stayed dead until the user manually re-enabled a channel.

Real-world hit

A Gmail "Personal" connection stopped ingesting after a June reconnect and produced zero mail for ~18 days across multiple deploys, while the same user's other Gmail connection (continuously healthy) was unaffected. The recovery paths shipped in #215 couldn't reach it because its watch state had never persisted / had expired with a dead chain.

Fix

upgrade() now always runs a recoverMailboxDelivery() backstop for any instance with enabled channels:

• Healthy watch (sentinel present and unexpired) → only re-assert the recurring tasks (unchanged behavior).
• Missing or expired watch → re-establish it and re-walk every enabled label/folder, so mail that accumulated while delivery was dead is backfilled. The backfill upserts by source (no duplicates) and uses initial-sync semantics (unread:false, archived:false), so it never spams notifications.

Gmail's legacy per-channel→mailbox migration is extracted into migrateLegacyPerChannelState(); the watch is then (re-)established by the shared recovery path (removing the old if (migratedAny) setupMailboxWebhook() tail).

Both connectors get the same shape (recoverMailboxDelivery + requeueInitialSync), since Outlook-mail had the identical gating gap (if (existing?.subscriptionId) re-assert, no else).

Why this fixes it for all affected users

It runs on every deploy for every non-archived instance (the deploy "Upgrading active twists" phase). Any Gmail/Outlook connection currently stranded with a missing/expired watch will re-establish live delivery and backfill its gap on the next deploy — no manual re-enable required.

Testing

• Gmail: 47 tests pass (4 new recoverMailboxDelivery cases: stranded, expired, healthy, no-channels)
• Outlook-mail: 35 tests pass (4 new equivalent cases)
• Both pnpm build (tsc) + plot lint clean
• Connector-only change → no changeset (per public/AGENTS.md)

Deploy / verify notes

• This is a public-submodule connector change. To reach production it needs the superproject (plotday/core) submodule pointer bumped to the merge commit, then a deploy (the deploy-connectors (gmail/outlook-mail) jobs).
• A connection that already self-healed its live watch (sentinel now fresh) takes the healthy path and is not re-backfilled; recovering its historical gap still needs a one-time channel re-enable (which triggers onChannelEnabled's full backfill). The auto-backfill here targets connections still stranded at deploy time.

🤖 Generated with Claude Code

[KrisBraun]

Problem

A mail connection (Gmail or Outlook) could be left permanently and silently dead with no self-recovery on deploy — the failure mode the durable-recurring-tasks work (#215) set out to eliminate, but with a gap.

upgrade() only re-asserted the recurring self-heal/renewal tasks when the watch/subscription sentinel (mailbox_webhook / mailbox_subscription) was already stored, and the cron maintenance sweep is gated on a recurring_meta.ever marker. Two stranded states slip through both recovery paths:

1. The sentinel never persisted — a prior setupWatch() / setupMailboxSubscription() threw before its set() (e.g. Gmail's 400 "Only one user push notification client allowed per developer"). There was nothing to re-assert, and because scheduleRecurring never ran, the sweep's ever marker was never set either.
2. The watch/subscription expired while the self-heal/renewal chain was dead.

Either way the connection stayed dead until the user manually re-enabled a channel.

Real-world hit

A Gmail "Personal" connection stopped ingesting after a June reconnect and produced zero mail for ~18 days across multiple deploys, while the same user's other Gmail connection (continuously healthy) was unaffected. The recovery paths shipped in #215 couldn't reach it because its watch state had never persisted / had expired with a dead chain.

Fix

upgrade() now always runs a recoverMailboxDelivery() backstop for any instance with enabled channels:

• Healthy watch (sentinel present and unexpired) → only re-assert the recurring tasks (unchanged behavior).
• Missing or expired watch → re-establish it and re-walk every enabled label/folder, so mail that accumulated while delivery was dead is backfilled. The backfill upserts by source (no duplicates) and uses initial-sync semantics (unread:false, archived:false), so it never spams notifications.

Gmail's legacy per-channel→mailbox migration is extracted into migrateLegacyPerChannelState(); the watch is then (re-)established by the shared recovery path (removing the old if (migratedAny) setupMailboxWebhook() tail).

Both connectors get the same shape (recoverMailboxDelivery + requeueInitialSync), since Outlook-mail had the identical gating gap (if (existing?.subscriptionId) re-assert, no else).

Why this fixes it for all affected users

It runs on every deploy for every non-archived instance (the deploy "Upgrading active twists" phase). Any Gmail/Outlook connection currently stranded with a missing/expired watch will re-establish live delivery and backfill its gap on the next deploy — no manual re-enable required.

Testing

• Gmail: 47 tests pass (4 new recoverMailboxDelivery cases: stranded, expired, healthy, no-channels)
• Outlook-mail: 35 tests pass (4 new equivalent cases)
• Both pnpm build (tsc) + plot lint clean
• Connector-only change → no changeset (per public/AGENTS.md)

Deploy / verify notes

• This is a public-submodule connector change. To reach production it needs the superproject (plotday/core) submodule pointer bumped to the merge commit, then a deploy (the deploy-connectors (gmail/outlook-mail) jobs).
• A connection that already self-healed its live watch (sentinel now fresh) takes the healthy path and is not re-backfilled; recovering its historical gap still needs a one-time channel re-enable (which triggers onChannelEnabled's full backfill). The auto-backfill here targets connections still stranded at deploy time.

🤖 Generated with Claude Code

[KrisBraun]

Problem

A mail connection (Gmail or Outlook) could be left permanently and silently dead with no self-recovery on deploy — the failure mode the durable-recurring-tasks work (#215) set out to eliminate, but with a gap.

upgrade() only re-asserted the recurring self-heal/renewal tasks when the watch/subscription sentinel (mailbox_webhook / mailbox_subscription) was already stored, and the cron maintenance sweep is gated on a recurring_meta.ever marker. Two stranded states slip through both recovery paths:

1. The sentinel never persisted — a prior setupWatch() / setupMailboxSubscription() threw before its set() (e.g. Gmail's 400 "Only one user push notification client allowed per developer"). There was nothing to re-assert, and because scheduleRecurring never ran, the sweep's ever marker was never set either.
2. The watch/subscription expired while the self-heal/renewal chain was dead.

Either way the connection stayed dead until the user manually re-enabled a channel.

Real-world hit

A Gmail "Personal" connection stopped ingesting after a June reconnect and produced zero mail for ~18 days across multiple deploys, while the same user's other Gmail connection (continuously healthy) was unaffected. The recovery paths shipped in #215 couldn't reach it because its watch state had never persisted / had expired with a dead chain.

Fix

upgrade() now always runs a recoverMailboxDelivery() backstop for any instance with enabled channels:

• Healthy watch (sentinel present and unexpired) → only re-assert the recurring tasks (unchanged behavior).
• Missing or expired watch → re-establish it and re-walk every enabled label/folder, so mail that accumulated while delivery was dead is backfilled. The backfill upserts by source (no duplicates) and uses initial-sync semantics (unread:false, archived:false), so it never spams notifications.

Gmail's legacy per-channel→mailbox migration is extracted into migrateLegacyPerChannelState(); the watch is then (re-)established by the shared recovery path (removing the old if (migratedAny) setupMailboxWebhook() tail).

Both connectors get the same shape (recoverMailboxDelivery + requeueInitialSync), since Outlook-mail had the identical gating gap (if (existing?.subscriptionId) re-assert, no else).

Why this fixes it for all affected users

It runs on every deploy for every non-archived instance (the deploy "Upgrading active twists" phase). Any Gmail/Outlook connection currently stranded with a missing/expired watch will re-establish live delivery and backfill its gap on the next deploy — no manual re-enable required.

Testing

• Gmail: 47 tests pass (4 new recoverMailboxDelivery cases: stranded, expired, healthy, no-channels)
• Outlook-mail: 35 tests pass (4 new equivalent cases)
• Both pnpm build (tsc) + plot lint clean
• Connector-only change → no changeset (per public/AGENTS.md)

Deploy / verify notes

• This is a public-submodule connector change. To reach production it needs the superproject (plotday/core) submodule pointer bumped to the merge commit, then a deploy (the deploy-connectors (gmail/outlook-mail) jobs).
• A connection that already self-healed its live watch (sentinel now fresh) takes the healthy path and is not re-backfilled; recovering its historical gap still needs a one-time channel re-enable (which triggers onChannelEnabled's full backfill). The auto-backfill here targets connections still stranded at deploy time.

🤖 Generated with Claude Code

[KrisBraun]

Problem

A mail connection (Gmail or Outlook) could be left permanently and silently dead with no self-recovery on deploy — the failure mode the durable-recurring-tasks work (#215) set out to eliminate, but with a gap.

upgrade() only re-asserted the recurring self-heal/renewal tasks when the watch/subscription sentinel (mailbox_webhook / mailbox_subscription) was already stored, and the cron maintenance sweep is gated on a recurring_meta.ever marker. Two stranded states slip through both recovery paths:

1. The sentinel never persisted — a prior setupWatch() / setupMailboxSubscription() threw before its set() (e.g. Gmail's 400 "Only one user push notification client allowed per developer"). There was nothing to re-assert, and because scheduleRecurring never ran, the sweep's ever marker was never set either.
2. The watch/subscription expired while the self-heal/renewal chain was dead.

Either way the connection stayed dead until the user manually re-enabled a channel.

Real-world hit

A Gmail "Personal" connection stopped ingesting after a June reconnect and produced zero mail for ~18 days across multiple deploys, while the same user's other Gmail connection (continuously healthy) was unaffected. The recovery paths shipped in #215 couldn't reach it because its watch state had never persisted / had expired with a dead chain.

Fix

upgrade() now always runs a recoverMailboxDelivery() backstop for any instance with enabled channels:

• Healthy watch (sentinel present and unexpired) → only re-assert the recurring tasks (unchanged behavior).
• Missing or expired watch → re-establish it and re-walk every enabled label/folder, so mail that accumulated while delivery was dead is backfilled. The backfill upserts by source (no duplicates) and uses initial-sync semantics (unread:false, archived:false), so it never spams notifications.

Gmail's legacy per-channel→mailbox migration is extracted into migrateLegacyPerChannelState(); the watch is then (re-)established by the shared recovery path (removing the old if (migratedAny) setupMailboxWebhook() tail).

Both connectors get the same shape (recoverMailboxDelivery + requeueInitialSync), since Outlook-mail had the identical gating gap (if (existing?.subscriptionId) re-assert, no else).

Why this fixes it for all affected users

It runs on every deploy for every non-archived instance (the deploy "Upgrading active twists" phase). Any Gmail/Outlook connection currently stranded with a missing/expired watch will re-establish live delivery and backfill its gap on the next deploy — no manual re-enable required.

Testing

• Gmail: 47 tests pass (4 new recoverMailboxDelivery cases: stranded, expired, healthy, no-channels)
• Outlook-mail: 35 tests pass (4 new equivalent cases)
• Both pnpm build (tsc) + plot lint clean
• Connector-only change → no changeset (per public/AGENTS.md)

Deploy / verify notes

• This is a public-submodule connector change. To reach production it needs the superproject (plotday/core) submodule pointer bumped to the merge commit, then a deploy (the deploy-connectors (gmail/outlook-mail) jobs).
• A connection that already self-healed its live watch (sentinel now fresh) takes the healthy path and is not re-backfilled; recovering its historical gap still needs a one-time channel re-enable (which triggers onChannelEnabled's full backfill). The auto-backfill here targets connections still stranded at deploy time.

🤖 Generated with Claude Code

[KrisBraun]

Problem

A mail connection (Gmail or Outlook) could be left permanently and silently dead with no self-recovery on deploy — the failure mode the durable-recurring-tasks work (#215) set out to eliminate, but with a gap.

upgrade() only re-asserted the recurring self-heal/renewal tasks when the watch/subscription sentinel (mailbox_webhook / mailbox_subscription) was already stored, and the cron maintenance sweep is gated on a recurring_meta.ever marker. Two stranded states slip through both recovery paths:

1. The sentinel never persisted — a prior setupWatch() / setupMailboxSubscription() threw before its set() (e.g. Gmail's 400 "Only one user push notification client allowed per developer"). There was nothing to re-assert, and because scheduleRecurring never ran, the sweep's ever marker was never set either.
2. The watch/subscription expired while the self-heal/renewal chain was dead.

Either way the connection stayed dead until the user manually re-enabled a channel.

Real-world hit

A Gmail "Personal" connection stopped ingesting after a June reconnect and produced zero mail for ~18 days across multiple deploys, while the same user's other Gmail connection (continuously healthy) was unaffected. The recovery paths shipped in #215 couldn't reach it because its watch state had never persisted / had expired with a dead chain.

Fix

upgrade() now always runs a recoverMailboxDelivery() backstop for any instance with enabled channels:

• Healthy watch (sentinel present and unexpired) → only re-assert the recurring tasks (unchanged behavior).
• Missing or expired watch → re-establish it and re-walk every enabled label/folder, so mail that accumulated while delivery was dead is backfilled. The backfill upserts by source (no duplicates) and uses initial-sync semantics (unread:false, archived:false), so it never spams notifications.

Gmail's legacy per-channel→mailbox migration is extracted into migrateLegacyPerChannelState(); the watch is then (re-)established by the shared recovery path (removing the old if (migratedAny) setupMailboxWebhook() tail).

Both connectors get the same shape (recoverMailboxDelivery + requeueInitialSync), since Outlook-mail had the identical gating gap (if (existing?.subscriptionId) re-assert, no else).

Why this fixes it for all affected users

It runs on every deploy for every non-archived instance (the deploy "Upgrading active twists" phase). Any Gmail/Outlook connection currently stranded with a missing/expired watch will re-establish live delivery and backfill its gap on the next deploy — no manual re-enable required.

Testing

• Gmail: 47 tests pass (4 new recoverMailboxDelivery cases: stranded, expired, healthy, no-channels)
• Outlook-mail: 35 tests pass (4 new equivalent cases)
• Both pnpm build (tsc) + plot lint clean
• Connector-only change → no changeset (per public/AGENTS.md)

Deploy / verify notes

• This is a public-submodule connector change. To reach production it needs the superproject (plotday/core) submodule pointer bumped to the merge commit, then a deploy (the deploy-connectors (gmail/outlook-mail) jobs).
• A connection that already self-healed its live watch (sentinel now fresh) takes the healthy path and is not re-backfilled; recovering its historical gap still needs a one-time channel re-enable (which triggers onChannelEnabled's full backfill). The auto-backfill here targets connections still stranded at deploy time.

🤖 Generated with Claude Code

[KrisBraun]

Problem

A mail connection (Gmail or Outlook) could be left permanently and silently dead with no self-recovery on deploy — the failure mode the durable-recurring-tasks work (#215) set out to eliminate, but with a gap.

upgrade() only re-asserted the recurring self-heal/renewal tasks when the watch/subscription sentinel (mailbox_webhook / mailbox_subscription) was already stored, and the cron maintenance sweep is gated on a recurring_meta.ever marker. Two stranded states slip through both recovery paths:

1. The sentinel never persisted — a prior setupWatch() / setupMailboxSubscription() threw before its set() (e.g. Gmail's 400 "Only one user push notification client allowed per developer"). There was nothing to re-assert, and because scheduleRecurring never ran, the sweep's ever marker was never set either.
2. The watch/subscription expired while the self-heal/renewal chain was dead.

Either way the connection stayed dead until the user manually re-enabled a channel.

Real-world hit

A Gmail "Personal" connection stopped ingesting after a June reconnect and produced zero mail for ~18 days across multiple deploys, while the same user's other Gmail connection (continuously healthy) was unaffected. The recovery paths shipped in #215 couldn't reach it because its watch state had never persisted / had expired with a dead chain.

Fix

upgrade() now always runs a recoverMailboxDelivery() backstop for any instance with enabled channels:

• Healthy watch (sentinel present and unexpired) → only re-assert the recurring tasks (unchanged behavior).
• Missing or expired watch → re-establish it and re-walk every enabled label/folder, so mail that accumulated while delivery was dead is backfilled. The backfill upserts by source (no duplicates) and uses initial-sync semantics (unread:false, archived:false), so it never spams notifications.

Gmail's legacy per-channel→mailbox migration is extracted into migrateLegacyPerChannelState(); the watch is then (re-)established by the shared recovery path (removing the old if (migratedAny) setupMailboxWebhook() tail).

Both connectors get the same shape (recoverMailboxDelivery + requeueInitialSync), since Outlook-mail had the identical gating gap (if (existing?.subscriptionId) re-assert, no else).

Why this fixes it for all affected users

It runs on every deploy for every non-archived instance (the deploy "Upgrading active twists" phase). Any Gmail/Outlook connection currently stranded with a missing/expired watch will re-establish live delivery and backfill its gap on the next deploy — no manual re-enable required.

Testing

• Gmail: 47 tests pass (4 new recoverMailboxDelivery cases: stranded, expired, healthy, no-channels)
• Outlook-mail: 35 tests pass (4 new equivalent cases)
• Both pnpm build (tsc) + plot lint clean
• Connector-only change → no changeset (per public/AGENTS.md)

Deploy / verify notes

• This is a public-submodule connector change. To reach production it needs the superproject (plotday/core) submodule pointer bumped to the merge commit, then a deploy (the deploy-connectors (gmail/outlook-mail) jobs).
• A connection that already self-healed its live watch (sentinel now fresh) takes the healthy path and is not re-backfilled; recovering its historical gap still needs a one-time channel re-enable (which triggers onChannelEnabled's full backfill). The auto-backfill here targets connections still stranded at deploy time.

🤖 Generated with Claude Code

[KrisBraun]

Problem

A mail connection (Gmail or Outlook) could be left permanently and silently dead with no self-recovery on deploy — the failure mode the durable-recurring-tasks work (#215) set out to eliminate, but with a gap.

upgrade() only re-asserted the recurring self-heal/renewal tasks when the watch/subscription sentinel (mailbox_webhook / mailbox_subscription) was already stored, and the cron maintenance sweep is gated on a recurring_meta.ever marker. Two stranded states slip through both recovery paths:

1. The sentinel never persisted — a prior setupWatch() / setupMailboxSubscription() threw before its set() (e.g. Gmail's 400 "Only one user push notification client allowed per developer"). There was nothing to re-assert, and because scheduleRecurring never ran, the sweep's ever marker was never set either.
2. The watch/subscription expired while the self-heal/renewal chain was dead.

Either way the connection stayed dead until the user manually re-enabled a channel.

Real-world hit

A Gmail "Personal" connection stopped ingesting after a June reconnect and produced zero mail for ~18 days across multiple deploys, while the same user's other Gmail connection (continuously healthy) was unaffected. The recovery paths shipped in #215 couldn't reach it because its watch state had never persisted / had expired with a dead chain.

Fix

upgrade() now always runs a recoverMailboxDelivery() backstop for any instance with enabled channels:

• Healthy watch (sentinel present and unexpired) → only re-assert the recurring tasks (unchanged behavior).
• Missing or expired watch → re-establish it and re-walk every enabled label/folder, so mail that accumulated while delivery was dead is backfilled. The backfill upserts by source (no duplicates) and uses initial-sync semantics (unread:false, archived:false), so it never spams notifications.

Gmail's legacy per-channel→mailbox migration is extracted into migrateLegacyPerChannelState(); the watch is then (re-)established by the shared recovery path (removing the old if (migratedAny) setupMailboxWebhook() tail).

Both connectors get the same shape (recoverMailboxDelivery + requeueInitialSync), since Outlook-mail had the identical gating gap (if (existing?.subscriptionId) re-assert, no else).

Why this fixes it for all affected users

It runs on every deploy for every non-archived instance (the deploy "Upgrading active twists" phase). Any Gmail/Outlook connection currently stranded with a missing/expired watch will re-establish live delivery and backfill its gap on the next deploy — no manual re-enable required.

Testing

• Gmail: 47 tests pass (4 new recoverMailboxDelivery cases: stranded, expired, healthy, no-channels)
• Outlook-mail: 35 tests pass (4 new equivalent cases)
• Both pnpm build (tsc) + plot lint clean
• Connector-only change → no changeset (per public/AGENTS.md)

Deploy / verify notes

• This is a public-submodule connector change. To reach production it needs the superproject (plotday/core) submodule pointer bumped to the merge commit, then a deploy (the deploy-connectors (gmail/outlook-mail) jobs).
• A connection that already self-healed its live watch (sentinel now fresh) takes the healthy path and is not re-backfilled; recovering its historical gap still needs a one-time channel re-enable (which triggers onChannelEnabled's full backfill). The auto-backfill here targets connections still stranded at deploy time.

🤖 Generated with Claude Code