Uh oh!
There was an error while loading. Please reload this page.
fix(google): read calendar parts Gmail stores as attachments - #355
Merged
Conversation
Gmail does not leave a message's calendar part inline. It treats the part as an attachment — synthesizing a filename such as `invite.ics` and moving the body out to `body.attachmentId` — so `messages.get` returns the part with no `data` to decode. The connector looked only for an inline body and explicitly skipped attachment parts, so it never found the iCalendar content of any invitation, update, cancellation or reply. Two behaviours depended on that content and were inert as a result: folding an attendee's response onto the event's thread, and bundling update and cancellation mail onto the event instead of a thread of its own. Read the calendar body in a separate step that fetches the attachment when the body is stored separately, and hand the result to the parsers, which stay pure and synchronous. Part detection also widens: senders deliver the part as `application/ics` about as often as `text/calendar`, and some label it `application/octet-stream`, where only the `.ics` filename identifies it. Ordinary mail costs nothing — a message with no calendar part issues no request — and a failed fetch degrades to syncing the conversation as plain email. The tests had built an inline, filename-less `text/calendar` part, a shape Gmail never produces, which is why they passed against code that could not work. Fixtures now use the real attachment shape and serve the body from a stubbed attachments endpoint.
An attendee response is folded onto the event's thread by addressing it as `icaluid:<uid>`. When the calendar has not synced that event yet the fold misses, and the response was left to sync as an ordinary email thread — a standalone "Accepted: …" that the fold exists to avoid. The window is small but real: an event created moments before someone responds to it. Retrying alone does not fix this. By the time the event arrives the email thread has already been saved, so a late fold has to retract it as well. Track a conversation whose responses all missed, retry it on later sync passes, and once every response has reached the event archive the email thread the responses no longer belong in. Only conversations that are nothing but responses are tracked, so archiving can never hide real correspondence — a conversation carrying a human reply keeps its thread and is left alone. The retry re-fetches and re-parses the conversation rather than storing a parsed response, so it always reflects the current state of the mail and shares one code path with first-pass sync. Entries stop being retried after a week; the email thread then simply stays as it is, so a response is never lost. Nothing pending costs one storage list per pass.
…t pass The retry omitted `unread` entirely, so a response folded late defaulted to unread — including an acceptance, which the first pass deliberately leaves read, and including responses first seen during the initial backfill, where nothing should be marked unread at all. A late fold was therefore noisier than a timely one. Carry the originating pass's `initialSync` on the pending entry and run the same `shouldMarkUnread` rule on retry.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Gmail does not leave a message's calendar part inline. It treats the part as an attachment — synthesizing a filename such as
invite.icsand moving the body out tobody.attachmentId— somessages.getreturns that part with nodatato decode. Per the Gmail API reference, whenattachmentIdis present the content is not indata; it has to be fetched withmessages.attachments.get.The connector looked only for an inline body, and explicitly skipped any part that had a filename and an attachment id — precisely the shape every real calendar part has. It therefore never found the iCalendar content of an invitation, update, cancellation or reply.
Two behaviours depended on that content and were inert as a result:
Fix
resolveIcsByMessage) that fetches the attachment when the body is stored separately, and hand the result toclassifyCalendarThread/extractCalendarReplies, which stay pure and synchronous.application/icsabout as often astext/calendar, and some label itapplication/octet-stream, where only the.icsfilename identifies it. Exact-matchingtext/calendaralone would still have missed a large share of real mail.Ordinary mail costs nothing: a message with no calendar part issues no request, and a batch with no calendar mail does not even resolve an API client. A failed fetch leaves the message out and the conversation syncs as plain email.
Follow-up commit: retry a response whose event had not synced yet
A response is folded by addressing the event as
icaluid:<uid>. If the calendar has not synced that event yet the fold misses and the response syncs as a standalone email thread — the very thing the fold avoids. The window is small but real: an event created moments before someone responds.Retrying alone does not resolve it, because by the time the event arrives the email thread has already been saved; a late fold has to retract it too. So a conversation whose responses all missed is tracked, retried on later passes, and once every response has reached the event the leftover email thread is archived.
Only conversations that are nothing but responses are tracked, so archiving can never hide real correspondence — a conversation carrying a human reply keeps its thread and is left alone. The retry re-fetches and re-parses the conversation rather than storing a parsed response, so it shares one code path with first-pass sync. Entries stop being retried after a week, leaving the email thread as-is, so a response is never lost. Nothing pending costs one storage list per pass.
Tests
The existing fixtures built an inline, filename-less
text/calendarpart — a shape Gmail never produces — which is why they passed against code that could not work. Fixtures now use the real attachment shape and serve the body from a stubbed attachments endpoint, so the sync tests exercise the same two-step read the connector performs at runtime.Each new behaviour was verified by sabotage — breaking the resolver fails the eight calendar tests; dropping the responses-only guard fails the test that protects real correspondence — so the tests genuinely guard the paths rather than passing vacuously. Full connector suite: 365 tests passing.
No new OAuth scope:
gmail.modifyalready permitsmessages.attachments.get, which the connector uses for ordinary attachment downloads.