Email demo mbox by bmerkle · Pull Request #198 · microsoft/typeagent-py

Bernhard Merkle (bmerkle) · 2026-02-15T22:53:52Z

added mbox handling

major refactoring
added mbox handling
added filters for first n, or last n mails of mbox
added filters for selecting mails with a date range, after or before
use verbose option to debug ingestion in detail
hardened mbox reading to cope with also invalid entries and other codecs, e.g.

testdata which i am using:

currently ingested fwts and bazzaar mbox.
unbunt mbox contain several 10k of mails hence we shall use selectors there (e.g. date ranges)

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 13/02/2026 21:18 2140627 bazaar-announce.mbox
-a--- 13/02/2026 21:55 648468 fwts-announce.mbox
-a--- 13/02/2026 20:19 269515931 kubuntu-users.mbox
-a--- 13/02/2026 20:18 195869391 ubuntu-devel.mbox

- hardened mbox reading to cope with also invalid entries and other codecs, e.g. - ubuntu-devel.mbox and kubuntu-users.mbox contained kind of special mails, which resulted in errors without this hardening first.

- added mbox handling - added filters for first n, or last n mails of mbox - added filters for selecting mails with a date range, after or before - use verbose option to debug ingestion in detail

Copilot

Pull request overview

This PR adds mbox file support to the email ingestion tool, enabling processing of mbox mailbox files in addition to individual .eml files. The changes include filtering capabilities (first/last N emails, date ranges) and hardened encoding/header handling for robustness with real-world email data.

Changes:

Added mbox file format support with index-based and date-based filtering
Refactored argument parsing to use mutually exclusive groups for --eml and --mbox sources
Enhanced email header handling to properly coerce Header objects to strings

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 7 comments.

File	Description
tools/ingest_email.py	Added mbox support, filtering options (--first/--last/--after/--before), refactored email iteration logic into _iter_emails function
src/typeagent/emails/email_import.py	Added import_emails_from_mbox and count_emails_in_mbox functions, improved header type handling with _header_to_str, enhanced encoding error handling
.gitignore	Added /tests/testdata/email-mbox to ignore test data files

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

more memory efficient for large mbox Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

- added testcases

Two changes: 1. Default charset: or "utf-8" → or "latin-1" — when no charset header is present, latin-1 preserves all bytes. 2. LookupError fallback: same switch to "latin-1" (no errors param needed since latin-1 never fails on any byte value).

Guido van Rossum (gvanrossum)

I haven't looked at the tests yet, but let's first get these comments out of the way. :-)

- mbox are expanded to eml - removed test data - fixed argument handling dates - removed first/last argument handling and replaced with limit etc

Bernhard Merkle (bmerkle) · 2026-02-17T11:59:03Z

we could also remove the --eml switch again, because it is now the only way to import

Guido van Rossum (gvanrossum)

Somehow I can't seem to view test_mbox.py on this flight. I'll look into it later.

Guido van Rossum (gvanrossum) · 2026-02-17T15:49:48Z

we could also remove the --eml switch again, because it is now the only way to import

Yes please.

Guido van Rossum (gvanrossum)

Here's the review for test_mbox.py.

- Updated .gitignore to reflect new email test data structure. - Modified demos.md to clarify usage of email ingestion tools and updated command-line arguments. - Changed references from `tools/gmail/` to `tools/mail/` in documentation and scripts. - Refactored test cases in test_mbox.py to use new date filtering parameters: --start-date and --stop-date. - Updated ingest_email.py to replace --after and --before with --start-date and --stop-date for date filtering. - Added new tools for downloading Gmail and Outlook emails as .eml files: gmail_dump.py and outlook_dump.py. - Introduced mbox_dump.py for extracting emails from mbox files into individual .eml files.

…in date filtering tests

Bernhard Merkle (bmerkle) · 2026-02-17T19:50:05Z

all review points should be implemented now

Guido van Rossum (gvanrossum)

Bunch more nits. Getting pretty close!

I don't have the courage to look at outlook_dump.py yet. :-)

- improve date parsing logic - add option for output-dir similar to gmail client - remove outlook-dump.py as this is handeled in microsoft#199 (copilot added the file arbitraritly, grrr.)

Bernhard Merkle (bmerkle) · 2026-02-19T10:41:52Z

outlook_dump.py

Copilot added the file arbitrarily and i did not notice it during commit.
Is removed now. we look at this in #199.

Guido van Rossum (gvanrossum)

LGTM

## Summary - Document the half-open `[start, stop)` interval convention for date/time range handling in AGENTS.md (agreed in #198) - Bump pyright dependency to 1.1.411 and widen the `uv_build` requirement to `<0.12.0` - Minor import reordering in `src/typeagent/knowpro/answers.py` ## Test plan - [ ] `make check` / `pytest` pass with the updated pyright version

…op)) A search time range whose stop_date carries no time compiled to that day's midnight, so "Jan 1 to Jan 5" became [Jan 1 00:00, Jan 5 00:00) and every message on Jan 5 was silently excluded. Reported in PR microsoft#296. Both timestamp index backends already filter half-open (get_in_range: "End is exclusive"; SQLite: start_timestamp >= ? AND start_timestamp < ?), so the fix is to make the compiled stop the exclusive bound the storage layer expects: for a bare date, midnight of the following day. A stop with an explicit time is already exclusive and is kept as-is. This follows the [start, stop) convention agreed in PR microsoft#198 and documented in AGENTS.md, rather than padding an inclusive end to 23:59:59.999999. Also fixes two boundary inconsistencies this uncovered: - DateRange documented and implemented its end as inclusive (start <= dt <= end) while both storage backends excluded it. It is now documented and implemented as half-open, so the two scope-filtering paths agree: with a timestamp index (lookup_range) and without one (get_text_range_for_date_range, which uses __contains__). - get_time_range_for_conversation builds an inclusive-looking end from the last message's timestamp; documented as prompt-display only, not for index lookups. get_enclosing_date_range_for_text_range already used the exclusive end ordinal's timestamp, so it needed no change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@LuShadowX

…op)) (#309) ## The bug `date_range_from_datetime_range` compiled a `stop_date` with no time to that day's midnight, so a query like "Jan 1 to Jan 5" became `[Jan 1 00:00, Jan 5 00:00)` and every message on Jan 5 was silently dropped. Reproduced on `main` before the fix: ``` compiled DateRange: DateRange(2023-01-01 00:00+00:00, 2023-01-05 00:00+00:00) index hits: ['2023-01-01T00:00:00+00:00', '2023-01-04T09:30:00+00:00'] # dropped: 2023-01-05T00:00, 2023-01-05T09:30, 2023-01-05T23:59 ``` This is the bug reported by @LuShadowX in #296 — credit for finding it goes to them. That PR fixed it by padding the (then inclusive) end to `23:59:59.999999`; this PR fixes it with the half-open `[start, stop)` convention agreed in #198 and documented in AGENTS.md instead. ## The fix Three source edits 1. searchlang.py — new exclusive_stop_from_date_time(); a stop_date with no time now rolls to next-day midnight. One if, plus docstrings. 2. interfaces_core.py — DateRange.end comment inclusive → exclusive, and __contains__ <= → <. Plus a docstring. 3. convutils.py — docstring only, no code. Tests: one regression test for the bug, unit tests for the new helper, two DateRange.__contains__ tests, and test_start_and_stop updated to expect Jan 1 instead of Dec 31. Full `make` is green: isort/black clean, pyright clean, 756 tests pass, build OK. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Bernhard Merkle (bmerkle) added 4 commits February 15, 2026 23:33

- added mbox handling
5d0a61d
- hardened mbox reading to cope with also invalid entries and other codecs, e.g. - ubuntu-devel.mbox and kubuntu-users.mbox contained kind of special mails, which resulted in errors without this hardening first.

- major refactoring
47d34cb
- added mbox handling - added filters for first n, or last n mails of mbox - added filters for selecting mails with a date range, after or before - use verbose option to debug ingestion in detail

fix and update docs
853d1b9

fixed format
89dc931

CopilotAI review requested due to automatic review settings February 15, 2026 22:53

Copilotstarted reviewing on behalf of Bernhard Merkle (bmerkle)February 15, 2026 22:54 View session

CopilotAI reviewed Feb 15, 2026
View reviewed changes

Bernhard Merkle (bmerkle)and others added 5 commits February 16, 2026 00:01

Update tools/ingest_email.py
38e6282
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/typeagent/emails/email_import.py
a3d2002
more memory efficient for large mbox Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

added test data for small mbox
85b14b1

- fixed copilot reviews
b2028a2
- added testcases

semantic of before is usually inclusive, so >= is ok
8947349

Guido van Rossum (gvanrossum) reviewed Feb 16, 2026
View reviewed changes

Comment threadsrc/typeagent/emails/email_import.py Outdated

Guido van Rossum (gvanrossum) reviewed Feb 16, 2026
View reviewed changes

Bernhard Merkle (bmerkle) added 3 commits February 16, 2026 20:50

removed 2 mbox data files
2a0335e

- implemented review feedback from Guido
969da13
- mbox are expanded to eml - removed test data - fixed argument handling dates - removed first/last argument handling and replaced with limit etc

fix: update email preview length in verbose output
360f7d7

Guido van Rossum (gvanrossum) reviewed Feb 17, 2026
View reviewed changes

Comment threadtests/test_mbox.py Outdated
Comment threadtests/test_mbox.py Outdated
Comment threadtests/test_mbox.py Outdated
Comment threadtests/test_mbox.py Outdated

Bernhard Merkle (bmerkle) added 4 commits February 17, 2026 20:29

moved email date filtering logic and update related tests
62f7cb9

fix: remove carriage returns from email test cases for consistency
56184d7

fix: adjust naive timestamp handling to ensure consistent UTC offset …
eefc8d7
…in date filtering tests

Guido van Rossum (gvanrossum) reviewed Feb 18, 2026
View reviewed changes

Comment threadsrc/typeagent/emails/email_import.py Outdated
Comment threadsrc/typeagent/emails/email_import.py
Comment threadtools/mail/mbox_dump.py
Comment threadtools/ingest_email.py Outdated
Comment threadtools/ingest_email.py Outdated

Guido van Rossum (gvanrossum) mentioned this pull request Feb 19, 2026
Add outlook email dump tool #199
Merged

- enhance email header handling
892480d
- improve date parsing logic - add option for output-dir similar to gmail client - remove outlook-dump.py as this is handeled in microsoft#199 (copilot added the file arbitraritly, grrr.)

Guido van Rossum (gvanrossum) approved these changes Feb 21, 2026
View reviewed changes

Guido van Rossum (gvanrossum) merged commit 3b6e29b into microsoft:mainFeb 21, 2026
14 checks passed

This was referenced Aug 1, 2026
Fix date-only range end dropping the final day #296
Closed
Update AGENTS.md date/time guidance, bump pyright, minor cleanup #297
Merged

Bernhard Merkle (bmerkle) mentioned this pull request Aug 4, 2026
Fix date-only range stop dropping the final day (half-open [start, stop)) #309
Merged

Uh oh!

Conversation

Bernhard Merkle (bmerkle) commented Feb 15, 2026• edited by gvanrossum Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

added mbox handling

testdata which i am using:

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Guido van Rossum (gvanrossum) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Bernhard Merkle (bmerkle) commented Feb 17, 2026

Uh oh!

Guido van Rossum (gvanrossum) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Guido van Rossum (gvanrossum) commented Feb 17, 2026

Uh oh!

Guido van Rossum (gvanrossum) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Bernhard Merkle (bmerkle) commented Feb 17, 2026

Uh oh!

Guido van Rossum (gvanrossum) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Bernhard Merkle (bmerkle) commented Feb 19, 2026

Uh oh!

Guido van Rossum (gvanrossum) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Bernhard Merkle (bmerkle) commented Feb 15, 2026•
edited by gvanrossum
Loading