mail message parsing glitch

[mcspang]

BPO	1590744
Nosy	@warsaw, @bitdancer, @akheron
Dependencies	bpo-14731: Enhance Policy framework in preparation for adding email6 policies as provisional
Files	mbox: mailbox preserve_leading_whitespace.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2006-11-05.09:21:04.000>
labels = ['type-bug', 'expert-email']
title = 'mail message parsing glitch'
updated_at = <Date 2012-06-18.08:03:55.567>
user = 'https://bugs.python.org/mcspang'

bugs.python.org fields:

activity = <Date 2012-06-18.08:03:55.567>
actor = 'petri.lehtinen'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['email']
creation = <Date 2006-11-05.09:21:04.000>
creator = 'mcspang'
dependencies = ['14731']
files = ['2204', '21370']
hgrepos = []
issue_num = 1590744
keywords = ['patch']
message_count = 4.0
messages = ['30445', '30446', '131976', '161476']
nosy_count = 4.0
nosy_names = ['barry', 'mcspang', 'r.david.murray', 'petri.lehtinen']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1590744'
versions = ['Python 3.3']

[mcspang]

There's something wrong with the handling of line
continuation in headers.

In the attached mbox the 'Message-id' header is read
and stored with a leading space. So message_id = '
<9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>'
when it's first read.

If you write this message out to a new mbox, it is
written with the leading space and without the newline.

THEN, if you read it in AGAIN, the parser ignores the
leading space.

One of these steps is buggy, I'm not sure which. It
seems to me that the value returned by
msg['Message_id'] should not change when the file is
written then re-read. In the initial read and final
reads above the value differs by a space.

[warsaw]

This makes some sense, although it may not be ideal. The "leading space" is really an RFC 2822 continuation line, so that whitespace at the start of the second line is folding whitespace. Technically those two lines should (and do) get collapsed.

The question is whether this message should be printed idempotently. In general the email package tries hard to maintain this principle, so I think it's probably a valid bug. However, it doesn't affect the semantics of the message so it's not a high priority fix.

[bitdancer]

I needed an airplane-trip-sized problem to work on on the way back from PyCon and the sprints, so I tried my hand at "fixing" this. The attached patch is really just a proof of concept. Because it is so invasive of the email package machinery I doubt that I will apply it, but it does serve to prove that it is quite practical, given the right design, to preserve the leading whitespace in message headers, and this does enable the email package to read and write the messages in the sample mbox without changing them. I will incorporate what I learned from this exercise into the header management in email6.

On the other hand, if anyone else thinks this *is* worth tidying up an applying I could be convinced.

Note that after this patch one test fails, but that test failure is actually a buggy test that hides a bug in the header formatter (a failure to provide folding white space at the start of a continuation line). That bug I may revisit.

[bitdancer]

I think I can actually fix this once the patch in bpo-14731 is applied.