imaplib: must not replace LF or CR by CRLF in literals

[memeplex]

BPO	5430
Nosy	@warsaw, @mcepl, @bitdancer, @rduplain, @JulienPalard, @csabella
PRs	bpo-5430: Preservation LF, CR in IMAP4 append method. #10901
Files	test_imaplib_cr_lf.diff: an attempt at testing IMAP4.append for CR, LF preservation

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2009-03-06.07:51:15.737>
labels = ['easy', 'type-bug', '3.8', 'expert-email', '3.10', 'library', '3.9']
title = 'imaplib: must not replace LF or CR by CRLF in literals'
updated_at = <Date 2020-11-07.01:52:35.851>
user = 'https://bugs.python.org/memeplex'

bugs.python.org fields:

activity = <Date 2020-11-07.01:52:35.851>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2009-03-06.07:51:15.737>
creator = 'memeplex'
dependencies = []
files = ['13789']
hgrepos = []
issue_num = 5430
keywords = ['patch', 'easy']
message_count = 8.0
messages = ['83241', '86572', '86602', '94578', '315575', '315577', '316170', '352292']
nosy_count = 8.0
nosy_names = ['barry', 'mcepl', 'r.david.murray', 'memeplex', 'ron.duplain', 'rajeshsr', 'mdk', 'cheryl.sabella']
pr_nums = ['10901']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5430'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-49680: Don't rewrite CR/LF in imaplib APPEND literals #151557
• gh-49680: Add translate_line_endings parameter to imaplib.IMAP4.append #152775
• [3.15] gh-49680: Test imaplib.IMAP4.append line-ending normalization #152877
• [3.14] gh-49680: Test imaplib.IMAP4.append line-ending normalization (GH-152877) #152878
• [3.13] gh-49680: Test imaplib.IMAP4.append line-ending normalization #152879

[memeplex]

For example, after that "normalization", quoted printable encoded
headers (as described at rfc 2047) longer than 76 characters are
splitted in two different ill-formed headers because the soft LF line
break becomes a "hard" CRLF one. This is clearly wrong.

rfc 2060 specifically allows CR and LF inside literals:

"""
A literal is a sequence of zero or more octets (including CR and LF),
prefix-quoted with an octet count in the form of an open brace ("{"),
the number of octets, close brace ("}"), and CRLF.
"""

[rduplain]

It looks like the IMAP4.append method is responsible for the CRLF
substitution (trunk/Lib/imaplib.py).

# defined near top of module:
MapCRLF = re.compile(r'\r\n|\r|\n')

# in append method:
self.literal = MapCRLF.sub(CRLF, message)

I'll work on a test for it this evening.

-Ron

[rduplain]

Module imaplib has pretty sparse test code. There is only 1 test case,
for imaplib.Time2Internaldate.
trunk/Lib/test/test_imaplib.py

The attached patch tests for LF, CR preservation with regard to the
IMAP4.append method, but more testing is necessary to make sure LF, CR
are preserved in IMAP literals throughout imaplib and that the
respective fix doesn't break anything. Mock IMAP interaction may be the
way to go for better test cases. Is anyone working on expanding imaplib
test coverage?

-Ron

[rajeshsr]

Am quite new here; just searching hard to contribute, would like to
patch this, if I can go ahead.

Don't we need to patch the original imaplib code also?

just remove the line:
self.literal = MapCRLF.sub(CRLF, message)
and have:
self.literal = message
Or am I missing something completely?

[mcepl]

Module imaplib has pretty sparse test code.

On that note: would anybody be willing to express in form of (simplified) test case, what's the problem reported here? I am not sure, I follow.

[mcepl]

Oh, this is 2.6 bug. This should be closed.

[csabella]

I'm not sure if this is a 2.6/2.7 only issue because the code mentioned in msg86572 still exists, although with a change for bpo-21800 (RFC 6855/UTF-8) applied.

For reference, bpo-25591 expanded the test coverage for the imaplib. However, I don't see any tests related to CR or LF in literals.

[JulienPalard]

It looks like a revert of 47404ff
Which is the fix for https://bugs.python.org/issue723962 which I'm currently reading.