imaplib: incorrect quoting in commands

[anadelonbrin]

BPO	917120
Nosy	@warsaw, @bitdancer, @soltysh, @xtsimpouris
Files	adjust-mustquote-re-in-imaplib.patch: exactly what dmbaggett described

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2004-03-16.06:36:58.000>
labels = ['easy', 'type-bug', 'library', 'expert-email']
title = 'imaplib: incorrect quoting in commands'
updated_at = <Date 2021-12-18.16:46:27.029>
user = 'https://bugs.python.org/anadelonbrin'

bugs.python.org fields:

activity = <Date 2021-12-18.16:46:27.029>
actor = 'xtsimpouris'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2004-03-16.06:36:58.000>
creator = 'anadelonbrin'
dependencies = []
files = ['18367']
hgrepos = []
issue_num = 917120
keywords = ['patch', 'easy']
message_count = 11.0
messages = ['20244', '20245', '87655', '87656', '112741', '112976', '114330', '181950', '181957', '181977', '408860']
nosy_count = 10.0
nosy_names = ['barry', 'anadelonbrin', 'dmbaggett', 'r.david.murray', 'meatballhat', 'maciej.szulik', 'Mauro.Cicognini', 'dveeden', 'bjshan', 'xtsimpouris']
pr_nums = []
priority = 'low'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue917120'
versions = ['Python 3.3']

Linked PRs

• gh-40038: Quote imaplib command arguments when necessary #152703
• [3.15] gh-40038: Quote imaplib command arguments when necessary (GH-152703) #153066
• [3.14] gh-40038: Quote imaplib command arguments when necessary (GH-152703) #153069
• [3.13] gh-40038: Quote imaplib command arguments when necessary (GH-152703) #153071

[anadelonbrin]

imaplib incorrectly chooses to quote some arguments.

In particular, doing "UID FETCH # BODY.PEEK[]" results
in the BODY.PEEK[] being quoted, and it should not
(according to the RFC), which means the command fails.
This is demonstrated below. It's possible (and
likely) that other UID FETCH arguments are incorrectly
quoted.

This occurs with anon cvs python of 16/3/04, and 2.3.3.
Windows XP SP1.

I'm happy to provide more info if required, just let me
know. I could try and work up a patch, but it would be
better from someone really familiar with imaplib so
that I don't screw up legitimate quoting.

>>> import imaplib
>>> i = imaplib.IMAP4("server")
>>> i.login("username", "password")
('OK', ['LOGIN Ok.'])
>>> i.select()
('OK', ['38'])
>>> i.debug = 4
>>> i.uid("FETCH", "96", "BODY")
 29:14.23 > GKGP7 UID FETCH 96 BODY
 29:14.40 < * 31 FETCH (UID 96 BODY (("text" "plain"
("charset" "iso-8859-1") NIL NIL "quoted-printable" 32
0)("text" "html" ("charset" "iso-8859-1") NIL NIL
"quoted-printable" 368 10) "alternative"))
 29:14.40 < GKGP7 OK FETCH completed.
('OK', ['31 (UID 96 BODY (("text" "plain" ("charset"
"iso-8859-1") NIL NIL "quoted-printable" 32 0)("text"
"html" ("charset" "iso-8859-1") NIL NIL
"quoted-printable" 368 10) "alternative"))'])
>>> i.uid("FETCH", "96", "BODY.PEEK[]")
 29:17.04 > GKGP8 UID FETCH 96 "BODY.PEEK[]"
 29:17.21 < GKGP8 NO Error in IMAP command received by
server.
 29:17.21 NO response: Error in IMAP command received
by server.
('NO', ['Error in IMAP command received by server.'])
>>> i.logout()
 29:31.26 > GKGP9 LOGOUT
 29:31.42 < * BYE Courier-IMAP server shutting down
 29:31.42 BYE response: Courier-IMAP server shutting down
 29:31.42 < GKGP9 OK LOGOUT completed
('BYE', ['Courier-IMAP server shutting down'])
>>>

[anadelonbrin]

Logged In: YES
user_id=552329

Sorry, I missed the bit in the docs that points out that
stuff is always quoted and that using () avoids it. Still,
it does seem that imaplib would be doing it's job better if
it followed correct quoting, rather than always quoting. It
would certainly be easier to use for people familiar with
IMAP, but unfamiliar with imaplib.

[dmbaggett]

I'm not sure this causes the behavior reported here, but I believe there
really is a bug in imaplib.

In particular, it seems wrong to me that this line:

mustquote = re.compile(r"[^\w!#$%&'*+,.:;<=>?^`|~-]")

has \w in it. Should that be \s?

I found this when I noticed that SELECT commands on mailboxes with
spaces in their names failed.

[dmbaggett]

OK, I missed the initial caret in the regex. The mustquote regex is
listing everything that needn't be quoted, and then negating. I still
think it's wrong, though. According to BNF given in the Formal Syntax
section of RFC 3501, you must must quote atom-specials, which are
defined thus:

atom-specials = "(" / ")" / "{" / SP / CTL / list-wildcards /
quoted-specials / resp-specials
list-wildcards = "%" / "*"
quoted-specials = DQUOTE / "\"
resp-specials = "]"

So I think this regex should do it:

mustquote = re.compile(r'[()\s%*"]|"{"|"\\"|"\]"')

Changing status to bug.

[meatballhat]

I'm attaching a patch which does exactly what dmbaggett recommended w.r.t. the mustquote regex. All current tests pass, but I'm not sure if the current tests even cover this code (how is coverage measured in the stdlib tests?)

On a related note, the _checkquote method which uses the mustquote regex is dead code in Python 3.2+, AFAICT.