Email MIME `base64` parser stops at the end of the first padding

[spencerwuwu]

Bug report

Bug description:

Hi Cpython Developers,

I was testing and comparing different email parsers, and found a parsing discrepancy that seems to be a problem.

MIME-Version: 1.0
Content-Type: application/zip
Content-Disposition: attachment; filename=archive.zip
Content-Transfer-Encoding: base64

UEsDBBQAAAAIAA==
emVkIGZpbGUgY29udGVudA==

With the python's email get_payload method, the return content would stopped at the first "==" as it seems to be the default behavior of base64.b64decode.
Meanwhile, peer implementations (e.g. apache.commons.mal (java), MimeKit (c#), PhpMimeMailParser (php)) will return the whole content.

Below is an running example in python.

import base64
import email

"""
 Parsing the mime format
"""
request = """MIME-Version: 1.0
Content-Type: application/zip
Content-Disposition: attachment; filename=archive.zip
Content-Transfer-Encoding: base64

UEsDBBQAAAAIAA==
emVkIGZpbGUgY29udGVudA==
"""
msg = email.message_from_string(request)
print("Part content:", repr(msg.get_payload(decode=True)))
print()

"""
 Examples of base64
"""
contents = [
 "UEsDBBQAAAAIAA==\nemVkIGZpbGUgY29udGVudA==",
 "UEsDBBQAAAAIAA==emVkIGZpbGUgY29udGVudA==",
 "UEsDBBQAAAAIAA=emVkIGZpbGUgY29udGVudA==",
 "UEsDBBQAAAAIAAemVkIGZpbGUgY29udGVudA==",
 "UEsDBBQAAAAIAA==",
 "emVkIGZpbGUgY29udGVudA=="
 ]
for content in contents:
 decoded_bytes = base64.b64decode(content)
 print(repr(content), " ->")
 print(" ", decoded_bytes)

Output:

Part content: b'PK\x03\x04\x14\x00\x00\x00\x08\x00'

'UEsDBBQAAAAIAA==\nemVkIGZpbGUgY29udGVudA==' ->
 b'PK\x03\x04\x14\x00\x00\x00\x08\x00'
'UEsDBBQAAAAIAA==emVkIGZpbGUgY29udGVudA==' ->
 b'PK\x03\x04\x14\x00\x00\x00\x08\x00'
'UEsDBBQAAAAIAA=emVkIGZpbGUgY29udGVudA==' ->
 b'PK\x03\x04\x14\x00\x00\x00\x08\x00\x07\xa6VB\x06f\x96\xc6R\x066\xf6\xe7FV\xe7@'
'UEsDBBQAAAAIAAemVkIGZpbGUgY29udGVudA==' ->
 b'PK\x03\x04\x14\x00\x00\x00\x08\x00\x07\xa6VB\x06f\x96\xc6R\x066\xf6\xe7FV\xe7@'
'UEsDBBQAAAAIAA==' ->
 b'PK\x03\x04\x14\x00\x00\x00\x08\x00'
'emVkIGZpbGUgY29udGVudA==' ->
 b'zed file content'

Thank you,
Wei-Cheng

CPython versions tested on:

3.15

Operating systems tested on:

Linux

[picnixz]

What about Perl? what about the RFC? the email module tries to stick to the RFC when possible so if it says "well stop when you should," then I wouldn't consider this a bug but a feature

[spencerwuwu]

Thanks @picnixz and yes I should have checked the rfc as well.

First, I checked that Perl and GNU's base64 will decode both lines. In Python, by calling msg.get_payload(decode=False)) we will also get the whole payload. So I would say if it's left for the user to decode, in most cases they will get the whole decoded content.

In the rfc, it's slightly ambiguous. I personally consider ignoring the first few = a better way as this is expected in other software, perfectly fine to consider this as either a bug or a feature. Seems to be more of a base64.b64decode usage problem by reading more.

Quoting some relevant contents from RFC2045 Sec6.8:

All line breaks or other characters not
found in Table 1 must be ignored by decoding software. In base64
data, characters other than those in Table 1, line breaks, and other
white space probably indicate a transmission error, about which a
warning message or even a message rejection might be appropriate
under some circumstances.

Because it is used only for padding at the end of the data, the
occurrence of any "=" characters may be taken as evidence that the
end of the data has been reached (without truncation in transit). No
such assurance is possible, however, when the number of octets
transmitted was a multiple of three and no "=" characters are
present.

[picnixz]

cc @bitdancer

[bitdancer]

Regardless of whether or not the email is out of spec with the RFC (I could argue it either way), the email module tries its best to recover from sender errors when a reasonable guess is possible, as it is in this case. Strictly speaking this isn't a bug, but considering that other email parsers do decode the entire content, I would treat it as a bug. It looks like this affects both the old API (get_payload) and the new API (get_content).

Question: what do the other email clients do with the newlines? Do they elide them, or do they preserve them? I would expect they would be discarded, but I can see the argument that the sender probably meant them to be preserved, so I'd prefer we follow de-facto practice here.

Now, is it a bug in the base64 module as well? Probably not, but it could be argued.

[serhiy-storchaka]

There is a bug in the Base64 decoder (#145264).

[bitdancer]

#145264 has been fixed, and that fixes the problem with get_payload. Closing this as a duplicate.