Skip to content

gh-121650 : detect newlines in headers - #121812

Closed
basbloemsaat wants to merge 12 commits into
python:mainfrom
basbloemsaat:fix-issue-121650-detect-newlines-in-headers
Closed

gh-121650 : detect newlines in headers#121812
basbloemsaat wants to merge 12 commits into
python:mainfrom
basbloemsaat:fix-issue-121650-detect-newlines-in-headers

Conversation

@basbloemsaat

@basbloemsaatbasbloemsaat commented Jul 15, 2024

Copy link
Copy Markdown
Contributor

@encukou as promised, the fix for this issue.

This PR fixes both issues addressed in the issue, both the newlines at the end as well as the encoded newlines.

As this has security implication, it may need to be backported as well.

@ZeroIntensityZeroIntensity left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't able to confirm that this PR fixes #121650. The original reproducer still contains the embedded newline:

fromemailimportmessage_from_stringfromemail.policyimportdefaultemail_in="""\To: incoming+tag@me.example.comFrom: External Sender <sender@them.example.com>Subject: Here's an =?UTF-8?Q?embedded_newline=0A?=Content-Type: text/html; charset=UTF-8Content-Transfer-Encoding: quoted-printableMIME-Version: 1.0<html><head><title>An embeded newline</title></head><body> <p>I sent you an embedded newline in the subject. How do you like that?!</p></body></html>"""msg=message_from_string(email_in, policy=default)
msg=message_from_string(email_in, policy=default)
forheader, valueinmsg.items():
delmsg[header]
msg[header] =valueemail_out=str(msg)
print(email_out)

Comment threadLib/email/policy.py Outdated
Comment threadMisc/NEWS.d/next/Library/2024-07-15-21-43-38.gh-issue-121650.A0FkCh.rst Outdated
basbloemsaatand others added 4 commits July 16, 2024 09:36
@basbloemsaat

basbloemsaat commented Jul 16, 2024

Copy link
Copy Markdown
ContributorAuthor

I wasn't able to confirm that this PR fixes #121650. The original reproducer still contains the embedded newline:
...

I missed headers that were parsed from a message, as in original issue. I updated the fix and the tests.

@ZeroIntensityZeroIntensity left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed that this fixes #121650. I'm pretty sure this is a security fix (as you could previously inject email headers using this method), so this should need a backport all the way to 3.8

@encukou

Copy link
Copy Markdown
Member

@warsaw, @bitdancer, @maxking: as the email experts, do you have any comments?

encukou
encukou previously approved these changes Jul 16, 2024

@encukouencukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks good to me! Thank you for digging into it!

@encukou

Copy link
Copy Markdown
Member

I take back the review. There's more to this, unfortunately :(

Here's another reproducer:

fromemailimportmessage_from_stringfromemail.policyimportdefaultemail_in="""\To: incoming+tag@me.example.comFrom: External Sender <sender@them.example.com> =?UTF-8?Q?embedded_newline=0A?=Smuggled-Data: BadSubject: foo <bar> Here's anContent-Type: text/html; charset=UTF-8Content-Transfer-Encoding: quoted-printableMIME-Version: 1.0<html><head><title>An embeded newline</title></head><body> <p>I sent you an embedded newline in the subject. How do you like that?!</p></body></html>"""msg=message_from_string(email_in, policy=default)
print(msg)
forheader, valueinmsg.items():
delmsg[header]
msg[header] =valueemail_out=str(msg)
print(email_out)

@basbloemsaat

Copy link
Copy Markdown
ContributorAuthor

I take back the review. There's more to this, unfortunately :(

I'll look into this...

@basbloemsaat

basbloemsaat commented Jul 19, 2024

Copy link
Copy Markdown
ContributorAuthor

@encukou I tried all header types. This eliminates all newlines. Two notes:

  • date headers (and derivatives) parse the date, and the offending code is eliminated that way
  • the MIME-Version header doesn't decode the encoded newline, so it doesn't break the message. Improving the parsing of that header would mean rewriting the parsing code to do so, but I think that goes beyond the scope of this ticket.

@encukou

Copy link
Copy Markdown
Member

After reading up on the email module, I propose to fix the issue in a different part of the code: see #122233.

@encukou

Copy link
Copy Markdown
Member

Closing in favour of #122233.
Thank you for the work here, @basbloemsaat! And sorry that I “stole” the issue.

@encukouencukou closed this Aug 1, 2024
@basbloemsaat
basbloemsaat deleted the fix-issue-121650-detect-newlines-in-headers branch August 1, 2024 15:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@basbloemsaat@encukou@ZeroIntensity