bpo-39040: Fix parsing of email mime headers with whitespace between encoded-words. #17620

maxking · 2019-12-16T02:51:29Z

In certain malformed content-disposition headers, parameter values are quoted
and split as encoded words on two lines with extra whitespaces. This fixes the
issue by removing the extra whitespace between the two encoded words.

https://bugs.python.org/issue39040

…ed-words. In certain malformed content-disposition headers, parameter values are quoted and split as encoded words on two lines with extra whitespaces. This fixes the issue by removing the extra whitespace between the two encoded words.

Lib/email/_header_value_parser.py

Lib/test/test_email/test_headerregistry.py

Misc/NEWS.d/next/Library/2019-12-15-18-47-20.bpo-39040.tKa0Qs.rst

bitdancer

Oops, I forgot to start a review :(

bedevere-bot · 2019-12-16T18:15:34Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

maxking · 2019-12-17T03:29:34Z

I have made the requested changes; please review again.

bedevere-bot · 2019-12-17T03:29:37Z

Thanks for making the requested changes!

@bitdancer: please review the changes made to this pull request.

bitdancer · 2019-12-17T20:31:26Z

Lib/test/test_email/test_headerregistry.py

+            [],
+            'attachment; filename="File Name With Spaces.pdf"',
+            ('Content-Disposition: attachment; '
+             'filename="File Name With Spaces.pdf"\n'),


Not quite. We need a test with something like "File =?utf-8?q?Name?= With Spaces.pdf". That should have spaces around Name...we want to make sure we aren't removing spaces around encoded words that aren't next to other encoded words.

I'm not seeing the changes?

You might be looking at the outdated diff, from a previous commit.

There should be 4 commits in the PR, and your requested changes are in the last 2 commits.

bedevere-bot · 2019-12-17T20:31:48Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

maxking · 2019-12-18T01:38:26Z

I have made the requested changes; please review again.

bedevere-bot · 2019-12-18T01:38:29Z

Thanks for making the requested changes!

@bitdancer: please review the changes made to this pull request.

miss-islington · 2020-05-29T00:05:02Z

Thanks @maxking for the PR, and @bitdancer for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8, 3.9.
🐍🍒⛏🤖

…encoded-words. (pythongh-17620) * bpo-39040: Fix parsing of email headers with encoded-words inside a quoted string. It is fairly common to find malformed mime headers (especially content-disposition headers) where the parameter values, instead of being encoded to RFC standards, are "encoded" by doing RFC 2047 "encoded word" encoding, and then enclosing the whole thing in quotes. The processing of these malformed headers was incorrectly leaving the spaces between encoded words in the decoded text (whitespace between adjacent encoded words is supposed to be stripped on decoding). This changeset fixes the encoded word processing inside quoted strings (bare-quoted-string) to do correct RFC 2047 decoding by stripping that whitespace. (cherry picked from commit 21017ed) Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>

miss-islington · 2020-05-29T00:07:00Z

Thanks @maxking for the PR, and @bitdancer for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

…encoded-words. (gh-17620) * bpo-39040: Fix parsing of email headers with encoded-words inside a quoted string. It is fairly common to find malformed mime headers (especially content-disposition headers) where the parameter values, instead of being encoded to RFC standards, are "encoded" by doing RFC 2047 "encoded word" encoding, and then enclosing the whole thing in quotes. The processing of these malformed headers was incorrectly leaving the spaces between encoded words in the decoded text (whitespace between adjacent encoded words is supposed to be stripped on decoding). This changeset fixes the encoded word processing inside quoted strings (bare-quoted-string) to do correct RFC 2047 decoding by stripping that whitespace. (cherry picked from commit 21017ed) Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>

* 'master' of github.com:python/cpython: (497 commits) bpo-40061: Fix a possible refleak in _asynciomodule.c (pythonGH-19748) bpo-40798: Generate a different message for already removed elements (pythonGH-20483) closes bpo-29017: Update the bindings for Qt information with PySide2 (pythonGH-20149) bpo-39885: Make IDLE context menu cut and copy work again (pythonGH-18951) bpo-29882: Add an efficient popcount method for integers (python#771) Further de-linting of zoneinfo module (python#20499) bpo-40780: Fix failure of _Py_dg_dtoa to remove trailing zeros (pythonGH-20435) Indicate that abs() method accept argument that implement __abs__(), just like call() method in the docs (pythonGH-20509) bpo-39040: Fix parsing of email mime headers with whitespace between encoded-words. (pythongh-17620) bpo-40784: Fix sqlite3 deterministic test (pythonGH-20448) bpo-30064: Properly skip unstable loop.sock_connect() racing test (pythonGH-20494) Note the output ordering of combinatoric functions (pythonGH-19732) bpo-40474: Updated coverage.yml to better report coverage stats (python#19851) bpo-40806: Clarify that itertools.product immediately consumes its inpt (pythonGH-20492) bpo-1294959: Try to clarify the meaning of platlibdir (pythonGH-20332) bpo-37878: PyThreadState_DeleteCurrent() was not removed (pythonGH-20489) bpo-40777: Initialize PyDateTime_IsoCalendarDateType.tp_base at run-time (pythonGH-20493) bpo-40755: Add missing multiset operations to Counter() (pythonGH-20339) bpo-25920: Remove socket.getaddrinfo() lock on macOS (pythonGH-20177) bpo-40275: Fix test.support.threading_helper (pythonGH-20488) ...

maxking requested a review from a team as a code owner December 16, 2019 02:51

the-knights-who-say-ni added the CLA signed label Dec 16, 2019

bedevere-bot added the awaiting core review label Dec 16, 2019

maxking added needs backport to 3.7 type-bug An unexpected behavior, bug, or error labels Dec 16, 2019

bitdancer reviewed Dec 16, 2019

View reviewed changes

Lib/email/_header_value_parser.py Outdated Show resolved Hide resolved

bitdancer reviewed Dec 16, 2019

View reviewed changes

Lib/email/_header_value_parser.py Outdated Show resolved Hide resolved

bitdancer reviewed Dec 16, 2019

View reviewed changes

Lib/test/test_email/test_headerregistry.py Outdated Show resolved Hide resolved

bitdancer reviewed Dec 16, 2019

View reviewed changes

Misc/NEWS.d/next/Library/2019-12-15-18-47-20.bpo-39040.tKa0Qs.rst Outdated Show resolved Hide resolved

bitdancer requested changes Dec 16, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting core review labels Dec 16, 2019

maxking changed the title ~~bpo-39040: Fix parsing of email headers with whitespace between encoded-words.~~ bpo-39040: Fix parsing of email mime headers with whitespace between encoded-words. Dec 17, 2019

Remove empty lines, add a new test and update the news entry.

bf2cb76

bedevere-bot added awaiting change review and removed awaiting changes labels Dec 17, 2019

bedevere-bot requested a review from bitdancer December 17, 2019 03:29

bitdancer requested changes Dec 17, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting change review labels Dec 17, 2019

maxking added 2 commits December 17, 2019 17:33

Update test to include spaces around an encoded word.

004f37d

Fix the encoded word syntax and resultant error.

016ceb3

bedevere-bot removed the awaiting changes label Dec 18, 2019

bedevere-bot added the awaiting change review label Dec 18, 2019

bedevere-bot requested a review from bitdancer December 18, 2019 01:38

csabella removed the request for review from bitdancer May 28, 2020 23:24

csabella added the needs backport to 3.9 only security fixes label May 28, 2020

csabella requested a review from bitdancer May 28, 2020 23:24

bitdancer approved these changes May 28, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels May 28, 2020

bitdancer merged commit 21017ed into python:master May 29, 2020

bedevere-bot removed the awaiting merge label May 29, 2020

bitdancer removed needs backport to 3.7 labels May 29, 2020

bitdancer added the needs backport to 3.8 label May 29, 2020

flokli mentioned this pull request Oct 15, 2021

bpo-28764: mailbox.mbox: handle lines with non-ascii more graceful #23553

Open

manfred-kaiser mannequin mentioned this pull request Jun 19, 2022

Wrong attachement filename when mail mime header was too long #83221

Closed

terryjreedy removed the needs backport to 3.9 only security fixes label Feb 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-39040: Fix parsing of email mime headers with whitespace between encoded-words. #17620

bpo-39040: Fix parsing of email mime headers with whitespace between encoded-words. #17620

maxking commented Dec 16, 2019 •

edited by bedevere-bot

Loading

bitdancer left a comment

bedevere-bot commented Dec 16, 2019

maxking commented Dec 17, 2019

bedevere-bot commented Dec 17, 2019

bitdancer Dec 17, 2019

maxking Dec 18, 2019

bitdancer Dec 24, 2019

maxking Dec 24, 2019

bedevere-bot commented Dec 17, 2019

maxking commented Dec 18, 2019

bedevere-bot commented Dec 18, 2019

miss-islington commented May 29, 2020

miss-islington commented May 29, 2020

bpo-39040: Fix parsing of email mime headers with whitespace between encoded-words. #17620

bpo-39040: Fix parsing of email mime headers with whitespace between encoded-words. #17620

Conversation

maxking commented Dec 16, 2019 • edited by bedevere-bot Loading

bitdancer left a comment

Choose a reason for hiding this comment

bedevere-bot commented Dec 16, 2019

maxking commented Dec 17, 2019

bedevere-bot commented Dec 17, 2019

bitdancer Dec 17, 2019

Choose a reason for hiding this comment

maxking Dec 18, 2019

Choose a reason for hiding this comment

bitdancer Dec 24, 2019

Choose a reason for hiding this comment

maxking Dec 24, 2019

Choose a reason for hiding this comment

bedevere-bot commented Dec 17, 2019

maxking commented Dec 18, 2019

bedevere-bot commented Dec 18, 2019

miss-islington commented May 29, 2020

miss-islington commented May 29, 2020

maxking commented Dec 16, 2019 •

edited by bedevere-bot

Loading