Defects when parsing email headers aren't raised or propagated to the email object

[me-and]

Bug report

When parsing an email using email.parser.Parser(policy=email.policy.strict), where the email has invalid headers, I expected exceptions to be raised when the email was parsed. Instead, the header objects within the generated EmailMessage record defects, but no exception is raised and the overall EmailMessage has no defects listed.

import email
import email.policy

email_str = '''\
Date: xxx
From: "Bad from header" <bad, value>
'''

parsed_email = email.message_from_string(email_str, policy=email.policy.strict)

# Should never get to this point as the above should already have raised an
# exception.

print(parsed_email['Date'].defects)  # Prints "(InvalidDateDefect('Invalid date value or format'),)", expected
print(parsed_email['From'].defects)  # Prints "(InvalidHeaderDefect('invalid address in address-list'), ..." , expected
print(parsed_email.defects)  # Prints "[]", not expected

Your environment

• Debian 12
• Raspberry Pi 4 (arm64)
• Python 3.11.2 (Debian package 3.11.2-1+b1)

[arhadthedev]

@python/email-team

[bitdancer]

You are correct, header defects are theoretically supposed to be copied to the message normally, and be raised in strict mode. It isn't quite as straightforward as that, though :(

Consider the following comment in EmailPolicy:

    # The logic of the next three methods is chosen such that it is possible to
    # switch a Message object between a Compat32 policy and a policy derived
    # from this class and have the results stay consistent.  This allows a
    # Message object constructed with this policy to be passed to a library
    # that only handles Compat32 objects, or to receive such an object and
    # convert it to use the newer style by just changing its policy.  It is
    # also chosen because it postpones the relatively expensive full rfc5322
    # parse until as late as possible when parsing from source, since in many
    # applications only a few headers will actually be inspected.

What this means is that the defects are never even produced until the header is accessed. An email message after parsing is just holding the un-parsed headers in its header list, and knows nothing about what defects it actually has because it hasn't parsed them yet.

So, to make strict work the way it is intended it actually needs to re-implement the header_source_parse method so that it parses the header using the header registry and calls register_defect on any header defects returned.

Clearly, it also needs to be documented that the default EmailPolicy does not gather header defects at message parse time; that defects in headers are only detected when the headers are accessed. And that shouldn't change for the reasons stated in the comment.

One thing we might consider, though, would be to change EmailPolicy's header_fetch_parse method to call register_defect. Then it would be possible to gather all the defects by serializing the message with refold_source set to 'all'. Or maybe we should just document how to create a policy, based on the fixed strict-that-parses-all-headers but with raise_on_defect set to False, for those people who want to gather all the defects in a message. I believe the latter aligns with my original thinking, but clearly I never followed through on this issue.

Oh, and another thing I never followed through on was improving the header parser defects to include more information about where the error occurred, to make the defects more useful when gatheedr into a big list. I also had thoughts about making more error types and/or putting some more structured data into the error object, but none of that ever got further than thoughts in the back of my head.