DKIM verification failing with mixed email formats in To field

[st-abarker]

I've run into an issue with trying to validate the DKIM signature on a message received from Gmail. The message had two recipients, both on an IDN domain. One of the recipient email addresses had a local part that included a non-ASCII character and the other recipient email address had an all ASCII local part. The message received from Gmail had a header-field like the following (the domain name has been anonymized):

To: =?utf-8?Q??= <renée@が大好き.com>,
	=?utf-8?Q??= <abarker@xn---s43jna5257l4bb.com>

When trying to use DkimVerifier, I have not been able to get a successful verification. I've tried using both of the following:

using var msg = MimeKit.MimeMessage.Load(stream, true);
var dkimHeader = msg.Headers.FirstOrDefault(x => x.Field.EqualsInvariant(DKIM_SIGNATURE))
if (dkimHeader is null)
    return;
dkimVerifier.Verify(msg, dkimHeader);

and

using var msg = MimeMessage.Load(stream, true);
var dkimHeader = msg.Headers.FirstOrDefault(x => x.Field.EqualsInvariant(DKIM_SIGNATURE))
if (dkimHeader is null)
    return;
var options = new FormatOptions { International = true };
dkimVerifier.Verify(options, msg, dkimHeader);

Can anyone suggest a way to verify the DKIM signature for a message like this?

[jstedfast]

The DKIM verifier logic doesn't reformat the headers international vs non-international because I don't think it is supposed to (am I wrong?).

My guess is that an MTA reformatted the headers at some point (which is why there is a mix) and that's what is causing DKIM to fail.

[st-abarker]

Based on my understanding of the relevant RFCs, the DKIM verifier should not be modifying the headers as far as international/non-international goes. However, it does appear to be doing so. I've run some tests with a few different messages, and got the following results:

• To header only contains a non-international address like abarker@xn---s43jna5257l4bb.com. DkimVerifier.Verify only returns true if I include a FormatOptions with International set to false. When I pass a FormatOptions with International set to true, stepping through the verification code shows that it successfully validates the body hash and fails when checking the header signature.
• To header only contains an international address like renée@が大好き.com. DkimVerifier.Verify only returns true if I include a FormatOptions with International set to true. Like in the previous scenario, passing a FormatOptions with International set to false successfully validates the body hash but fails when checking the header signature.

One thing to note is that, due to how Gmail formats parameters in the Content-Type and Content-Disposition headers, when using a FormatOptions with International set to true, I also set AlwaysQuoteParameterValues to true. Without that, the body hash was not properly validating.

As for an intermediate MTA modifying the headers in transit, that could only be the case if one of Google's own servers is modifying the message. The received headers indicate that it was received by a Google server which assigned an SMTP id, then sent directly to the server doing the DKIM verification.

I did another walk through the code trying to verify the message I originally described, and I can confirm that the headers are being converted in some scenarios during DKIM verification. The problem appears to be in DkimVerifierBase.WriteHeaderRelaxed() when it calls header.GetRawValue(). If the FormatOptions has International set to true, some of the headers can be converted from non-international to international - including the headers that are expected to include an email address. That converts the To header value in my initial post to renée@が大好き.com, abarker@が大好き.com. If International is false, that doesn't seem to happen but I haven't been able to figure out why it still fails validation.

[jstedfast]

when it calls header.GetRawValue(). If the FormatOptions has International set to true, some of the headers can be converted from non-international to international - including the headers that are expected to include an email address.

Ugh, yes, you are correct. When FormatOptions.International is set to false, though, it shouldn't reformat anything (which is the default value). That said, ultimately this code should not be using GetRawValue(options) when verifying signatures (but should when constructing signatures).

If International is false, that doesn't seem to happen but I haven't been able to figure out why it still fails validation.

Right. It should be verifying correctly in this scenario.

[jstedfast]

FWIW, this issue is very closely related to this: #993

[st-abarker]

I think I figured out why International needs to be set to true for validating a message from Gmail to an address like renée@が大好き.com. When sending such a message, Gmail encodes the header as:

To: =?utf-8?Q??= <renée@が大好き.com>

When attempting to verify the DKIM signature with International = false, header.GetRawValue() returns the raw value =?utf-8?Q??= <renée@が大好き.com> and verification fails. However, when attempting to verify with International = true, header.GetRawValue() returns only renée@が大好き.com and verification succeeds.

[jstedfast]

I can see why MimeKit does that^[1], but I don't understand why that needs to happen in order for the signature to verify.

1. Decoding =?utf-8?Q??= results in an empty string and so when MimeKit re-serializes the address, it doesn't need to write anything for the display-name text (it's an empty string) and it doesn't need the <>'s either.

[st-abarker]

I figured out what was causing the behavior I described. It appears to be due to a service I was unaware was running on our system. That service did not properly support 8BITMIME and was trying to ensure the headers were properly encoded. Due to timing, this was happening between when the message was received over SMTP and when the DKIM verification was done. Removing that service fixed the issue, and all my test cases are now verifying correctly with the default FormatOptions.

[jstedfast]

Glad everything worked out! I think, ultimately, I want to get rid of the option to provide FormatOptions for verifying DKIM/ARC signatures (as you discovered, all it really does is provide a way for developers to shoot themselves in the foot) and figure out a better way for these formatting options to work in general. Issue #993 was a good start, but it broke current expectations and I need to figure out a way of making that patch work better.

Appreciate that you took the time to investigate this for me. Thanks!