Emails not sending to UTF8 recipients

[bogdanst24]

Describe the bug
We noticed that on our solution using MailKit, sending emails to UTF8 recipients is not working as expected if the non-ASCII part is in the local part of the address.
e.g.: úßerñame@domain.com

What we are seeing is that it actually tries to send the email to the Punycode converted address. For example, instead of the address from above, it will send to xn--sserame@domain.com.

Results from the investigation on the issue:

• isInternational field on mailbox addresses is set to true as expected
• setting formatOptions.International to true - nothing changes
• we are using the SendAsync with sender and recipients parameters. We've tried also the simplest one only with format options and mimeMessage - nothing changes
• the solution communicates with Exchange Online and we have tried disabling it and sending the emails directly without touching them - it worked and from there we deduced the issue is on our component's side.

Platform:

• OS: docker container
• .NET Framework: .Net Core 3.1
• MailKit Version: 2.6

To Reproduce
Steps to reproduce the behavior:

• send an email to a valid Gmail account (e.g. yourgmailaccount@gmail.com) by adding an ignored part containing utf8 characters - yourgmailaccoun+ø@gmail.com.

Thank you,

[jstedfast]

Can you get the EHLO response from the SMTP server using the ProtocolLogger?

new SmtpClient (new ProtocolLogger ("smtp.log"))

[bogdanst24]

S: 250 2.6.0 <DF638101-4BA0-4F27-92C4-1A10F0A296AB@xxx.onmicrosoft.com> [InternalId=14516989460721, Hostname=AM0PR05MB4162.eurprd05.prod.outlook.com] 370002 bytes in 0.207, 1741.395 KB/sec Queued mail for delivery
Connected to smtp://xxx.mail.protection.outlook.com:25/?starttls=when-available
S: 220 DB5EUR03FT032.mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Fri, 22 May 2020 10:01:01 +0000
C: EHLO [10.244.4.117]
S: 250-DB5EUR03FT032.mail.protection.outlook.com Hello [13.69.192.166]
S: 250-SIZE 157286400
S: 250-PIPELINING
S: 250-DSN
S: 250-ENHANCEDSTATUSCODES
S: 250-STARTTLS
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250 SMTPUTF8
C: STARTTLS
S: 220 2.0.0 SMTP server ready
C: EHLO [10.244.4.117]
S: 250-DB5EUR03FT032.mail.protection.outlook.com Hello [13.69.192.166]
S: 250-SIZE 157286400
S: 250-PIPELINING
S: 250-DSN
S: 250-ENHANCEDSTATUSCODES
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250 SMTPUTF8
C: MAIL FROM:<bfs@xxx.onmicrosoft.com> SIZE=61966
C: RCPT TO:<bogdan.xn--stupariu0+-9cb@gmail.com>
S: 250 2.1.0 Sender OK
S: 250 2.1.5 Recipient OK
C: DATA
S: 354 Start mail input; end with <CRLF>.<CRLF>
C: Received: from DB6PR0501MB2741.eurprd05.prod.outlook.com (2603:10a6:4:80::14)
C:  by DB6PR0501MB2679.eurprd05.prod.outlook.com (2603:10a6:4:81::8) with
C:  Microsoft SMTP Server (version=TLS1_2,
C:  cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23; Fri, 22 May
C:  2020 10:01:00 +0000
C: Received: from DB6PR0501MB2741.eurprd05.prod.outlook.com
C:  ([fe80::cdfc:b26b:144e:eaf]) by DB6PR0501MB2741.eurprd05.prod.outlook.com
C:  ([fe80::cdfc:b26b:144e:eaf%12]) with mapi id 15.20.3021.020; Fri, 22 May 2020
C:  10:01:00 +0000
C: From: Bogdan Test <bfs@xxx.onmicrosoft.com>
C: To: =?utf-8?B?Ym9nZGFuLnN0dXBhcml1MCvDuEBnbWFpbC5jb20=?=
C:      <bogdan.stupariu0+ø@gmail.com>
C: Subject: Test
C: Thread-Topic: Test
C: Thread-Index: AdYwH+MRuSJ08lDuRLejT73LIbbuoQ==
C: Date: Fri, 22 May 2020 10:00:59 +0000
C: Message-ID:
C:  <DB6PR0501MB27411E035B7970C71F51D7FD91B40@DB6PR0501MB2741.eurprd05.prod.outlook.com>
C: Accept-Language: en-US
C: X-MS-Has-Attach:
C: X-MS-TNEF-Correlator:
C:  <DB6PR0501MB27411E035B7970C71F51D7FD91B40@DB6PR0501MB2741.eurprd05.prod.outlook.com>
C: authentication-results: gmail.com; dkim=none (message not signed)
C:  header.d=none;gmail.com; dmarc=none action=none
C:  header.from=xxx.onmicrosoft.com;
C: x-ms-publictraffictype: Email
C: x-ms-office365-filtering-correlation-id: 647c1aa6-32fa-4ac2-a27b-08d7fe3707aa
C: x-ms-traffictypediagnostic: DB6PR0501MB2679:
C: x-ms-oob-tlc-oobclassifiers: OLM:1728;
C: x-forefront-prvs: 04111BAC64
C: x-ms-exchange-senderadcheck: 1
C: x-ms-exchange-transport-forked: True
C: MIME-Version: 1.0
C: X-MS-Exchange-CrossTenant-Network-Message-Id: 647c1aa6-32fa-4ac2-a27b-08d7fe3707aa
C: X-MS-Exchange-CrossTenant-originalarrivaltime: 22 May 2020 10:00:59.9944
C:  (UTC)
C: X-MS-Exchange-CrossTenant-fromentityheader: Hosted
C: X-MS-Exchange-CrossTenant-id: 7463db36-b374-4866-8373-b76e3c2e49f7
C: X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
C: X-MS-Exchange-CrossTenant-userprincipalname: zlcYOj4oXbEPTNX2pPM2lv76K7fov3ZF+pFcXdfnecpsGhcN/2WYhSkeZNVLUPFEHr4lINYU//qlz+vPft5EeGx+TqZ+aOY/qjNEF4t3Su7Q9B6TTvbjJ88dMEvZXzDB
C: X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0501MB2679

[jstedfast]

Oh, I misunderstood the issue - I thought that the To: header was using the punycode version of the address.

Is this email ending up being sent to the wrong person? I mean, I understand that MailKit could technically use the UTF-8 version of the address in the RCPT TO command, but it doesn't really matter at the end of the day because it should still be sent to the right address.

[jstedfast]

Oddly, according to the unit tests that I just wrote for this, it works fine...

[jstedfast]

How are you calling the Send/SendAsync method?

Are you passing in a custom FormatOptions? Does it set International=true?

If you aren't doing that, then that's the problem.

[bogdanst24]

How are you calling the Send/SendAsync method?

await _smtpClient.SendAsync(mimeMessage, sender, recipientsToSend);

Are you passing in a custom FormatOptions? Does it set International=true?

As I was also saying in the original message, that was the first thing I've tried (following issue 649 and the others related). I have just tested it again and it doesn't fix it. Actually, nothing changes.

FormatOptions formatOptions = new FormatOptions {International = true}; await _smtpClient.SendAsync(formatOptions, mimeMessage, sender, recipientsToSend);

Oh, I misunderstood the issue - I thought that the To: header was using the punycode version of the address.
Is this email ending up being sent to the wrong person?

The To: header comes right, in utf8. Let me give you another example. I'm sending an email to account.test+ø@gmail.com.. The email reaches Exchange and then it comes to our solution. Here, let's say we are just sending it with _smtpClient.SendAsync(). Nothing was changed in the To: header or with recipients encoding. It then reaches back Exchange that then would send it to the destination (we are making sure it does not come back to us again). Then the email is not sent and we are getting from office 365 an NDR saying account.xn--test+-9cb wasn't found at gmail.com.

Another example, tested with a mailinator recipient address (acount.testø@mailinator.com), results in the email actually being sent to acount.xn--test-jra@mailinator.com.

Both cases tested with formatOptions {International = true}

[jstedfast]

That's really odd because the unit tests I wrote for this work as you expect.

The only difference is that I do:

var options = FormatOptions.Default.Clone ();
options.International = true;

[jstedfast]

I just modified SmtpClient to use the SMTPUTF8 extension even if FormatOptions.International isn't set if the sender or any of the recipients are International addresses (and assuming that the SMTP server advertises the SMTPUTF8 extension, obviously).

I cannot reproduce the problem that you describe so I'm not sure what's going on. If the above patch doesn't fix this for you, I may need you to step through with a debugger to see if you can figure out where things are going wrong.

[jstedfast]

How are you creating the address? Maybe that's the key to reproducing this?

[jstedfast]

Hmmm, if I create the MailboxAddress like this, then it fails just like it does for you in your test:

var mailbox = new MailboxAddress ("User Name", "xn--sserame@domain.com");

but if I create it like this, then it works:

var mailbox = new MailboxAddress ("User Name", "úßerñame@domain.com");

[jstedfast]

Oops, didn't mean for it to close.

That said, I suspect this might be fixed now (although your original comment suggested that IsInternational was returning true for you, but if I create the MailboxAddress using an IDN-encoded mailbox address, then it is false for me).

I dunno... that said, what I found & fixed was certainly a bug and it would have caused this issue.

[bogdanst24]

It works now, with the build version! Those changes definitely fixed it. Could you create a release for this fix? Thanks a lot for the quickly figuring out the issue

[bogdanst24]

I can also confirm that without setting the International in formatOptions, the mail is rejected. So it is working as expected now on the build version.

How are you creating the address? Maybe that's the key to reproducing this?

We are using another library for the SmtpServer which has an extension method for casting as MailboxAddress. However, the mailboxes seem to be generated with the right encoding, and IsInternational is set to true. I've just tested on latest released version (not containing the fix), with format options set by manually initializing the MailboxAddress as you suggested var mailbox = new MailboxAddress ("User Name", "úßerñame@domain.com");. The issue is still there.

[jstedfast]

Awesome! I'm glad my changes have fixed your issue!

I'm definitely planning to make a new release very soon, but I want to (hopefully) hear back from someone else to make sure I haven't broken something for their use-case. I expect to be able to make a release this week or by no later than this coming weekend.

My goal is to have everything ready to release by the end of today so as soon as I hear back, I can make a release.

[jstedfast]

MimeKit 2.8.0 released.
MailKit 2.7.0 released.