MimeKit hangs when parsing specific emails

[ctbrewski]

I am using MimeKit/MailKit to process hundreds of email replies daily that come from marketing emails. It works perfectly 99% of the time, but I have encountered several emails that hang MimeKit when it tries to parse the message. I have tracked this down to the ParseMessage function in the MimeParser, but I have no idea what is causing these messages to hang the parser. I can provide several emails that cause the parser to hang.

[jstedfast]

If you could email them to jeff@xamarin.com, that would be awesome - I'll look into them this weekend.

[jstedfast]

Sorry I didn't respond sooner, I hadn't thought to look in my spam folder until just a few minutes ago.

I can't get any of the messages to hang the MimeKit parser. I've tried both Unix and DOS line-endings and the MimeKit parser parses them fine without any hangs.

What version of MimeKit are you using?

Here's my test program:

using System;

using MimeKit;

namespace ParserTest {
    class MainClass
    {
        public static void Main (string[] args)
        {
            var message = MimeMessage.Load ("noname.eml");
            var message1 = MimeMessage.Load ("noname-1.eml");
            var message2 = MimeMessage.Load ("noname-2.eml");
            var message3 = MimeMessage.Load ("noname-3.eml");
            var message4 = MimeMessage.Load ("noname-4.eml");

            Console.WriteLine ("parsed!");
        }
    }
}

[ctbrewski]

I am using MailKit to read from an Exchange server using POP when this occurs. Let me see if I can narrow this down and give you a simple code example to replicate.

I am using the latest version of MimeKit and MailKit, and have had the issue for several versions.

[jstedfast]

Well, the first step is to figure out why the Pop3Stream.Read() method is not returning End Of Stream (the MimeKit parser presumably thinks there is more data but the Pop3Stream times out waiting for it).

This means that either the Exchange server is broken and not sending the termination sequence (.\r\n on a line by itself)... or Pop3Stream.Read() is failing to properly find/handle it.

The easiest way to figure this out is to create the Pop3Client instance with a ProtocolLogger, like so:

using (var client = new Pop3Client (new ProtocolLogger ("pop3.txt"))) {
    ...
}

What you should see at the end of the log file is something like this:

S: .

followed by a newline.

[jstedfast]

Any luck getting the log?

[ctbrewski]

So, after trying different things, I think I found where/how the issue is happening. It only happens when retrieving messages by UniqueID with Pop3. The emails I sent you specifically hang the process when I try and retrieve them using Unique ID's on my Exchange server. I can replicate this over and over using these specific messages.

Here is the code I am using:

using (var cancel = new CancellationTokenSource())
{
    client.Connect(uri, cancel.Token);
    client.Authenticate(credentials, cancel.Token);

    List<string> sList = new List<string>(client.GetMessageUids(cancel.Token));
    for (int i = 0; i < sList.Count; i++)
    {
        MimeMessage msg = client.GetMessage(sList[i], cancel.Token);
        list.Add(msg);
    }
}

The problem occurs when I call client.GetMessage(sList[i], cancel.Token);. There is no error, it just hangs in the ParseMessage function in the MimeParser. Often I can process 2000 emails before a failure, but eventually an email comes along that will hang it, and continue to hang it until I remove it from the list and restart the process.

I have switched to using message index rather than unique ID and the 'bad' messages now work fine, so I hope this has resolved my issue. I will know for sure in a day or two.

[jstedfast]

FWIW, unless you are actually going to use the cancellation token, you don't need to use it (the cancellation token argument is optional)

[jstedfast]

This is very strange news because all the GetMessage(string uid, ...) method does is map the string uid to the index and then retrieves the message based on that :-\

[jstedfast]

Could you try https://files.xamarin.com/~jeff/MailKit.dll and let me know if that fixes the problem?

I've replaced my custom MemMove() logic with System.Buffer.BlockCopy()... perhaps my custom logic was wrong?

[jstedfast]

Another user that was hitting this problem says that this new dll fixes things for him.

Might be coincidence or it might really fix things (hard to tell for sure since this bug seems sort of "random").

Would love to know if this works for you as well or not.

[ctbrewski]

The new DLL does work. Emails that previously failed are now working.

BTW, I LOVE MailKit/MimeKit. I have used several other options to access mail in the past, but all seemed a bit hokey. MailKit and MimeKit are very fast and top-notch and I have been VERY happy with them.

[jstedfast]

Glad it's working for you now, but I actually just discovered the real bug that you've been hitting.

I think the new dll is just working for you out of pure coincidence due to network packets being broken up in just the right way so as to not break apart the terminating ".\r\n" sequence.

Here's the fix: jstedfast/MailKit@07a5987

I'll be releasing new NuGet packages of MimeKit and MailKit tonight.

[jstedfast]

I released MailKit 0.92.0.0 last night with the fix.