Unexpected token in IMAP response: NIL - during FETCH from mail.ru IMAP server

[vzhusel]

Using Mailkit with Mail.ru imap server following exception is thrown on messages fetching:

Debug:[ResilientMailService] Call failed 'Unexpected token in IMAP response: NIL'.
Warn:[MailDataSourcesModel] LoadNewMessages, MailKit.Net.Imap.ImapProtocolException: Unexpected token in IMAP response: NIL
at MailKit.Net.Imap.ImapUtils.ReadStringToken (MailKit.Net.Imap.ImapEngine engine, CancellationToken cancellationToken) [0x00000] in :0
at MailKit.Net.Imap.ImapUtils.ParseParameterList (System.Text.StringBuilder builder, MailKit.Net.Imap.ImapEngine engine, CancellationToken cancellationToken) [0x00000] in :0
at MailKit.Net.Imap.ImapUtils.ParseMultipart (MailKit.Net.Imap.ImapEngine engine, System.String path, CancellationToken cancellationToken) [0x00000] in :0
at MailKit.Net.Imap.ImapUtils.ParseBody (MailKit.Net.Imap.ImapEngine engine, System.String path, CancellationToken cancellationToken) [0x00000] in :0
at MailKit.Net.Imap.ImapUtils.ParseMultipart (MailKit.Net.Imap.ImapEngine engine, System.String path, CancellationToken cancellationToken) [0x00000] in :0
at MailKit.Net.Imap.ImapUtils.ParseBody (MailKit.Net.Imap.ImapEngine engine, System.String path, CancellationToken cancellationToken) [0x00000] in :0
at MailKit.Net.Imap.ImapFolder.FetchSummaryItems (MailKit.Net.Imap.ImapEngine engine, MailKit.Net.Imap.ImapCommand ic, Int32 index) [0x00000] in :0
at MailKit.Net.Imap.ImapEngine.ProcessUntaggedResponse (CancellationToken cancellationToken) [0x00000] in :0
at MailKit.Net.Imap.ImapCommand.Step () [0x00000] in :0
at MailKit.Net.Imap.ImapEngine.Iterate () [0x00000] in :0

Protocol log:

Connected to imaps://imap.mail.ru:993/
S: * OK Welcome
C: A00000000 CAPABILITY
S: * CAPABILITY IMAP4rev1 ID XLIST UIDPLUS UNSELECT MOVE AUTH=PLAIN AUTH=XOAUTH2
S: A00000000 OK CAPABILITY completed
C: A00000001 AUTHENTICATE XOAUTH2
S: A00000001 BAD [CLIENTBUG] Invalid number of arguments for command
C: A00000002 AUTHENTICATE PLAIN
S: +
C: removed by user
S: * CAPABILITY IMAP4rev1 ID XLIST UIDPLUS UNSELECT MOVE
S: A00000002 OK Authentication successful
C: A00000003 LIST "" ""
S: * LIST (\NoSelect) "/" ""
S: A00000003 OK LIST done
C: A00000004 LIST "" "INBOX"
S: * LIST (\Inbox) "/" "INBOX"
S: A00000004 OK LIST done
C: A00000005 XLIST "" "*"
S: * XLIST (\Inbox) "/" "INBOX"
S: * XLIST (\Spam) "/" "&BCEEPwQwBDw-"
S: * XLIST (\Sent) "/" "&BB4EQgQ,BEAEMAQyBDsENQQ9BD0ESwQ1-"
S: * XLIST (\Drafts) "/" "&BCcENQRABD0EPgQyBDgEOgQ4-"
S: * XLIST (\Trash) "/" "&BBoEPgRABDcEOAQ9BDA-"
S: A00000005 OK XLIST done
C: A00000006 SELECT INBOX
S: * FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
S: * 125 EXISTS
S: * 0 RECENT
S: * OK [UNSEEN 100]
S: * OK [UIDVALIDITY 1364574861]
S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen)]
S: * OK [UIDNEXT 451]
S: A00000006 OK [READ-WRITE] SELECT completed
C: A00000007 CHECK
S: A00000007 OK CHECK completed
C: A00000008 FETCH 26:125 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODYSTRUCTURE)
S: * 26 FETCH (UID 94 FLAGS (\Seen) INTERNALDATE "07-Mar-2013 23:24:09 +0400" RFC822.SIZE 5090 ENVELOPE ("Thu, 07 Mar 2013 23:23:00 +0400" "=?utf-8?B?0J/QvtC00LDRgNC+0Log0L3QsCA4INC80LDRgNGC0LAg0Lgg0YDQsNGB0L/RgNC+0LTQsNC20LAgQnJpbmsh?=" (("YUPLAY" NIL "no-reply" "yuplay.ru")) NIL (("YUPLAY" NIL "no-reply" "yuplay.ru")) ((NIL NIL "fish54" "bk.ru")) NIL NIL NIL "E1UDgP6-000644-9M@mailer.yuplay.com") BODYSTRUCTURE ("text" "html" ("charset" "utf-8") NIL NIL "7-bit" 3332 0 NIL NIL NIL NIL))
S: * 27 FETCH (UID 95 FLAGS (\Seen) INTERNALDATE "15-Mar-2013 19:18:37 +0400" RFC822.SIZE 5283 ENVELOPE ("Fri, 15 Mar 2013 19:17:23 +0400" "=?utf-8?B?0KHQutC40LTQutC4INC90LAgQmF0dGxlZmllbGQsIE1hc3MgRWZmZWN0LCBEcmFnb24gQWdlINC4INC60LDRgNGC0YsgUGxheXN0YXRpb24gTmV0d29yayAyNTAwIQ==?=" (("YUPLAY" NIL "no-reply" "yuplay.ru")) NIL (("YUPLAY" NIL "no-reply" "yuplay.ru")) ((NIL NIL "fish54" "bk.ru")) NIL NIL NIL "E1UGWNn-0003aY-3n@mailer.yuplay.com") BODYSTRUCTURE ("text" "html" ("charset" "utf-8") NIL NIL "7-bit" 3484 0 NIL NIL NIL NIL))
S: * 28 FETCH (UID 96 FLAGS (\Seen) INTERNALDATE "23-Mar-2013 01:01:02 +0400" RFC822.SIZE 51043 ENVELOPE ("Sat, 23 Mar 2013 01:01:02 +0400" "=?utf-8?B?0J3QvtCy0L7RgdGC0Lgg0L/RgNC+0LXQutGC0LAg0J/QvtGH0YLQsCBNYWlsLlJ1?=" (("=?utf-8?B?0JrQvtC80LDQvdC00LAg0J/QvtGH0YLRiyBNYWlsLlJ1?=" NIL "mail.news" "corp.mail.ru")) NIL NIL ((NIL NIL "fish54" "bk.ru")) NIL NIL NIL "E1UJ95C-0001uV-28.root-mailer7-mail-ru@mailer7.mail.ru") BODYSTRUCTURE (("text" "plain" ("charset" "utf-8") NIL NIL "8bit" 3341 0 NIL NIL NIL NIL)(("text" "html" ("charset" "utf-8") NIL NIL "8bit" 9856 0 NIL NIL NIL NIL)("image" "png" ("name" "logo.png") "part1.00030305.02030104@mail.ru" NIL "base64" 7232 NIL ("inline" ("filename" "logo.png")) NIL NIL)("image" "png" ("name" "p1_pic.png") "part2.07050406.00040503@mail.ru" NIL "base64" 7418 NIL ("inline" ("filename" "p1_pic.png")) NIL NIL)("image" "png" ("name" "p1_btn.png") "part3.05000004.06060304@mail.ru" NIL "base64" 2334 NIL ("inline" ("filename" "p1_btn.png")) NIL NIL)("image" "png" ("name" "p2_pic.png") "part5.08010908.01060501@mail.ru" NIL "base64" 7336 NIL ("inline" ("filename" "p2_pic.png")) NIL NIL)("image" "png" ("name" "p2_btn.png") "part6.06060800.04090408@mail.ru" NIL "base64" 2582 NIL ("inline" ("filename" "p2_btn.png")) NIL NIL)("image" "png" ("name" "p3_pic.png") "part8.01030208.07010007@mail.ru" NIL "base64" 6218 NIL ("inline" ("filename" "p3_pic.png")) NIL NIL)("image" "png" ("name" "p3_btn.png") "part9.03080803.02040702@mail.ru" NIL "base64" 2798 NIL ("inline" ("filename" "p3_btn.png")) NIL NIL)("image" "png" ("name" "p4_pic.png") "part11.05010601.04000807@mail.ru" NIL "base64" 8558 NIL ("inline" ("filename" "p4_pic.png"

[jstedfast]

I'm not able to reproduce the problem with what you've given me in the log.

The stack trace suggests that it is trying to parse the parameter list and one of the 2 tokens is NIL, which is illegal.

This is the syntax:

body-fld-param  = "(" string SP string *(SP string SP string) ")" / nil

instead of one of the string tokens, the server is sending NIL.

At least, that's as much as I can figure out based on the stack trace. From the log, I tried feeding the responses to MailKit but it was able to parse every single one of those responses without any problems (except the last one, but that's because it is incomplete, not due to a NIL token).

[vzhusel]

Thanks for the response. I tried to spend more time with this issue and it seems that we don't get full response from server, maybe connection is closed, timeout or something else.

C: A00000008 FETCH 12:126 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODYSTRUCTURE)

Sometimes the last we get is this:
S: * 29 FETCH (UID 108 FLAGS (\Seen) INTERNALDATE "30-Mar-2013 04:20:38 +0400" RFC822.SIZE 4340 ENVELOPE ("Sat, 30 Mar 2013 04:19:14 +0400" "=?utf-8?B?0KHQutC40LTQutCwIDUwJSDQvdCwIERpc2hvbm9yZWQh?=" (("YUPLAY" NIL "no-reply" "yuplay.ru")) NIL (("YUPLAY" NIL "no-reply" "yupla

Sometimes this:
S: * 17 FETCH (UID 85 FLAGS (\Seen) INTERNALDATE "01-Feb-2013 20:37:53 +0400" RFC822.SIZE 7883 ENVELOPE ("Fri, 01 Feb 2013 20:37:10 +04

Could you suggest in which direction i need to dig to understand this problem more?

Please note I am using mailkit in iOS application

[jstedfast]

Well, the log being "truncated" is to be expected since MailKit's ImapClient buffers data (it reads 4k at a time from the socket), then parses that data before reading the next 4k chunk.

This means that, in general, the parse failure will be within the last ~4k of output (slightly more than 4k since the logger prefixes each line with "S: ").

Make sure you're using MailKit 1.0.2 if you aren't already, as that version makes sure to be a bit more aggressive about flushing the log to disk. I suspect you're already using 1.0.2, but figured I'd make sure.

If you could try narrowing it down to a single message summary (i.e. by requesting 1 uid at a time until you get the failure), that would probably be helpful.

[jstedfast]

Have you made any progress on trying to get a log of fetching the info for individual messages?

[vzhusel]

Hi, sorry for the delay with response. Yes, I got it. Basically, if I do fetch without BODYSTRUCTURE then everything is ok. Here's details:

C: A00000008 UID FETCH 96 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODYSTRUCTURE)
S: * 28 FETCH (UID 96 FLAGS (\Seen) INTERNALDATE "23-Mar-2013 01:01:02 +0400" RFC822.SIZE 51043 ENVELOPE ("Sat, 23 Mar 2013 01:01:02 +0400" "=?utf-8?B?0J3QvtCy0L7RgdGC0Lgg0L/RgNC+0LXQutGC0LAg0J/QvtGH0YLQsCBNYWlsLlJ1?=" (("=?utf-8?B?0JrQvtC80LDQvdC00LAg0J/QvtGH0YLRiyBNYWlsLlJ1?=" NIL "mail.news" "corp.mail.ru")) NIL NIL ((NIL NIL "fish54" "bk.ru")) NIL NIL NIL "E1UJ95C-0001uV-28.root-mailer7-mail-ru@mailer7.mail.ru") BODYSTRUCTURE (("text" "plain" ("charset" "utf-8") NIL NIL "8bit" 3341 0 NIL NIL NIL NIL)(("text" "html" ("charset" "utf-8") NIL NIL "8bit" 9856 0 NIL NIL NIL NIL)("image" "png" ("name" "logo.png") "part1.00030305.02030104@mail.ru" NIL "base64" 7232 NIL ("inline" ("filename" "logo.png")) NIL NIL)("image" "png" ("name" "p1_pic.png") "part2.07050406.00040503@mail.ru" NIL "base64" 7418 NIL ("inline" ("filename" "p1_pic.png")) NIL NIL)("image" "png" ("name" "p1_btn.png") "part3.05000004.06060304@mail.ru" NIL "base64" 2334 NIL ("inline" ("filename" "p1_btn.png")) NIL NIL)("image" "png" ("name" "p2_pic.png") "part5.08010908.01060501@mail.ru" NIL "base64" 7336 NIL ("inline" ("filename" "p2_pic.png")) NIL NIL)("image" "png" ("name" "p2_btn.png") "part6.06060800.04090408@mail.ru" NIL "base64" 2582 NIL ("inline" ("filename" "p2_btn.png")) NIL NIL)("image" "png" ("name" "p3_pic.png") "part8.01030208.07010007@mail.ru" NIL "base64" 6218 NIL ("inline" ("filename" "p3_pic.png")) NIL NIL)("image" "png" ("name" "p3_btn.png") "part9.03080803.02040702@mail.ru" NIL "base64" 2798 NIL ("inline" ("filename" "p3_btn.png")) NIL NIL)("image" "png" ("name" "p4_pic.png") "part11.05010601.04000807@mail.ru" NIL "base64" 8558 NIL ("inline" ("filename" "p4_pic.png")) NIL NIL)("image" "png" ("name" "p4_btn.png") "part12.08090508.05040702@mail.ru" NIL "base64" 3054 NIL ("inline" ("filename" "p4_btn.png")) NIL NIL) "related" ("boundary" NIL)) "alternative" ("boundary" NIL)))
S: A00000008 OK FETCH done

Do you think this is a problem with IMAP server?

[jstedfast]

These look like they would cause the problem you had:

"related" ("boundary" NIL)) "alternative" ("boundary" NIL)))

It does look like a bug in the IMAP server, but what the bug is depends on the message source.

For example, maybe the raw message has something like:

Content-Type: multipart/alternative; boundary=

And so the server is returning that the boundary value is NULL. What it should be doing in this case, though, is sending "" (i.e. empty string).

Or it could be a bug in the IMAP server's MIME parser.

I can probably work around this issue in MailKit by just treating the value token as an nstring which would make it not fail if it gets a NIL token.

[jstedfast]

Can I ask what IMAP server this was? Preferably with version info if you have it handy - I'd just like to keep track of server bugs if I can. Thanks.

[vzhusel]

Got it, thank you for the explanation. I just checked the message and it looks like it has boundary:

Content-Type: multipart/related;
boundary="------------020605010006060003000801"

So, I guess this is a server bug. Thanks again for the explanation.

[jstedfast]

No problem. The patch I just committed should work around the issue (as far as the IMAP protocol goes), but I would not depend on the Content-Type parameter values being reliable in the MessageSummary objects.

[vzhusel]

Thanks again. Regarding IMAP server, unfortunately i do not know IMAP server info, tried to google no results, two things I know about it URL and which protocol it supports :)

[jstedfast]

Ok, no problem.