Syntax error in BODYSTRUCTURE. Unexpected token: NIL

[Sbaia]

Describe the bug
I detected an invalid response in PreviewText Fetch response

To Reproduce
Execute a Fetch command including PreviewText flag of MessageSummary. This cause a parse
From the logs:

CONNECT to imap://webmail.*****.it:143/?starttls=when-available
S:* OK ****** Mail IMAP4rev1 Tue, 19 Nov 2019 10:10:23 +0100
C:B00000000 CAPABILITY
S:* CAPABILITY IMAP4rev1 AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SORT THREAD=ORDEREDSUBJECT UIDPLUS QUOTA ACL NAMESPACE CHILDREN IDLE ID UNSELECT METADATA X-ICEWARP-SERVER X-MOVE MSEARCH XLIST CREATE-SPECIAL-USE X-ICEWARP-PRIVATEBUSY X-ICEWARP-PROVISIONS X-ICEWARP-CHANGEPASSWORD X-ICEWARP-DIAL STARTTLS
B00000000 OK CAPABILITY Completed
---- AUTHENTICATION TRANSCRIPT AND SELECT INBOX FOLDER

C:B00000009 FETCH 4873:4877 (UID FLAGS INTERNALDATE RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER])
--- SOME VALID HEADERS UNTIL
* 4876 FETCH (UID 5156 FLAGS (\Seen) INTERNALDATE "17-Nov-2019 03:02:44 +0100" RFC822.SIZE 713586 BODYSTRUCTURE ((("text" "plain" ("format" "flowed" "charset" "UTF-8") NIL NIL "8bit" 314 8 NIL NIL NIL NIL)("text" "html" ("charset" "UTF-8") NIL NIL "8bit" 763 18 NIL NIL NIL NIL) "alternative" ("boundary" "b3_f0dcbd2fdb06033cba91309b09af1cd8") NIL NIL NIL NIL)("image" "jpeg" ("name" "18e5ca259ceb18af6dd3ea0659f83a4c") "<18e5ca259ceb18af6dd3ea0659f83a4c>" NIL "base64" 334384 NIL NIL NIL NIL)("image" "png" ("name" "87c487a1ff757e32ee27ff267d28af35") "<87c487a1ff757e32ee27ff267d28af35>" NIL "base64" 375634 NIL NIL NIL NIL) "related" ("type" "multipart/alternative" "charset" "UTF-8" "boundary" "b1_f0dcbd2fdb06033cba91309b09af1cd8") NIL ("inline" NIL) NIL NIL) BODY[HEADER] 

I get the following exception:

{"Syntax error in BODYSTRUCTURE. Unexpected token: NIL"}
    Data: {System.Collections.ListDictionaryInternal}
    HResult: -2146233088
    HelpLink: "https://github.com/jstedfast/MailKit/blob/master/FAQ.md#ProtocolLog"
    InnerException: null
    Message: "Syntax error in BODYSTRUCTURE. Unexpected token: NIL"
    Source: "MailKit"
    StackTrace: "   at MailKit.Net.Imap.ImapUtils.<ReadStringTokenAsync>d__24.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at MailKit.Net.Imap.ImapUtils.<ParseContentLanguageAsync>d__31.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at MailKit.Net.Imap.ImapUtils.<ParseMultipartAsync>d__34.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at MailKit.Net.Imap.ImapUtils.<ParseBodyAsync>d__35.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at MailKit.Net.Imap.ImapFolder.<FetchSummaryItemsAsync>d__172.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSucc
essAndDebuggerNotification(Task task)\r\n   at MailKit.Net.Imap.ImapFolder.<FetchSummaryItemsAsync>d__173.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at MailKit.Net.Imap.ImapEngine.<ProcessUntaggedResponseAsync>d__169.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at MailKit.Net.Imap.ImapCommand.<StepAsync>d__83.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at MailKit.Net.Imap.ImapEngine.<IterateAsync>d__170.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at MailKit.Net.Imap.ImapEngine.<RunAsync>d__171.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at MailKit.Net.Imap.ImapFolder.<FetchAsync>d__213

Expected behavior
Getting no exception error

Desktop (please complete the following information):

• OS: Windows 10
• .Net Framework 4.7.2
• Version MimeKit.2.4.1

Additional context
Thanks!

[jstedfast]

The problem is this part of the response:

... "related" ("type" "multipart/alternative" "charset" "UTF-8" "boundary" "b1_f0dcbd2fdb06033cba91309b09af1cd8") NIL ("inline" NIL) NIL NIL)

What we've got above is what should be a body-type-mpart (starting with "related" which is the media-subtype token):

body-type-mpart = 1*body SP media-subtype
                  [SP body-ext-mpart]
body-ext-mpart  = body-fld-param [SP body-fld-dsp [SP body-fld-lang
                  [SP body-fld-loc *(SP body-extension)]]]
                    ; MUST NOT be returned on non-extensible
                    ; "BODY" fetch
body-fld-param  = "(" string SP string *(SP string SP string) ")" / nil
body-fld-dsp    = "(" string SP body-fld-param ")" / nil
body-fld-lang   = nstring / "(" string *(SP string) ")"
body-fld-loc    = nstring
body-extension  = nstring / number /
                   "(" body-extension *(SP body-extension) ")"
                    ; Future expansion.  Client implementations
                    ; MUST accept body-extension fields.  Server
                    ; implementations MUST NOT generate
                    ; body-extension fields except as defined by
                    ; future standard or standards-track
                    ; revisions of this specification.

That first paren-list is the body-fld-params and that's all good.

Next up is the body-fld-dsp which is NIL. That's an acceptable token as well.

But then we get ("inline" NIL) which is supposedly the body-fld-lang tokens, but it contains a NIL which is illegal and even the "inline" string doesn't make sense since body-fld-lang represents the parsed Content-Language header value which is supposed to be strings like "es" (spanish), "en" (english), or "ja" (japanese).

[jstedfast]

I've added a work-around that will work for this specific case, but it will fail if there is a nested paren list (would that even happen? I don't know).

[Sbaia]

Hi J,
I understand the problem, the mime message header is:

Content-Type: multipart/related;
	type="multipart/alternative"; charset="UTF-8"; boundary="b1_f0dcbd2fdb06033cba91309b09af1cd8"
Content-Transfer-Encoding: 8bit
Content-Disposition: inline

--b1_f0dcbd2fdb06033cba91309b09af1cd8
Content-Type: multipart/alternative;
	boundary="b3_f0dcbd2fdb06033cba91309b09af1cd8"

It seems that IceWarp interpretate Content-Disposition header as Language Header?

I don't understand if you consider this as a case to handle, or as a IMAP server bug, not resolvable?

[Sbaia]

Sorry J, I read your response too much later..

[jstedfast]

Right, it looks like the server is getting the Content-Language and Content-Disposition tokens backwards or something.

What server is this? Is it Exchange?

[Sbaia]

I don't know, I have catch this error in production. I think that could be IceWarp, because in CAPABILITIES there are some special capability as X-ICEWARP-SERVER X-ICEWARP-PRIVATEBUSY X-ICEWARP-PROVISIONS X-ICEWARP-CHANGEPASSWORD X-ICEWARP-DIAL.

Thanks.

[jstedfast]

Ah, right, I missed that and for some reason thought of Exchange. Thanks.