Add Annotations support

[pm64]

While MailKit exposes the ImapEvent.AnnotationChange field ("An IMAP event notification for message annotation changes"), the documentation states that "Annotations are currently not supported by MailKit". I'm opening this feature request to document this gap as an issue.

[jstedfast]

Do you actually have reason to need it? Seems that servers that implement it are extremely rare and that clients that implement it are even rarer.

Just seems like an epic waste of time to me.

[pm64]

I would like to implement a client that supports it. I believe there are many uses cases and hope to see adoption broaden. Of course I do understand why it's not been a top priority for this project.

[jstedfast]

Just to save anyone (including myself) the trouble later, the specification for this is https://tools.ietf.org/html/rfc5257

[jstedfast]

FWIW, ANNOTATE extension support is not likely to broaden among clients and servers. It's been over a decade since its introduction and has seen very little acceptance. Based on what I've seen while working on implementing this feature so far, the reason is likely the complexity and incompleteness of the extension.

For example, the only real annotations that are standardized are /comment and /altsubject for messages with the addition of the /flags/... annotations for the various flags on sub-parts of the message for clients that want to mark individual MIME parts as seen/answered/etc.

Anyway, I've implemented the ANNOTATE support classes and the ANNOTATE and ANNOTATIONS resp-codes for IMAP.

The number of new public methods that I'll have to add to IMailFolder is a bit daunting since pretty much every Fetch() and [Add,Remove,Set]Flags() method will need to be duplicated with an IList<Annotation> annotations argument.

[jstedfast]

I ended up just adding a Store() method for annotations.

Even though the IMAP STORE (and UID STORE) commands are used for storing annotations, the syntax doesn't work quite like FLAGS or (GMail) LABELS in that there is no equivalent of +FLAGS or -FLAGS or a .SILENT modifier to block notifications. In other words, you can't do things like +ANNOTATION.SILENT or -ANNOTATION.SILENT.

Seems that the way you "remove" an annotation is to simply set the value to NIL and of course, set vs add doesn't make sense in the context of annotations since setting and adding would have the same effect anyway.

Still left to do are overloading the Append() method to allow setting annotations as well as adding a dozen or so more Fetch() methods to allow fine-tuned querying of annotations. Right now, there is a new MessageSummaryItems.Annotations flag that will fetch all annotations. Maybe that's good enough for now?

Here's what I'm thinking...

Since I have to pretty drastically break API/ABI by introducing overloaded Append() methods and the Store() methods to the IMailFolder interface and having to add an Annotations property to the IMessageSummary interface, perhaps this feature should wait until MailKit 3.0.

If I wait until MailKit 3.0 (for which there were no plans until now), I can make the following changes:

Redesign the Fetch() API so that it works more like this:

var request = new FetchRequest {
    AnnotationEntries = new AnnotationEntry[] { new AnnotationEntry ("/*") },
    AnnotationAttributes = new AnnotationAttribute[] { AnnotationAttribute.Value, AnnotationAttribute.Size },
    BodyStructure = true,
    Envelope = true,
    Flags = true,
    InternalDate = true,
};
var fetched = folder.Fetch (0, -1, request);

For Add/RemoveSetFlags() (and Labels), the API could look something like this:

var options = new StoreOptions {
    UnchangedSince = modseq,
    Silent = true
};
folder.Store (options, uids, StoreAction.Set, flags, keywords);

StoreOptions.Default and StoreOptions.Silent could be 2 default static fields. There could also be a Store() that doesn't take an options argument and just uses StoreOptions.Default internally.

Anyway... the above Store() methods don't really have anything to do with the ANNOTATE-EXPERIMENT-1 extension, but it's just some API consistency ideas that I have for 3.0 that I figured I'd document.

Once that is all done, I could remove all of the existing Add/Remove/SetFlags() methods and move them into an IMailFolderExtensions class to keep from cluttering up the IMailFolder interface while still more-or-less maintaining compatibility with the 2.0 API. I could do the same for the existing Fetch() methods.

[pm64]

@jstedfast I've been reviewing the annotate branch code and am thrilled to see this being pursued! The MessageSummaryItems.Annotations flag provides a good way for client implementers to start getting their feet wet right now while retaining backward compatibility.

I agree that the best API for a complete implementation would demand breaking compatibility in a way that is similar to what is illustrated in your Fetch() and Store() examples, so yes it would seem reasonable to earmark this work for v3.0.

Just to briefly comment on the issue of adoption - while there's been an obvious "failure to launch" over the past decade, efforts like this could have a profound effect on what happens next. I'd like to see adoption broaden and a growth in standardized annotations. Support from key libraries like MailKit could easily bring this about.

As always, your hard, thoughtful work is sincerely appreciated.

[jstedfast]

@pm64 thanks for taking a look over the changes and providing some feedback, that's always nice to get when I'm working on such large features. I like knowing whether the API's I'm hashing out will work for the needs of the developers that want to use them.

I'm hoping to finish this up and make a 2.3.0 release with the existing Fetch() API in the next 2 weeks or so. I think I shouldn't have too much difficulty finishing up the Append() methods. Then I just need to write unit tests to make sure everything is working properly.

[jstedfast]

Status Update:

1. new Append() methods have been added that take an AnnotationCollection argument, but only for single-message APPEND commands. MUTLIAPPEND has not been implemented yet.
2. Added a new OrderBy subclass, OrderByAnnotation, and the necessary changes to ImapFolder to respect this new OrderBy which allows developers to specify an annotation-based sort order for use with the IMAP SORT command.
3. Updated the MessageSorter class to implement client-side sorting based on the above OrderByAnnotation as well.