Two-way syncing between X-GM-LABELS and IMAP Keywords

[guyest]

This is a quite an impressive project. I am still trying to get my head around just how you handle tag implementation in Dovecot, but it seems to be smooth and that is something that I have wanted to accomplish for quite some time now as I use Dovecot locally on my machine with mbsync to pull down from Gmail.

Your Python implementation of fetching messages from Gmail is very nice (and it sounds like its significantly faster than mbsync though I haven't compared it myself - just taking your word for it). However, it does not seem like you have grabbed the X-GM-LABELS attribute and mapped that IMAP Keywords. This would be very nice because it would enable the Dovecot backed storage and the web-based Gmail to remain in sync while your web interface is still in transition and allow easier mobile interaction before that interface is fully implemented as well. Furthermore, it would allow for the use case of company email on GSuite where it is not possible to abandon it altogether, in which case you would really want the two versions of the inbox and labels to be identical.

I think this enhancement is feasible and would be an important improvement. I am curious to hear whether you see it as possible to implement and of interest to you. Regardless, nice work!

[naspeh]

Thank you for bringing this here!

I changed import/syncing mechanism since I wrote #9 and haven't measured the time with last implementation :), but it should be still pretty good.

IMAP Keywords have some limitations, so If X-GM-LABELS have special characters it saves metadata about such tags and stores hash_id in IMAP Keywords on particular messages (details here). Metadata in mailur is other emails in special folder.

Two-way synchronization is pretty delicate task. I have been thinking about this though, because of reasons you specified. So yes I would like to see it someday, but I don't know when I can work on it yet. I've just got a new full-time job and I fully dedicated to my work now.

[guyest]

OK, I was mistaken about your not grabbing X-GM-LABELS and I see that happening now. But I was alerted to it through an error with \Muted not being a valid IMAP system flag so I hacked the imap.py function clean_recent to clean the \Muted flag away too. However, I am not sure why the \Recent flag is being removed since it is a valid IMAP system flag... Perhaps it should only be the \Muted flag that is removed.

I understand you are focused on your job right now - such is the case with open source projects. It is very good to hear you have the two-way sync problem in mind, though. I may play around with it a bit if I get the chance, but that would also mean better understanding how it is you have implemented everything. I am only part way to that...

[naspeh]

\Recent flag has special meaning for Dovecot and can't be added to messages.

\Muted should be mapped to #muted probably here.

Thank you for trying it!

[naspeh]

I also was thinking about partial two-way syncing. Sync only specified gmail flags and labels (like \Seen, \Junk, \Trash, \Inbox, etc.), so it would be useful to read/spam/delete/archive emails from Gmail mobile app and Gmail interface, but other advanced tags are handled only by mailur.

What do you think about such option?

[guyest]

That is a significant start! Honestly, that is the main advantage. Even if organizational tags dont go up to the Gmail that is not the end of the world. At least your inbox doesn't reach into the thousands. You have probably heard of the similar project that does this for notmuch and Gmail tags: https://github.com/gauteh/lieer. It is also done in Python so I wonder if some of the code/logic would be reusable.

[naspeh]

I haven't seen lieer yet, thank you for sharing it with me. I was considering to use notmuch as storage and backend, but dovecot won the battle :).

I think partial syncing can be done just faster, probably it can be the first step in this direction and then I can see how hard it would be to implement the full sync.

[guyest]

I think Dovecot was the right choice for sure. Notmuch is nice, but IMAP is standardized and with something like email standards are preferable (which is what makes Gmail annoying in how they go about it).

[naspeh]

Some technical details.

I have two functions using IMAP4 IDLE:

1. one imports messages from gmail constantly here
2. another sync flags between two imap folders here (one folder contain original messages and another one contain parsed messages)

So syncing flags will be in a similar way as above. More events would be handled from gmail side, and flags would be synced in a way they synced between local folders.

[naspeh]

Implemented partial two-way syncing for Gmail here: 52109d0 and 42b2b15

Five flags and labels stay in sync: \Seen, \Flagged, \Inbox, \Trash, \Junk.

Official Gmail apps can be used for some workflows.