IMAP import job hangs due to large starting UID and inefficient UID iteration

[prinsss]

Describe the bug

When the IMAP server uses very large UIDs (e.g. { seq: 1, uid: 1377008892 }), the import job in ImapConnector iterates from UID 0 up to the actual lowest valid UID. This causes many unnecessary fetch commands, significantly slowing down the import process or causing it to hang indefinitely.

According to the IMAP standard, UIDs do not always start from 0 or 1; servers may assign any positive integer as the lowest UID. The current logic assumes a starting UID of 0, which is inefficient for such cases.

OpenArchiver/packages/backend/src/services/ingestion-connectors/ImapConnector.ts

Line 207 in 1e048fd

let startUid = (lastUid || 0) + 1;

To Reproduce

1. Connect to an IMAP server where mailbox messages have very large UIDs (e.g., starting at 1377008892).
2. Run the import job with ImapConnector.
3. Observe the import job sending many fetch commands for UIDs that do not exist, causing delays or hanging.

Expected behavior

The import job should detect the actual lowest UID available in the mailbox and start iteration from there, rather than assuming UID 0.

System:

• Open Archiver Version: 0.3.4

Relevant logs:

if (mailbox.exists > 0) {
  const lastMessage = await this.client.fetchOne(String(mailbox.exists), {
    uid: true,
  });
+ console.log('lastMessage', lastMessage);
  if (lastMessage && lastMessage.uid > currentMaxUid) {
    currentMaxUid = lastMessage.uid;
+   logger.info('Updating currentMaxUid to %d for mailbox %s', currentMaxUid, mailboxPath);
  }
+
+ const firstMessage = await this.client.fetchOne('1', { uid: true });
+ console.log('firstMessage', firstMessage);
}

👉 Debug logs:

[0] lastMessage { seq: 326, uid: 1377009217, id: '63b60e410a1db7d600852a7e6f27e208' }
[0] [15:15:50.718] DEBUG (47981): 9 OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.718] INFO (47981): Updating currentMaxUid to 1377009217 for mailbox INBOX
[0] [15:15:50.718] DEBUG (47981): A FETCH 1 UID
[0]     module: "ImapFlow"
[0]     src: "s"
[0] firstMessage { seq: 1, uid: 1377008892, id: '4e3f698708ca14de5e28cc2a7a307ab5' }
[0] [15:15:50.753] DEBUG (47981): A OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.753] INFO (47981): startUid <= maxUidToFetch: 1 <= 1377009217
[0] [15:15:50.754] DEBUG (47981): B UID FETCH 1:250 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[])
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.787] DEBUG (47981): B OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.788] INFO (47981): startUid <= maxUidToFetch: 251 <= 1377009217
[0] [15:15:50.788] DEBUG (47981): C UID FETCH 251:500 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[])
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.823] DEBUG (47981): C OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.824] INFO (47981): startUid <= maxUidToFetch: 501 <= 1377009217
[0] [15:15:50.824] DEBUG (47981): D UID FETCH 501:750 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[])
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:15:50.856] DEBUG (47981): D OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"

[... 5000+ lines of logs ...]

[0] [15:16:10.178] INFO (47981): startUid <= maxUidToFetch: 136501 <= 1377009217
[0] [15:16:10.178] DEBUG (47981): 22D UID FETCH 136501:136750 (UID BODYSTRUCTURE ENVELOPE INTERNALDATE BODY.PEEK[])
[0]     module: "ImapFlow"
[0]     src: "s"
[0] [15:16:10.217] DEBUG (47981): 22D OK Fetch completed
[0]     module: "ImapFlow"
[0]     src: "s"

Additional context

AFAIK, imap.163.com is using very large UIDs.