import: keep what an interrupted IMAP or Maildir import wrote

jcoffey-dev commented

2026-09-30 19:24:48 +00:00

Owner

IMAP import wrote a whole folder in one transaction and stopped the folder
at the first message it could not import. A crash near the end of a large
INBOX kept nothing, and one bad INTERNALDATE lost the rest of the folder.
Worse, when a folder stopped early, fetches still in flight for it could be
filed into the next folder's mailbox.

The transaction is committed after every fetch chunk. Each message is
written in its own savepoint, so what is committed is always whole, and a
rerun fetches only the UIDs still missing.
A message that cannot be imported is rolled back on its own, logged with
its folder and UID, and counted as failed; the folder carries on, and the
message stays out of the UID map so the next run tries it again. Archive
and I/O errors still stop the run.
Fetch jobs and events carry a folder generation. Moving to a new folder
cancels queued work for older ones, and any event from an older
generation is dropped, never filed. Shutdown drains in-flight events
before joining the workers, so it cannot hang on a blocked worker.
INTERNALDATE month names are matched in any case.
Maildir import gets the same per-message savepoint, and commits every 500
new messages instead of once per folder.

Tests: 1382 pass (7 new), fmt and clippy clean. Durability after each chunk cannot be observed from outside a running import (the archive is opened with an exclusive lock); the tests check its effects: a failed chunk keeps the ones before it, and a rerun fetches only the missing UIDs.

IMAP import wrote a whole folder in one transaction and stopped the folder at the first message it could not import. A crash near the end of a large INBOX kept nothing, and one bad INTERNALDATE lost the rest of the folder. Worse, when a folder stopped early, fetches still in flight for it could be filed into the next folder's mailbox. - The transaction is committed after every fetch chunk. Each message is written in its own savepoint, so what is committed is always whole, and a rerun fetches only the UIDs still missing. - A message that cannot be imported is rolled back on its own, logged with its folder and UID, and counted as failed; the folder carries on, and the message stays out of the UID map so the next run tries it again. Archive and I/O errors still stop the run. - Fetch jobs and events carry a folder generation. Moving to a new folder cancels queued work for older ones, and any event from an older generation is dropped, never filed. Shutdown drains in-flight events before joining the workers, so it cannot hang on a blocked worker. - INTERNALDATE month names are matched in any case. - Maildir import gets the same per-message savepoint, and commits every 500 new messages instead of once per folder. Tests: 1382 pass (7 new), fmt and clippy clean. Durability after each chunk cannot be observed from outside a running import (the archive is opened with an exclusive lock); the tests check its effects: a failed chunk keeps the ones before it, and a rerun fetches only the missing UIDs.

jcoffey-dev added 1 commit 2026-09-30 19:24:49 +00:00

import: keep what an interrupted IMAP or Maildir import wrote

ci / test (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 2m51s

Details

ci / announce (pull_request) Skipped

Details

2f33cd76a1

IMAP import wrote a whole folder in one transaction and stopped the folder
at the first message it could not import. A crash near the end of a large
INBOX kept nothing, and one bad INTERNALDATE lost the rest of the folder.
Worse, when a folder stopped early, fetches still in flight for it could be
filed into the next folder's mailbox.

- The transaction is committed after every fetch chunk. Each message is
  written in its own savepoint, so what is committed is always whole, and a
  rerun fetches only the UIDs still missing.
- A message that cannot be imported is rolled back on its own, logged with
  its folder and UID, and counted as failed; the folder carries on, and the
  message stays out of the UID map so the next run tries it again. Archive
  and I/O errors still stop the run.
- Fetch jobs and events carry a folder generation. Moving to a new folder
  cancels queued work for older ones, and any event from an older
  generation is dropped, never filed. Shutdown drains in-flight events
  before joining the workers, so it cannot hang on a blocked worker.
- INTERNALDATE month names are matched in any case.
- Maildir import gets the same per-message savepoint, and commits every 500
  new messages instead of once per folder.

jcoffey-dev referenced this pull request

2026-09-30 19:32:32 +00:00

import: bound the memory an IMAP or EWS import holds #9

jcoffey-dev merged commit 0a0d9b4e12 into main

2026-09-30 19:33:18 +00:00

jcoffey-dev deleted branch fix/imap-import-resilience

2026-09-30 19:33:18 +00:00

import: keep what an interrupted IMAP or Maildir import wrote #8