diff --git a/CHANGELOG.md b/CHANGELOG.md index d51adc4..ff68edd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,26 @@ # Change Log -All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](http://semver.org/). +All notable changes to this project are recorded here. Versions are dates: +release `v2026.9.30` is version `2026.9.30`. + +## [Unreleased] -- 2026.9.30 + +### Changed +- Renamed to inbuxa-migrate: the binary, the crate, and the credential + variables, now `INBUXA_MIGRATE_PASSWORD`, `INBUXA_MIGRATE_TOKEN`, + `INBUXA_MIGRATE_EWS_CLIENT_SECRET` and `INBUXA_MIGRATE_GRAPH_TOKEN`. The old + names are not read. Archives written by Vandelay are not read either. +- Registry calls (`x:Account`, `x:Domain` and the other `x:` types) use the + capability the server advertises: `urn:inbuxa:jmap:registry` on inbuxa, + `urn:stalwart:jmap` on a Stalwart source. A server that advertises neither + gets a clear error instead of a request it never offered to accept. +- Released as Linux archives for amd64 and arm64 on the Gitea release page, + with `SHA256SUMS`. The npm, Homebrew, shell and PowerShell installers and + the MSI are gone. + +--- + +# Earlier releases (upstream Vandelay) ## [1.0.11] - 2026-09-28 diff --git a/README.md b/README.md index 2b6c1e7..673a425 100644 --- a/README.md +++ b/README.md @@ -1,355 +1,103 @@ -

Vandelay

+# inbuxa-migrate -

- The JMAP importer-exporter -
- One-shot account migration and backup across JMAP, IMAP, CalDAV, CardDAV, WebDAV, ManageSieve, Maildir, Google Takeout and Microsoft Exchange. -

+> [!NOTE] +> Development happens on [git.coffeylabs.org/inbuxa/inbuxa-migrate](https://git.coffeylabs.org/inbuxa/inbuxa-migrate); the copy on GitHub is a read-only mirror. +> Report issues at **[git.coffeylabs.org/inbuxa/inbuxa-migrate/issues](https://git.coffeylabs.org/inbuxa/inbuxa-migrate/issues)**, and join discussions at **[community.coffeylabs.org](https://community.coffeylabs.org)**. -
+One program that moves an account into **inbuxa**: its mail, calendars, +contacts, identities, Sieve scripts and files. -

- build -   - release -   - npm -   - downloads -   - platforms -   - license -

+ inbuxa-migrate import imap … alice.sqlite an account into an archive + inbuxa-migrate inspect alice.sqlite what landed + inbuxa-migrate export … alice.sqlite the archive into inbuxa -
+Between the two sides sits an archive: one SQLite file holding one account. +`import` fills it from the old server; `export` writes it into the new one. +Import and export never talk to each other, only to the archive, so the old +server can be gone before the new one exists. - - - - - -
+Both commands converge. An interrupted run picks up where it stopped, and a +later `import` into the same archive adds what arrived since. Every command +takes `--dry-run` and then only reports the plan. `inspect` reads an archive +and changes nothing. -- Jerry: Well, what does **he** do? -- George: He's an importer. -- Jerry: Just imports, no exports? -- George: He's an **importer/exporter**, okay? +The archive is also a backup. Import on a schedule, keep the file, and +restore it later with `export`. An archive remembers which account filled it +and refuses a different one unless told otherwise. - -He's an importer-exporter. -
+Export speaks JMAP only. It writes into inbuxa, or into any JMAP server that +advertises the types being written. The full command reference is in +[docs/usage.md](docs/usage.md). -  +## Sources -## About +- **JMAP** -- any JMAP account, including an inbuxa or Stalwart server. +- **IMAP** -- mail only. Folders are chosen by name or by pattern. +- **CalDAV, CardDAV** -- calendars and events, address books and contacts. +- **WebDAV** -- a plain file collection, kept as a file tree. +- **ManageSieve** -- Sieve scripts only, with the active one recorded. +- **Maildir** -- a local Maildir++ tree. No network. +- **Google Takeout** -- the `.mbox`, `.ics` and `.vcf` files in a Takeout + export, or in any directory tree laid out that way. +- **Exchange Server** -- an on-premises mailbox, through EWS. +- **Exchange Online** -- through Microsoft Graph. EWS is being retired in + Exchange Online: blocked from 1 October 2026 unless an administrator allows + the client, and switched off on 1 April 2027. Public folders are only + reachable through EWS. -Vandelay is a one-shot account-migration utility for JMAP, the JMAP analogue of `imapsync` generalised to every JMAP data type (mail, contacts, calendars, identities, sieve scripts, file storage). It imports an account from a wide range of source protocols into a local SQLite "archive", then exports that archive into a target JMAP server. Import and export never talk to each other, only to the SQLite archive. One archive holds exactly one account. +## Credentials -Because the archive is a self-contained SQLite file that fully describes one account, vandelay doubles as a per-account backup tool: run an import on a schedule to capture a fresh snapshot, keep the resulting SQLite file as your backup, and restore it later by running an export against a JMAP target. +Secrets are read from the environment, or asked for at a prompt: -## Features + INBUXA_MIGRATE_PASSWORD a password for --auth-basic + INBUXA_MIGRATE_TOKEN a bearer token for --auth-bearer + INBUXA_MIGRATE_EWS_CLIENT_SECRET the OAuth client secret for app-only EWS + INBUXA_MIGRATE_GRAPH_TOKEN a Microsoft Graph access token -- **Many source protocols:** - - JMAP - - IMAP - - CalDAV - - CardDAV - - WebDAV - - ManageSieve - - Maildir++ - - Google Takeout - - Microsoft Exchange Server (on-premises) via EWS - - Microsoft Exchange Online via Graph -- **One target protocol:** JMAP, with type-by-type stateless re-matching on every run. -- **Convergent:** Re-running an interrupted import or export picks up where it left off without bookkeeping flags. -- **Multi-threaded, no async runtime:** Blocking HTTP with per-server concurrency caps respected automatically. -- **Content-addressed blobs:** Emails, sieve scripts and file-storage payloads are stored once by BLAKE3 hash and deduplicated across the archive. -- **Dry-run everywhere:** Every command supports `--dry-run` to compute the full plan without writing. -- **Source-change protection:** An archive remembers which account it was filled from; pointing it at a different one fails unless explicitly permitted. -- **Read-only inspection:** A built-in `inspect` command dumps any object type from an archive for verification. +The command line takes them too, but a secret there ends up in the shell's +history and in the process list. Use the environment or the prompt. -## Install +## Installing -```sh -# macOS / Linux -curl --proto '=https' --tlsv1.2 -LsSf \ - https://github.com/stalwartlabs/vandelay/releases/latest/download/vandelay-installer.sh | sh +Linux, on amd64 or arm64. Each release carries one archive per architecture +and a `SHA256SUMS` file: -# Homebrew -brew install stalwartlabs/tap/vandelay + base=https://git.coffeylabs.org/inbuxa/inbuxa-migrate/releases/latest/download + curl -fLO "$base/inbuxa-migrate-linux-amd64.tar.gz" + curl -fLO "$base/SHA256SUMS" + sha256sum --check --ignore-missing SHA256SUMS + tar -xzf inbuxa-migrate-linux-amd64.tar.gz + ./inbuxa-migrate --version -# Windows -powershell -ExecutionPolicy Bypass -c "irm https://github.com/stalwartlabs/vandelay/releases/latest/download/vandelay-installer.ps1 | iex" +On arm64, fetch `inbuxa-migrate-linux-arm64.tar.gz` instead. The binary is +the whole program; put it anywhere on the path. -# npm -npm install -g @stalwartlabs/vandelay +## Building and testing -# From source -cargo install --path . -``` + cargo build --release the binary is target/release/inbuxa-migrate + cargo test the default suite -A signed `.msi` is also published with each release. +The default suite needs no network and no Docker. Unit tests and scripted +mocks stand in for JMAP, DAV, EWS and Graph servers. -## Quick start +The live tests are marked `#[ignore]`. Each boots a throwaway container -- +Stalwart, Dovecot, Cyrus, Radicale, Baikal or Apache `mod_dav` -- so they +need a working Docker daemon. Run them one binary at a time, and always with +one test thread: -A typical run is two commands, one to capture a source account into a local SQLite archive and one to push that archive into a JMAP target. + cargo test --test sync_jmap -- --ignored --test-threads=1 + cargo test --test integration_dovecot -- --ignored --test-threads=1 -```sh -# 1. Import an IMAP mailbox into a fresh archive. -export VANDELAY_PASSWORD='source-app-password' -vandelay import imap \ - --url imaps://imap.example.com \ - --auth-basic alice@example.com \ - alice.sqlite - -# 2. Peek at what landed. -vandelay inspect alice.sqlite # per-type summary -vandelay inspect alice.sqlite mailbox # mailbox tree -vandelay inspect alice.sqlite email --limit 5 - -# 3. Push the archive into a target JMAP server. -export VANDELAY_PASSWORD='target-password' -vandelay export \ - --url https://jmap.example.org \ - --auth-basic alice@example.org \ - --account-name alice@example.org \ - alice.sqlite -``` - -Both commands are convergent: rerun either to resume an interrupted run, or rerun `import` later to pick up new mail since the last snapshot. Use `--dry-run` on either side to compute the full plan without writing. - -## CLI quick reference - -``` -vandelay [args...] -``` - -All actions accept a set of global flags (verbosity, worker pool size, retry policy, TLS handling) in addition to action-specific ones. The most useful are: - -| Flag | Purpose | -| --- | --- | -| `-j, --threads ` | Worker pool size (default: logical CPUs). | -| `--dry-run` | Compute the full plan; perform no writes. | -| `-v`, `-vv`, `-vvv` | Increase log verbosity. | -| `-q, --quiet` | Warnings and errors only. | -| `--max-retries ` | Max retries per request on transient failures (default 5). | -| `--allow-invalid-certs` | Accept self-signed / invalid TLS certs. | - -Credentials should be supplied via the `VANDELAY_PASSWORD` / `VANDELAY_TOKEN` / `VANDELAY_EWS_CLIENT_SECRET` / `VANDELAY_GRAPH_TOKEN` environment variables, or via an interactive prompt; passing them on the command line is supported but not recommended. - -### Import - -``` -vandelay import [source-args...] -``` - -Reads a source account into the local SQLite `ARCHIVE` (created if absent). Every importer accepts `--allow-source-change` to override the archive's source-identity guard. - -#### JMAP - -``` -vandelay import jmap \ - --url \ - (--auth-basic [--auth-password ] | --auth-bearer [TOKEN]) \ - (--account-id | --account-name ) \ - [--objects ] \ - -``` - -Imports a single JMAP account. `--objects` accepts a comma-separated list of object tokens (`mailbox,email,calendar,calendarevent,addressbook,contactcard,identity,sievescript,participantidentity,filenode`); default is everything the server advertises. - -#### IMAP - -``` -vandelay import imap \ - --url imap(s)://host[:port] \ - (--auth-basic [--auth-password ] | --auth-bearer [TOKEN] --auth-user ) \ - [--include ...] [--exclude ...] [--exclude-special ...] \ - [--folder ...] [--subscribed-only] [--noautomap] \ - [--include-deleted] [--allow-cleartext] [--compress] \ - [--fetch-batch ] [--imap-connections <1..8>] \ - -``` - -Imports mail (and only mail) from any IMAP server. Folder selection is via `--include`/`--exclude` regexes (mutually exclusive with the exact-match `--folder`); `--exclude-special` drops by SPECIAL-USE role. - -#### CalDAV - -``` -vandelay import caldav \ - --url \ - (--auth-basic [--auth-password ] | --auth-bearer [TOKEN]) \ - [--allow-cleartext] [--dav-connections <1..8>] [--multiget-batch ] \ - -``` - -Discovers the user's CalDAV principal (or accepts a URL pointing straight at a calendar-home or calendar), then imports calendars and events. - -#### CardDAV - -``` -vandelay import carddav \ - --url \ - (--auth-basic [--auth-password ] | --auth-bearer [TOKEN]) \ - [--allow-cleartext] [--dav-connections <1..8>] [--multiget-batch ] \ - -``` - -Same shape as `caldav`, but for address books and contacts. - -#### WebDAV - -``` -vandelay import webdav \ - --url \ - (--auth-basic [--auth-password ] | --auth-bearer [TOKEN]) \ - [--allow-cleartext] [--dav-connections <1..8>] [--multiget-batch ] \ - -``` - -Imports a plain WebDAV file collection as a JMAP `FileNode` tree. - -#### ManageSieve - -``` -vandelay import managesieve \ - --url sieve(s)://host[:port] \ - (--auth-basic [--auth-password ] | --auth-bearer [TOKEN] --auth-user ) \ - [--allow-cleartext] \ - -``` - -Imports sieve scripts only. Each script is content-addressed in the blob table; the active script is recorded. - -#### Maildir - -``` -vandelay import maildir \ - [--include ...] [--exclude ...] [--folder ...] \ - [--noautomap] [--include-deleted] -``` - -Reads a local Maildir++ tree (a directory with `cur/`, `new/`, `tmp/`). No network. Folder selection mirrors the IMAP importer. - -#### Google Takeout - -``` -vandelay import takeout [--noautomap] -``` - -Scans a directory tree recursively for `.mbox`, `.ics` and `.vcf` files and imports them. Tailored to Google Takeout layouts but works on any such tree; system-label role assignment can be disabled with `--noautomap`. - -#### Microsoft Exchange (EWS) - -``` -vandelay import exchange-ews \ - [--url ] [--mailbox ] \ - [--mailbox-kind primary|archive|public-folders] \ - (--auth-basic [--auth-password ] \ - | --auth-bearer [TOKEN] [--ews-tenant --ews-client-id \ - (--ews-device-code | --ews-client-secret )]) \ - [--ews-connections <1..8>] [--ews-getitem-batch ] [--ews-attachment-batch ] \ - [--ews-no-syncfolderitems] \ - -``` - -Imports a mailbox via EWS from an on-premises Exchange Server. Autodiscover is used when `--url` is omitted (a `--mailbox` SMTP address is then required). Supports Basic, pre-acquired bearer, interactive device-code OAuth, and app-only client-credentials OAuth. - -For Exchange Online, use `exchange-graph` instead. Microsoft is retiring EWS in Exchange Online: from 1 October 2026 it is blocked unless a tenant administrator sets `EwsEnabled` to `True` and adds the client id to `EwsAllowedAppIDs`, and on 1 April 2027 it is switched off for every tenant. On-premises Exchange Server is not affected. - -#### Microsoft Exchange (Graph) - -``` -vandelay import exchange-graph \ - (--client-id [--tenant ] | --access-token [TOKEN]) \ - [--user ] \ - [--mailbox-kind primary|archive] \ - [--objects mail,calendar,contacts] \ - [--event-body-format text|html] \ - [--graph-connections <1..16>] [--top <1..1000>] \ - -``` - -Imports a mailbox from Exchange Online via Microsoft Graph. Without `--access-token`, the interactive device-code flow is used. `public-folders` is rejected here: Graph does not expose public folders, so they can only be imported with `exchange-ews`, which for Exchange Online is subject to the retirement described above. - -### Export - -``` -vandelay export \ - --url \ - (--auth-basic [--auth-password ] | --auth-bearer [TOKEN]) \ - (--account-id | --account-name ) \ - [--objects ] [--prune [--yes]] \ - -``` - -Stateless re-export of `ARCHIVE` into a target JMAP server account. The default behaviour is upsert-only: matched items are updated, unmatched local items are created, but pre-existing target items not covered by the archive are left alone. - -`--prune` enables destructive reconciliation: target objects that do not match anything in the archive are deleted. The confirmation prompt can be skipped with `--yes` for automation. Export speaks JMAP only; no other target protocols are currently supported. - -### Inspect - -``` -vandelay inspect [TYPE] [--limit ] [--offset ] -``` - -Read-only dump of a local archive. This command never opens a network connection and never writes to the archive. - -- Omit `TYPE` for a per-type summary (counts of every object kind plus blob storage stats). -- Pass an object type to dump it: `mailbox`, `email`, `identity`, `sievescript`, `addressbook`, `contactcard`, `calendar`, `calendarevent`, `participantidentity`, `filenode`. -- `mailbox` and `filenode` render as a tree (`--limit`/`--offset` are ignored); all other types use a paginated list and respect `--limit` and `--offset`. - -## Testing - -The default suite is hermetic (unit tests plus `mockito`-scripted JMAP/DAV/EWS/Graph behaviours) and needs no network or Docker: - -```sh -cargo build -cargo clippy --all-targets -cargo test -``` - -### Live and integration tests (Docker required) - -Live integration tests against a Stalwart server and container-based tests against third-party servers (Dovecot, Cyrus, Radicale, Baikal, Apache `mod_dav`) are gated behind `--ignored`. **They require a running Docker daemon**: each test binary boots its own throwaway container via `testcontainers` (images are pulled automatically on first run), so Docker must be installed and `docker info` must succeed before invoking them. - -Run them per binary, and always with `--test-threads=1`: - -```sh -cargo test --test sync_jmap -- --ignored --test-threads=1 # live JMAP import/export/convergence/prune -cargo test --test sync_imap -- --ignored --test-threads=1 -cargo test --test sync_managesieve -- --ignored --test-threads=1 -cargo test --test sync_maildir -- --ignored --test-threads=1 -cargo test --test sync_caldav -- --ignored --test-threads=1 -cargo test --test sync_carddav -- --ignored --test-threads=1 -cargo test --test sync_webdav -- --ignored --test-threads=1 -cargo test --test live_stalwart -- --ignored --test-threads=1 -cargo test --test seed_smoke -- --ignored --test-threads=1 -cargo test --test seed_only -- --ignored --test-threads=1 - -# Third-party-server tests (one container each): -cargo test --test integration_radicale -- --ignored --test-threads=1 -cargo test --test integration_baikal -- --ignored --test-threads=1 -cargo test --test integration_webdav -- --ignored --test-threads=1 -cargo test --test integration_dovecot -- --ignored --test-threads=1 -cargo test --test integration_cyrus -- --ignored --test-threads=1 - -# Slow tests -cargo test --test mock_jmap -- --ignored -``` - -`--test-threads=1` is mandatory, not just advisory: within a binary every test shares a single per-binary container, and each test provisions then tears down the same disposable `vandelay.org` domain (and opens the archive with SQLite `EXCLUSIVE` locking). Separate binaries are isolated (each boots its own container on dynamic host ports), so plain `cargo test --test ` invocations are safe to run one after another. +The tests in a binary share one container and one disposable domain, so a +second thread would trip over the first. [docs/usage.md](docs/usage.md) lists +every live test binary. ## License -Licensed under either of +Apache-2.0 OR MIT, at your option. The texts are in [LICENSES](LICENSES). - * Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0) - * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT) +Copyright (C) 2020, Stalwart Labs LLC
+Copyright (C) 2026, John Coffey -at your option. - -## Copyright - -Copyright (C) 2020, Stalwart Labs LLC +Forked from Vandelay, originally developed by Stalwart Labs, and distributed +under the same Apache-2.0 OR MIT terms.