import jmap read the shared ~/.config/inbuxa profile, which names the inbuxa server and holds its API key, and fell back to that key whenever no flag or INBUXA_MIGRATE_* variable gave one -- even with a bare --auth-bearer, which is documented to prompt. But import jmap reads FROM a source server, often another provider's, so the inbuxa key went as a bearer token to whatever --url named. An operator with a default profile who ran import jmap against another server handed that server (or anyone who controlled or watched the URL) an inbuxa administrator key. import jmap no longer reads the profile: --url is required again, --profile is gone from it, and the source's credentials come from a flag, an INBUXA_MIGRATE_* variable or the prompt. import-directory, which does write to the inbuxa server, keeps the profile, but now sends its key only to the profile's own origin; a --url with another scheme, host or port gets no key from the profile. The --url help, the README, usage.md and the missing-secret message no longer describe import as writing to inbuxa. Test: import_jmap_never_takes_the_url_or_key_from_the_profile and import_directory_sends_the_profile_key_only_to_the_profile_url failed before (the profile's url and key were used for import jmap, and the key went to another --url on import-directory); both pass now.
50 KiB
Using inbuxa-migrate
The full command reference. The README says what the tool is and how to install it.
Commands and global flags
inbuxa-migrate <import|export|export-imap|export-managesieve|export-caldav|export-carddav|export-files|export-directory|import-directory|migrate|guide|plan|inspect|reconcile> [args...]
Every command takes a few global flags -- verbosity, worker pool size, retry policy, TLS handling -- besides its own. The ones that matter most:
| Flag | Purpose |
|---|---|
-j, --threads <N> |
Worker pool size (default: logical CPUs). |
--dry-run |
Compute the full plan; perform no writes. |
-v, -vv, -vvv |
Increase log verbosity. |
-q, --quiet |
Warnings and errors only. |
--max-retries <N> |
Max retries per request on transient failures (default 5). |
--allow-invalid-certs |
Accept a self-signed or otherwise invalid certificate from the server named by --url (see below). |
Invalid certificates
--allow-invalid-certs is for a server with a self-signed certificate, and
it applies to that server only: the host in --url, whether that is the
source of an import or the target of an export. For an Exchange import with
no --url, it covers the mailbox's own domain and the hosts under it, which
is where on-premises Autodiscover looks, and then only the EWS endpoint
Autodiscover finds.
Every other host is verified as usual, including a host the server redirects to or names for its API, uploads or downloads.
Staying on the address given
A JMAP server's session names the addresses for its API, uploads and
downloads, and the tool uses them, warning when they're on another host
than --url. --stay-on-url sends every request to the --url host
instead, keeping each path. It's for a server reached at an address it
doesn't advertise: its own loopback, an SSH tunnel, a node behind a load
balancer. import jmap, export, export-directory and
import-directory take it. The Microsoft and
Google sign-in and cloud endpoints are always verified, with or without the
flag: a certificate that fails there is an attack or a broken network, never
a server to trust.
Connections time out rather than wait forever: 30 seconds to connect, 5 minutes for the server's first byte, and 30 minutes to read a whole response. A transfer that goes silent for 5 minutes in either direction also times out. A timed-out request is retried like any other transient failure.
Secrets come from the INBUXA_MIGRATE_* environment variables, a file named
by the same variable with _FILE appended (the Docker and systemd secrets
convention), or a prompt; see Credentials. The
command line takes them too, but should not be given them.
import-directory writes to an inbuxa server. Its address and API key can be
named once in a shared profile, ~/.config/inbuxa/config.toml (which
inbuxa-cli reads too), so --url and the key need not be repeated:
[profile.default]
url = "https://mail.example.org"
api-key = "API_…"
--profile or $INBUXA_PROFILE pick a [profile.<name>] section, and the
default is default; a section gives url and either api-key or
api-key-file (a path to a file holding the key). A --url, or any
INBUXA_MIGRATE_* variable, still takes precedence, so --url is optional
only when a profile supplies it. The key goes only to the profile's own
server: a --url with another scheme, host or port gets no key from the
profile. The import commands read from a source server, not the inbuxa
server, so they never read the profile; give them the source's own
credentials. See
Credentials.
Import
inbuxa-migrate import <source> [source-args...] <ARCHIVE>
Reads a source account into the SQLite ARCHIVE, creating it if it is absent.
An archive remembers which account filled it; every importer takes
--allow-source-change to fill it from a different one anyway. The DAV
importers (caldav, carddav, webdav) also take --replace-source, which
first removes the other sources of the same kind, and what came from them, so
a share that moved leaves one tree in the archive, not two.
JMAP
inbuxa-migrate import jmap \
--url <URL> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN]) \
(--account-id <ID> | --account-name <NAME>) \
[--objects <list>] \
<ARCHIVE>
Imports one JMAP account, from inbuxa, Stalwart or any other JMAP server. --objects accepts a comma-separated list of object tokens (mailbox,email,calendar,calendarevent,addressbook,contactcard,identity,sievescript,participantidentity,filenode). The default is everything the server advertises.
The out-of-office reply, the server's VacationResponse, comes with
sievescript. Files are read in either shape servers spell them: nodeType
and type, as inbuxa and other servers on the 0.16 line do, or @type and
mediaType, as draft-ietf-jmap-filenode-18 servers such as Stalwart 1.0 do.
Fastmail
inbuxa-migrate import fastmail \
[--token [TOKEN]] \
[--url <URL>] \
[--account-id <ID> | --account-name <NAME>] \
[--objects <list>] \
<ARCHIVE>
The same as import jmap, with Fastmail's server
(https://api.fastmail.com/jmap/session) and bearer-token sign-in already
filled in, so it needs only a token. Create one in Fastmail at Settings >
Privacy & Security > API tokens with read access to Mail, Calendars and
Contacts; pass it with --token, in $INBUXA_MIGRATE_API_KEY, or at the
prompt. With no --account-* it imports the account Fastmail marks primary,
which is the one you want unless you have several. What crosses over is
whatever that account's session advertises: mail always, calendars and
contacts where Fastmail offers them over JMAP. --url overrides the server
for a Fastmail-compatible host.
IMAP
inbuxa-migrate import imap \
--url imap(s)://host[:port] \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN] --auth-user <USER> \
| --google-key <FILE> --auth-user <USER>) \
[--include <REGEX>...] [--exclude <REGEX>...] [--exclude-special <ROLE>...] \
[--folder <NAME>...] [--subscribed-only] [--noautomap] \
[--include-deleted] [--allow-cleartext] [--compress] \
[--fetch-batch <N>] [--fetch-batch-mib <MIB>] [--imap-connections <1..8>] \
<ARCHIVE>
Imports mail, and only mail, from any IMAP server. Folders are chosen with
--include and --exclude patterns, or by exact name with --folder, but
not both. --exclude-special drops folders by SPECIAL-USE role.
Messages are fetched in chunks of at most --fetch-batch messages and
--fetch-batch-mib MiB (32 by default), so a folder of large attachments is
fetched a little at a time, like any other. A single message larger than the
cap is fetched on its own. Each chunk is written to the archive as it
arrives: an interrupted import keeps what it fetched, and the next run picks
up from there. A message that can't be imported is reported with its folder
and UID, and the rest of the folder carries on.
Gmail, a whole Google Workspace domain. With --google-key the tool
signs in as a Workspace service account acting for the user --auth-user
names, so no one's password is needed. In a plan that is
--url imaps://imap.gmail.com --google-key key.json --auth-user {email}.
--google-key-from-env reads the key's JSON from
$INBUXA_MIGRATE_GOOGLE_KEY instead of a file; in a plan, the step names
the variable that holds it as "secrets": {"googleKey": "..."}.
- In Google Cloud, create a service account and a JSON key for it.
- In the Workspace Admin console, under domain-wide delegation, allow the
service account's client id the scope
https://mail.google.com/. IMAP has to be allowed for the users, in the Admin console's Gmail settings. - The key file holds the account's private key and reaches every mailbox in the domain: keep it readable by you alone, and delete the key in the Cloud console once the move is done.
Each connection signs in with a token for that user, and takes a new one when the last is close to running out.
POP3
inbuxa-migrate import pop3 \
--url pop3s://host[:port] | pop3://host[:port] \
--user <USER> [--password <PASS>] \
[--allow-cleartext] [--allow-source-change] \
<ARCHIVE>
Imports mail, and only mail, from a POP3 server. POP3 has no folders and no flags, so the whole maildrop lands in one INBOX. Each message is keyed by its UIDL -- the stable unique-id POP3 gives it -- so a later run into the same archive adds only what arrived since.
pop3s:// is TLS from the first byte (port 995 by default). pop3:// (port
110) upgrades with STLS before the password is sent; --allow-cleartext
permits a plain pop3:// connection with no STLS, sending the password in the
clear, for a trusted local network only. The password comes from --password,
$INBUXA_MIGRATE_PASSWORD, or the prompt (prefer the last two; a password on
the command line is visible in the shell history and the process list).
Two things set POP3 apart from the other sources:
- It is additive. A message that is in the archive but no longer on the server is kept, not deleted. POP3 mailboxes are routinely emptied after download -- that is the protocol's own model -- so a message leaving the server says nothing about whether the archive should still hold it. This is the one importer that does not mirror deletions.
- It never changes the source. There is no
DELE: a run only reads, so it is safe to repeat and leaves the server mailbox exactly as it found it.
OAuth (SASL XOAUTH2) is not offered here; use --user/password over TLS. A
provider that requires an app-specific password (for an account with
two-factor sign-in) takes that password the same way.
Zoho Mail
inbuxa-migrate import zoho \
[--region com|eu|in|au|jp] [--host <HOST>] \
--user <EMAIL> [--password <PASS>] \
[--allow-source-change] \
<ARCHIVE>
A preset for import imap that fills in Zoho's IMAP server for your region, so
it needs only your address and password. It imports mail only, over IMAPS on
993.
Your IMAP host matches the data center your account signs in at: if you sign in
at mail.zoho.eu, your host is imap.zoho.eu. Pick it with --region
(com is the default, for the US). For a data center not in that list -- a
Zoho Workplace host such as imappro.zoho.com, or another region -- give
--host instead, which overrides --region.
In Zoho Mail, enable IMAP under Settings > Mail Accounts > IMAP Access. If
the account signs in with two-factor authentication, create an app-specific
password and use that. For folder filters and the other IMAP options, use
import imap with --url imaps://<host>:993.
Zimbra
inbuxa-migrate import zimbra \
--host <HOST> --user <EMAIL> [--password <PASS>] \
[--starttls [--allow-cleartext]] [--allow-source-change] \
<ARCHIVE>
A preset for import imap pointed at your own Zimbra server, so it needs only
the host, your address and password. It imports mail only, over IMAPS on 993 by
default; --starttls connects on 143 and upgrades, and --allow-cleartext
(only with --starttls) permits a plain connection where STARTTLS is
unavailable.
IMAP has to be enabled for the account -- zimbraImapEnabled, in its class of
service or account features. For folder filters and the other IMAP options, use
import imap with the same URL.
CalDAV
inbuxa-migrate import caldav \
--url <http(s)://host[/path]> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN]) \
[--allow-cleartext] [--dav-connections <1..8>] [--multiget-batch <N>] \
<ARCHIVE>
Discovers the user's CalDAV principal (or accepts a URL pointing straight at a calendar-home or calendar), then imports calendars and events.
A recurring event comes in with its changed occurrences. Someone invited to
single occurrences of a series, without the series itself, has each of those
occurrences as its own event, and export, export-caldav and
export-files write them back the same way.
CardDAV
inbuxa-migrate import carddav \
--url <http(s)://host[/path]> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN]) \
[--allow-cleartext] [--dav-connections <1..8>] [--multiget-batch <N>] \
<ARCHIVE>
Same shape as caldav, but for address books and contacts. A card the
server keeps in two address books, by the same UID, is one card in both. If
it leaves one book, or that book is deleted, it stays in the other; if the
copies differ, the one read last is kept.
WebDAV
inbuxa-migrate import webdav \
--url <http(s)://host[/path]> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN]) \
[--allow-cleartext] [--dav-connections <1..8>] [--multiget-batch <N>] \
<ARCHIVE>
Imports a plain WebDAV file collection as a JMAP FileNode tree.
ManageSieve
inbuxa-migrate import managesieve \
--url sieve(s)://host[:port] \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN] --auth-user <USER>) \
[--allow-cleartext] \
<ARCHIVE>
Imports Sieve scripts only. Each script is stored once by content, and the active one is recorded.
Maildir
inbuxa-migrate import maildir <MAILDIR> <ARCHIVE> \
[--include <REGEX>...] [--exclude <REGEX>...] [--folder <NAME>...] \
[--noautomap] [--include-deleted]
Reads a local Maildir++ tree: a directory with cur/, new/ and tmp/. No
network. Folders are chosen as for IMAP.
Google Takeout
inbuxa-migrate import takeout <PATH> <ARCHIVE> [--noautomap]
Finds every .mbox, .ics and .vcf file under a directory and imports it.
It is shaped for Google Takeout but reads any such tree. --noautomap stops
it giving Gmail's system labels their mailbox roles.
An account export
inbuxa-migrate import files <DIR> <ARCHIVE>
Reads an account export back: the directory export-files writes, laid out
as https://coffeylabs.org/specs/account-export/ specifies, from this tool
or any other that writes the layout. No network. Where import maildir and
import takeout read the open formats on their own, this also reads the
manifest, so what those formats can't hold comes back too:
- every folder's real name, its place in the tree, its role and whether it was subscribed, in place of the names Maildir++ could give it on disk;
- which address book and calendar were the default, and each calendar's description, color and time zone;
- which Sieve script was active;
- the files' media types and links, and the real name of any file or directory whose name had to change on disk;
- the identities with their reply-to, bcc and signatures, and the calendar identities.
A message kept in several folders is written to each folder's directory; copies with the same content come back as one message in all of those folders. The same goes for a card in two address books and an event in two calendars. A message's arrival time is its file's modification time.
The manifest's paths are data, so each is checked to stay inside its part of
the export, and nothing that is a symbolic link is followed: a link in the
tree is skipped with a warning, and the links an account had are made from
the manifest as archive entries. A directory without manifest.json was
interrupted, and is refused, as is a later version of the layout than this
release reads. 27.1.0-rc.2 named the format inbuxa-migrate-files; that is
read as the same format.
Run again, it brings the archive up to date with the export: flags that
changed, a different active script, a renamed folder. It never deletes.
With --dry-run it checks the export and counts what it holds, and opens no
archive.
Microsoft Exchange (EWS)
inbuxa-migrate import exchange-ews \
[--url <EWS-ENDPOINT>] [--mailbox <SMTP>] \
[--mailbox-kind primary|archive|public-folders] \
(--auth-basic <USER> [--auth-password <PASS>] \
| --auth-bearer [TOKEN] [--ews-tenant <T> --ews-client-id <ID> \
(--ews-device-code | --ews-client-secret <SECRET>)]) \
[--ews-connections <1..8>] [--max-rps <RPS>] \
[--ews-getitem-batch <N>] [--ews-getitem-batch-mib <MIB>] \
[--ews-attachment-batch <N>] \
[--ews-no-syncfolderitems] \
<ARCHIVE>
Imports a mailbox from an on-premises Exchange Server through EWS. Without
--url it uses Autodiscover, and then needs --mailbox. It signs in with
Basic, with a bearer token acquired beforehand, with OAuth's interactive
device-code flow, or with app-only client credentials.
Items are fetched in GetItem batches of at most --ews-getitem-batch items
and --ews-getitem-batch-mib MiB (32 by default), --ews-connections at a
time. The byte cap applies where Exchange reports each item's size, which it
does on a full listing; items found through an incremental sync are batched
by count.
The server throttles on its own, and a 429 or 503 is honored (its
Retry-After, then a shared backoff across the workers). To stay under a
known limit in the first place, --max-rps caps how many requests start per
second across all connections; leave it off to run uncapped.
For Exchange Online, use exchange-graph instead. Microsoft is retiring EWS in Exchange Online: from October 1, 2026 it is blocked unless a tenant administrator sets EwsEnabled to True and adds the client id to EwsAllowedAppIDs, and on April 1, 2027 it is switched off for every tenant. On-premises Exchange Server is not affected.
Microsoft Exchange (Graph)
inbuxa-migrate import exchange-graph \
(--client-id <UUID> [--tenant <ID>] [--app-only] | --access-token [TOKEN]) \
[--user <UPN|UUID>] \
[--mailbox-kind primary|archive] \
[--objects mail,calendar,contacts] \
[--event-body-format text|html] \
[--graph-connections <1..16>] [--max-rps <RPS>] [--top <1..1000>] \
<ARCHIVE>
Imports a mailbox from Exchange Online through Microsoft Graph. Without
--access-token it signs in with the interactive device-code flow. public-folders is rejected here: Graph does not expose public folders, so they can only be imported with exchange-ews, which for Exchange Online is subject to the retirement described above.
Graph throttles per app and per mailbox, and a 429 or 503 is honored (its
Retry-After, then a shared backoff across the workers). --max-rps caps how
many requests start per second across all connections, to stay under a tenant's
limit before it pushes back; leave it off to run uncapped.
With --app-only the app signs in as itself, with its client secret, and
reads the mailbox --user names. No one signs in, so a whole tenant moves
with one app, in a plan with --user {email}.
- The app. Register it in Microsoft Entra, add a client secret, and
grant it these Microsoft Graph application permissions with admin
consent:
User.Read.All,Mail.Read,MailboxSettings.Read,Calendars.ReadandContacts.Read, andFiles.Read.Allfor files. To keep it to some mailboxes, scope it with Exchange Online's RBAC for Applications or an application access policy. - The tenant.
--tenantnames it, as its id orname.onmicrosoft.com; the sharedcommonandorganizationsare refused. - The secret is read from
$INBUXA_MIGRATE_CLIENT_SECRET, or asked for. In a plan, the step names it asclientSecret. - Its token lasts about an hour and is renewed the same way before it runs out.
Export
inbuxa-migrate export \
--url <URL> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN]) \
(--account-id <ID> | --account-name <NAME>) \
[--objects <list>] [--prune [--yes]] \
<ARCHIVE>
Writes ARCHIVE into an account on a JMAP server, usually inbuxa. It keeps no
state of its own: every run matches the archive against the target afresh.
By default it only adds and updates: what the target lacks is created, what
it has is brought up to date, and anything on the target that the archive
does not cover is left alone. So a second run after a later import carries
what changed at the source in between.
- Email is matched by Message-ID, or without one by sender, subject, date and recipients; where several messages share one, size decides. A message the source kept in several folders -- IMAP and Maildir copies, Gmail labels -- is written once, in all of them. On a match, its keywords (read, flagged and the rest) are set to the archive's, and so are its memberships of folders this run migrated. Folders that exist only on the target are left alone, and a message is never left in no folder.
- Contacts and events are matched by UID. When both copies carry an
updatedtime, the archive's is written only if it is newer; otherwise the properties that differ are written. - Sieve scripts are matched by name, and the target's is replaced when
its content differs from the archive's. Where the target has a
VacationResponse, the out-of-office reply is written there, and the script a server keeps for it is not copied as a script. - Files are written in the shape the target takes, which
exportasks it once per run: inbuxa takesnodeTypeandtype.
Messages go in batches, as many to an Email/import as the target's
maxObjectsInSet allows, up to 50, with their blobs uploaded several at a
time -- the target's maxConcurrentUpload, and no more than --threads.
Each message is still counted on its own: one the target rejects fails
alone, and the rest of its batch lands. A batch that ends without a clear
answer -- a dropped connection, a gateway timeout -- is never sent again as
it was. The target is read first, the messages that arrived are counted as
created, and only the rest are imported again, so none is ever doubled.
While it runs, export prints a line every few seconds for mail, contacts and events: how many of how many, how fast, and about how long is left. Each type ends with a line of what was created, updated, left unchanged and failed.
--prune also deletes what is on the target and not in the archive. It asks
first; --yes answers for it, for scripts. Export speaks JMAP only; for
other servers, see Export to IMAP and
Export to files.
With --dry-run, export reads the target and writes nothing to the account.
It prints the plan in plain words: for each type, how much would be
created, updated, left unchanged or deleted, and what would fail. It
catches in advance the failures a real run would hit:
- a message larger than the target's
maxSizeUpload; - a contact, event or other object too large for one request under its
maxSizeRequest, as happens when a photo is carried inline; - a Sieve script the target would reject. Each script that would be written
is checked with
SieveScript/validate, after anyvnd.stalwart.*names are renamed, so what is checked is what would be uploaded. The check needs the script as a blob, so the dry run uploads Sieve scripts, and only them; a blob that nothing uses is discarded by the server. A target withoutSieveScript/validategets one warning, and its scripts are not checked.
The plan lists each predicted failure and its reason. When anything would fail, the dry run exits 5, as the real run would, so a script can stop before it starts.
Export to IMAP
inbuxa-migrate export-imap \
--url <imap://host[:port]|imaps://host[:port]> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN] --auth-user <USER>) \
[--allow-cleartext] [--dry-run] \
<ARCHIVE>
Writes an archive's mail into an account on any IMAP server, for a target
that doesn't speak JMAP. It signs in the way import imap does, with the
same flags. Contacts, calendars and filters don't go over IMAP; write the
filters with export-managesieve, and contacts and calendars with
export-carddav and export-caldav.
Like export, it keeps no state. Each run reads the target folder first,
matching messages by Message-ID, or without one by sender, subject, date and
recipients, with size deciding between equals. What's missing is appended,
with its flags and the time it arrived; what's there gets the archive's
flags if they differ. So a second run writes nothing, and an interrupted run
is finished by running it again: a message that landed before the
connection dropped is found, not doubled.
- Folders. The inbox goes to INBOX. A sent, drafts, trash, junk or
archive folder goes to the one the server marks for it with SPECIAL-USE,
such as Gmail's
[Gmail]/Sent Mail, and is created with that mark when the server allows it. Every other folder is created under the account's personal namespace with its path, using the server's separator; that separator inside a name becomes_. Names go as UTF-8 to a server that accepts it, in modified UTF-7 otherwise. - Flags. Read, answered, flagged and draft are IMAP's own. Other keywords (Junk, labels) go along where the folder keeps keywords, and the run says how many were left off where it doesn't. Deleted isn't carried, so the next client to expunge removes nothing that was written.
- A message the server refuses (too large, over quota) is counted as failed and the run goes on; a dropped connection ends the run.
With --dry-run it signs in, reads the target with EXAMINE, and counts
what it would create, append and update, without writing anything.
Export to ManageSieve
inbuxa-migrate export-managesieve \
--url <sieve://host[:port]|sieves://host[:port]> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN] --auth-user <USER>) \
[--allow-cleartext] [--dry-run] \
<ARCHIVE>
Writes an archive's Sieve scripts into an account on any ManageSieve server
(RFC 5804), with the flags import managesieve takes. It keeps no state:
each run lists the target's scripts, writes each one that is missing or
differs with PUTSCRIPT, and leaves alone the ones that match, so a second
run writes nothing. A script with no name is written as script-<id>.
- The archive's active script is made active on the target. If no script in the archive is active, the target's active script stays as it is.
- A script the server refuses, for instance one using an extension it lacks, is counted as failed with the server's reason, and the run goes on. If the active script is refused, it is not made active.
- Scripts are never deleted from the target, and names that match are overwritten only when their content differs.
With --dry-run it signs in, lists and compares, and checks each script it
would write with CHECKSCRIPT, so a script the server would refuse is
counted as failed before anything is written. A server without
CHECKSCRIPT (one that advertises no VERSION) gets the comparison only.
Export to CalDAV and CardDAV
inbuxa-migrate export-caldav \
--url <http(s)://host[:port][/path]> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN]) \
[--allow-cleartext] [--dry-run] \
<ARCHIVE>
inbuxa-migrate export-carddav …same flags… <ARCHIVE>
Writes an archive's calendars into any CalDAV server, or its address books
into any CardDAV server, with the flags import caldav and import carddav
take. --url can be the server's root: the account's home is found from
the signed-in user's principal, through /.well-known/caldav or
/.well-known/carddav if need be, so a fresh account with nothing in it
works. Like the other exports it keeps no state, and a second run writes
nothing.
- Collections. A calendar or address book is matched to the target's
by its display name. One that isn't there is made with
MKCALENDARor an extendedMKCOL, with its name, description and (for a calendar) color, at a path made from the name that nothing in the home uses yet. - Items. Each event or card goes to
<UID>.icsor<UID>.vcfin its collection; a UID that isn't a plain name is hashed instead. A new item is written withIf-None-Match: *. One that is there is read and compared, and written again withIf-Matchon its etag only when it differs. The comparison leaves out what servers rewrite on their own: line folding, the order and case of parameters,PRODID,DTSTAMPandREV. A card'sNandADRare compared on the parts vCard 3 has, since some servers drop the rest; a change only in those newer parts is not sent. - An item in two collections is written to both. An item, or a collection, the server refuses is counted as failed with the reason, and the run goes on; a collection that can't be made has its items counted as failed too. An item whose server gives no etag isn't overwritten.
- Nothing is deleted from the target.
With --dry-run it signs in, finds the home, reads and compares what is
there, and counts what it would create and update, without writing.
Export to files
inbuxa-migrate export-files [--objects <list>] [--dry-run] <ARCHIVE> <DIR>
Writes ARCHIVE out as plain files in open formats, for a server or app
that doesn't speak JMAP, or to keep a copy that needs no tool to read. It
never opens a network connection. DIR must be new or empty, and is made
readable by its owner only, like the archive.
DIR/
mail/ Maildir++: the inbox here, other folders as .Parent.Child
contacts/*.vcf one file per address book
calendars/*.ics one file per calendar
sieve/*.sieve one file per script
files/ the file tree
manifest.json what the formats above don't hold
- Mail is laid out as Dovecot and Courier keep it, so either can serve
it as it is. Folder names are in
modified UTF-7. A
.or/inside a folder's name becomes_on disk, since Maildir++ uses the dot between levels. A message kept in several folders is written to each. Read, answered, flagged, draft, forwarded and deleted are the standard Maildir flags; other keywords (Junk, labels) are the lettersatoz, named in each folder'sdovecot-keywords, up to 26 per folder. Each file is dated when the message arrived. - Contacts are vCard 4.0, and a card in two address books is in both
files. Calendars are iCalendar, one VCALENDAR each, named by
X-WR-CALNAME, with each time zone once. Photos and attachments the archive holds go inside asdata:URIs, asexportsends them. - Sieve scripts are written as they are; the manifest says which is active.
- Files keep their names and dates. Links are listed in the manifest and never made, since the source chose where they point.
Every name comes from the source, so none is used as a path as it stands:
separators and names Windows refuses become _, and names that clash, even
only by case, are numbered. The manifest keeps the real name of every file
and directory whose name changed, so a reader can restore it.
manifest.json records the source account, every folder's real name,
path, role and message count, each address book and calendar with its
file, color and time zone, which Sieve script is active, the identities,
and the links. It is written last: a tree without one was interrupted, and
the way to finish it is to run again into an empty directory.
The layout and the manifest are specified at
https://coffeylabs.org/specs/account-export/, so other tools can read and
write them too. import files reads the whole tree back, manifest and all
(An account export); import maildir and import takeout read the mail, contacts and calendars without it.
--objects takes the same types as export; naming a folder or a message
type writes all of mail, and the same for contacts and calendars. With
--dry-run it counts what it would write and creates nothing.
Export a directory
inbuxa-migrate export-directory \
--url <URL> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN]) \
[--domain <NAME>]... [--dry-run] \
<DIR>
Writes who has an address on a server, so the same accounts can be set up
on another one: domains, users, groups, mailing lists, their aliases and
quotas, and the tenants and roles they belong to. It reads inbuxa's
registry, or Stalwart's, so the login must be an administrator's. Mail and
the rest of each account's data move with import and the exports above.
Nothing secret is written, because nothing secret is asked for: every read
names the fields it wants, and credentials, app passwords, OTP seeds, API
keys, encryption keys and DKIM keys are not among them. Each new server
sets its own passwords and DKIM keys; import-directory, below, does that
on inbuxa.
DIR/
directory.json everything below, exactly, with roles and tenants
domains.csv name, enabled, aliases, catch-all, sub-addressing, tenant
users.csv address, description, roles, groups, aliases, disk quota
groups.csv address, members, aliases, disk quota
lists.csv address, recipients, aliases
aliases.csv each alias with the address it delivers to
- Accounts, groups and lists are named by address, and roles and tenants by name, never by the server's ids. A group lists its members, a user the groups it is in.
- Roles are
user,adminordefault, or the custom roles by name;permissionsholds what an account adds to its roles (merge) or has in their place (replace). - Quotas are by name, in bytes or counts (
maxDiskQuotaand the rest); the CSV files showmaxDiskQuotaasdisk_quota. - In the CSV files, a cell with several values joins them with
;. The text comes from the server and these files are made to be opened in a spreadsheet, so a cell that one would run as a formula starts with'.directory.jsonkeeps every value as it is.
--domain keeps the export to the domains named, with what they use:
their tenants and the roles their accounts hold. A group's members on
other domains are still listed by address. DIR must be new or empty, and
is made readable by its owner only. directory.json is written last, so a
directory without one was interrupted. With --dry-run it reads the
registry and counts what it would write, and creates nothing.
Import a directory
inbuxa-migrate import-directory \
--url <URL> \
(--auth-basic <USER> [--auth-password <PASS>] | --auth-bearer [TOKEN]) \
--passwords <FILE> [--domain <NAME>]... [--dry-run] \
<DIR>
Sets up on an inbuxa server what export-directory wrote: tenants, roles,
domains, groups, users and mailing lists, with their aliases, quotas, roles
and group memberships. It works through the server's registry, so the
login must be an administrator's. Each kind is made before the ones that
point at it, so a user finds its domain, its groups and its roles.
- What's already there stays as it is. A domain, tenant or role with the same name, or an account or list with the same address, is counted as skipped and not changed. So a second run creates nothing, and a run that stopped is finished by running it again. It also means an account made on a run where one of its alias domains was refused keeps no alias there: add that alias by hand.
- Passwords. Each user it creates gets a generated password: 24
letters and digits, with look-alikes such as
0andOleft out. The passwords go into--passwords FILEasemail,passwordrows, written as each batch of accounts is made. The file must not exist yet and is made readable by its owner only. Hand the passwords out, then delete the file. Accounts that were already there get no password and aren't listed, so each run needs a new file. - DKIM. A new domain gets inbuxa's defaults: DKIM keys are made for it, and DNS and certificates are left to you.
- Anything the server refuses is counted as failed, with the server's reason, and the rest are still made. An alias or custom role that isn't on the server is left off with a warning. Creates are sent once and not retried, so a lost reply never makes an account twice; run again to pick up what didn't land.
--domain keeps it to the domains named. With --dry-run it reads the
server and counts what it would create, sends nothing and writes no file;
--passwords isn't needed then.
Migrate many accounts
inbuxa-migrate migrate --plan <FILE> [--parallel <N>] [--again]
[--repeat <DURATION> [--max-passes <N>] [--until-quiet]]
[--account <EMAIL>]... [--dry-run] [-q] [--progress json] <DIR>
Moves a list of accounts with one command. The plan file names the steps
and the accounts. Each step is one of the commands on this page, written
the same way, and migrate runs it for every account, several accounts at
once. <DIR> is the work directory. It holds an archive and a log per
account, the state a later run resumes from, and the report.
{
"format": "inbuxa-migrate-plan",
"formatVersion": 1,
"setup": [
{"name": "directory",
"args": ["import-directory", "--url", "https://mail.example.org",
"--auth-bearer", "--passwords", "passwords.csv", "directory"],
"secrets": {"token": "INBUXA_ADMIN_TOKEN"}}
],
"accounts": {"directory": "directory/directory.json"},
"steps": [
{"name": "mail",
"args": ["import", "imap", "--url", "imaps://old.example.org",
"--auth-basic", "{email}*master", "{archive}"],
"secrets": {"password": "OLD_MASTER_PASSWORD"}},
{"name": "inbuxa",
"args": ["export", "--url", "https://mail.example.org", "--auth-bearer",
"--account-name", "{email}", "{archive}"],
"secrets": {"token": "INBUXA_ADMIN_TOKEN"}}
]
}
- Steps.
setupsteps run once, before any account:import-directoryorexport-directory. Thestepsrun for each account, in order, and each has to name{archive}.{email},{local}and{domain}are the account's address and its two halves. Steps run in the plan's directory, so a relative path means the same as it does in the plan. - Accounts. A list of addresses, or
{"directory": "…/directory.json"}for the users thatexport-directorywrote. - Secrets stay out of the plan. A step names the environment variable
that holds each secret it needs:
password,token,clientSecret(EWS, or Graph with--app-only) orgraphToken. A step sees only the secrets it names. A plan that writes a password or token intoargsis refused. A step that needs a secret it doesn't name fails at once instead of waiting for a prompt. - Signing in for each account. Each step signs in the way its command
does, so the plan decides how. To avoid collecting every user's password,
use a login that can act for each account. The example uses Dovecot's
master user,
{email}*master. Use the source's own form where it has one: Exchange impersonation through--mailbox {email}, a Graph app with--app-only --user {email}, a Google service account with--google-key key.json --auth-user {email}, or an administrator on a JMAP source with--account-name {email}. - One account at a time, several at once.
--parallel(default 4) sets how many accounts run at the same time. Each step runs as its own process. Its output goes to<DIR>/<email>/<step>.log, and one account failing doesn't stop the others. A step that fails stops its account. A step that leaves some objects behind (exit code 5) doesn't. A failed setup step stops the run. - Resuming. A second run with the same plan and work directory skips
every account that finished. For the rest, it skips the steps that were
done and runs everything from the first one that wasn't.
--againruns every account step once more, done or not. Use it for the final pass once the old server stops taking mail: the steps converge, so it only moves what changed.--accountlimits a run to the accounts named. - Keep syncing until cutover.
--repeat <DURATION>runs the accounts again and again, waiting that long between passes (30s,5m,1h), each pass moving only what has arrived at the source since the last. The setup runs once, before the first pass. Leave it running while the old server still takes mail, so the new one stays close behind; then change the MX and stop it, and the last pass is a short catch-up instead of the whole mailbox. Stop it by interrupting it (Ctrl-C): a pass in flight is resumable, so a later run picks up where it left off.--max-passes <N>stops after N passes, and--until-quietstops on the first pass that moves nothing new -- the source has gone quiet and the two servers are in step. Each--repeatpass re-runs every step, like--again. - The report.
<DIR>/report.csvhas one row per step per account: how it ended (done,partial,failed,not run, ordone earlier), the counts summed over the step's types, and, for one that didn't finish, its last error. Exit code 5 means some account didn't finish. - Progress for another program.
--progress jsonwrites one JSON object per line on stdout, and nothing else there.eventsays which:start(accounts,parallel,dryRun),setup(a setup step ended),step(an account's step began, or ended withresult,countsanddetail),account(an account ended:email,noftotal,result) andend(pass,done,unfinished,report). Under--repeat,endcloses each pass and carries itspassnumber, and awaitingevent (seconds) comes between passes. Messages still go to stderr.
The work directory is made readable by its owner only, and so are the
logs. With --dry-run, each step runs with --dry-run, nothing is
recorded, and the report goes to dry-run-report.csv. An export step
reads the archive as it is, so on a first dry run it has nothing to show.
The guide
inbuxa-migrate guide [--port <PORT>] [--no-open] <DIR>
Walks through moving a domain, or a few accounts, into inbuxa in the
browser. guide serves a page on 127.0.0.1 from the binary itself and
opens it. The page asks five things in turn:
- Where the mail is now: an IMAP server, a JMAP server (Stalwart or
inbuxa), Maildir folders on this machine, Microsoft 365 or Google
Workspace. It also asks for a login that can act for every account. On
IMAP that's a master user, such as Dovecot's
{email}*master. On JMAP it's an administrator, with a password or an API key. On Microsoft 365 it's an Entra app with application permissions (--app-only), and on Google Workspace a service account's key (--google-key). Check signs the app in and says which permissions it lacks, or asks Google for a token for the first account. - Which accounts: pasted one per line or as a CSV, or read from a JMAP server's account list. The list can be edited before anything runs.
- Where to: the inbuxa server and an administrator's API key, and whether to create the accounts inbuxa doesn't have yet. Each new account gets a generated password, written to a file only its owner can read.
- Preview: a dry run. It signs in for each account, counts what would move, and creates nothing.
- Move, with a progress bar that fills as accounts finish and a running count and percentage beside it, a report, and a final pass for when the old server stops taking mail. When a run ends the page says how many accounts didn't finish and points at the report, whose failed rows are marked.
The guide writes an ordinary plan, <DIR>/plan.json, and runs it with
migrate, so a move started in the browser can be finished from a
terminal. The page shows the command. Everything else goes into <DIR>
too: the archives, the logs, work/report.csv, the passwords file, and the
answers so far, in guide.json. A guide started again on the same
directory picks up where it was. The guide process also outlives the
browser: close the tab or reload while a move runs and the page reconnects
to it and says so. A move that stopped part way starts again from where
each account left off -- start it once more, or run the final pass.
- Only this browser. The page is served on 127.0.0.1 alone, at an address with a random token that changes each start. A request without the token, for another host name, or from another site's page is refused.
- Secrets stay in memory. Passwords and API keys never go into
guide.jsonor the plan. The runs get them through environment variables,MIGRATE_SOURCE_SECRETandMIGRATE_INBUXA_TOKEN, as a plan run from a terminal does. After the guide restarts, the page asks for them again. --no-openprints the address instead of opening a browser.--portpicks the port; by default any free one is used.
Plans from answers
inbuxa-migrate plan --answers <FILE> [--source-directory <DIR>] <DIR>
Writes what the guide would, for a program that asks the questions itself
and runs migrate: <DIR>/plan.json, <DIR>/preview-plan.json and,
when accounts are to be created, the directory the setup reads. The
answers are the guide's, as JSON: source (kind, url, login,
path, cleartext, tenant, clientId, keyFile), accounts,
create, and target (url, stayOnUrl, allowInvalidCerts).
--source-directory is what export-directory wrote from a JMAP source,
so its accounts are set up as they were.
The answers hold no secrets, and plan refuses them if they do. It says
on stdout, as one JSON object, which plans it wrote, where the move's
passwords will go, and which variables to set when running them:
MIGRATE_SOURCE_SECRET for the source's password, API key, client secret
or Google key (for Google, a keyFile left empty means the key is the
secret), and MIGRATE_INBUXA_TOKEN for inbuxa's.
Inspect
inbuxa-migrate inspect <ARCHIVE> [TYPE] [--limit <N>] [--offset <N>]
Read-only dump of a local archive. This command never opens a network connection and never writes to the archive: a path with no archive is an error rather than a new, empty one, and an archive from an older release is shown as it is or refused, never upgraded. While another run holds the archive, it says the archive is in use instead of reading it.
- Omit
TYPEfor a per-type summary (counts of every object kind plus blob storage stats). - Pass an object type to dump it:
mailbox,email,identity,sievescript,addressbook,contactcard,calendar,calendarevent,participantidentity,filenode. mailboxandfilenoderender as a tree (--limit/--offsetare ignored); all other types use a paginated list and respect--limitand--offset.
Reconcile
inbuxa-migrate reconcile <ARCHIVE>
Read-only report that an archive holds what it should, folder by folder. Like inspect, it never opens a network connection and never writes to the archive; while another run holds the archive it says the archive is in use instead of reading it.
It prints, in order:
- Folders -- one row per mailbox, by full path, with its message count and a content checksum. A message filed in more than one folder is counted in each. The checksum is over the content of the messages in that folder, so two folders -- or two archives -- with the same mail share a checksum, regardless of order.
- Totals -- the message count, the number of folders, the stored content size and blob count, and a single checksum over the whole archive's mail.
- Integrity -- either
ok, or a line for each problem found: messages that point at content not in the archive, empty stored blobs, messages filed in no folder, and messages filed in a folder that is not in the archive.
Two uses. To prove a migration was complete, compare the per-folder counts against the old mailbox. To prove two archives hold the very same mail -- an export and a re-import, or a backup and its source -- compare their checksums.
Exit codes: 0 when the archive reconciles cleanly, 8 when an integrity check found a problem (past the 1--7 the other commands use, so a script can tell a reconciliation failure from any other error), and 7 when the archive can't be opened or read.
Live tests
The tests against real servers -- a Stalwart server as a JMAP source, and
Dovecot, Cyrus, Radicale, Baikal and Apache mod_dav -- are marked
#[ignore]. Each test binary boots its own throwaway container through
testcontainers, pulling the image on first use, so Docker must be running
(docker info succeeds) before they start.
Run them one binary at a time, always with one test thread:
cargo test --test sync_jmap -- --ignored --test-threads=1 # live JMAP import/export/convergence/prune
cargo test --test sync_imap -- --ignored --test-threads=1
cargo test --test sync_managesieve -- --ignored --test-threads=1
cargo test --test sync_maildir -- --ignored --test-threads=1
cargo test --test sync_caldav -- --ignored --test-threads=1
cargo test --test sync_carddav -- --ignored --test-threads=1
cargo test --test sync_webdav -- --ignored --test-threads=1
cargo test --test live_stalwart -- --ignored --test-threads=1
cargo test --test seed_smoke -- --ignored --test-threads=1
cargo test --test seed_only -- --ignored --test-threads=1
# Third-party-server tests (one container each):
cargo test --test integration_radicale -- --ignored --test-threads=1
cargo test --test integration_baikal -- --ignored --test-threads=1
cargo test --test integration_webdav -- --ignored --test-threads=1
cargo test --test integration_dovecot -- --ignored --test-threads=1
cargo test --test integration_cyrus -- --ignored --test-threads=1
cargo test --test integration_export_imap -- --ignored --test-threads=1
cargo test --test integration_export_managesieve -- --ignored --test-threads=1
cargo test --test integration_export_dav -- --ignored --test-threads=1
# inbuxa itself, an image pinned by digest (run before each release):
cargo test --test integration_inbuxa -- --ignored
# Slow tests
cargo test --test mock_jmap -- --ignored
One thread is required, not advised. The tests in a binary share its
container, and each one creates and removes the same disposable domain,
inbuxa-migrate.org, and opens its archive with SQLite's EXCLUSIVE lock.
Separate binaries each boot their own container on their own ports, so
running them one after another is safe.
integration_inbuxa is the exception: each of its tests starts its own
inbuxa server on its own volume, so its tests run in parallel. A test starts
the server once in recovery mode to make a domain and an administrator, then
again normally, where the administrator makes an API key. It then checks:
- a
migrate --planrun that sets up the accounts withimport-directoryand exports each one with that one key; - a message in two folders arriving once, in both;
- a second export carrying over a changed flag;
- Sieve scripts getting inbuxa's names.
To test another release, change the digest in
tests/integration/inbuxa.rs.