diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5e5ab2a..c5c5049 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,20 @@ candidates before a series ships (`27.1-rc.1`). The tag is the version with a
## [Unreleased]
+### Added
+- `import files` reads an account export back into an archive: the tree
+ `export-files` writes, specified at
+ . Besides the mail,
+ contacts, calendars, Sieve scripts and files, it reads what only the
+ manifest holds: real folder names, roles and subscriptions, the default
+ address book and calendar, calendar colors and time zones, the active
+ script, media types, links and identities. Copies of a message in several
+ folders come back as one message in each. Paths in the manifest are
+ checked to stay inside the export, and no symbolic link is followed. An
+ archive gains the source kind `files` and a table for what it read at its
+ next import; until then `inspect` and the other read-only commands call it
+ an older layout.
+
## [27.1-rc.8]
The eighth release candidate of the 27.1 series: what a server needs to
diff --git a/README.md b/README.md
index c85a009..e44db02 100644
--- a/README.md
+++ b/README.md
@@ -18,6 +18,7 @@ contacts, identities, Sieve scripts and files.
inbuxa-migrate inspect alice.sqlite what landed
inbuxa-migrate export … alice.sqlite the archive into inbuxa
inbuxa-migrate export-files alice.sqlite out the archive as plain files
+ inbuxa-migrate import files out alice.sqlite those files read back, manifest and all
inbuxa-migrate export-imap … alice.sqlite its mail into any IMAP server
inbuxa-migrate export-managesieve … alice.sqlite its filters, likewise
inbuxa-migrate export-caldav … alice.sqlite its calendars into any CalDAV server
@@ -48,7 +49,8 @@ server, `export-caldav` and `export-carddav` the calendars and contacts into
any CalDAV and CardDAV server, and `export-files` is the way out to everything else: it writes
the archive as a Maildir++ tree, vCard, iCalendar and Sieve files that other
servers and mail, calendar and contacts apps open, with no network and no
-store format in the way. For a whole server, `export-directory` writes the
+store format in the way, and `import files` reads such a tree back whole.
+For a whole server, `export-directory` writes the
list of domains, accounts, groups, lists and aliases to set up again
elsewhere, without passwords, and `import-directory` sets them up on an
inbuxa server, with a new password for each user. `migrate` moves many
diff --git a/docs/usage.md b/docs/usage.md
index 0a63f44..47d2aef 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -215,6 +215,45 @@ Finds every `.mbox`, `.ics` and `.vcf` file under a directory and imports it.
It is shaped for Google Takeout but reads any such tree. `--noautomap` stops
it giving Gmail's system labels their mailbox roles.
+### An account export
+
+```
+inbuxa-migrate import files
+```
+
+Reads an account export back: the directory `export-files` writes, laid out
+as specifies, from this tool
+or any other that writes the layout. No network. Where `import maildir` and
+`import takeout` read the open formats on their own, this also reads the
+manifest, so what those formats can't hold comes back too:
+
+- every folder's real name, its place in the tree, its role and whether it
+ was subscribed, in place of the names Maildir++ could give it on disk;
+- which address book and calendar were the default, and each calendar's
+ description, color and time zone;
+- which Sieve script was active;
+- the files' media types and links;
+- the identities with their reply-to, bcc and signatures, and the calendar
+ identities.
+
+A message kept in several folders is written to each folder's directory;
+copies with the same content come back as one message in all of those
+folders. The same goes for a card in two address books and an event in two
+calendars. A message's arrival time is its file's modification time.
+
+The manifest's paths are data, so each is checked to stay inside its part of
+the export, and nothing that is a symbolic link is followed: a link in the
+tree is skipped with a warning, and the links an account had are made from
+the manifest as archive entries. A directory without `manifest.json` was
+interrupted, and is refused, as is a later version of the layout than this
+release reads. 27.1.0-rc.2 named the format `inbuxa-migrate-files`; that is
+read as the same format.
+
+Run again, it brings the archive up to date with the export: flags that
+changed, a different active script, a renamed folder. It never deletes.
+With `--dry-run` it checks the export and counts what it holds, and opens no
+archive.
+
### Microsoft Exchange (EWS)
```
@@ -484,7 +523,7 @@ DIR/
```
- **Mail** is laid out as Dovecot and Courier keep it, so either can serve
- it as it is, and `import maildir` reads it back. Folder names are in
+ it as it is. Folder names are in
modified UTF-7. A `.` or `/` inside a folder's name becomes `_` on disk,
since Maildir++ uses the dot between levels. A message kept in several
folders is written to each. Read, answered, flagged, draft, forwarded and
@@ -495,7 +534,6 @@ DIR/
files. **Calendars** are iCalendar, one VCALENDAR each, named by
`X-WR-CALNAME`, with each time zone once. Photos and attachments the
archive holds go inside as `data:` URIs, as `export` sends them.
- `import takeout` reads both back.
- **Sieve scripts** are written as they are; the manifest says which is
active.
- **Files** keep their names and dates. Links are listed in the manifest
@@ -513,7 +551,9 @@ the way to finish it is to run again into an empty directory.
The layout and the manifest are specified at
, so other tools can read and
-write them too.
+write them too. `import files` reads the whole tree back, manifest and all
+([An account export](#an-account-export)); `import maildir` and `import
+takeout` read the mail, contacts and calendars without it.
`--objects` takes the same types as `export`; naming a folder or a message
type writes all of mail, and the same for contacts and calendars. With
diff --git a/src/cli.rs b/src/cli.rs
index 68a343f..151a34c 100644
--- a/src/cli.rs
+++ b/src/cli.rs
@@ -30,6 +30,7 @@ use crate::sync::export_managesieve::ExportManageSieveConfig;
use crate::sync::import_dav::{DavAuth, DavImportConfig, DavKindArg, SourceChange};
use crate::sync::import_exchange_ews::{EwsAuth, EwsImportConfig};
use crate::sync::import_exchange_graph::{GraphAuth, GraphImportConfig};
+use crate::sync::import_files::FilesImportConfig;
use crate::sync::import_imap::{ImapAuth, ImapImportConfig};
use crate::sync::import_maildir::MaildirImportConfig;
use crate::sync::import_managesieve::{ManageSieveAuth, ManageSieveImportConfig};
@@ -126,6 +127,8 @@ enum Source {
Maildir(Box),
#[command(about = "Google Takeout (or any mbox/ics/vcf directory tree)")]
Takeout(Box),
+ #[command(about = "An account export, as export-files writes it, manifest and all")]
+ Files(Box),
#[command(about = "MS Exchange via EWS (SOAP/XML)", name = "exchange-ews")]
ExchangeEws(Box),
#[command(
@@ -864,6 +867,7 @@ pub enum Action {
ImportManageSieve(CommonConfig, ManageSieveImportConfig),
ImportMaildir(CommonConfig, MaildirImportConfig),
ImportTakeout(CommonConfig, TakeoutImportConfig),
+ ImportFiles(CommonConfig, FilesImportConfig),
ImportExchangeEws(CommonConfig, EwsImportConfig),
ImportExchangeGraph(CommonConfig, GraphImportConfig),
Export(CommonConfig, ExportConfig),
@@ -1042,6 +1046,7 @@ fn resolve_import(source: Source) -> Result {
Source::Managesieve(args) => resolve_managesieve_import(*args),
Source::Maildir(args) => resolve_maildir_import(*args),
Source::Takeout(args) => resolve_takeout_import(*args),
+ Source::Files(args) => resolve_files_import(*args),
Source::ExchangeEws(args) => resolve_exchange_ews_import(*args),
Source::ExchangeGraph(args) => resolve_exchange_graph_import(*args),
}
@@ -1219,6 +1224,41 @@ fn resolve_takeout_import(args: TakeoutImportArgs) -> Result {
))
}
+#[derive(Args)]
+pub struct FilesImportArgs {
+ #[arg(
+ value_name = "DIR",
+ help = "An account export: the directory export-files wrote, with its manifest.json"
+ )]
+ dir: PathBuf,
+
+ #[arg(
+ value_name = "ARCHIVE",
+ help = "Local SQLite archive (created if absent)"
+ )]
+ archive: PathBuf,
+
+ #[arg(
+ long,
+ help = "Permit importing when the archive records an account export at a different path"
+ )]
+ allow_source_change: bool,
+
+ #[command(flatten)]
+ global: GlobalArgs,
+}
+
+fn resolve_files_import(args: FilesImportArgs) -> Result {
+ let common = common_config(&args.global, args.archive);
+ Ok(Action::ImportFiles(
+ common,
+ FilesImportConfig {
+ dir: args.dir,
+ allow_source_change: args.allow_source_change,
+ },
+ ))
+}
+
fn resolve_dav_import(args: DavImportArgs, kind: DavKindArg) -> Result {
let common = common_config(&args.global, args.archive);
let auth = resolve_dav_auth(
diff --git a/src/db/files_ids.rs b/src/db/files_ids.rs
new file mode 100644
index 0000000..20f9ad7
--- /dev/null
+++ b/src/db/files_ids.rs
@@ -0,0 +1,90 @@
+/*
+ * SPDX-FileCopyrightText: 2026 Coffey Labs LLC
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ */
+
+//! What `import files` has read from an account export, by a key of its
+//! own for each thing: a folder's path of real names, a message's content
+//! hash, a collection's file, a card's uid. Several keys may lead to one
+//! archive row: a message kept in two folders, a card in two address books.
+
+use rusqlite::{Connection, OptionalExtension, params};
+
+pub const MAILBOX: &str = "mailbox";
+pub const EMAIL: &str = "email";
+pub const ADDRESS_BOOK: &str = "addressbook";
+pub const CONTACT_CARD: &str = "contactcard";
+pub const CALENDAR: &str = "calendar";
+pub const CALENDAR_EVENT: &str = "calendarevent";
+pub const SIEVE_SCRIPT: &str = "sievescript";
+pub const FILE_NODE: &str = "filenode";
+pub const IDENTITY: &str = "identity";
+pub const PARTICIPANT_IDENTITY: &str = "participantidentity";
+
+pub fn local_for(
+ conn: &Connection,
+ source_id: i64,
+ type_name: &str,
+ key: &str,
+) -> Result, rusqlite::Error> {
+ conn.query_row(
+ "SELECT local_id FROM sync_id_files
+ WHERE source_id = ?1 AND type_name = ?2 AND item_key = ?3",
+ params![source_id, type_name, key],
+ |row| row.get(0),
+ )
+ .optional()
+}
+
+/// Records `key` as `local_id`; a key already recorded is pointed at it.
+pub fn record(
+ conn: &Connection,
+ source_id: i64,
+ type_name: &str,
+ key: &str,
+ local_id: i64,
+) -> Result<(), rusqlite::Error> {
+ conn.execute(
+ "INSERT INTO sync_id_files (source_id, type_name, item_key, local_id)
+ VALUES (?1, ?2, ?3, ?4)
+ ON CONFLICT (source_id, type_name, item_key) DO UPDATE SET local_id = excluded.local_id",
+ params![source_id, type_name, key, local_id],
+ )?;
+ Ok(())
+}
+
+pub fn count(conn: &Connection, source_id: i64, type_name: &str) -> Result {
+ conn.query_row(
+ "SELECT count(DISTINCT local_id) FROM sync_id_files
+ WHERE source_id = ?1 AND type_name = ?2",
+ params![source_id, type_name],
+ |row| row.get::<_, i64>(0),
+ )
+ .map(|n| n.max(0) as u64)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::db::init;
+
+ #[test]
+ fn two_keys_may_lead_to_one_row_and_a_key_moves() {
+ let conn = Connection::open_in_memory().unwrap();
+ init::apply_schema(&conn).unwrap();
+ conn.execute(
+ "INSERT INTO sources (id, kind, session_url, account_id, username)
+ VALUES (1, 'files', 'file:///e', '/e', '')",
+ [],
+ )
+ .unwrap();
+ record(&conn, 1, EMAIL, "a", 7).unwrap();
+ record(&conn, 1, EMAIL, "b", 7).unwrap();
+ assert_eq!(local_for(&conn, 1, EMAIL, "a").unwrap(), Some(7));
+ assert_eq!(count(&conn, 1, EMAIL).unwrap(), 1);
+ record(&conn, 1, EMAIL, "a", 8).unwrap();
+ assert_eq!(local_for(&conn, 1, EMAIL, "a").unwrap(), Some(8));
+ assert_eq!(local_for(&conn, 1, MAILBOX, "a").unwrap(), None);
+ }
+}
diff --git a/src/db/init.rs b/src/db/init.rs
index f25ed96..ec6248b 100644
--- a/src/db/init.rs
+++ b/src/db/init.rs
@@ -5,7 +5,7 @@
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
-use rusqlite::{Connection, OpenFlags};
+use rusqlite::{Connection, OpenFlags, OptionalExtension};
pub const SCHEMA_SQL: &str = include_str!("schema.sql");
@@ -126,6 +126,9 @@ fn schema_tables() -> impl Iterator- {
}
pub fn apply_schema(conn: &Connection) -> Result<(), OpenError> {
+ // Before the transaction: it turns foreign keys off, which SQLite only
+ // allows outside one.
+ ensure_sources_accept_files(conn)?;
let tx = conn.unchecked_transaction()?;
tx.execute_batch(SCHEMA_SQL)?;
ensure_calendar_events_data_type(&tx)?;
@@ -245,6 +248,64 @@ fn ensure_dav_ids_share_local_rows(conn: &Connection) -> Result<(), OpenError> {
Ok(())
}
+/// `import files` records its source as the kind `files`, which the
+/// `sources` table's CHECK didn't allow before. SQLite can't change a CHECK
+/// in place, so an older archive's table is rebuilt. Foreign keys are off
+/// while it is: with them on, dropping the old table would cascade into
+/// every row that came from a source.
+fn ensure_sources_accept_files(conn: &Connection) -> Result<(), OpenError> {
+ let sql: Option
= conn
+ .query_row(
+ "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'sources'",
+ [],
+ |row| row.get(0),
+ )
+ .optional()?;
+ // Only the table as earlier releases made it: its kind list ended at
+ // `takeout`.
+ let Some(sql) = sql else { return Ok(()) };
+ if sql.contains("'files'") || !sql.contains("'takeout')") {
+ return Ok(());
+ }
+ let foreign_keys: i64 = conn.query_row("PRAGMA foreign_keys", [], |row| row.get(0))?;
+ conn.pragma_update(None, "foreign_keys", "OFF")?;
+ let rebuilt = rebuild_sources(conn);
+ if foreign_keys != 0 {
+ conn.pragma_update(None, "foreign_keys", "ON")?;
+ }
+ rebuilt
+}
+
+fn rebuild_sources(conn: &Connection) -> Result<(), OpenError> {
+ let tx = conn.unchecked_transaction()?;
+ tx.execute_batch(
+ "CREATE TABLE sources_new (
+ id INTEGER PRIMARY KEY,
+ kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout','files')),
+ session_url TEXT NOT NULL,
+ account_id TEXT NOT NULL,
+ account_name TEXT,
+ username TEXT NOT NULL,
+ UNIQUE (kind, session_url, account_id)
+ );
+ INSERT INTO sources_new (id, kind, session_url, account_id, account_name, username)
+ SELECT id, kind, session_url, account_id, account_name, username FROM sources;
+ DROP TABLE sources;
+ ALTER TABLE sources_new RENAME TO sources;",
+ )?;
+ // Every row that names a source still finds it.
+ let dangling = tx.prepare("PRAGMA foreign_key_check")?.exists([])?;
+ if dangling {
+ return Err(OpenError::NotAnArchive(
+ "rebuilding the archive's sources table left rows without a source; \
+ nothing was changed"
+ .to_owned(),
+ ));
+ }
+ tx.commit()?;
+ Ok(())
+}
+
/// An archive holds a whole mailbox, so it is created readable by its owner
/// only. SQLite gives its `-wal` and `-shm` files the database file's mode,
/// so they follow. An existing archive others can read is left as it is,
@@ -494,6 +555,65 @@ mod permission_tests {
mod migration_tests {
use super::*;
+ #[test]
+ fn an_older_sources_table_is_rebuilt_to_accept_files_and_keeps_every_row() {
+ let conn = Connection::open_in_memory().unwrap();
+ // The table as releases before `import files` made it.
+ let old = SCHEMA_SQL.replace("'takeout','files')", "'takeout')");
+ assert_ne!(old, SCHEMA_SQL);
+ conn.execute_batch(&old).unwrap();
+ conn.pragma_update(None, "foreign_keys", "ON").unwrap();
+ conn.execute_batch(
+ "INSERT INTO sources (id, kind, session_url, account_id, account_name, username)
+ VALUES (3, 'maildir', 'file:///m', '/m', 'm', '');
+ INSERT INTO sync_id_maildir VALUES (3, 'mailbox', 'INBOX', 'x', 9);",
+ )
+ .unwrap();
+ assert!(
+ conn.execute(
+ "INSERT INTO sources (kind, session_url, account_id, username)
+ VALUES ('files', 'file:///e', '/e', '')",
+ [],
+ )
+ .is_err(),
+ "the old CHECK refuses it"
+ );
+
+ apply_schema(&conn).unwrap();
+
+ conn.execute(
+ "INSERT INTO sources (kind, session_url, account_id, username)
+ VALUES ('files', 'file:///e', '/e', '')",
+ [],
+ )
+ .expect("files is a source kind now");
+ let kept: (String, i64) = conn
+ .query_row(
+ "SELECT s.kind, m.local_id FROM sync_id_maildir m JOIN sources s ON s.id = m.source_id",
+ [],
+ |r| Ok((r.get(0)?, r.get(1)?)),
+ )
+ .unwrap();
+ assert_eq!(
+ kept,
+ ("maildir".to_owned(), 9),
+ "rows that came from it stayed"
+ );
+ let on: i64 = conn
+ .query_row("PRAGMA foreign_keys", [], |r| r.get(0))
+ .unwrap();
+ assert_eq!(on, 1, "foreign keys are back on");
+ // Deleting the source still cascades, so the reference survived.
+ conn.execute("DELETE FROM sources WHERE id = 3", [])
+ .unwrap();
+ let left: i64 = conn
+ .query_row("SELECT count(*) FROM sync_id_maildir", [], |r| r.get(0))
+ .unwrap();
+ assert_eq!(left, 0);
+ // A second pass finds nothing to do.
+ apply_schema(&conn).unwrap();
+ }
+
#[test]
fn an_older_dav_map_is_rebuilt_to_share_local_rows() {
let conn = Connection::open_in_memory().unwrap();
diff --git a/src/db/mod.rs b/src/db/mod.rs
index 351d35a..96abfeb 100644
--- a/src/db/mod.rs
+++ b/src/db/mod.rs
@@ -10,6 +10,7 @@ pub mod dav_ids;
pub mod defaults;
pub mod exchange_ews_ids;
pub mod exchange_graph_ids;
+pub mod files_ids;
pub mod ids;
pub mod imap_ids;
pub mod imap_state;
diff --git a/src/db/schema.sql b/src/db/schema.sql
index 6603294..cfc56de 100644
--- a/src/db/schema.sql
+++ b/src/db/schema.sql
@@ -1,6 +1,6 @@
CREATE TABLE IF NOT EXISTS sources (
id INTEGER PRIMARY KEY,
- kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout')),
+ kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout','files')),
session_url TEXT NOT NULL,
account_id TEXT NOT NULL,
account_name TEXT,
@@ -268,3 +268,14 @@ CREATE TABLE IF NOT EXISTS sync_id_takeout (
CREATE INDEX IF NOT EXISTS sync_id_takeout_type_idx
ON sync_id_takeout (source_id, type_name);
+
+CREATE TABLE IF NOT EXISTS sync_id_files (
+ source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
+ type_name TEXT NOT NULL,
+ item_key TEXT NOT NULL,
+ local_id INTEGER NOT NULL,
+ PRIMARY KEY (source_id, type_name, item_key)
+);
+
+CREATE INDEX IF NOT EXISTS sync_id_files_local_idx
+ ON sync_id_files (source_id, type_name, local_id);
diff --git a/src/main.rs b/src/main.rs
index 8085124..5f94ff7 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -61,6 +61,10 @@ fn run() -> i32 {
let logger = common.logger;
(sync::import_takeout::run_reporting(common, config), logger)
}
+ Action::ImportFiles(common, config) => {
+ let logger = common.logger;
+ (sync::import_files::run_reporting(common, config), logger)
+ }
Action::ImportExchangeEws(common, config) => {
let logger = common.logger;
(
diff --git a/src/sync/import_files.rs b/src/sync/import_files.rs
new file mode 100644
index 0000000..480407f
--- /dev/null
+++ b/src/sync/import_files.rs
@@ -0,0 +1,402 @@
+/*
+ * SPDX-FileCopyrightText: 2026 Coffey Labs LLC
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ */
+
+//! `import files`: an account export (the layout `export-files` writes,
+//! specified at ) read back
+//! into an archive, manifest and all. `import maildir` and `import takeout`
+//! read the open formats; this reads what only the manifest holds as well:
+//! real folder names, roles and subscriptions, which address book and
+//! calendar were the default, their colors and time zones, the active Sieve
+//! script, media types and links, and the identities with their signatures.
+//!
+//! Paths in the manifest are data. Each one is checked to stay inside its
+//! part of the export, and nothing that is a symbolic link is followed. A
+//! run that is repeated brings the archive up to date with the export; it
+//! never deletes.
+
+use std::collections::HashMap;
+use std::fs;
+use std::path::{Path, PathBuf};
+use std::time::UNIX_EPOCH;
+
+use rusqlite::{Connection, OptionalExtension, params};
+use time::OffsetDateTime;
+use time::format_description::well_known::Rfc3339;
+use url::Url;
+
+use crate::db;
+use crate::db::sources::SourceKey;
+use crate::error::Error;
+use crate::logging::{LEVEL_DEFAULT, LEVEL_PROGRESS, Logger};
+use crate::sync::import_dav::calcard;
+use crate::sync::import_takeout::contacts::split_vcards;
+use crate::sync::{CommonConfig, RunOutcome, Summary, TypeCounts};
+
+pub mod cards;
+pub mod mail;
+pub mod manifest;
+pub mod other;
+pub mod paths;
+
+use manifest::Manifest;
+
+pub const SOURCE_KIND: &str = "files";
+
+#[derive(Debug)]
+pub struct FilesImportConfig {
+ pub dir: PathBuf,
+ pub allow_source_change: bool,
+}
+
+pub fn run(common: CommonConfig, config: FilesImportConfig) -> Result {
+ run_reporting(common, config).into_result()
+}
+
+pub fn run_reporting(common: CommonConfig, config: FilesImportConfig) -> RunOutcome {
+ let mut summary = Summary::default();
+ let error = run_into(common, config, &mut summary).err();
+ RunOutcome { summary, error }
+}
+
+fn run_into(
+ common: CommonConfig,
+ config: FilesImportConfig,
+ summary: &mut Summary,
+) -> Result<(), Error> {
+ let logger = common.logger;
+ let root = fs::canonicalize(&config.dir)
+ .map_err(|e| Error::Usage(format!("{}: {e}", config.dir.display())))?;
+ if !root.is_dir() {
+ return Err(Error::Usage(format!("not a directory: {}", root.display())));
+ }
+ let manifest = manifest::read(&root)?;
+ if logger.enabled(LEVEL_PROGRESS) {
+ eprintln!(
+ "account export {} written by {} at {}",
+ root.display(),
+ manifest
+ .created_by
+ .as_deref()
+ .unwrap_or("an unnamed program"),
+ manifest.created.as_deref().unwrap_or("an unrecorded time"),
+ );
+ }
+
+ if common.dry_run {
+ *summary = count_only(&root, &manifest, logger)?;
+ logger.warn("dry run: nothing was written; the counts are what the export holds");
+ return Ok(());
+ }
+
+ let mut conn = db::init::open(&common.archive)?;
+ let session_url = Url::from_file_path(&root)
+ .map(|u| u.to_string())
+ .map_err(|_| Error::Usage(format!("cannot name {} as a file: URL", root.display())))?;
+ let account_id = root.to_string_lossy().into_owned();
+ if let Some((existing_url, _)) =
+ db::sources::conflicting_source(&conn, SOURCE_KIND, &session_url, &account_id)?
+ && !config.allow_source_change
+ {
+ return Err(Error::SourceChange(format!(
+ "archive already records the account export at {existing_url}; re-run with \
+ --allow-source-change or use a fresh archive"
+ )));
+ }
+ let original = manifest.sources.first();
+ let account_name = original
+ .and_then(|s| s.account_name.clone().or_else(|| s.username.clone()))
+ .or_else(|| root.file_name().map(|n| n.to_string_lossy().into_owned()));
+ if logger.enabled(LEVEL_PROGRESS)
+ && let Some(source) = original
+ {
+ eprintln!(
+ "the account came from {} ({})",
+ source.url.as_deref().unwrap_or("an unrecorded address"),
+ source.kind.as_deref().unwrap_or("an unrecorded kind"),
+ );
+ }
+ let source_id = db::sources::upsert_source(
+ &conn,
+ &SourceKey {
+ kind: SOURCE_KIND.to_owned(),
+ session_url,
+ account_id,
+ },
+ account_name.as_deref(),
+ "",
+ )?;
+
+ let result = import_parts(&mut conn, &root, source_id, &manifest, logger, summary);
+ if result.is_ok() && !summary.any_failed() {
+ let tx = conn.unchecked_transaction()?;
+ db::blobs::gc_orphan_blobs(&tx)?;
+ tx.commit()?;
+ }
+ if logger.enabled(LEVEL_DEFAULT) {
+ let line: Vec = summary
+ .per_type
+ .iter()
+ .map(|(name, c)| {
+ format!(
+ "{name} c={} u={} s={} f={}",
+ c.created, c.updated, c.skipped, c.failed
+ )
+ })
+ .collect();
+ eprintln!("files: {}", line.join(" | "));
+ }
+ result
+}
+
+/// Each part the manifest lists, in turn; one that is missing wasn't
+/// exported and is left as the archive has it.
+fn import_parts(
+ conn: &mut Connection,
+ root: &Path,
+ source_id: i64,
+ manifest: &Manifest,
+ logger: Logger,
+ summary: &mut Summary,
+) -> Result<(), Error> {
+ if let Some(mail) = &manifest.mail {
+ let out = mail::import(conn, root, source_id, mail, logger)?;
+ summary.per_type.push(("mailbox", out.mailbox));
+ summary.per_type.push(("email", out.email));
+ }
+ if let Some(books) = &manifest.contacts {
+ let out = cards::import_contacts(conn, root, source_id, books, logger)?;
+ summary.per_type.push(("addressbook", out.collection));
+ summary.per_type.push(("contactcard", out.item));
+ }
+ if let Some(calendars) = &manifest.calendars {
+ let out = cards::import_calendars(conn, root, source_id, calendars, logger)?;
+ summary.per_type.push(("calendar", out.collection));
+ summary.per_type.push(("calendarevent", out.item));
+ }
+ if let Some(scripts) = &manifest.sieve {
+ let counts = other::import_sieve(conn, root, source_id, scripts, logger)?;
+ summary.per_type.push(("sievescript", counts));
+ }
+ if let Some(files) = &manifest.files {
+ let created = manifest
+ .created
+ .clone()
+ .filter(|c| OffsetDateTime::parse(c, &Rfc3339).is_ok())
+ .unwrap_or_else(now_rfc3339);
+ let counts = other::import_files(conn, root, source_id, files, &created, logger)?;
+ summary.per_type.push(("filenode", counts));
+ }
+ if let Some(identities) = &manifest.identities {
+ let counts = other::import_identities(conn, source_id, identities)?;
+ summary.per_type.push(("identity", counts));
+ }
+ if let Some(identities) = &manifest.participant_identities {
+ let counts = other::import_participant_identities(conn, source_id, identities, logger)?;
+ summary.per_type.push(("participantidentity", counts));
+ }
+ Ok(())
+}
+
+/// What a run would read, counted from the export without opening the
+/// archive. A message kept in several folders counts once per folder.
+fn count_only(root: &Path, manifest: &Manifest, logger: Logger) -> Result {
+ let found = |n: usize| TypeCounts {
+ fetched: n as u64,
+ ..Default::default()
+ };
+ let mut summary = Summary::default();
+ if let Some(mail) = &manifest.mail {
+ let mut messages = 0;
+ let mut failed = 0;
+ for folder in &mail.folders {
+ match paths::resolve(root, &folder.dir, &mail.path) {
+ Ok(dir) => {
+ for sub in ["cur", "new"] {
+ let sub_dir = dir.join(sub);
+ if paths::is_real_dir(&sub_dir) {
+ messages += fs::read_dir(&sub_dir)?
+ .filter_map(Result::ok)
+ .filter(|e| e.file_type().is_ok_and(|t| t.is_file()))
+ .filter(|e| !e.file_name().to_string_lossy().starts_with('.'))
+ .count();
+ }
+ }
+ }
+ Err(reason) => {
+ logger.warn(&format!("folder {:?}: {reason}", folder.name));
+ failed += 1;
+ }
+ }
+ }
+ summary
+ .per_type
+ .push(("mailbox", found(mail.folders.len())));
+ let mut email = found(messages);
+ email.failed = failed;
+ summary.per_type.push(("email", email));
+ }
+ for (part, books, collection, item) in [
+ ("contacts", &manifest.contacts, "addressbook", "contactcard"),
+ (
+ "calendars",
+ &manifest.calendars,
+ "calendar",
+ "calendarevent",
+ ),
+ ] {
+ let Some(books) = books else { continue };
+ let mut items = TypeCounts::default();
+ for book in books {
+ match cards::read_part_file(root, &book.file, part, logger) {
+ Some(text) if part == "contacts" => {
+ items.fetched += split_vcards(&text).len() as u64
+ }
+ Some(text) => match calcard::ical_to_jscalendar_entries(&text) {
+ Ok(entries) => items.fetched += entries.len() as u64,
+ Err(calcard::CalcardError::NoEntries) => {}
+ Err(e) => {
+ logger.warn(&format!("{}: {e}", book.file));
+ items.failed += 1;
+ }
+ },
+ None => items.failed += 1,
+ }
+ }
+ summary.per_type.push((collection, found(books.len())));
+ summary.per_type.push((item, items));
+ }
+ if let Some(scripts) = &manifest.sieve {
+ let mut counts = TypeCounts::default();
+ for script in scripts {
+ match paths::resolve(root, &script.file, "sieve") {
+ Ok(path) if paths::is_real_file(&path) => counts.fetched += 1,
+ Ok(_) => counts.failed += 1,
+ Err(reason) => {
+ logger.warn(&reason);
+ counts.failed += 1;
+ }
+ }
+ }
+ summary.per_type.push(("sievescript", counts));
+ }
+ if let Some(files) = &manifest.files {
+ let base = root.join(&files.path);
+ let nodes = if paths::segments(&files.path).map(|s| s.len()) == Ok(1)
+ && paths::is_real_dir(&base)
+ {
+ count_tree(&base)?
+ } else {
+ 0
+ };
+ summary
+ .per_type
+ .push(("filenode", found(nodes + files.links.len())));
+ }
+ if let Some(identities) = &manifest.identities {
+ summary.per_type.push(("identity", found(identities.len())));
+ }
+ if let Some(identities) = &manifest.participant_identities {
+ summary
+ .per_type
+ .push(("participantidentity", found(identities.len())));
+ }
+ Ok(summary)
+}
+
+fn count_tree(dir: &Path) -> Result {
+ let mut n = 0;
+ for entry in fs::read_dir(dir)? {
+ let entry = entry?;
+ let file_type = entry.file_type()?;
+ if file_type.is_dir() {
+ n += 1 + count_tree(&entry.path())?;
+ } else if file_type.is_file() {
+ n += 1;
+ }
+ }
+ Ok(n)
+}
+
+/// Rows a run found already in the archive, as they were before it changed
+/// them, so the end of the run can count which it actually changed.
+#[derive(Default)]
+pub(crate) struct Rerun {
+ before: HashMap>,
+}
+
+impl Rerun {
+ pub(crate) fn before(
+ &mut self,
+ conn: &Connection,
+ table: &str,
+ columns: &str,
+ id: i64,
+ ) -> Result<(), Error> {
+ if self.before.contains_key(&id) {
+ return Ok(());
+ }
+ if let Some(row) = read_row(conn, table, columns, id)? {
+ self.before.insert(id, row);
+ }
+ Ok(())
+ }
+
+ pub(crate) fn tally(
+ &self,
+ conn: &Connection,
+ table: &str,
+ columns: &str,
+ counts: &mut TypeCounts,
+ ) -> Result<(), Error> {
+ for (id, was) in &self.before {
+ match read_row(conn, table, columns, *id)? {
+ Some(now) if now == *was => counts.skipped += 1,
+ _ => counts.updated += 1,
+ }
+ }
+ Ok(())
+ }
+}
+
+fn read_row(
+ conn: &Connection,
+ table: &str,
+ columns: &str,
+ id: i64,
+) -> Result>, Error> {
+ let mut stmt = conn.prepare(&format!("SELECT {columns} FROM {table} WHERE id = ?1"))?;
+ let n = stmt.column_count();
+ Ok(stmt
+ .query_row(params![id], |r| {
+ (0..n)
+ .map(|i| r.get::<_, rusqlite::types::Value>(i))
+ .collect()
+ })
+ .optional()?)
+}
+
+/// A file's modification time as RFC 3339, if it has a usable one.
+pub(crate) fn mtime_rfc3339(meta: &fs::Metadata) -> Option {
+ let secs = meta
+ .modified()
+ .ok()?
+ .duration_since(UNIX_EPOCH)
+ .ok()?
+ .as_secs();
+ if secs == 0 {
+ return None;
+ }
+ OffsetDateTime::from_unix_timestamp(i64::try_from(secs).ok()?)
+ .ok()?
+ .format(&Rfc3339)
+ .ok()
+}
+
+pub(crate) fn now_rfc3339() -> String {
+ OffsetDateTime::now_utc()
+ .format(&Rfc3339)
+ .unwrap_or_else(|_| "1970-01-01T00:00:00Z".to_owned())
+}
diff --git a/src/sync/import_files/cards.rs b/src/sync/import_files/cards.rs
new file mode 100644
index 0000000..1af442d
--- /dev/null
+++ b/src/sync/import_files/cards.rs
@@ -0,0 +1,441 @@
+/*
+ * SPDX-FileCopyrightText: 2026 Coffey Labs LLC
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ */
+
+//! Address books from `contacts/*.vcf` and calendars from
+//! `calendars/*.ics`, each named and set up as the manifest says: its real
+//! name, description, color, time zone and whether it was the default. A
+//! card or event in two files is one row in both collections.
+
+use std::collections::HashMap;
+use std::fs;
+use std::path::Path;
+
+use rusqlite::{Connection, OptionalExtension, params};
+use serde_json::Value;
+
+use crate::db::{cards, defaults, files_ids};
+use crate::error::Error;
+use crate::logging::Logger;
+use crate::sync::TypeCounts;
+use crate::sync::import_dav::calcard;
+use crate::sync::import_takeout::contacts::split_vcards;
+use crate::types::ObjectType;
+
+use super::mail::row_exists;
+use super::manifest::Collection;
+use super::{Rerun, paths};
+
+pub struct Outcome {
+ pub collection: TypeCounts,
+ pub item: TypeCounts,
+}
+
+const BOOK_COLUMNS: &str = "name, description, sort_order, is_default";
+const CARD_COLUMNS: &str = "address_book_ids, data";
+const CALENDAR_COLUMNS: &str = "name, description, color, time_zone, sort_order, is_default";
+const EVENT_COLUMNS: &str = "calendar_ids, is_draft, use_default_alerts, data, data_type";
+
+pub fn import_contacts(
+ conn: &mut Connection,
+ root: &Path,
+ source_id: i64,
+ books: &[Collection],
+ logger: Logger,
+) -> Result {
+ let mut out = Outcome {
+ collection: TypeCounts::default(),
+ item: TypeCounts::default(),
+ };
+ let mut collections = Rerun::default();
+ let mut items = Rerun::default();
+ let mut seen: HashMap = HashMap::new();
+ for (order, book) in books.iter().enumerate() {
+ let tx = conn.transaction()?;
+ let book_id = ensure_collection(
+ &tx,
+ source_id,
+ Kind::Contacts,
+ book,
+ order,
+ &mut collections,
+ &mut out.collection,
+ logger,
+ )?;
+ let Some(text) = read_part_file(root, &book.file, "contacts", logger) else {
+ out.item.failed += 1;
+ tx.commit()?;
+ continue;
+ };
+ let chunks = split_vcards(&text);
+ for (ordinal, chunk) in chunks.iter().enumerate() {
+ let card = match calcard::vcard_to_jscontact(chunk, &format!("{}#{ordinal}", book.file))
+ {
+ Ok(card) => card,
+ Err(e) => {
+ logger.warn(&format!("{} card {ordinal}: {e}", book.file));
+ out.item.failed += 1;
+ continue;
+ }
+ };
+ let data = card.data.to_string();
+ if let Some(&id) = seen.get(&card.uid) {
+ // The same card in another address book.
+ add_to_collection(&tx, "contact_cards", "address_book_ids", id, book_id)?;
+ continue;
+ }
+ let ours =
+ match files_ids::local_for(&tx, source_id, files_ids::CONTACT_CARD, &card.uid)? {
+ Some(id) if row_exists(&tx, "contact_cards", id)? => Some(id),
+ _ => None,
+ };
+ // The uid is the archive's key for a card, whoever put it there.
+ let theirs = match ours {
+ Some(_) => None,
+ None => tx
+ .query_row(
+ "SELECT id, address_book_ids FROM contact_cards WHERE uid = ?1",
+ params![card.uid],
+ |r| Ok((r.get::<_, i64>(0)?, r.get::<_, String>(1)?)),
+ )
+ .optional()?,
+ };
+ let id = match (ours, theirs) {
+ (Some(id), _) => {
+ // Read on an earlier run: now in the books it is in now.
+ items.before(&tx, "contact_cards", CARD_COLUMNS, id)?;
+ tx.execute(
+ "UPDATE contact_cards SET address_book_ids = ?1, data = ?2 WHERE id = ?3",
+ params![format!("[{book_id}]"), data, id],
+ )?;
+ id
+ }
+ (None, Some((id, books))) => {
+ // Another import's card: it keeps the books it was in.
+ items.before(&tx, "contact_cards", CARD_COLUMNS, id)?;
+ tx.execute(
+ "UPDATE contact_cards SET address_book_ids = ?1, data = ?2 WHERE id = ?3",
+ params![cards::with_book(&books, book_id), data, id],
+ )?;
+ id
+ }
+ (None, None) => {
+ tx.execute(
+ "INSERT INTO contact_cards (uid, address_book_ids, data) VALUES (?1, ?2, ?3)",
+ params![card.uid, format!("[{book_id}]"), data],
+ )?;
+ out.item.created += 1;
+ out.item.fetched += 1;
+ tx.last_insert_rowid()
+ }
+ };
+ files_ids::record(&tx, source_id, files_ids::CONTACT_CARD, &card.uid, id)?;
+ seen.insert(card.uid, id);
+ }
+ tx.commit()?;
+ listed_and_found(logger, &book.file, "cards", book.cards, chunks.len());
+ }
+ let tx = conn.transaction()?;
+ collections.tally(&tx, "address_books", BOOK_COLUMNS, &mut out.collection)?;
+ items.tally(&tx, "contact_cards", CARD_COLUMNS, &mut out.item)?;
+ tx.commit()?;
+ Ok(out)
+}
+
+pub fn import_calendars(
+ conn: &mut Connection,
+ root: &Path,
+ source_id: i64,
+ calendars: &[Collection],
+ logger: Logger,
+) -> Result {
+ let mut out = Outcome {
+ collection: TypeCounts::default(),
+ item: TypeCounts::default(),
+ };
+ let mut collections = Rerun::default();
+ let mut items = Rerun::default();
+ let mut seen: HashMap = HashMap::new();
+ for (order, calendar) in calendars.iter().enumerate() {
+ let tx = conn.transaction()?;
+ let calendar_id = ensure_collection(
+ &tx,
+ source_id,
+ Kind::Calendars,
+ calendar,
+ order,
+ &mut collections,
+ &mut out.collection,
+ logger,
+ )?;
+ let Some(text) = read_part_file(root, &calendar.file, "calendars", logger) else {
+ out.item.failed += 1;
+ tx.commit()?;
+ continue;
+ };
+ let entries = match calcard::ical_to_jscalendar_entries(&text) {
+ Ok(entries) => entries,
+ // A calendar with no events is written as an empty VCALENDAR.
+ Err(calcard::CalcardError::NoEntries) => Vec::new(),
+ Err(e) => {
+ logger.warn(&format!("{}: {e}", calendar.file));
+ out.item.failed += 1;
+ tx.commit()?;
+ continue;
+ }
+ };
+ let found = entries.len();
+ for (ordinal, mut entry) in entries.into_iter().enumerate() {
+ let (is_draft, use_default_alerts, uid) =
+ calcard::strip_extracted_fields_from_event(&mut entry.data);
+ let uid = uid
+ .map(|u| u.trim().to_owned())
+ .filter(|u| !u.is_empty())
+ .unwrap_or_else(|| {
+ calcard::synthesise_uid(&format!("{}#{ordinal}", calendar.file))
+ });
+ // An occurrence sent on its own shares its series' uid.
+ let key = match entry.data.get("recurrenceId").and_then(Value::as_str) {
+ Some(recurrence) => serde_json::to_string(&[&uid, recurrence]).unwrap_or_default(),
+ None => uid,
+ };
+ if let Some(&id) = seen.get(&key) {
+ add_to_collection(&tx, "calendar_events", "calendar_ids", id, calendar_id)?;
+ continue;
+ }
+ let ids = format!("[{calendar_id}]");
+ let data = entry.data.to_string();
+ let data_type = entry.data_type.as_column();
+ let id = match files_ids::local_for(&tx, source_id, files_ids::CALENDAR_EVENT, &key)? {
+ Some(id) if row_exists(&tx, "calendar_events", id)? => {
+ items.before(&tx, "calendar_events", EVENT_COLUMNS, id)?;
+ tx.execute(
+ "UPDATE calendar_events SET calendar_ids = ?1, is_draft = ?2,
+ use_default_alerts = ?3, data = ?4, data_type = ?5
+ WHERE id = ?6",
+ params![
+ ids,
+ is_draft as i64,
+ use_default_alerts as i64,
+ data,
+ data_type,
+ id
+ ],
+ )?;
+ id
+ }
+ _ => {
+ tx.execute(
+ "INSERT INTO calendar_events
+ (calendar_ids, is_draft, use_default_alerts, data, data_type)
+ VALUES (?1, ?2, ?3, ?4, ?5)",
+ params![
+ ids,
+ is_draft as i64,
+ use_default_alerts as i64,
+ data,
+ data_type
+ ],
+ )?;
+ out.item.created += 1;
+ out.item.fetched += 1;
+ tx.last_insert_rowid()
+ }
+ };
+ files_ids::record(&tx, source_id, files_ids::CALENDAR_EVENT, &key, id)?;
+ seen.insert(key, id);
+ }
+ tx.commit()?;
+ listed_and_found(logger, &calendar.file, "events", calendar.events, found);
+ }
+ let tx = conn.transaction()?;
+ collections.tally(&tx, "calendars", CALENDAR_COLUMNS, &mut out.collection)?;
+ items.tally(&tx, "calendar_events", EVENT_COLUMNS, &mut out.item)?;
+ tx.commit()?;
+ Ok(out)
+}
+
+#[derive(Clone, Copy)]
+enum Kind {
+ Contacts,
+ Calendars,
+}
+
+impl Kind {
+ fn table(self) -> &'static str {
+ match self {
+ Kind::Contacts => "address_books",
+ Kind::Calendars => "calendars",
+ }
+ }
+
+ fn type_name(self) -> &'static str {
+ match self {
+ Kind::Contacts => files_ids::ADDRESS_BOOK,
+ Kind::Calendars => files_ids::CALENDAR,
+ }
+ }
+
+ fn object(self) -> ObjectType {
+ match self {
+ Kind::Contacts => ObjectType::AddressBook,
+ Kind::Calendars => ObjectType::Calendar,
+ }
+ }
+}
+
+/// The address book or calendar for one manifest entry, keyed by its file.
+#[allow(clippy::too_many_arguments)]
+fn ensure_collection(
+ conn: &Connection,
+ source_id: i64,
+ kind: Kind,
+ entry: &Collection,
+ order: usize,
+ rerun: &mut Rerun,
+ counts: &mut TypeCounts,
+ logger: Logger,
+) -> Result {
+ let table = kind.table();
+ let existing = match files_ids::local_for(conn, source_id, kind.type_name(), &entry.file)? {
+ Some(id) if row_exists(conn, table, id)? => Some(id),
+ _ => None,
+ };
+ let is_default = defaults::unique_default(conn, kind.object(), entry.default, existing)?;
+ if entry.default && !is_default {
+ logger.warn(&format!(
+ "{}: another {} is already the default, so this one is not",
+ entry.file,
+ match kind {
+ Kind::Contacts => "address book",
+ Kind::Calendars => "calendar",
+ }
+ ));
+ }
+ let blank = |v: &Option| v.clone().filter(|s| !s.is_empty());
+ let id = match (kind, existing) {
+ (Kind::Contacts, Some(id)) => {
+ rerun.before(conn, table, BOOK_COLUMNS, id)?;
+ conn.execute(
+ "UPDATE address_books SET name = ?1, description = ?2, sort_order = ?3,
+ is_default = ?4
+ WHERE id = ?5",
+ params![
+ entry.name,
+ blank(&entry.description),
+ order as i64,
+ is_default as i64,
+ id
+ ],
+ )?;
+ id
+ }
+ (Kind::Contacts, None) => {
+ conn.execute(
+ "INSERT INTO address_books (name, description, sort_order, is_default, is_subscribed)
+ VALUES (?1, ?2, ?3, ?4, 1)",
+ params![
+ entry.name,
+ blank(&entry.description),
+ order as i64,
+ is_default as i64
+ ],
+ )?;
+ counts.created += 1;
+ counts.fetched += 1;
+ conn.last_insert_rowid()
+ }
+ (Kind::Calendars, Some(id)) => {
+ rerun.before(conn, table, CALENDAR_COLUMNS, id)?;
+ conn.execute(
+ "UPDATE calendars SET name = ?1, description = ?2, color = ?3, time_zone = ?4,
+ sort_order = ?5, is_default = ?6
+ WHERE id = ?7",
+ params![
+ entry.name,
+ blank(&entry.description),
+ blank(&entry.color),
+ blank(&entry.time_zone),
+ order as i64,
+ is_default as i64,
+ id
+ ],
+ )?;
+ id
+ }
+ (Kind::Calendars, None) => {
+ conn.execute(
+ "INSERT INTO calendars
+ (name, description, color, time_zone, sort_order, is_subscribed,
+ is_visible, is_default)
+ VALUES (?1, ?2, ?3, ?4, ?5, 1, 1, ?6)",
+ params![
+ entry.name,
+ blank(&entry.description),
+ blank(&entry.color),
+ blank(&entry.time_zone),
+ order as i64,
+ is_default as i64
+ ],
+ )?;
+ counts.created += 1;
+ counts.fetched += 1;
+ conn.last_insert_rowid()
+ }
+ };
+ files_ids::record(conn, source_id, kind.type_name(), &entry.file, id)?;
+ Ok(id)
+}
+
+fn add_to_collection(
+ conn: &Connection,
+ table: &str,
+ column: &str,
+ id: i64,
+ collection: i64,
+) -> Result<(), Error> {
+ let ids: String = conn.query_row(
+ &format!("SELECT {column} FROM {table} WHERE id = ?1"),
+ params![id],
+ |r| r.get(0),
+ )?;
+ conn.execute(
+ &format!("UPDATE {table} SET {column} = ?1 WHERE id = ?2"),
+ params![cards::with_book(&ids, collection), id],
+ )?;
+ Ok(())
+}
+
+/// A text file the manifest names, if it is safely inside `part`.
+pub(super) fn read_part_file(root: &Path, rel: &str, part: &str, logger: Logger) -> Option {
+ let path = match paths::resolve(root, rel, part) {
+ Ok(path) if paths::is_real_file(&path) => path,
+ Ok(_) => {
+ logger.warn(&format!("{rel}: not a file"));
+ return None;
+ }
+ Err(reason) => {
+ logger.warn(&reason);
+ return None;
+ }
+ };
+ match fs::read_to_string(&path) {
+ Ok(text) => Some(text),
+ Err(e) => {
+ logger.warn(&format!("{rel}: {e}"));
+ None
+ }
+ }
+}
+
+fn listed_and_found(logger: Logger, file: &str, what: &str, listed: Option, found: usize) {
+ if let Some(listed) = listed
+ && listed != found as u64
+ {
+ logger.warn(&format!(
+ "{file}: the manifest lists {listed} {what} and {found} were found"
+ ));
+ }
+}
diff --git a/src/sync/import_files/mail.rs b/src/sync/import_files/mail.rs
new file mode 100644
index 0000000..3ed25e4
--- /dev/null
+++ b/src/sync/import_files/mail.rs
@@ -0,0 +1,453 @@
+/*
+ * SPDX-FileCopyrightText: 2026 Coffey Labs LLC
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ */
+
+//! The mail part: a Maildir++ tree, read with the manifest's real folder
+//! names, hierarchy, roles and subscriptions in place of what the directory
+//! names could hold. A message kept in several folders is written to each;
+//! copies with the same content become one message in several mailboxes,
+//! as the archive holds them. File names are only read for their Maildir
+//! flags: the specification says nothing else in them can be relied on.
+
+use std::collections::{BTreeSet, HashMap};
+use std::fs;
+use std::path::Path;
+
+use rusqlite::{Connection, OptionalExtension, params};
+
+use crate::db::{blobs, cards, files_ids, roles};
+use crate::error::Error;
+use crate::logging::{LEVEL_PROGRESS, Logger};
+use crate::sync::TypeCounts;
+use crate::sync::emailmeta::email_meta_from_blob;
+use crate::sync::import_maildir::keywords::{
+ custom_keywords_from_filename, flags_from_filename, read_keyword_names, translate_flags,
+ with_custom,
+};
+use crate::sync::keys::index_to_json;
+
+use super::manifest::{Folder, Mail};
+use super::{Rerun, mtime_rfc3339, now_rfc3339, paths};
+
+pub struct Outcome {
+ pub mailbox: TypeCounts,
+ pub email: TypeCounts,
+}
+
+pub fn import(
+ conn: &mut Connection,
+ root: &Path,
+ source_id: i64,
+ mail: &Mail,
+ logger: Logger,
+) -> Result {
+ let mut out = Outcome {
+ mailbox: TypeCounts::default(),
+ email: TypeCounts::default(),
+ };
+ if paths::segments(&mail.path).map(|s| s.len()) != Ok(1) {
+ return Err(Error::Usage(format!(
+ "manifest.json: mail path {:?} is not a directory at the export's top",
+ mail.path
+ )));
+ }
+
+ // Parents before children, so each folder finds its parent made.
+ let mut folders: Vec<&Folder> = mail.folders.iter().collect();
+ folders.sort_by_key(|f| real_path(f).len());
+ let mut planned: Vec<(&Folder, i64)> = Vec::new();
+ {
+ let tx = conn.transaction()?;
+ let mut tree = Tree {
+ conn: &tx,
+ source_id,
+ made: HashMap::new(),
+ rerun: Rerun::default(),
+ logger,
+ };
+ for folder in folders {
+ match tree.ensure(&real_path(folder), Some(folder), &mut out.mailbox) {
+ Ok(id) => planned.push((folder, id)),
+ Err(e) if e.aborts_run() => return Err(e),
+ Err(e) => {
+ logger.warn(&format!("folder {:?}: {e}", folder.name));
+ out.mailbox.failed += 1;
+ }
+ }
+ }
+ tree.rerun
+ .tally(&tx, "mailboxes", MAILBOX_COLUMNS, &mut out.mailbox)?;
+ tx.commit()?;
+ }
+
+ let mut messages = Messages {
+ source_id,
+ seen: HashMap::new(),
+ rerun: Rerun::default(),
+ copies: 0,
+ logger,
+ };
+ for (folder, mailbox) in &planned {
+ let dir = match paths::resolve(root, &folder.dir, &mail.path) {
+ Ok(dir) if paths::is_real_dir(&dir) => dir,
+ Ok(_) => {
+ logger.warn(&format!(
+ "folder {:?}: {} is not a directory",
+ folder.name, folder.dir
+ ));
+ out.email.failed += 1;
+ continue;
+ }
+ Err(reason) => {
+ logger.warn(&format!("folder {:?}: {reason}", folder.name));
+ out.email.failed += 1;
+ continue;
+ }
+ };
+ let tx = conn.transaction()?;
+ let found = match messages.folder(&tx, &dir, *mailbox, &folder.name, &mut out.email) {
+ Ok(found) => found,
+ Err(e) if e.aborts_run() => return Err(e),
+ Err(e) => {
+ logger.warn(&format!("folder {:?}: {e}", folder.name));
+ out.email.failed += 1;
+ 0
+ }
+ };
+ tx.commit()?;
+ if let Some(listed) = folder.messages
+ && listed != found
+ {
+ logger.warn(&format!(
+ "folder {:?}: the manifest lists {listed} messages and {found} were found",
+ folder.name
+ ));
+ }
+ if let Some(dropped) = folder.keywords_dropped.filter(|n| *n > 0) {
+ logger.warn(&format!(
+ "folder {:?}: the export had no room for {dropped} keywords on its messages \
+ (Maildir holds 26 per folder), so those are not on them",
+ folder.name
+ ));
+ }
+ }
+ {
+ let tx = conn.transaction()?;
+ messages
+ .rerun
+ .tally(&tx, "emails", EMAIL_COLUMNS, &mut out.email)?;
+ tx.commit()?;
+ }
+ if messages.copies > 0 && logger.enabled(LEVEL_PROGRESS) {
+ eprintln!(
+ "mail: {} copies of messages kept in more than one folder were read as one \
+ message in each of those folders",
+ messages.copies
+ );
+ }
+ Ok(out)
+}
+
+const MAILBOX_COLUMNS: &str = "name, parent_id, role, is_subscribed";
+const EMAIL_COLUMNS: &str = "mailbox_ids, keywords";
+
+/// The folder's real names from the top down; the manifest may leave the
+/// path out of a top-level folder.
+fn real_path(folder: &Folder) -> Vec {
+ if folder.path.is_empty() {
+ vec![folder.name.clone()]
+ } else {
+ folder.path.clone()
+ }
+}
+
+struct Tree<'a> {
+ conn: &'a Connection,
+ source_id: i64,
+ made: HashMap, i64>,
+ rerun: Rerun,
+ logger: Logger,
+}
+
+impl Tree<'_> {
+ /// The mailbox for `path`, made or brought up to date. A parent the
+ /// manifest doesn't list is made too, so its children have one.
+ fn ensure(
+ &mut self,
+ path: &[String],
+ folder: Option<&Folder>,
+ counts: &mut TypeCounts,
+ ) -> Result {
+ if folder.is_none()
+ && let Some(id) = self.made.get(path)
+ {
+ return Ok(*id);
+ }
+ let Some((name, parents)) = path.split_last() else {
+ return Err(Error::Partial("a folder with no name".to_owned()));
+ };
+ let parent = if parents.is_empty() {
+ None
+ } else {
+ Some(self.ensure(parents, None, counts)?)
+ };
+ let key = serde_json::to_string(path).unwrap_or_default();
+ let existing =
+ match files_ids::local_for(self.conn, self.source_id, files_ids::MAILBOX, &key)? {
+ Some(id) if row_exists(self.conn, "mailboxes", id)? => Some(id),
+ _ => self
+ .conn
+ .query_row(
+ "SELECT id FROM mailboxes WHERE parent_id IS ?1 AND name = ?2",
+ params![parent, name],
+ |r| r.get::<_, i64>(0),
+ )
+ .optional()?,
+ };
+ let Some(folder) = folder else {
+ // A parent only: one the manifest doesn't list.
+ if let Some(id) = existing {
+ self.made.insert(path.to_vec(), id);
+ files_ids::record(self.conn, self.source_id, files_ids::MAILBOX, &key, id)?;
+ return Ok(id);
+ }
+ if self.logger.enabled(LEVEL_PROGRESS) {
+ eprintln!(
+ "folder {:?}: not in the manifest; made so its folders have a parent",
+ path.join("/")
+ );
+ }
+ self.conn.execute(
+ "INSERT INTO mailboxes (name, parent_id, role, sort_order, is_subscribed)
+ VALUES (?1, ?2, NULL, 0, 1)",
+ params![name, parent],
+ )?;
+ let id = self.conn.last_insert_rowid();
+ files_ids::record(self.conn, self.source_id, files_ids::MAILBOX, &key, id)?;
+ self.made.insert(path.to_vec(), id);
+ counts.created += 1;
+ counts.fetched += 1;
+ return Ok(id);
+ };
+
+ let wanted_role = folder
+ .role
+ .as_deref()
+ .map(str::trim)
+ .filter(|r| !r.is_empty())
+ .map(str::to_ascii_lowercase);
+ let role = roles::unique_role(self.conn, wanted_role.as_deref(), existing)?;
+ if let (Some(wanted), None) = (&wanted_role, &role) {
+ self.logger.warn(&format!(
+ "folder {:?}: another folder already has the role {wanted:?}, so this one \
+ is left without it",
+ path.join("/")
+ ));
+ }
+ let subscribed = folder.subscribed as i64;
+ let id = match existing {
+ Some(id) => {
+ self.rerun
+ .before(self.conn, "mailboxes", MAILBOX_COLUMNS, id)?;
+ self.conn.execute(
+ "UPDATE mailboxes SET name = ?1, parent_id = ?2, role = ?3, is_subscribed = ?4
+ WHERE id = ?5",
+ params![name, parent, role, subscribed, id],
+ )?;
+ id
+ }
+ None => {
+ self.conn.execute(
+ "INSERT INTO mailboxes (name, parent_id, role, sort_order, is_subscribed)
+ VALUES (?1, ?2, ?3, 0, ?4)",
+ params![name, parent, role, subscribed],
+ )?;
+ counts.created += 1;
+ counts.fetched += 1;
+ self.conn.last_insert_rowid()
+ }
+ };
+ files_ids::record(self.conn, self.source_id, files_ids::MAILBOX, &key, id)?;
+ self.made.insert(path.to_vec(), id);
+ Ok(id)
+ }
+}
+
+struct Messages {
+ source_id: i64,
+ /// Each message read this run, by content hash.
+ seen: HashMap,
+ rerun: Rerun,
+ copies: u64,
+ logger: Logger,
+}
+
+impl Messages {
+ /// Every message in one folder's `cur/` and `new/`; returns how many
+ /// were found.
+ fn folder(
+ &mut self,
+ conn: &Connection,
+ dir: &Path,
+ mailbox: i64,
+ folder_name: &str,
+ counts: &mut TypeCounts,
+ ) -> Result {
+ let keyword_file = dir.join("dovecot-keywords");
+ let names = if paths::is_real_file(&keyword_file) {
+ read_keyword_names(dir)
+ } else {
+ if keyword_file.symlink_metadata().is_ok() {
+ self.logger.warn(&format!(
+ "folder {folder_name:?}: its dovecot-keywords is not a plain file and \
+ is not read"
+ ));
+ }
+ vec![None; 26]
+ };
+ let mut found = 0;
+ for sub in ["cur", "new"] {
+ let sub_dir = dir.join(sub);
+ if !paths::is_real_dir(&sub_dir) {
+ if sub_dir.symlink_metadata().is_ok() {
+ self.logger.warn(&format!(
+ "folder {folder_name:?}: {sub}/ is not a directory and is not read"
+ ));
+ counts.failed += 1;
+ }
+ continue;
+ }
+ let mut entries: Vec = fs::read_dir(&sub_dir)
+ .and_then(|d| d.collect::>())
+ .map_err(|e| Error::Partial(format!("reading {sub}/: {e}")))?;
+ entries.sort_by_key(fs::DirEntry::file_name);
+ for entry in entries {
+ let file_name = entry.file_name().to_string_lossy().into_owned();
+ if file_name.starts_with('.') {
+ continue;
+ }
+ let file_type = entry
+ .file_type()
+ .map_err(|e| Error::Partial(format!("{sub}/{file_name}: {e}")))?;
+ if file_type.is_symlink() {
+ self.logger.warn(&format!(
+ "folder {folder_name:?}: {sub}/{file_name} is a symbolic link and is \
+ not followed"
+ ));
+ counts.skipped += 1;
+ continue;
+ }
+ if !file_type.is_file() {
+ continue;
+ }
+ found += 1;
+ if let Err(e) = self.message(conn, &entry, &file_name, &names, mailbox, counts) {
+ if e.aborts_run() {
+ return Err(e);
+ }
+ self.logger
+ .warn(&format!("folder {folder_name:?}: {sub}/{file_name}: {e}"));
+ counts.failed += 1;
+ }
+ }
+ }
+ Ok(found)
+ }
+
+ fn message(
+ &mut self,
+ conn: &Connection,
+ entry: &fs::DirEntry,
+ file_name: &str,
+ keyword_names: &[Option],
+ mailbox: i64,
+ counts: &mut TypeCounts,
+ ) -> Result<(), Error> {
+ let bytes = fs::read(entry.path()).map_err(|e| Error::Partial(e.to_string()))?;
+ let hash = blake3::hash(&bytes).to_hex().to_string();
+ let keywords: BTreeSet = with_custom(
+ translate_flags(&flags_from_filename(file_name), true),
+ &custom_keywords_from_filename(file_name, keyword_names),
+ )
+ .keywords
+ .into_iter()
+ .collect();
+
+ if let Some(&id) = self.seen.get(&hash) {
+ // Another copy of a message already read this run.
+ let (mailbox_ids, current): (String, String) = conn.query_row(
+ "SELECT mailbox_ids, keywords FROM emails WHERE id = ?1",
+ params![id],
+ |r| Ok((r.get(0)?, r.get(1)?)),
+ )?;
+ let mut merged: BTreeSet = serde_json::from_str::>(¤t)
+ .unwrap_or_default()
+ .into_iter()
+ .collect();
+ merged.extend(keywords);
+ conn.execute(
+ "UPDATE emails SET mailbox_ids = ?1, keywords = ?2 WHERE id = ?3",
+ params![
+ cards::with_book(&mailbox_ids, mailbox),
+ keywords_json(&merged),
+ id
+ ],
+ )?;
+ self.copies += 1;
+ return Ok(());
+ }
+
+ let mailbox_ids = format!("[{mailbox}]");
+ if let Some(id) = files_ids::local_for(conn, self.source_id, files_ids::EMAIL, &hash)?
+ && row_exists(conn, "emails", id)?
+ {
+ // Read on an earlier run: now in the folders it is in now.
+ self.rerun.before(conn, "emails", EMAIL_COLUMNS, id)?;
+ conn.execute(
+ "UPDATE emails SET mailbox_ids = ?1, keywords = ?2 WHERE id = ?3",
+ params![mailbox_ids, keywords_json(&keywords), id],
+ )?;
+ self.seen.insert(hash, id);
+ return Ok(());
+ }
+
+ let (index, date_header) = email_meta_from_blob(&bytes);
+ // The file's modification time is when the message arrived.
+ let received_at = entry
+ .metadata()
+ .ok()
+ .and_then(|m| mtime_rfc3339(&m))
+ .or(date_header)
+ .unwrap_or_else(now_rfc3339);
+ let blob = blobs::intern_blob(conn, &bytes)?;
+ conn.execute(
+ "INSERT INTO emails (blob_id, received_at, mailbox_ids, keywords, message_match)
+ VALUES (?1, ?2, ?3, ?4, ?5)",
+ params![
+ blob,
+ received_at,
+ mailbox_ids,
+ keywords_json(&keywords),
+ index_to_json(&index)
+ ],
+ )?;
+ let id = conn.last_insert_rowid();
+ files_ids::record(conn, self.source_id, files_ids::EMAIL, &hash, id)?;
+ self.seen.insert(hash, id);
+ counts.created += 1;
+ counts.fetched += 1;
+ Ok(())
+ }
+}
+
+fn keywords_json(keywords: &BTreeSet) -> String {
+ serde_json::to_string(keywords).unwrap_or_else(|_| "[]".to_owned())
+}
+
+pub(super) fn row_exists(conn: &Connection, table: &str, id: i64) -> Result {
+ Ok(conn
+ .prepare(&format!("SELECT 1 FROM {table} WHERE id = ?1"))?
+ .exists(params![id])?)
+}
diff --git a/src/sync/import_files/manifest.rs b/src/sync/import_files/manifest.rs
new file mode 100644
index 0000000..2c9d01e
--- /dev/null
+++ b/src/sync/import_files/manifest.rs
@@ -0,0 +1,317 @@
+/*
+ * SPDX-FileCopyrightText: 2026 Coffey Labs LLC
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ */
+
+//! `manifest.json`, as
+//! defines it. A part that is missing wasn't exported; a part that is there
+//! and empty says the account had none. Members this release doesn't know
+//! are ignored, so a later writer can add them.
+
+use std::collections::BTreeMap;
+use std::fs;
+use std::path::Path;
+
+use serde::Deserialize;
+use serde_json::Value;
+
+use crate::error::Error;
+use crate::export_files::{FORMAT, FORMAT_VERSION};
+
+/// What inbuxa-migrate 27.1.0-rc.2 wrote as the format before it was named
+/// by the specification's address. The specification lets readers accept it.
+pub const LEGACY_FORMAT: &str = "inbuxa-migrate-files";
+
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Manifest {
+ #[serde(default)]
+ pub created_by: Option,
+ #[serde(default)]
+ pub created: Option,
+ #[serde(default)]
+ pub sources: Vec,
+ #[serde(default)]
+ pub mail: Option,
+ #[serde(default)]
+ pub contacts: Option>,
+ #[serde(default)]
+ pub calendars: Option>,
+ #[serde(default)]
+ pub sieve: Option>,
+ #[serde(default)]
+ pub files: Option,
+ #[serde(default)]
+ pub identities: Option>,
+ #[serde(default)]
+ pub participant_identities: Option>,
+}
+
+#[derive(Debug, Default, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Source {
+ #[serde(default)]
+ pub kind: Option,
+ #[serde(default)]
+ pub url: Option,
+ #[serde(default)]
+ pub account_name: Option,
+ #[serde(default)]
+ pub username: Option,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Mail {
+ pub layout: String,
+ pub path: String,
+ #[serde(default)]
+ pub folders: Vec,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Folder {
+ pub name: String,
+ #[serde(default)]
+ pub path: Vec,
+ pub dir: String,
+ #[serde(default)]
+ pub role: Option,
+ #[serde(default = "yes")]
+ pub subscribed: bool,
+ #[serde(default)]
+ pub messages: Option,
+ #[serde(default)]
+ pub keywords_dropped: Option,
+}
+
+/// An address book or a calendar: the members both have, and those only a
+/// calendar has.
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Collection {
+ pub name: String,
+ #[serde(default)]
+ pub description: Option,
+ pub file: String,
+ #[serde(default)]
+ pub default: bool,
+ #[serde(default)]
+ pub color: Option,
+ #[serde(default)]
+ pub time_zone: Option,
+ #[serde(default)]
+ pub cards: Option,
+ #[serde(default)]
+ pub events: Option,
+}
+
+#[derive(Debug, Deserialize)]
+pub struct Script {
+ pub name: String,
+ pub file: String,
+ #[serde(default)]
+ pub active: bool,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Files {
+ pub path: String,
+ #[serde(default)]
+ pub media_types: BTreeMap,
+ #[serde(default)]
+ pub links: Vec ,
+}
+
+#[derive(Debug, Deserialize)]
+pub struct Link {
+ pub path: String,
+ #[serde(default)]
+ pub target: Value,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Identity {
+ #[serde(default)]
+ pub name: String,
+ pub email: String,
+ #[serde(default)]
+ pub reply_to: Value,
+ #[serde(default)]
+ pub bcc: Value,
+ #[serde(default)]
+ pub text_signature: String,
+ #[serde(default)]
+ pub html_signature: String,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct ParticipantIdentity {
+ #[serde(default)]
+ pub name: String,
+ pub calendar_address: String,
+ #[serde(default)]
+ pub default: bool,
+}
+
+fn yes() -> bool {
+ true
+}
+
+/// The manifest of the export at `root`. A tree without one is unfinished
+/// and is refused, and so is a format or version this release doesn't read.
+pub fn read(root: &Path) -> Result {
+ let path = root.join("manifest.json");
+ match fs::symlink_metadata(&path) {
+ Ok(meta) if meta.file_type().is_symlink() => {
+ return Err(Error::Usage(format!(
+ "{}: manifest.json is a symbolic link; an export's files are never links",
+ root.display()
+ )));
+ }
+ Ok(meta) if !meta.is_file() => {
+ return Err(Error::Usage(format!(
+ "{}: manifest.json is not a file",
+ root.display()
+ )));
+ }
+ Ok(_) => {}
+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
+ return Err(Error::Usage(format!(
+ "{} has no manifest.json: either the export was interrupted (a writer \
+ adds the manifest last) or this is not an account export",
+ root.display()
+ )));
+ }
+ Err(e) => return Err(Error::Io(e)),
+ }
+ let text = fs::read_to_string(&path)?;
+ parse(&text)
+}
+
+pub fn parse(text: &str) -> Result {
+ let value: Value = serde_json::from_str(text)
+ .map_err(|e| Error::Usage(format!("manifest.json is not valid JSON: {e}")))?;
+ let Value::Object(map) = &value else {
+ return Err(Error::Usage(
+ "manifest.json is not a JSON object".to_owned(),
+ ));
+ };
+ match map.get("format").and_then(Value::as_str) {
+ Some(f) if f == FORMAT || f == LEGACY_FORMAT => {}
+ Some(other) => {
+ return Err(Error::Usage(format!(
+ "manifest.json names the format {other:?}, not the account export layout \
+ ({FORMAT})"
+ )));
+ }
+ None => {
+ return Err(Error::Usage(
+ "manifest.json has no format; this is not an account export".to_owned(),
+ ));
+ }
+ }
+ match map.get("formatVersion").and_then(Value::as_u64) {
+ Some(v) if v == u64::from(FORMAT_VERSION) => {}
+ Some(v) if v > u64::from(FORMAT_VERSION) => {
+ return Err(Error::Usage(format!(
+ "this export follows version {v} of the layout, and this release reads \
+ version {FORMAT_VERSION}; use a newer inbuxa-migrate"
+ )));
+ }
+ _ => {
+ return Err(Error::Usage(
+ "manifest.json has no usable formatVersion".to_owned(),
+ ));
+ }
+ }
+ let manifest: Manifest =
+ serde_json::from_value(value).map_err(|e| Error::Usage(format!("manifest.json: {e}")))?;
+ if let Some(mail) = &manifest.mail
+ && !mail.layout.eq_ignore_ascii_case("maildir++")
+ {
+ return Err(Error::Usage(format!(
+ "manifest.json: mail layout {:?} is not one this release reads (maildir++)",
+ mail.layout
+ )));
+ }
+ Ok(manifest)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn err(text: &str) -> String {
+ parse(text).unwrap_err().to_string()
+ }
+
+ #[test]
+ fn the_format_is_the_spec_address_or_the_old_name() {
+ let m = parse(&format!(r#"{{"format":"{FORMAT}","formatVersion":1}}"#)).unwrap();
+ assert!(m.mail.is_none() && m.contacts.is_none(), "nothing exported");
+ parse(r#"{"format":"inbuxa-migrate-files","formatVersion":1}"#).unwrap();
+ assert!(
+ err(r#"{"format":"something-else","formatVersion":1}"#).contains("not the account")
+ );
+ assert!(err(r#"{"formatVersion":1}"#).contains("no format"));
+ }
+
+ #[test]
+ fn a_later_version_is_refused_rather_than_guessed_at() {
+ let e = err(&format!(r#"{{"format":"{FORMAT}","formatVersion":2}}"#));
+ assert!(e.contains("version 2") && e.contains("newer"), "{e}");
+ assert!(
+ err(&format!(r#"{{"format":"{FORMAT}","formatVersion":"1"}}"#))
+ .contains("formatVersion")
+ );
+ assert!(err(&format!(r#"{{"format":"{FORMAT}"}}"#)).contains("formatVersion"));
+ }
+
+ #[test]
+ fn unknown_members_are_ignored_and_empty_parts_kept_apart_from_missing_ones() {
+ let m = parse(&format!(
+ r#"{{"format":"{FORMAT}","formatVersion":1,"futureThing":{{"x":1}},
+ "contacts":[],"mail":{{"layout":"maildir++","path":"mail","folders":[
+ {{"name":"Inbox","path":["Inbox"],"dir":"mail","role":"inbox",
+ "subscribed":true,"messages":3,"newMember":true}}]}}}}"#
+ ))
+ .unwrap();
+ assert_eq!(m.contacts.as_deref().map(<[Collection]>::len), Some(0));
+ assert!(m.calendars.is_none());
+ let mail = m.mail.unwrap();
+ assert_eq!(mail.folders[0].role.as_deref(), Some("inbox"));
+ }
+
+ #[test]
+ fn another_mail_layout_is_refused() {
+ let e = err(&format!(
+ r#"{{"format":"{FORMAT}","formatVersion":1,"mail":{{"layout":"mbox","path":"mail"}}}}"#
+ ));
+ assert!(e.contains("mbox"), "{e}");
+ }
+
+ #[test]
+ fn a_missing_or_linked_manifest_is_refused() {
+ let td = tempfile::tempdir().unwrap();
+ let e = read(td.path()).unwrap_err().to_string();
+ assert!(e.contains("interrupted"), "{e}");
+ #[cfg(unix)]
+ {
+ let real = td.path().join("real.json");
+ fs::write(
+ &real,
+ format!(r#"{{"format":"{FORMAT}","formatVersion":1}}"#),
+ )
+ .unwrap();
+ std::os::unix::fs::symlink(&real, td.path().join("manifest.json")).unwrap();
+ let e = read(td.path()).unwrap_err().to_string();
+ assert!(e.contains("symbolic link"), "{e}");
+ }
+ }
+}
diff --git a/src/sync/import_files/other.rs b/src/sync/import_files/other.rs
new file mode 100644
index 0000000..c3d1214
--- /dev/null
+++ b/src/sync/import_files/other.rs
@@ -0,0 +1,532 @@
+/*
+ * SPDX-FileCopyrightText: 2026 Coffey Labs LLC
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ */
+
+//! Sieve scripts with the one that was active, the file tree with its
+//! media types and links, and the identities, which only the manifest
+//! holds.
+
+use std::collections::HashMap;
+use std::fs;
+use std::path::Path;
+
+use rusqlite::{Connection, OptionalExtension, params};
+use serde_json::Value;
+
+use crate::db::{blobs, defaults, files_ids};
+use crate::error::Error;
+use crate::logging::Logger;
+use crate::sync::TypeCounts;
+use crate::sync::keys::{identity_key, participant_identity_key};
+use crate::types::ObjectType;
+
+use super::mail::row_exists;
+use super::manifest::{Files, Identity, ParticipantIdentity, Script};
+use super::{Rerun, mtime_rfc3339, paths};
+
+const SCRIPT_COLUMNS: &str = "blob_id, is_active";
+const NODE_COLUMNS: &str = "parent_id, node_type, blob_id, target, name, media_type, modified";
+const IDENTITY_COLUMNS: &str = "name, email, reply_to, bcc, text_signature, html_signature";
+const PARTICIPANT_COLUMNS: &str = "name, calendar_address, is_default";
+
+pub fn import_sieve(
+ conn: &mut Connection,
+ root: &Path,
+ source_id: i64,
+ scripts: &[Script],
+ logger: Logger,
+) -> Result {
+ let mut counts = TypeCounts::default();
+ let mut rerun = Rerun::default();
+ let tx = conn.transaction()?;
+ let mut active: Option<&str> = None;
+ for script in scripts {
+ let path = match paths::resolve(root, &script.file, "sieve") {
+ Ok(path) if paths::is_real_file(&path) => path,
+ Ok(_) => {
+ logger.warn(&format!("{}: not a file", script.file));
+ counts.failed += 1;
+ continue;
+ }
+ Err(reason) => {
+ logger.warn(&format!("Sieve script {:?}: {reason}", script.name));
+ counts.failed += 1;
+ continue;
+ }
+ };
+ let bytes = match fs::read(&path) {
+ Ok(bytes) => bytes,
+ Err(e) => {
+ logger.warn(&format!("{}: {e}", script.file));
+ counts.failed += 1;
+ continue;
+ }
+ };
+ let is_active = script.active && active.is_none();
+ if script.active && !is_active {
+ logger.warn(&format!(
+ "Sieve script {:?}: the manifest marks more than one script active; only \
+ {:?} is",
+ script.name,
+ active.unwrap_or_default()
+ ));
+ }
+ if is_active {
+ active = Some(&script.name);
+ }
+ let blob = blobs::intern_blob(&tx, &bytes)?;
+ let existing =
+ match files_ids::local_for(&tx, source_id, files_ids::SIEVE_SCRIPT, &script.name)? {
+ Some(id) if row_exists(&tx, "sieve_scripts", id)? => Some(id),
+ // Names are unique in the archive.
+ _ => tx
+ .query_row(
+ "SELECT id FROM sieve_scripts WHERE name = ?1",
+ params![script.name],
+ |r| r.get::<_, i64>(0),
+ )
+ .optional()?,
+ };
+ if let Some(id) = existing {
+ rerun.before(&tx, "sieve_scripts", SCRIPT_COLUMNS, id)?;
+ }
+ if is_active {
+ // One script runs at a time.
+ tx.execute(
+ "UPDATE sieve_scripts SET is_active = 0 WHERE is_active = 1 AND id IS NOT ?1",
+ params![existing],
+ )?;
+ }
+ let id = match existing {
+ Some(id) => {
+ tx.execute(
+ "UPDATE sieve_scripts SET blob_id = ?1, is_active = ?2 WHERE id = ?3",
+ params![blob, is_active as i64, id],
+ )?;
+ id
+ }
+ None => {
+ tx.execute(
+ "INSERT INTO sieve_scripts (name, is_active, blob_id) VALUES (?1, ?2, ?3)",
+ params![script.name, is_active as i64, blob],
+ )?;
+ counts.created += 1;
+ counts.fetched += 1;
+ tx.last_insert_rowid()
+ }
+ };
+ files_ids::record(&tx, source_id, files_ids::SIEVE_SCRIPT, &script.name, id)?;
+ }
+ rerun.tally(&tx, "sieve_scripts", SCRIPT_COLUMNS, &mut counts)?;
+ tx.commit()?;
+ Ok(counts)
+}
+
+/// The file tree, walked without following links, then the links the
+/// manifest lists, made as archive nodes and never on disk.
+pub fn import_files(
+ conn: &mut Connection,
+ root: &Path,
+ source_id: i64,
+ files: &Files,
+ created: &str,
+ logger: Logger,
+) -> Result {
+ let mut counts = TypeCounts::default();
+ let part = match paths::segments(&files.path).as_deref() {
+ Ok([part]) => (*part).to_owned(),
+ _ => {
+ return Err(Error::Usage(format!(
+ "manifest.json: files path {:?} is not a directory at the export's top",
+ files.path
+ )));
+ }
+ };
+ let tx = conn.transaction()?;
+ let mut tree = FileTree {
+ conn: &tx,
+ source_id,
+ files,
+ dirs: HashMap::new(),
+ rerun: Rerun::default(),
+ created,
+ logger,
+ };
+ // A writer with no files to write may not make the directory at all.
+ let base = root.join(&part);
+ match fs::symlink_metadata(&base) {
+ Ok(meta) if meta.is_dir() => tree.walk(&base, &part, None, &mut counts)?,
+ Ok(_) => {
+ logger.warn(&format!("{part}/ is not a directory and is not read"));
+ counts.failed += 1;
+ }
+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
+ Err(e) => return Err(Error::Io(e)),
+ }
+ for link in &files.links {
+ if let Err(reason) = tree.link(&part, &link.path, &link.target, &mut counts) {
+ logger.warn(&format!("link {:?}: {reason}", link.path));
+ counts.failed += 1;
+ }
+ }
+ let rerun = std::mem::take(&mut tree.rerun);
+ rerun.tally(&tx, "file_nodes", NODE_COLUMNS, &mut counts)?;
+ tx.commit()?;
+ Ok(counts)
+}
+
+struct FileTree<'a> {
+ conn: &'a Connection,
+ source_id: i64,
+ files: &'a Files,
+ /// Each directory made, by its path in the export.
+ dirs: HashMap,
+ rerun: Rerun,
+ /// When the export was written: the time for what has none of its own.
+ created: &'a str,
+ logger: Logger,
+}
+
+struct Node<'a> {
+ parent: Option,
+ node_type: &'a str,
+ blob: Option,
+ target: Option,
+ name: &'a str,
+ media_type: Option<&'a str>,
+ created: String,
+ modified: Option,
+}
+
+impl FileTree<'_> {
+ fn walk(
+ &mut self,
+ dir: &Path,
+ rel: &str,
+ parent: Option,
+ counts: &mut TypeCounts,
+ ) -> Result<(), Error> {
+ let mut entries: Vec =
+ match fs::read_dir(dir).and_then(|d| d.collect::>()) {
+ Ok(entries) => entries,
+ Err(e) => {
+ self.logger.warn(&format!("{rel}/: {e}"));
+ counts.failed += 1;
+ return Ok(());
+ }
+ };
+ entries.sort_by_key(fs::DirEntry::file_name);
+ for entry in entries {
+ let Some(name) = entry.file_name().to_str().map(str::to_owned) else {
+ self.logger.warn(&format!(
+ "{rel}/{}: the name is not UTF-8 and is skipped",
+ entry.file_name().to_string_lossy()
+ ));
+ counts.failed += 1;
+ continue;
+ };
+ let rel_path = format!("{rel}/{name}");
+ let Ok(file_type) = entry.file_type() else {
+ self.logger
+ .warn(&format!("{rel_path}: its type can't be read"));
+ counts.failed += 1;
+ continue;
+ };
+ if file_type.is_symlink() {
+ self.logger.warn(&format!(
+ "{rel_path} is a symbolic link and is not followed; an export's links \
+ are listed in its manifest"
+ ));
+ counts.skipped += 1;
+ continue;
+ }
+ if file_type.is_dir() {
+ // A writer sets the times of files, not of directories, so a
+ // directory's time is only when the export was written.
+ let id = self.node(
+ &rel_path,
+ Node {
+ parent,
+ node_type: "directory",
+ blob: None,
+ target: None,
+ name: &name,
+ media_type: None,
+ created: self.created.to_owned(),
+ modified: None,
+ },
+ counts,
+ )?;
+ self.dirs.insert(rel_path.clone(), id);
+ self.walk(&entry.path(), &rel_path, Some(id), counts)?;
+ } else if file_type.is_file() {
+ let modified = entry.metadata().ok().and_then(|m| mtime_rfc3339(&m));
+ let created = modified.clone().unwrap_or_else(|| self.created.to_owned());
+ let bytes = match fs::read(entry.path()) {
+ Ok(bytes) => bytes,
+ Err(e) => {
+ self.logger.warn(&format!("{rel_path}: {e}"));
+ counts.failed += 1;
+ continue;
+ }
+ };
+ let blob = blobs::intern_blob(self.conn, &bytes)?;
+ let media_type = self.files.media_types.get(&rel_path).map(String::as_str);
+ self.node(
+ &rel_path,
+ Node {
+ parent,
+ node_type: "file",
+ blob: Some(blob),
+ target: None,
+ name: &name,
+ media_type,
+ created,
+ modified,
+ },
+ counts,
+ )?;
+ } else {
+ self.logger
+ .warn(&format!("{rel_path} is not a file or directory; skipped"));
+ counts.skipped += 1;
+ }
+ }
+ Ok(())
+ }
+
+ fn link(
+ &mut self,
+ part: &str,
+ rel: &str,
+ target: &Value,
+ counts: &mut TypeCounts,
+ ) -> Result<(), String> {
+ let segments = paths::segments(rel)?;
+ let [first, parents @ .., name] = segments.as_slice() else {
+ return Err("names no file".to_owned());
+ };
+ if *first != part {
+ return Err(format!("is outside {part}/"));
+ }
+ let parent = if parents.is_empty() {
+ None
+ } else {
+ let parent_rel = format!("{part}/{}", parents.join("/"));
+ match self.dirs.get(&parent_rel) {
+ Some(id) => Some(*id),
+ None => return Err(format!("its directory {parent_rel} is not in the export")),
+ }
+ };
+ let target = (!target.is_null()).then(|| target.to_string());
+ self.node(
+ rel,
+ Node {
+ parent,
+ node_type: "symlink",
+ blob: None,
+ target,
+ name,
+ media_type: None,
+ created: self.created.to_owned(),
+ modified: None,
+ },
+ counts,
+ )
+ .map(|_| ())
+ .map_err(|e| e.to_string())
+ }
+
+ fn node(&mut self, key: &str, node: Node<'_>, counts: &mut TypeCounts) -> Result {
+ let existing =
+ match files_ids::local_for(self.conn, self.source_id, files_ids::FILE_NODE, key)? {
+ Some(id) if row_exists(self.conn, "file_nodes", id)? => Some(id),
+ // Names are unique within a directory.
+ _ => self
+ .conn
+ .query_row(
+ "SELECT id FROM file_nodes WHERE parent_id IS ?1 AND name = ?2",
+ params![node.parent, node.name],
+ |r| r.get::<_, i64>(0),
+ )
+ .optional()?,
+ };
+ let id = match existing {
+ Some(id) => {
+ self.rerun
+ .before(self.conn, "file_nodes", NODE_COLUMNS, id)?;
+ self.conn.execute(
+ "UPDATE file_nodes SET parent_id = ?1, node_type = ?2, blob_id = ?3,
+ target = ?4, name = ?5, media_type = ?6, modified = ?7
+ WHERE id = ?8",
+ params![
+ node.parent,
+ node.node_type,
+ node.blob,
+ node.target,
+ node.name,
+ node.media_type,
+ node.modified,
+ id
+ ],
+ )?;
+ id
+ }
+ None => {
+ self.conn.execute(
+ "INSERT INTO file_nodes
+ (parent_id, node_type, blob_id, target, name, media_type, created, modified)
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
+ params![
+ node.parent,
+ node.node_type,
+ node.blob,
+ node.target,
+ node.name,
+ node.media_type,
+ node.created,
+ node.modified
+ ],
+ )?;
+ counts.created += 1;
+ counts.fetched += 1;
+ self.conn.last_insert_rowid()
+ }
+ };
+ files_ids::record(self.conn, self.source_id, files_ids::FILE_NODE, key, id)?;
+ Ok(id)
+ }
+}
+
+pub fn import_identities(
+ conn: &mut Connection,
+ source_id: i64,
+ identities: &[Identity],
+) -> Result {
+ let mut counts = TypeCounts::default();
+ let mut rerun = Rerun::default();
+ let tx = conn.transaction()?;
+ for identity in identities {
+ let key = hex(&identity_key(&identity.name, &identity.email));
+ let addresses = |v: &Value| (!v.is_null()).then(|| v.to_string());
+ let (reply_to, bcc) = (addresses(&identity.reply_to), addresses(&identity.bcc));
+ let id = match files_ids::local_for(&tx, source_id, files_ids::IDENTITY, &key)? {
+ Some(id) if row_exists(&tx, "identities", id)? => {
+ rerun.before(&tx, "identities", IDENTITY_COLUMNS, id)?;
+ tx.execute(
+ "UPDATE identities SET name = ?1, email = ?2, reply_to = ?3, bcc = ?4,
+ text_signature = ?5, html_signature = ?6
+ WHERE id = ?7",
+ params![
+ identity.name,
+ identity.email,
+ reply_to,
+ bcc,
+ identity.text_signature,
+ identity.html_signature,
+ id
+ ],
+ )?;
+ id
+ }
+ _ => {
+ tx.execute(
+ "INSERT INTO identities
+ (name, email, reply_to, bcc, text_signature, html_signature)
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
+ params![
+ identity.name,
+ identity.email,
+ reply_to,
+ bcc,
+ identity.text_signature,
+ identity.html_signature
+ ],
+ )?;
+ counts.created += 1;
+ counts.fetched += 1;
+ tx.last_insert_rowid()
+ }
+ };
+ files_ids::record(&tx, source_id, files_ids::IDENTITY, &key, id)?;
+ }
+ rerun.tally(&tx, "identities", IDENTITY_COLUMNS, &mut counts)?;
+ tx.commit()?;
+ Ok(counts)
+}
+
+pub fn import_participant_identities(
+ conn: &mut Connection,
+ source_id: i64,
+ identities: &[ParticipantIdentity],
+ logger: Logger,
+) -> Result {
+ let mut counts = TypeCounts::default();
+ let mut rerun = Rerun::default();
+ let tx = conn.transaction()?;
+ for identity in identities {
+ let key = hex(&participant_identity_key(
+ &identity.calendar_address,
+ &identity.name,
+ ));
+ let existing =
+ match files_ids::local_for(&tx, source_id, files_ids::PARTICIPANT_IDENTITY, &key)? {
+ Some(id) if row_exists(&tx, "participant_identities", id)? => Some(id),
+ _ => None,
+ };
+ let is_default = defaults::unique_default(
+ &tx,
+ ObjectType::ParticipantIdentity,
+ identity.default,
+ existing,
+ )?;
+ if identity.default && !is_default {
+ logger.warn(&format!(
+ "calendar identity {:?}: another is already the default, so this one is not",
+ identity.calendar_address
+ ));
+ }
+ let id = match existing {
+ Some(id) => {
+ rerun.before(&tx, "participant_identities", PARTICIPANT_COLUMNS, id)?;
+ tx.execute(
+ "UPDATE participant_identities SET name = ?1, calendar_address = ?2,
+ is_default = ?3
+ WHERE id = ?4",
+ params![
+ identity.name,
+ identity.calendar_address,
+ is_default as i64,
+ id
+ ],
+ )?;
+ id
+ }
+ None => {
+ tx.execute(
+ "INSERT INTO participant_identities (name, calendar_address, is_default)
+ VALUES (?1, ?2, ?3)",
+ params![identity.name, identity.calendar_address, is_default as i64],
+ )?;
+ counts.created += 1;
+ counts.fetched += 1;
+ tx.last_insert_rowid()
+ }
+ };
+ files_ids::record(&tx, source_id, files_ids::PARTICIPANT_IDENTITY, &key, id)?;
+ }
+ rerun.tally(
+ &tx,
+ "participant_identities",
+ PARTICIPANT_COLUMNS,
+ &mut counts,
+ )?;
+ tx.commit()?;
+ Ok(counts)
+}
+
+fn hex(bytes: &[u8]) -> String {
+ bytes.iter().map(|b| format!("{b:02x}")).collect()
+}
diff --git a/src/sync/import_files/paths.rs b/src/sync/import_files/paths.rs
new file mode 100644
index 0000000..a93c72a
--- /dev/null
+++ b/src/sync/import_files/paths.rs
@@ -0,0 +1,132 @@
+/*
+ * SPDX-FileCopyrightText: 2026 Coffey Labs LLC
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ */
+
+//! Paths from the manifest are data: whoever wrote the export chose them.
+//! Each is checked to stay inside its part of the tree, and no part of it
+//! may be a symbolic link, before anything is opened.
+
+use std::fs;
+use std::path::{Component, Path, PathBuf};
+
+/// `rel`, a `/`-separated path from the manifest, under `root`. It must
+/// start with `part` (`mail`, `contacts` and so on), climb nowhere, and
+/// pass through no symbolic link. What it names must exist.
+pub fn resolve(root: &Path, rel: &str, part: &str) -> Result {
+ let segments = segments(rel)?;
+ if segments.first() != Some(&part) {
+ return Err(format!("{rel:?} is outside {part}/"));
+ }
+ let mut path = root.to_path_buf();
+ for segment in &segments {
+ path.push(segment);
+ match fs::symlink_metadata(&path) {
+ Ok(meta) if meta.file_type().is_symlink() => {
+ return Err(format!(
+ "{rel:?} passes through a symbolic link, which is never followed"
+ ));
+ }
+ Ok(_) => {}
+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
+ return Err(format!("{rel:?} is not in the export"));
+ }
+ Err(e) => return Err(format!("{rel:?}: {e}")),
+ }
+ }
+ Ok(path)
+}
+
+/// The segments of a manifest path, refused if any could climb out or
+/// mean something other than a plain name.
+pub fn segments(rel: &str) -> Result, String> {
+ if rel.is_empty() {
+ return Err("an empty path".to_owned());
+ }
+ let mut out = Vec::new();
+ for segment in rel.split('/') {
+ let plain = !segment.is_empty()
+ && segment != "."
+ && segment != ".."
+ && !segment.contains(['\\', '\0'])
+ && matches!(
+ Path::new(segment)
+ .components()
+ .collect::>()
+ .as_slice(),
+ [Component::Normal(_)]
+ );
+ if !plain {
+ return Err(format!("{rel:?} is not a plain relative path"));
+ }
+ out.push(segment);
+ }
+ Ok(out)
+}
+
+/// A directory, and not a link to one.
+pub fn is_real_dir(path: &Path) -> bool {
+ fs::symlink_metadata(path).is_ok_and(|m| m.is_dir())
+}
+
+/// A regular file, and not a link to one.
+pub fn is_real_file(path: &Path) -> bool {
+ fs::symlink_metadata(path).is_ok_and(|m| m.is_file())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn paths_that_climb_or_jump_are_refused() {
+ for bad in [
+ "",
+ "/etc/passwd",
+ "mail/../../etc",
+ "mail/./x",
+ "mail//x",
+ "mail/a\\..\\b",
+ "..",
+ ] {
+ assert!(segments(bad).is_err(), "{bad:?}");
+ }
+ assert_eq!(segments("mail/.INBOX.x").unwrap(), ["mail", ".INBOX.x"]);
+ }
+
+ #[test]
+ fn a_path_must_stay_in_its_part_and_exist() {
+ let td = tempfile::tempdir().unwrap();
+ fs::create_dir_all(td.path().join("contacts")).unwrap();
+ fs::write(td.path().join("contacts/a.vcf"), "").unwrap();
+ fs::create_dir_all(td.path().join("calendars")).unwrap();
+ fs::write(td.path().join("calendars/b.ics"), "").unwrap();
+ assert!(resolve(td.path(), "contacts/a.vcf", "contacts").is_ok());
+ let e = resolve(td.path(), "calendars/b.ics", "contacts").unwrap_err();
+ assert!(e.contains("outside contacts/"), "{e}");
+ let e = resolve(td.path(), "contacts/missing.vcf", "contacts").unwrap_err();
+ assert!(e.contains("not in the export"), "{e}");
+ }
+
+ #[cfg(unix)]
+ #[test]
+ fn a_link_anywhere_on_the_way_is_refused() {
+ let td = tempfile::tempdir().unwrap();
+ let outside = tempfile::tempdir().unwrap();
+ fs::write(outside.path().join("secret"), "x").unwrap();
+ fs::create_dir_all(td.path().join("sieve")).unwrap();
+ std::os::unix::fs::symlink(
+ outside.path().join("secret"),
+ td.path().join("sieve/a.sieve"),
+ )
+ .unwrap();
+ std::os::unix::fs::symlink(outside.path(), td.path().join("contacts")).unwrap();
+ let e = resolve(td.path(), "sieve/a.sieve", "sieve").unwrap_err();
+ assert!(e.contains("symbolic link"), "{e}");
+ let e = resolve(td.path(), "contacts/secret", "contacts").unwrap_err();
+ assert!(e.contains("symbolic link"), "{e}");
+ assert!(!is_real_file(&td.path().join("sieve/a.sieve")));
+ assert!(!is_real_dir(&td.path().join("contacts")));
+ }
+}
diff --git a/src/sync/import_takeout/contacts.rs b/src/sync/import_takeout/contacts.rs
index 4d13149..231eb71 100644
--- a/src/sync/import_takeout/contacts.rs
+++ b/src/sync/import_takeout/contacts.rs
@@ -160,7 +160,8 @@ fn insert_card(
Ok(())
}
-fn split_vcards(text: &str) -> Vec {
+/// The vCards in one file, each as its own text.
+pub fn split_vcards(text: &str) -> Vec {
let mut chunks: Vec = Vec::new();
let mut current: Option = None;
for raw_line in text.lines() {
diff --git a/src/sync/mod.rs b/src/sync/mod.rs
index 90d93a1..264cf1d 100644
--- a/src/sync/mod.rs
+++ b/src/sync/mod.rs
@@ -14,6 +14,7 @@ pub mod export_managesieve;
pub mod import_dav;
pub mod import_exchange_ews;
pub mod import_exchange_graph;
+pub mod import_files;
pub mod import_imap;
pub mod import_jmap;
pub mod import_maildir;
diff --git a/tests/import_files.rs b/tests/import_files.rs
new file mode 100644
index 0000000..00fa61c
--- /dev/null
+++ b/tests/import_files.rs
@@ -0,0 +1,789 @@
+/*
+ * SPDX-FileCopyrightText: 2026 Coffey Labs LLC
+ *
+ * SPDX-License-Identifier: Apache-2.0 OR MIT
+ */
+
+//! `import files`: an account export read back, manifest and all, so what
+//! the open formats can't hold comes back too. An archive exported and
+//! imported again holds the same account, and exporting that gives the
+//! same tree.
+
+use std::collections::{BTreeMap, BTreeSet};
+use std::fs;
+use std::path::{Path, PathBuf};
+
+use inbuxa_migrate::db::{blobs, init};
+use inbuxa_migrate::export_files::{self, ExportFilesConfig};
+use inbuxa_migrate::logging::Logger;
+use inbuxa_migrate::sync::import_dav::calcard;
+use inbuxa_migrate::sync::import_files::{self, FilesImportConfig};
+use inbuxa_migrate::sync::{CommonConfig, Summary, TypeCounts};
+use rusqlite::{Connection, params};
+use serde_json::{Value, json};
+use tempfile::TempDir;
+
+const VCARD_ALICE: &str = "BEGIN:VCARD\r\nVERSION:4.0\r\nUID:card-alice\r\nFN:Alice Smith\r\n\
+EMAIL:alice@example.com\r\nEND:VCARD\r\n";
+const VCARD_BOB: &str = "BEGIN:VCARD\r\nVERSION:4.0\r\nUID:card-bob\r\nFN:Bob Jones\r\n\
+TEL:+1 555 0100\r\nEND:VCARD\r\n";
+const STANDUP: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:x\r\nBEGIN:VEVENT\r\nUID:ev-standup\r\n\
+DTSTAMP:20240101T000000Z\r\nDTSTART;TZID=Europe/Berlin:20240301T090000\r\nDURATION:PT15M\r\n\
+RRULE:FREQ=WEEKLY;COUNT=4\r\nSUMMARY:Standup\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n";
+const OFFSITE: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:x\r\nBEGIN:VEVENT\r\nUID:ev-offsite\r\n\
+DTSTAMP:20240101T000000Z\r\nDTSTART;VALUE=DATE:20240510\r\nSUMMARY:Offsite\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n";
+
+fn message(subject: &str) -> Vec {
+ format!(
+ "From: dave@example.net\r\nTo: alice@example.org\r\nSubject: {subject}\r\n\
+ Message-ID: <{}@example.net>\r\nDate: Fri, 01 Mar 2024 12:00:00 +0000\r\n\r\n{subject}\r\n",
+ subject.replace(' ', "-")
+ )
+ .into_bytes()
+}
+
+/// An account with everything the manifest exists for: folder names Maildir
+/// can't hold, roles, an unsubscribed folder, a message in two folders,
+/// custom keywords, default address book and calendar, a card and an event
+/// in two collections, colors and time zones, an active Sieve script,
+/// media types, a link, identities with signatures, a calendar identity.
+fn seed(path: &Path) {
+ let conn = init::open(path).unwrap();
+ conn.execute(
+ "INSERT INTO sources (id, kind, session_url, account_id, account_name, username)
+ VALUES (1, 'jmap', 'https://mail.example.org/jmap/session', 'a1', 'alice@example.org', 'alice')",
+ [],
+ )
+ .unwrap();
+ for (id, name, parent, role, subscribed) in [
+ (1, "Inbox", None, Some("inbox"), 1),
+ (2, "Clients", Some(1), None, 1),
+ (3, "Archive", None, Some("archive"), 1),
+ (4, "2024", Some(3), None, 1),
+ (5, "Équipe & co", None, None, 1),
+ (6, "v1.0 notes", None, None, 0),
+ (7, "Sent Items", None, Some("sent"), 1),
+ ] {
+ conn.execute(
+ "INSERT INTO mailboxes (id, name, parent_id, role, is_subscribed)
+ VALUES (?1, ?2, ?3, ?4, ?5)",
+ params![id, name, parent, role, subscribed],
+ )
+ .unwrap();
+ }
+ for (subject, mailboxes, keywords) in [
+ ("Seen and flagged", "[1]", r#"["$flagged","$seen"]"#),
+ ("Junk in two folders", "[1,3]", r#"["$Junk"]"#),
+ ("Client note", "[2]", r#"["$answered","project-x"]"#),
+ ("Old report", "[4]", "[]"),
+ ("Bonjour", "[5]", r#"["$seen"]"#),
+ ("Release notes", "[6]", "[]"),
+ ("Sent reply", "[7]", r#"["$seen"]"#),
+ ] {
+ let blob = blobs::intern_blob(&conn, &message(subject)).unwrap();
+ conn.execute(
+ "INSERT INTO emails (blob_id, received_at, mailbox_ids, keywords)
+ VALUES (?1, '2024-03-01T12:00:00Z', ?2, ?3)",
+ params![blob, mailboxes, keywords],
+ )
+ .unwrap();
+ }
+
+ conn.execute(
+ "INSERT INTO address_books (id, name, description, is_default)
+ VALUES (1, 'Personal', 'Friends and family', 0), (2, 'Work', NULL, 1)",
+ [],
+ )
+ .unwrap();
+ for (vcard, books) in [(VCARD_ALICE, "[1,2]"), (VCARD_BOB, "[2]")] {
+ let card = calcard::vcard_to_jscontact(vcard, "/x").unwrap();
+ conn.execute(
+ "INSERT INTO contact_cards (uid, address_book_ids, data) VALUES (?1, ?2, ?3)",
+ params![card.uid, books, card.data.to_string()],
+ )
+ .unwrap();
+ }
+
+ conn.execute(
+ "INSERT INTO calendars (id, name, description, color, time_zone, is_default)
+ VALUES (1, 'Team, Berlin', 'Stand-ups', '#3366ff', 'Europe/Berlin', 1),
+ (2, 'Company', NULL, '#ff9900', 'America/Phoenix', 0)",
+ [],
+ )
+ .unwrap();
+ for (ics, calendars) in [(STANDUP, "[1]"), (OFFSITE, "[1,2]")] {
+ let mut entry = calcard::ical_to_jscalendar_entries(ics).unwrap().remove(0);
+ calcard::strip_extracted_fields_from_event(&mut entry.data);
+ conn.execute(
+ "INSERT INTO calendar_events (calendar_ids, data) VALUES (?1, ?2)",
+ params![calendars, entry.data.to_string()],
+ )
+ .unwrap();
+ }
+
+ for (name, active, script) in [
+ (
+ "filters",
+ 1,
+ "require \"fileinto\";\r\nif header :contains \"subject\" \"report\" { fileinto \"Archive\"; }\r\n",
+ ),
+ ("old", 0, "keep;\r\n"),
+ ] {
+ let blob = blobs::intern_blob(&conn, script.as_bytes()).unwrap();
+ conn.execute(
+ "INSERT INTO sieve_scripts (name, is_active, blob_id) VALUES (?1, ?2, ?3)",
+ params![name, active, blob],
+ )
+ .unwrap();
+ }
+
+ let text = blobs::intern_blob(&conn, b"hello\n").unwrap();
+ let pdf = blobs::intern_blob(&conn, b"%PDF-1.7 not really\n").unwrap();
+ conn.execute(
+ "INSERT INTO file_nodes (id, parent_id, node_type, blob_id, target, name, media_type, created, modified)
+ VALUES (1, NULL, 'directory', NULL, NULL, 'Docs', NULL, '2024-01-01T00:00:00Z', NULL),
+ (2, 1, 'file', ?1, NULL, 'a.txt', 'text/plain', '2024-01-01T00:00:00Z', '2024-02-01T10:00:00Z'),
+ (3, 1, 'directory', NULL, NULL, 'Taxes', NULL, '2024-01-01T00:00:00Z', NULL),
+ (4, 3, 'file', ?2, NULL, '2025.pdf', 'application/pdf', '2024-01-01T00:00:00Z', '2024-04-15T09:30:00Z'),
+ (5, 1, 'symlink', NULL, '{\"path\":\"Taxes/2025.pdf\"}', 'latest', NULL, '2024-01-01T00:00:00Z', NULL)",
+ params![text, pdf],
+ )
+ .unwrap();
+ conn.execute(
+ "INSERT INTO identities (name, email, reply_to, bcc, text_signature, html_signature)
+ VALUES ('Alice Smith', 'alice@example.org', '[{\"name\":\"Alice\",\"email\":\"replies@example.org\"}]',
+ NULL, 'Alice\n-- \nExample Org', 'Alice
'),
+ ('Alice (support)', 'support@example.org', NULL, '[{\"name\":null,\"email\":\"log@example.org\"}]', '', '')",
+ [],
+ )
+ .unwrap();
+ conn.execute(
+ "INSERT INTO participant_identities (name, calendar_address, is_default)
+ VALUES ('Alice Smith', 'mailto:alice@example.org', 1),
+ ('Alice (support)', 'mailto:support@example.org', 0)",
+ [],
+ )
+ .unwrap();
+}
+
+struct Fixture {
+ _td: TempDir,
+ root: PathBuf,
+ archive: PathBuf,
+ out: PathBuf,
+}
+
+fn exported() -> Fixture {
+ let td = tempfile::tempdir().unwrap();
+ let root = td.path().to_path_buf();
+ let archive = root.join("archive.sqlite");
+ let out = root.join("out");
+ seed(&archive);
+ export(&archive, &out);
+ Fixture {
+ _td: td,
+ root,
+ archive,
+ out,
+ }
+}
+
+fn export(archive: &Path, out: &Path) {
+ let summary = export_files::run(ExportFilesConfig {
+ archive: archive.to_path_buf(),
+ dir: out.to_path_buf(),
+ objects: None,
+ dry_run: false,
+ logger: Logger::from_flags(true, 0),
+ })
+ .unwrap();
+ assert!(!summary.any_failed(), "{:?}", summary.per_type);
+}
+
+fn common(archive: &Path, dry_run: bool) -> CommonConfig {
+ CommonConfig {
+ archive: archive.to_path_buf(),
+ threads: 1,
+ dry_run,
+ max_retries: 0,
+ allow_invalid_certs: false,
+ logger: Logger::from_flags(true, 0),
+ }
+}
+
+fn import(dir: &Path, archive: &Path) -> Result {
+ import_files::run(
+ common(archive, false),
+ FilesImportConfig {
+ dir: dir.to_path_buf(),
+ allow_source_change: false,
+ },
+ )
+}
+
+fn counts<'a>(summary: &'a Summary, type_name: &str) -> &'a TypeCounts {
+ &summary
+ .per_type
+ .iter()
+ .find(|(t, _)| *t == type_name)
+ .unwrap_or_else(|| panic!("no {type_name} in {:?}", summary.per_type))
+ .1
+}
+
+fn rows(conn: &Connection, sql: &str) -> Vec> {
+ let mut stmt = conn.prepare(sql).unwrap();
+ let n = stmt.column_count();
+ stmt.query_map([], |r| {
+ (0..n)
+ .map(|i| {
+ Ok(match r.get::<_, rusqlite::types::Value>(i)? {
+ rusqlite::types::Value::Null => Value::Null,
+ rusqlite::types::Value::Integer(v) => json!(v),
+ rusqlite::types::Value::Real(v) => json!(v),
+ rusqlite::types::Value::Text(v) => json!(v),
+ rusqlite::types::Value::Blob(v) => json!(blake3::hash(&v).to_hex().to_string()),
+ })
+ })
+ .collect()
+ })
+ .unwrap()
+ .map(Result::unwrap)
+ .collect()
+}
+
+/// An archive's account in terms no row id appears in: every collection by
+/// its names, every item by what it is and where it is.
+fn account(path: &Path) -> BTreeMap<&'static str, BTreeSet> {
+ let conn = Connection::open(path).unwrap();
+ let mut out: BTreeMap<&'static str, BTreeSet> = BTreeMap::new();
+
+ let boxes: BTreeMap)> =
+ rows(&conn, "SELECT id, name, parent_id FROM mailboxes")
+ .into_iter()
+ .map(|r| {
+ (
+ r[0].as_i64().unwrap(),
+ (r[1].as_str().unwrap().to_owned(), r[2].as_i64()),
+ )
+ })
+ .collect();
+ let path_of = |mut id: i64| {
+ let mut parts = Vec::new();
+ loop {
+ let (name, parent) = &boxes[&id];
+ parts.insert(0, name.clone());
+ match parent {
+ Some(p) => id = *p,
+ None => break parts.join("/"),
+ }
+ }
+ };
+ for r in rows(&conn, "SELECT id, role, is_subscribed FROM mailboxes") {
+ out.entry("mailboxes").or_default().insert(format!(
+ "{} role={} subscribed={}",
+ path_of(r[0].as_i64().unwrap()),
+ r[1],
+ r[2]
+ ));
+ }
+ let ids = |v: &Value| -> Vec { serde_json::from_str(v.as_str().unwrap()).unwrap() };
+ for r in rows(
+ &conn,
+ "SELECT b.data, e.mailbox_ids, e.keywords, e.received_at FROM emails e JOIN blobs b ON b.id = e.blob_id",
+ ) {
+ let mut folders: Vec = ids(&r[1]).into_iter().map(path_of).collect();
+ folders.sort();
+ let mut keywords: Vec = serde_json::from_str(r[2].as_str().unwrap()).unwrap();
+ keywords.sort();
+ out.entry("emails").or_default().insert(format!(
+ "{} in {folders:?} keywords={keywords:?} received={}",
+ r[0], r[3]
+ ));
+ }
+
+ let names = |sql: &str| -> BTreeMap {
+ rows(&conn, sql)
+ .into_iter()
+ .map(|r| (r[0].as_i64().unwrap(), r[1].as_str().unwrap().to_owned()))
+ .collect()
+ };
+ let books = names("SELECT id, name FROM address_books");
+ for r in rows(
+ &conn,
+ "SELECT name, description, is_default FROM address_books",
+ ) {
+ out.entry("address_books")
+ .or_default()
+ .insert(format!("{r:?}"));
+ }
+ for r in rows(
+ &conn,
+ "SELECT uid, address_book_ids, data FROM contact_cards",
+ ) {
+ let mut in_books: Vec<&str> = ids(&r[1]).iter().map(|b| books[b].as_str()).collect();
+ in_books.sort();
+ out.entry("cards")
+ .or_default()
+ .insert(format!("{} in {in_books:?}", r[0]));
+ }
+ let calendars = names("SELECT id, name FROM calendars");
+ for r in rows(
+ &conn,
+ "SELECT name, description, color, time_zone, is_default FROM calendars",
+ ) {
+ out.entry("calendars").or_default().insert(format!("{r:?}"));
+ }
+ for r in rows(
+ &conn,
+ "SELECT calendar_ids, data, data_type FROM calendar_events",
+ ) {
+ let data: Value = serde_json::from_str(r[1].as_str().unwrap()).unwrap();
+ let mut in_calendars: Vec<&str> =
+ ids(&r[0]).iter().map(|c| calendars[c].as_str()).collect();
+ in_calendars.sort();
+ out.entry("events").or_default().insert(format!(
+ "{} {} {} in {in_calendars:?}",
+ data["uid"], data["title"], r[2]
+ ));
+ }
+ for r in rows(
+ &conn,
+ "SELECT s.name, s.is_active, b.data FROM sieve_scripts s JOIN blobs b ON b.id = s.blob_id",
+ ) {
+ out.entry("sieve").or_default().insert(format!("{r:?}"));
+ }
+ let nodes = names("SELECT id, name FROM file_nodes");
+ let parents: BTreeMap> = rows(&conn, "SELECT id, parent_id FROM file_nodes")
+ .into_iter()
+ .map(|r| (r[0].as_i64().unwrap(), r[1].as_i64()))
+ .collect();
+ for r in rows(
+ &conn,
+ "SELECT n.id, n.node_type, n.media_type, n.target, b.data, n.modified
+ FROM file_nodes n LEFT JOIN blobs b ON b.id = n.blob_id",
+ ) {
+ let mut id = r[0].as_i64().unwrap();
+ let mut path = nodes[&id].clone();
+ while let Some(Some(p)) = parents.get(&id) {
+ path = format!("{}/{path}", nodes[p]);
+ id = *p;
+ }
+ // A link has no file whose time the export could set.
+ let modified = if r[1] == "symlink" {
+ Value::Null
+ } else {
+ r[5].clone()
+ };
+ out.entry("files").or_default().insert(format!(
+ "{path} {} {} {} {} {modified}",
+ r[1], r[2], r[3], r[4]
+ ));
+ }
+ for r in rows(
+ &conn,
+ "SELECT name, email, reply_to, bcc, text_signature, html_signature FROM identities",
+ ) {
+ // Addresses are JSON; the same addresses may be written in any key order.
+ let mut r = r;
+ for v in &mut r[2..4] {
+ if let Some(text) = v.as_str() {
+ *v = serde_json::from_str(text).unwrap();
+ }
+ }
+ out.entry("identities")
+ .or_default()
+ .insert(format!("{r:?}"));
+ }
+ for r in rows(
+ &conn,
+ "SELECT name, calendar_address, is_default FROM participant_identities",
+ ) {
+ out.entry("participant_identities")
+ .or_default()
+ .insert(format!("{r:?}"));
+ }
+ out
+}
+
+#[test]
+fn an_exported_account_comes_back_whole() {
+ let f = exported();
+ let back = f.root.join("back.sqlite");
+ let summary = import(&f.out, &back).unwrap();
+ assert!(!summary.any_failed(), "{:?}", summary.per_type);
+
+ let original = account(&f.archive);
+ let read_back = account(&back);
+ for part in original.keys() {
+ assert_eq!(read_back.get(part), original.get(part), "{part}");
+ }
+ assert_eq!(read_back, original);
+
+ // One message in two folders came back as one message, not two.
+ assert_eq!(counts(&summary, "email").created, 7);
+ // Folders Maildir can't name kept their real names.
+ let mailboxes = &read_back["mailboxes"];
+ assert!(
+ mailboxes.contains("v1.0 notes role=null subscribed=0"),
+ "{mailboxes:?}"
+ );
+ assert!(
+ mailboxes.contains("Équipe & co role=null subscribed=1"),
+ "{mailboxes:?}"
+ );
+
+ let conn = Connection::open(&back).unwrap();
+ let source: (String, String) = conn
+ .query_row("SELECT kind, account_name FROM sources", [], |r| {
+ Ok((r.get(0)?, r.get(1)?))
+ })
+ .unwrap();
+ assert_eq!(source, ("files".to_owned(), "alice@example.org".to_owned()));
+}
+
+#[test]
+fn exporting_what_was_imported_gives_the_same_tree() {
+ let f = exported();
+ let back = f.root.join("back.sqlite");
+ import(&f.out, &back).unwrap();
+ let again = f.root.join("again");
+ export(&back, &again);
+
+ let mut first = manifest(&f.out);
+ let mut second = manifest(&again);
+ for m in [&mut first, &mut second] {
+ let m = m.as_object_mut().unwrap();
+ m.remove("created");
+ m.remove("sources");
+ }
+ assert_eq!(second, first);
+ assert_eq!(tree(&again), tree(&f.out));
+}
+
+/// Every file in an export by path, with its content; messages by folder
+/// and content, since their names carry archive ids.
+fn tree(root: &Path) -> BTreeMap {
+ let mut out = BTreeMap::new();
+ let mut stack = vec![root.to_path_buf()];
+ while let Some(dir) = stack.pop() {
+ for entry in fs::read_dir(&dir).unwrap() {
+ let path = entry.unwrap().path();
+ if path.is_dir() {
+ stack.push(path);
+ continue;
+ }
+ let rel = path
+ .strip_prefix(root)
+ .unwrap()
+ .to_string_lossy()
+ .into_owned();
+ if rel == "manifest.json" {
+ continue;
+ }
+ let bytes = fs::read(&path).unwrap();
+ // iCalendar gives the order of a calendar's events no meaning.
+ let bytes = if rel.ends_with(".ics") {
+ components_sorted(&String::from_utf8(bytes).unwrap()).into_bytes()
+ } else {
+ bytes
+ };
+ let content = blake3::hash(&bytes).to_hex().to_string();
+ let key = match rel.split_once("/cur/") {
+ Some((folder, name)) => {
+ let flags = name.rsplit_once(":2,").map(|(_, f)| f).unwrap_or("");
+ format!("{folder}/cur/{content}:2,{flags}")
+ }
+ None => rel,
+ };
+ out.insert(key, content);
+ }
+ }
+ out
+}
+
+/// An iCalendar object with its components in sorted order, each kept as
+/// written.
+fn components_sorted(ics: &str) -> String {
+ let mut head = String::new();
+ let mut components: Vec = Vec::new();
+ let mut current: Option = None;
+ for line in ics.split_inclusive('\n') {
+ let bare = line.trim_end();
+ match &mut current {
+ None if bare.starts_with("BEGIN:") && bare != "BEGIN:VCALENDAR" => {
+ current = Some(line.to_owned());
+ }
+ None => head.push_str(line),
+ Some(text) => {
+ text.push_str(line);
+ if bare.starts_with("END:") && text.starts_with(&format!("BEGIN:{}", &bare[4..])) {
+ components.push(current.take().unwrap());
+ }
+ }
+ }
+ }
+ components.sort();
+ head + &components.concat()
+}
+
+fn manifest(out: &Path) -> Value {
+ serde_json::from_str(&fs::read_to_string(out.join("manifest.json")).unwrap()).unwrap()
+}
+
+#[test]
+fn a_second_run_finds_everything_in_place() {
+ let f = exported();
+ let back = f.root.join("back.sqlite");
+ import(&f.out, &back).unwrap();
+ let before = account(&back);
+ let summary = import(&f.out, &back).unwrap();
+ for (type_name, c) in &summary.per_type {
+ assert_eq!(
+ (c.created, c.updated, c.failed),
+ (0, 0, 0),
+ "{type_name}: {c:?}"
+ );
+ }
+ assert_eq!(counts(&summary, "email").skipped, 7);
+ assert_eq!(account(&back), before);
+}
+
+#[test]
+fn a_changed_export_updates_the_archive_and_deletes_nothing() {
+ let f = exported();
+ let back = f.root.join("back.sqlite");
+ import(&f.out, &back).unwrap();
+
+ // Read in Dovecot since: one message in the archive folder is now seen.
+ let archive_cur = f.out.join("mail/.Archive/cur");
+ let name = fs::read_dir(&archive_cur)
+ .unwrap()
+ .next()
+ .unwrap()
+ .unwrap()
+ .file_name()
+ .into_string()
+ .unwrap();
+ fs::rename(
+ archive_cur.join(&name),
+ archive_cur.join(format!("{name}S")),
+ )
+ .unwrap();
+ // And the Sieve script that runs is now the other one.
+ let mut m = manifest(&f.out);
+ for script in m["sieve"].as_array_mut().unwrap() {
+ let active = script["name"] == "old";
+ script["active"] = json!(active);
+ }
+ fs::write(f.out.join("manifest.json"), m.to_string()).unwrap();
+
+ let summary = import(&f.out, &back).unwrap();
+ assert_eq!(
+ counts(&summary, "email").updated,
+ 1,
+ "{:?}",
+ summary.per_type
+ );
+ assert_eq!(counts(&summary, "sievescript").updated, 2);
+ let conn = Connection::open(&back).unwrap();
+ let active: String = conn
+ .query_row(
+ "SELECT name FROM sieve_scripts WHERE is_active = 1",
+ [],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(active, "old");
+ let junk: String = conn
+ .query_row(
+ "SELECT keywords FROM emails WHERE json_array_length(mailbox_ids) = 2",
+ [],
+ |r| r.get(0),
+ )
+ .unwrap();
+ // The copy in the inbox wasn't marked; the one in Archive was, and a
+ // message has one set of keywords.
+ assert_eq!(junk, r#"["$Junk","$seen"]"#);
+ let emails: i64 = conn
+ .query_row("SELECT count(*) FROM emails", [], |r| r.get(0))
+ .unwrap();
+ assert_eq!(emails, 7);
+}
+
+#[test]
+fn an_unfinished_export_or_a_later_version_is_refused() {
+ let f = exported();
+ let back = f.root.join("back.sqlite");
+ let text = fs::read_to_string(f.out.join("manifest.json")).unwrap();
+
+ fs::remove_file(f.out.join("manifest.json")).unwrap();
+ let e = import(&f.out, &back).unwrap_err();
+ assert_eq!(e.exit_code(), 1);
+ assert!(e.to_string().contains("interrupted"), "{e}");
+ assert!(
+ !back.exists(),
+ "nothing is created for an export that can't be read"
+ );
+
+ let mut m: Value = serde_json::from_str(&text).unwrap();
+ m["formatVersion"] = json!(2);
+ fs::write(f.out.join("manifest.json"), m.to_string()).unwrap();
+ let e = import(&f.out, &back).unwrap_err();
+ assert!(e.to_string().contains("version 2"), "{e}");
+
+ // What 27.1.0-rc.2 wrote is the same format under its old name.
+ m["formatVersion"] = json!(1);
+ m["format"] = json!("inbuxa-migrate-files");
+ fs::write(f.out.join("manifest.json"), m.to_string()).unwrap();
+ import(&f.out, &back).unwrap();
+}
+
+#[test]
+fn a_dry_run_reads_the_export_and_writes_nothing() {
+ let f = exported();
+ let back = f.root.join("back.sqlite");
+ let summary = import_files::run(
+ common(&back, true),
+ FilesImportConfig {
+ dir: f.out.clone(),
+ allow_source_change: false,
+ },
+ )
+ .unwrap();
+ assert!(!back.exists());
+ assert_eq!(counts(&summary, "mailbox").fetched, 7);
+ // The message in two folders is a file in each.
+ assert_eq!(counts(&summary, "email").fetched, 8);
+ assert_eq!(counts(&summary, "contactcard").fetched, 3);
+ assert_eq!(counts(&summary, "calendarevent").fetched, 3);
+ assert_eq!(counts(&summary, "filenode").fetched, 5);
+}
+
+#[test]
+fn another_export_needs_allow_source_change() {
+ let f = exported();
+ let back = f.root.join("back.sqlite");
+ import(&f.out, &back).unwrap();
+ let copy = f.root.join("copy");
+ export(&back, ©);
+ let e = import(©, &back).unwrap_err();
+ assert_eq!(e.exit_code(), 4, "{e}");
+}
+
+#[cfg(unix)]
+#[test]
+fn nothing_outside_the_export_is_read() {
+ use std::os::unix::fs::symlink;
+
+ let f = exported();
+ let outside = f.root.join("outside");
+ fs::create_dir_all(outside.join("cur")).unwrap();
+ fs::create_dir_all(outside.join("new")).unwrap();
+ fs::create_dir_all(outside.join("tmp")).unwrap();
+ let secret = b"Subject: not yours\r\n\r\nsecret\r\n";
+ fs::write(outside.join("cur/1.x:2,S"), secret).unwrap();
+ fs::write(
+ outside.join("secret.vcf"),
+ VCARD_BOB.replace("card-bob", "card-secret"),
+ )
+ .unwrap();
+
+ let mut m = manifest(&f.out);
+ // Paths that climb out, by name and through links.
+ m["mail"]["folders"].as_array_mut().unwrap().push(json!({
+ "name": "Escape", "path": ["Escape"], "dir": "mail/../outside",
+ "role": null, "subscribed": true, "messages": 1
+ }));
+ symlink(&outside, f.out.join("mail/.Linked")).unwrap();
+ m["mail"]["folders"].as_array_mut().unwrap().push(json!({
+ "name": "Linked", "path": ["Linked"], "dir": "mail/.Linked",
+ "role": null, "subscribed": true, "messages": 1
+ }));
+ symlink(outside.join("cur/1.x:2,S"), f.out.join("mail/cur/2.x:2,S")).unwrap();
+ m["contacts"].as_array_mut().unwrap().push(json!({
+ "name": "Stolen", "description": null, "file": "/etc/passwd", "default": false, "cards": 1
+ }));
+ symlink(
+ outside.join("secret.vcf"),
+ f.out.join("contacts/Linked.vcf"),
+ )
+ .unwrap();
+ m["contacts"].as_array_mut().unwrap().push(json!({
+ "name": "Linked", "description": null, "file": "contacts/Linked.vcf", "default": false, "cards": 1
+ }));
+ symlink(&outside, f.out.join("files/Docs/elsewhere")).unwrap();
+ m["files"]["links"].as_array_mut().unwrap().push(json!({
+ "path": "files/../../x", "target": {"path": "/"}
+ }));
+ fs::write(f.out.join("manifest.json"), m.to_string()).unwrap();
+
+ let back = f.root.join("back.sqlite");
+ let summary = import(&f.out, &back).unwrap();
+ assert!(
+ counts(&summary, "email").failed >= 2,
+ "{:?}",
+ summary.per_type
+ );
+ assert!(counts(&summary, "contactcard").failed >= 2);
+ assert!(counts(&summary, "filenode").failed >= 1);
+
+ let conn = Connection::open(&back).unwrap();
+ let leaked: i64 = conn
+ .query_row(
+ "SELECT count(*) FROM blobs WHERE instr(data, 'secret') > 0",
+ [],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(leaked, 0);
+ let cards: i64 = conn
+ .query_row(
+ "SELECT count(*) FROM contact_cards WHERE uid = 'card-secret'",
+ [],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(cards, 0);
+ let elsewhere: i64 = conn
+ .query_row(
+ "SELECT count(*) FROM file_nodes WHERE name = 'elsewhere'",
+ [],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(elsewhere, 0);
+ // Everything that was safely inside still came in.
+ assert_eq!(counts(&summary, "email").created, 7);
+}
+
+#[test]
+fn a_card_another_import_put_in_the_archive_keeps_its_books() {
+ let f = exported();
+ let back = f.root.join("back.sqlite");
+ {
+ let conn = init::open(&back).unwrap();
+ conn.execute(
+ "INSERT INTO sources (id, kind, session_url, account_id, account_name, username)
+ VALUES (1, 'carddav', 'https://dav.example.org/', 'alice', 'alice', 'alice')",
+ [],
+ )
+ .unwrap();
+ conn.execute(
+ "INSERT INTO address_books (id, name) VALUES (1, 'Phone')",
+ [],
+ )
+ .unwrap();
+ let card = calcard::vcard_to_jscontact(VCARD_ALICE, "/x").unwrap();
+ conn.execute(
+ "INSERT INTO contact_cards (uid, address_book_ids, data) VALUES (?1, '[1]', ?2)",
+ params![card.uid, card.data.to_string()],
+ )
+ .unwrap();
+ }
+ import(&f.out, &back).unwrap();
+ let cards = &account(&back)["cards"];
+ assert!(
+ cards.contains(r#""card-alice" in ["Personal", "Phone", "Work"]"#),
+ "{cards:?}"
+ );
+}