import files: read an account export back, manifest and all #54
No files matched your search
@@ -7,6 +7,20 @@ candidates before a series ships (`27.1-rc.1`). The tag is the version with a
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- `import files` reads an account export back into an archive: the tree
|
||||
`export-files` writes, specified at
|
||||
<https://coffeylabs.org/specs/account-export/>. Besides the mail,
|
||||
contacts, calendars, Sieve scripts and files, it reads what only the
|
||||
manifest holds: real folder names, roles and subscriptions, the default
|
||||
address book and calendar, calendar colors and time zones, the active
|
||||
script, media types, links and identities. Copies of a message in several
|
||||
folders come back as one message in each. Paths in the manifest are
|
||||
checked to stay inside the export, and no symbolic link is followed. An
|
||||
archive gains the source kind `files` and a table for what it read at its
|
||||
next import; until then `inspect` and the other read-only commands call it
|
||||
an older layout.
|
||||
|
||||
## [27.1-rc.8]
|
||||
|
||||
The eighth release candidate of the 27.1 series: what a server needs to
|
||||
|
||||
@@ -18,6 +18,7 @@ contacts, identities, Sieve scripts and files.
|
||||
inbuxa-migrate inspect alice.sqlite what landed
|
||||
inbuxa-migrate export … alice.sqlite the archive into inbuxa
|
||||
inbuxa-migrate export-files alice.sqlite out the archive as plain files
|
||||
inbuxa-migrate import files out alice.sqlite those files read back, manifest and all
|
||||
inbuxa-migrate export-imap … alice.sqlite its mail into any IMAP server
|
||||
inbuxa-migrate export-managesieve … alice.sqlite its filters, likewise
|
||||
inbuxa-migrate export-caldav … alice.sqlite its calendars into any CalDAV server
|
||||
@@ -48,7 +49,8 @@ server, `export-caldav` and `export-carddav` the calendars and contacts into
|
||||
any CalDAV and CardDAV server, and `export-files` is the way out to everything else: it writes
|
||||
the archive as a Maildir++ tree, vCard, iCalendar and Sieve files that other
|
||||
servers and mail, calendar and contacts apps open, with no network and no
|
||||
store format in the way. For a whole server, `export-directory` writes the
|
||||
store format in the way, and `import files` reads such a tree back whole.
|
||||
For a whole server, `export-directory` writes the
|
||||
list of domains, accounts, groups, lists and aliases to set up again
|
||||
elsewhere, without passwords, and `import-directory` sets them up on an
|
||||
inbuxa server, with a new password for each user. `migrate` moves many
|
||||
|
||||
+43
-3
@@ -215,6 +215,45 @@ Finds every `.mbox`, `.ics` and `.vcf` file under a directory and imports it.
|
||||
It is shaped for Google Takeout but reads any such tree. `--noautomap` stops
|
||||
it giving Gmail's system labels their mailbox roles.
|
||||
|
||||
### An account export
|
||||
|
||||
```
|
||||
inbuxa-migrate import files <DIR> <ARCHIVE>
|
||||
```
|
||||
|
||||
Reads an account export back: the directory `export-files` writes, laid out
|
||||
as <https://coffeylabs.org/specs/account-export/> specifies, from this tool
|
||||
or any other that writes the layout. No network. Where `import maildir` and
|
||||
`import takeout` read the open formats on their own, this also reads the
|
||||
manifest, so what those formats can't hold comes back too:
|
||||
|
||||
- every folder's real name, its place in the tree, its role and whether it
|
||||
was subscribed, in place of the names Maildir++ could give it on disk;
|
||||
- which address book and calendar were the default, and each calendar's
|
||||
description, color and time zone;
|
||||
- which Sieve script was active;
|
||||
- the files' media types and links;
|
||||
- the identities with their reply-to, bcc and signatures, and the calendar
|
||||
identities.
|
||||
|
||||
A message kept in several folders is written to each folder's directory;
|
||||
copies with the same content come back as one message in all of those
|
||||
folders. The same goes for a card in two address books and an event in two
|
||||
calendars. A message's arrival time is its file's modification time.
|
||||
|
||||
The manifest's paths are data, so each is checked to stay inside its part of
|
||||
the export, and nothing that is a symbolic link is followed: a link in the
|
||||
tree is skipped with a warning, and the links an account had are made from
|
||||
the manifest as archive entries. A directory without `manifest.json` was
|
||||
interrupted, and is refused, as is a later version of the layout than this
|
||||
release reads. 27.1.0-rc.2 named the format `inbuxa-migrate-files`; that is
|
||||
read as the same format.
|
||||
|
||||
Run again, it brings the archive up to date with the export: flags that
|
||||
changed, a different active script, a renamed folder. It never deletes.
|
||||
With `--dry-run` it checks the export and counts what it holds, and opens no
|
||||
archive.
|
||||
|
||||
### Microsoft Exchange (EWS)
|
||||
|
||||
```
|
||||
@@ -484,7 +523,7 @@ DIR/
|
||||
```
|
||||
|
||||
- **Mail** is laid out as Dovecot and Courier keep it, so either can serve
|
||||
it as it is, and `import maildir` reads it back. Folder names are in
|
||||
it as it is. Folder names are in
|
||||
modified UTF-7. A `.` or `/` inside a folder's name becomes `_` on disk,
|
||||
since Maildir++ uses the dot between levels. A message kept in several
|
||||
folders is written to each. Read, answered, flagged, draft, forwarded and
|
||||
@@ -495,7 +534,6 @@ DIR/
|
||||
files. **Calendars** are iCalendar, one VCALENDAR each, named by
|
||||
`X-WR-CALNAME`, with each time zone once. Photos and attachments the
|
||||
archive holds go inside as `data:` URIs, as `export` sends them.
|
||||
`import takeout` reads both back.
|
||||
- **Sieve scripts** are written as they are; the manifest says which is
|
||||
active.
|
||||
- **Files** keep their names and dates. Links are listed in the manifest
|
||||
@@ -513,7 +551,9 @@ the way to finish it is to run again into an empty directory.
|
||||
|
||||
The layout and the manifest are specified at
|
||||
<https://coffeylabs.org/specs/account-export/>, so other tools can read and
|
||||
write them too.
|
||||
write them too. `import files` reads the whole tree back, manifest and all
|
||||
([An account export](#an-account-export)); `import maildir` and `import
|
||||
takeout` read the mail, contacts and calendars without it.
|
||||
|
||||
`--objects` takes the same types as `export`; naming a folder or a message
|
||||
type writes all of mail, and the same for contacts and calendars. With
|
||||
|
||||
+40
@@ -30,6 +30,7 @@ use crate::sync::export_managesieve::ExportManageSieveConfig;
|
||||
use crate::sync::import_dav::{DavAuth, DavImportConfig, DavKindArg, SourceChange};
|
||||
use crate::sync::import_exchange_ews::{EwsAuth, EwsImportConfig};
|
||||
use crate::sync::import_exchange_graph::{GraphAuth, GraphImportConfig};
|
||||
use crate::sync::import_files::FilesImportConfig;
|
||||
use crate::sync::import_imap::{ImapAuth, ImapImportConfig};
|
||||
use crate::sync::import_maildir::MaildirImportConfig;
|
||||
use crate::sync::import_managesieve::{ManageSieveAuth, ManageSieveImportConfig};
|
||||
@@ -126,6 +127,8 @@ enum Source {
|
||||
Maildir(Box<MaildirImportArgs>),
|
||||
#[command(about = "Google Takeout (or any mbox/ics/vcf directory tree)")]
|
||||
Takeout(Box<TakeoutImportArgs>),
|
||||
#[command(about = "An account export, as export-files writes it, manifest and all")]
|
||||
Files(Box<FilesImportArgs>),
|
||||
#[command(about = "MS Exchange via EWS (SOAP/XML)", name = "exchange-ews")]
|
||||
ExchangeEws(Box<ExchangeEwsImportArgs>),
|
||||
#[command(
|
||||
@@ -864,6 +867,7 @@ pub enum Action {
|
||||
ImportManageSieve(CommonConfig, ManageSieveImportConfig),
|
||||
ImportMaildir(CommonConfig, MaildirImportConfig),
|
||||
ImportTakeout(CommonConfig, TakeoutImportConfig),
|
||||
ImportFiles(CommonConfig, FilesImportConfig),
|
||||
ImportExchangeEws(CommonConfig, EwsImportConfig),
|
||||
ImportExchangeGraph(CommonConfig, GraphImportConfig),
|
||||
Export(CommonConfig, ExportConfig),
|
||||
@@ -1042,6 +1046,7 @@ fn resolve_import(source: Source) -> Result<Action, Error> {
|
||||
Source::Managesieve(args) => resolve_managesieve_import(*args),
|
||||
Source::Maildir(args) => resolve_maildir_import(*args),
|
||||
Source::Takeout(args) => resolve_takeout_import(*args),
|
||||
Source::Files(args) => resolve_files_import(*args),
|
||||
Source::ExchangeEws(args) => resolve_exchange_ews_import(*args),
|
||||
Source::ExchangeGraph(args) => resolve_exchange_graph_import(*args),
|
||||
}
|
||||
@@ -1219,6 +1224,41 @@ fn resolve_takeout_import(args: TakeoutImportArgs) -> Result<Action, Error> {
|
||||
))
|
||||
}
|
||||
|
||||
#[derive(Args)]
|
||||
pub struct FilesImportArgs {
|
||||
#[arg(
|
||||
value_name = "DIR",
|
||||
help = "An account export: the directory export-files wrote, with its manifest.json"
|
||||
)]
|
||||
dir: PathBuf,
|
||||
|
||||
#[arg(
|
||||
value_name = "ARCHIVE",
|
||||
help = "Local SQLite archive (created if absent)"
|
||||
)]
|
||||
archive: PathBuf,
|
||||
|
||||
#[arg(
|
||||
long,
|
||||
help = "Permit importing when the archive records an account export at a different path"
|
||||
)]
|
||||
allow_source_change: bool,
|
||||
|
||||
#[command(flatten)]
|
||||
global: GlobalArgs,
|
||||
}
|
||||
|
||||
fn resolve_files_import(args: FilesImportArgs) -> Result<Action, Error> {
|
||||
let common = common_config(&args.global, args.archive);
|
||||
Ok(Action::ImportFiles(
|
||||
common,
|
||||
FilesImportConfig {
|
||||
dir: args.dir,
|
||||
allow_source_change: args.allow_source_change,
|
||||
},
|
||||
))
|
||||
}
|
||||
|
||||
fn resolve_dav_import(args: DavImportArgs, kind: DavKindArg) -> Result<Action, Error> {
|
||||
let common = common_config(&args.global, args.archive);
|
||||
let auth = resolve_dav_auth(
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! What `import files` has read from an account export, by a key of its
|
||||
//! own for each thing: a folder's path of real names, a message's content
|
||||
//! hash, a collection's file, a card's uid. Several keys may lead to one
|
||||
//! archive row: a message kept in two folders, a card in two address books.
|
||||
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
|
||||
pub const MAILBOX: &str = "mailbox";
|
||||
pub const EMAIL: &str = "email";
|
||||
pub const ADDRESS_BOOK: &str = "addressbook";
|
||||
pub const CONTACT_CARD: &str = "contactcard";
|
||||
pub const CALENDAR: &str = "calendar";
|
||||
pub const CALENDAR_EVENT: &str = "calendarevent";
|
||||
pub const SIEVE_SCRIPT: &str = "sievescript";
|
||||
pub const FILE_NODE: &str = "filenode";
|
||||
pub const IDENTITY: &str = "identity";
|
||||
pub const PARTICIPANT_IDENTITY: &str = "participantidentity";
|
||||
|
||||
pub fn local_for(
|
||||
conn: &Connection,
|
||||
source_id: i64,
|
||||
type_name: &str,
|
||||
key: &str,
|
||||
) -> Result<Option<i64>, rusqlite::Error> {
|
||||
conn.query_row(
|
||||
"SELECT local_id FROM sync_id_files
|
||||
WHERE source_id = ?1 AND type_name = ?2 AND item_key = ?3",
|
||||
params![source_id, type_name, key],
|
||||
|row| row.get(0),
|
||||
)
|
||||
.optional()
|
||||
}
|
||||
|
||||
/// Records `key` as `local_id`; a key already recorded is pointed at it.
|
||||
pub fn record(
|
||||
conn: &Connection,
|
||||
source_id: i64,
|
||||
type_name: &str,
|
||||
key: &str,
|
||||
local_id: i64,
|
||||
) -> Result<(), rusqlite::Error> {
|
||||
conn.execute(
|
||||
"INSERT INTO sync_id_files (source_id, type_name, item_key, local_id)
|
||||
VALUES (?1, ?2, ?3, ?4)
|
||||
ON CONFLICT (source_id, type_name, item_key) DO UPDATE SET local_id = excluded.local_id",
|
||||
params![source_id, type_name, key, local_id],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn count(conn: &Connection, source_id: i64, type_name: &str) -> Result<u64, rusqlite::Error> {
|
||||
conn.query_row(
|
||||
"SELECT count(DISTINCT local_id) FROM sync_id_files
|
||||
WHERE source_id = ?1 AND type_name = ?2",
|
||||
params![source_id, type_name],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)
|
||||
.map(|n| n.max(0) as u64)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::db::init;
|
||||
|
||||
#[test]
|
||||
fn two_keys_may_lead_to_one_row_and_a_key_moves() {
|
||||
let conn = Connection::open_in_memory().unwrap();
|
||||
init::apply_schema(&conn).unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO sources (id, kind, session_url, account_id, username)
|
||||
VALUES (1, 'files', 'file:///e', '/e', '')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
record(&conn, 1, EMAIL, "a", 7).unwrap();
|
||||
record(&conn, 1, EMAIL, "b", 7).unwrap();
|
||||
assert_eq!(local_for(&conn, 1, EMAIL, "a").unwrap(), Some(7));
|
||||
assert_eq!(count(&conn, 1, EMAIL).unwrap(), 1);
|
||||
record(&conn, 1, EMAIL, "a", 8).unwrap();
|
||||
assert_eq!(local_for(&conn, 1, EMAIL, "a").unwrap(), Some(8));
|
||||
assert_eq!(local_for(&conn, 1, MAILBOX, "a").unwrap(), None);
|
||||
}
|
||||
}
|
||||
+121
-1
@@ -5,7 +5,7 @@
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
use rusqlite::{Connection, OpenFlags};
|
||||
use rusqlite::{Connection, OpenFlags, OptionalExtension};
|
||||
|
||||
pub const SCHEMA_SQL: &str = include_str!("schema.sql");
|
||||
|
||||
@@ -126,6 +126,9 @@ fn schema_tables() -> impl Iterator<Item = &'static str> {
|
||||
}
|
||||
|
||||
pub fn apply_schema(conn: &Connection) -> Result<(), OpenError> {
|
||||
// Before the transaction: it turns foreign keys off, which SQLite only
|
||||
// allows outside one.
|
||||
ensure_sources_accept_files(conn)?;
|
||||
let tx = conn.unchecked_transaction()?;
|
||||
tx.execute_batch(SCHEMA_SQL)?;
|
||||
ensure_calendar_events_data_type(&tx)?;
|
||||
@@ -245,6 +248,64 @@ fn ensure_dav_ids_share_local_rows(conn: &Connection) -> Result<(), OpenError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `import files` records its source as the kind `files`, which the
|
||||
/// `sources` table's CHECK didn't allow before. SQLite can't change a CHECK
|
||||
/// in place, so an older archive's table is rebuilt. Foreign keys are off
|
||||
/// while it is: with them on, dropping the old table would cascade into
|
||||
/// every row that came from a source.
|
||||
fn ensure_sources_accept_files(conn: &Connection) -> Result<(), OpenError> {
|
||||
let sql: Option<String> = conn
|
||||
.query_row(
|
||||
"SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'sources'",
|
||||
[],
|
||||
|row| row.get(0),
|
||||
)
|
||||
.optional()?;
|
||||
// Only the table as earlier releases made it: its kind list ended at
|
||||
// `takeout`.
|
||||
let Some(sql) = sql else { return Ok(()) };
|
||||
if sql.contains("'files'") || !sql.contains("'takeout')") {
|
||||
return Ok(());
|
||||
}
|
||||
let foreign_keys: i64 = conn.query_row("PRAGMA foreign_keys", [], |row| row.get(0))?;
|
||||
conn.pragma_update(None, "foreign_keys", "OFF")?;
|
||||
let rebuilt = rebuild_sources(conn);
|
||||
if foreign_keys != 0 {
|
||||
conn.pragma_update(None, "foreign_keys", "ON")?;
|
||||
}
|
||||
rebuilt
|
||||
}
|
||||
|
||||
fn rebuild_sources(conn: &Connection) -> Result<(), OpenError> {
|
||||
let tx = conn.unchecked_transaction()?;
|
||||
tx.execute_batch(
|
||||
"CREATE TABLE sources_new (
|
||||
id INTEGER PRIMARY KEY,
|
||||
kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout','files')),
|
||||
session_url TEXT NOT NULL,
|
||||
account_id TEXT NOT NULL,
|
||||
account_name TEXT,
|
||||
username TEXT NOT NULL,
|
||||
UNIQUE (kind, session_url, account_id)
|
||||
);
|
||||
INSERT INTO sources_new (id, kind, session_url, account_id, account_name, username)
|
||||
SELECT id, kind, session_url, account_id, account_name, username FROM sources;
|
||||
DROP TABLE sources;
|
||||
ALTER TABLE sources_new RENAME TO sources;",
|
||||
)?;
|
||||
// Every row that names a source still finds it.
|
||||
let dangling = tx.prepare("PRAGMA foreign_key_check")?.exists([])?;
|
||||
if dangling {
|
||||
return Err(OpenError::NotAnArchive(
|
||||
"rebuilding the archive's sources table left rows without a source; \
|
||||
nothing was changed"
|
||||
.to_owned(),
|
||||
));
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// An archive holds a whole mailbox, so it is created readable by its owner
|
||||
/// only. SQLite gives its `-wal` and `-shm` files the database file's mode,
|
||||
/// so they follow. An existing archive others can read is left as it is,
|
||||
@@ -494,6 +555,65 @@ mod permission_tests {
|
||||
mod migration_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn an_older_sources_table_is_rebuilt_to_accept_files_and_keeps_every_row() {
|
||||
let conn = Connection::open_in_memory().unwrap();
|
||||
// The table as releases before `import files` made it.
|
||||
let old = SCHEMA_SQL.replace("'takeout','files')", "'takeout')");
|
||||
assert_ne!(old, SCHEMA_SQL);
|
||||
conn.execute_batch(&old).unwrap();
|
||||
conn.pragma_update(None, "foreign_keys", "ON").unwrap();
|
||||
conn.execute_batch(
|
||||
"INSERT INTO sources (id, kind, session_url, account_id, account_name, username)
|
||||
VALUES (3, 'maildir', 'file:///m', '/m', 'm', '');
|
||||
INSERT INTO sync_id_maildir VALUES (3, 'mailbox', 'INBOX', 'x', 9);",
|
||||
)
|
||||
.unwrap();
|
||||
assert!(
|
||||
conn.execute(
|
||||
"INSERT INTO sources (kind, session_url, account_id, username)
|
||||
VALUES ('files', 'file:///e', '/e', '')",
|
||||
[],
|
||||
)
|
||||
.is_err(),
|
||||
"the old CHECK refuses it"
|
||||
);
|
||||
|
||||
apply_schema(&conn).unwrap();
|
||||
|
||||
conn.execute(
|
||||
"INSERT INTO sources (kind, session_url, account_id, username)
|
||||
VALUES ('files', 'file:///e', '/e', '')",
|
||||
[],
|
||||
)
|
||||
.expect("files is a source kind now");
|
||||
let kept: (String, i64) = conn
|
||||
.query_row(
|
||||
"SELECT s.kind, m.local_id FROM sync_id_maildir m JOIN sources s ON s.id = m.source_id",
|
||||
[],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
kept,
|
||||
("maildir".to_owned(), 9),
|
||||
"rows that came from it stayed"
|
||||
);
|
||||
let on: i64 = conn
|
||||
.query_row("PRAGMA foreign_keys", [], |r| r.get(0))
|
||||
.unwrap();
|
||||
assert_eq!(on, 1, "foreign keys are back on");
|
||||
// Deleting the source still cascades, so the reference survived.
|
||||
conn.execute("DELETE FROM sources WHERE id = 3", [])
|
||||
.unwrap();
|
||||
let left: i64 = conn
|
||||
.query_row("SELECT count(*) FROM sync_id_maildir", [], |r| r.get(0))
|
||||
.unwrap();
|
||||
assert_eq!(left, 0);
|
||||
// A second pass finds nothing to do.
|
||||
apply_schema(&conn).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_older_dav_map_is_rebuilt_to_share_local_rows() {
|
||||
let conn = Connection::open_in_memory().unwrap();
|
||||
|
||||
@@ -10,6 +10,7 @@ pub mod dav_ids;
|
||||
pub mod defaults;
|
||||
pub mod exchange_ews_ids;
|
||||
pub mod exchange_graph_ids;
|
||||
pub mod files_ids;
|
||||
pub mod ids;
|
||||
pub mod imap_ids;
|
||||
pub mod imap_state;
|
||||
|
||||
+12
-1
@@ -1,6 +1,6 @@
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id INTEGER PRIMARY KEY,
|
||||
kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout')),
|
||||
kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout','files')),
|
||||
session_url TEXT NOT NULL,
|
||||
account_id TEXT NOT NULL,
|
||||
account_name TEXT,
|
||||
@@ -268,3 +268,14 @@ CREATE TABLE IF NOT EXISTS sync_id_takeout (
|
||||
|
||||
CREATE INDEX IF NOT EXISTS sync_id_takeout_type_idx
|
||||
ON sync_id_takeout (source_id, type_name);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sync_id_files (
|
||||
source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
|
||||
type_name TEXT NOT NULL,
|
||||
item_key TEXT NOT NULL,
|
||||
local_id INTEGER NOT NULL,
|
||||
PRIMARY KEY (source_id, type_name, item_key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS sync_id_files_local_idx
|
||||
ON sync_id_files (source_id, type_name, local_id);
|
||||
@@ -61,6 +61,10 @@ fn run() -> i32 {
|
||||
let logger = common.logger;
|
||||
(sync::import_takeout::run_reporting(common, config), logger)
|
||||
}
|
||||
Action::ImportFiles(common, config) => {
|
||||
let logger = common.logger;
|
||||
(sync::import_files::run_reporting(common, config), logger)
|
||||
}
|
||||
Action::ImportExchangeEws(common, config) => {
|
||||
let logger = common.logger;
|
||||
(
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! `import files`: an account export (the layout `export-files` writes,
|
||||
//! specified at <https://coffeylabs.org/specs/account-export/>) read back
|
||||
//! into an archive, manifest and all. `import maildir` and `import takeout`
|
||||
//! read the open formats; this reads what only the manifest holds as well:
|
||||
//! real folder names, roles and subscriptions, which address book and
|
||||
//! calendar were the default, their colors and time zones, the active Sieve
|
||||
//! script, media types and links, and the identities with their signatures.
|
||||
//!
|
||||
//! Paths in the manifest are data. Each one is checked to stay inside its
|
||||
//! part of the export, and nothing that is a symbolic link is followed. A
|
||||
//! run that is repeated brings the archive up to date with the export; it
|
||||
//! never deletes.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::UNIX_EPOCH;
|
||||
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
use time::OffsetDateTime;
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
use url::Url;
|
||||
|
||||
use crate::db;
|
||||
use crate::db::sources::SourceKey;
|
||||
use crate::error::Error;
|
||||
use crate::logging::{LEVEL_DEFAULT, LEVEL_PROGRESS, Logger};
|
||||
use crate::sync::import_dav::calcard;
|
||||
use crate::sync::import_takeout::contacts::split_vcards;
|
||||
use crate::sync::{CommonConfig, RunOutcome, Summary, TypeCounts};
|
||||
|
||||
pub mod cards;
|
||||
pub mod mail;
|
||||
pub mod manifest;
|
||||
pub mod other;
|
||||
pub mod paths;
|
||||
|
||||
use manifest::Manifest;
|
||||
|
||||
pub const SOURCE_KIND: &str = "files";
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct FilesImportConfig {
|
||||
pub dir: PathBuf,
|
||||
pub allow_source_change: bool,
|
||||
}
|
||||
|
||||
pub fn run(common: CommonConfig, config: FilesImportConfig) -> Result<Summary, Error> {
|
||||
run_reporting(common, config).into_result()
|
||||
}
|
||||
|
||||
pub fn run_reporting(common: CommonConfig, config: FilesImportConfig) -> RunOutcome {
|
||||
let mut summary = Summary::default();
|
||||
let error = run_into(common, config, &mut summary).err();
|
||||
RunOutcome { summary, error }
|
||||
}
|
||||
|
||||
fn run_into(
|
||||
common: CommonConfig,
|
||||
config: FilesImportConfig,
|
||||
summary: &mut Summary,
|
||||
) -> Result<(), Error> {
|
||||
let logger = common.logger;
|
||||
let root = fs::canonicalize(&config.dir)
|
||||
.map_err(|e| Error::Usage(format!("{}: {e}", config.dir.display())))?;
|
||||
if !root.is_dir() {
|
||||
return Err(Error::Usage(format!("not a directory: {}", root.display())));
|
||||
}
|
||||
let manifest = manifest::read(&root)?;
|
||||
if logger.enabled(LEVEL_PROGRESS) {
|
||||
eprintln!(
|
||||
"account export {} written by {} at {}",
|
||||
root.display(),
|
||||
manifest
|
||||
.created_by
|
||||
.as_deref()
|
||||
.unwrap_or("an unnamed program"),
|
||||
manifest.created.as_deref().unwrap_or("an unrecorded time"),
|
||||
);
|
||||
}
|
||||
|
||||
if common.dry_run {
|
||||
*summary = count_only(&root, &manifest, logger)?;
|
||||
logger.warn("dry run: nothing was written; the counts are what the export holds");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let mut conn = db::init::open(&common.archive)?;
|
||||
let session_url = Url::from_file_path(&root)
|
||||
.map(|u| u.to_string())
|
||||
.map_err(|_| Error::Usage(format!("cannot name {} as a file: URL", root.display())))?;
|
||||
let account_id = root.to_string_lossy().into_owned();
|
||||
if let Some((existing_url, _)) =
|
||||
db::sources::conflicting_source(&conn, SOURCE_KIND, &session_url, &account_id)?
|
||||
&& !config.allow_source_change
|
||||
{
|
||||
return Err(Error::SourceChange(format!(
|
||||
"archive already records the account export at {existing_url}; re-run with \
|
||||
--allow-source-change or use a fresh archive"
|
||||
)));
|
||||
}
|
||||
let original = manifest.sources.first();
|
||||
let account_name = original
|
||||
.and_then(|s| s.account_name.clone().or_else(|| s.username.clone()))
|
||||
.or_else(|| root.file_name().map(|n| n.to_string_lossy().into_owned()));
|
||||
if logger.enabled(LEVEL_PROGRESS)
|
||||
&& let Some(source) = original
|
||||
{
|
||||
eprintln!(
|
||||
"the account came from {} ({})",
|
||||
source.url.as_deref().unwrap_or("an unrecorded address"),
|
||||
source.kind.as_deref().unwrap_or("an unrecorded kind"),
|
||||
);
|
||||
}
|
||||
let source_id = db::sources::upsert_source(
|
||||
&conn,
|
||||
&SourceKey {
|
||||
kind: SOURCE_KIND.to_owned(),
|
||||
session_url,
|
||||
account_id,
|
||||
},
|
||||
account_name.as_deref(),
|
||||
"",
|
||||
)?;
|
||||
|
||||
let result = import_parts(&mut conn, &root, source_id, &manifest, logger, summary);
|
||||
if result.is_ok() && !summary.any_failed() {
|
||||
let tx = conn.unchecked_transaction()?;
|
||||
db::blobs::gc_orphan_blobs(&tx)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
if logger.enabled(LEVEL_DEFAULT) {
|
||||
let line: Vec<String> = summary
|
||||
.per_type
|
||||
.iter()
|
||||
.map(|(name, c)| {
|
||||
format!(
|
||||
"{name} c={} u={} s={} f={}",
|
||||
c.created, c.updated, c.skipped, c.failed
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
eprintln!("files: {}", line.join(" | "));
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
/// Each part the manifest lists, in turn; one that is missing wasn't
|
||||
/// exported and is left as the archive has it.
|
||||
fn import_parts(
|
||||
conn: &mut Connection,
|
||||
root: &Path,
|
||||
source_id: i64,
|
||||
manifest: &Manifest,
|
||||
logger: Logger,
|
||||
summary: &mut Summary,
|
||||
) -> Result<(), Error> {
|
||||
if let Some(mail) = &manifest.mail {
|
||||
let out = mail::import(conn, root, source_id, mail, logger)?;
|
||||
summary.per_type.push(("mailbox", out.mailbox));
|
||||
summary.per_type.push(("email", out.email));
|
||||
}
|
||||
if let Some(books) = &manifest.contacts {
|
||||
let out = cards::import_contacts(conn, root, source_id, books, logger)?;
|
||||
summary.per_type.push(("addressbook", out.collection));
|
||||
summary.per_type.push(("contactcard", out.item));
|
||||
}
|
||||
if let Some(calendars) = &manifest.calendars {
|
||||
let out = cards::import_calendars(conn, root, source_id, calendars, logger)?;
|
||||
summary.per_type.push(("calendar", out.collection));
|
||||
summary.per_type.push(("calendarevent", out.item));
|
||||
}
|
||||
if let Some(scripts) = &manifest.sieve {
|
||||
let counts = other::import_sieve(conn, root, source_id, scripts, logger)?;
|
||||
summary.per_type.push(("sievescript", counts));
|
||||
}
|
||||
if let Some(files) = &manifest.files {
|
||||
let created = manifest
|
||||
.created
|
||||
.clone()
|
||||
.filter(|c| OffsetDateTime::parse(c, &Rfc3339).is_ok())
|
||||
.unwrap_or_else(now_rfc3339);
|
||||
let counts = other::import_files(conn, root, source_id, files, &created, logger)?;
|
||||
summary.per_type.push(("filenode", counts));
|
||||
}
|
||||
if let Some(identities) = &manifest.identities {
|
||||
let counts = other::import_identities(conn, source_id, identities)?;
|
||||
summary.per_type.push(("identity", counts));
|
||||
}
|
||||
if let Some(identities) = &manifest.participant_identities {
|
||||
let counts = other::import_participant_identities(conn, source_id, identities, logger)?;
|
||||
summary.per_type.push(("participantidentity", counts));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// What a run would read, counted from the export without opening the
|
||||
/// archive. A message kept in several folders counts once per folder.
|
||||
fn count_only(root: &Path, manifest: &Manifest, logger: Logger) -> Result<Summary, Error> {
|
||||
let found = |n: usize| TypeCounts {
|
||||
fetched: n as u64,
|
||||
..Default::default()
|
||||
};
|
||||
let mut summary = Summary::default();
|
||||
if let Some(mail) = &manifest.mail {
|
||||
let mut messages = 0;
|
||||
let mut failed = 0;
|
||||
for folder in &mail.folders {
|
||||
match paths::resolve(root, &folder.dir, &mail.path) {
|
||||
Ok(dir) => {
|
||||
for sub in ["cur", "new"] {
|
||||
let sub_dir = dir.join(sub);
|
||||
if paths::is_real_dir(&sub_dir) {
|
||||
messages += fs::read_dir(&sub_dir)?
|
||||
.filter_map(Result::ok)
|
||||
.filter(|e| e.file_type().is_ok_and(|t| t.is_file()))
|
||||
.filter(|e| !e.file_name().to_string_lossy().starts_with('.'))
|
||||
.count();
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(reason) => {
|
||||
logger.warn(&format!("folder {:?}: {reason}", folder.name));
|
||||
failed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
summary
|
||||
.per_type
|
||||
.push(("mailbox", found(mail.folders.len())));
|
||||
let mut email = found(messages);
|
||||
email.failed = failed;
|
||||
summary.per_type.push(("email", email));
|
||||
}
|
||||
for (part, books, collection, item) in [
|
||||
("contacts", &manifest.contacts, "addressbook", "contactcard"),
|
||||
(
|
||||
"calendars",
|
||||
&manifest.calendars,
|
||||
"calendar",
|
||||
"calendarevent",
|
||||
),
|
||||
] {
|
||||
let Some(books) = books else { continue };
|
||||
let mut items = TypeCounts::default();
|
||||
for book in books {
|
||||
match cards::read_part_file(root, &book.file, part, logger) {
|
||||
Some(text) if part == "contacts" => {
|
||||
items.fetched += split_vcards(&text).len() as u64
|
||||
}
|
||||
Some(text) => match calcard::ical_to_jscalendar_entries(&text) {
|
||||
Ok(entries) => items.fetched += entries.len() as u64,
|
||||
Err(calcard::CalcardError::NoEntries) => {}
|
||||
Err(e) => {
|
||||
logger.warn(&format!("{}: {e}", book.file));
|
||||
items.failed += 1;
|
||||
}
|
||||
},
|
||||
None => items.failed += 1,
|
||||
}
|
||||
}
|
||||
summary.per_type.push((collection, found(books.len())));
|
||||
summary.per_type.push((item, items));
|
||||
}
|
||||
if let Some(scripts) = &manifest.sieve {
|
||||
let mut counts = TypeCounts::default();
|
||||
for script in scripts {
|
||||
match paths::resolve(root, &script.file, "sieve") {
|
||||
Ok(path) if paths::is_real_file(&path) => counts.fetched += 1,
|
||||
Ok(_) => counts.failed += 1,
|
||||
Err(reason) => {
|
||||
logger.warn(&reason);
|
||||
counts.failed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
summary.per_type.push(("sievescript", counts));
|
||||
}
|
||||
if let Some(files) = &manifest.files {
|
||||
let base = root.join(&files.path);
|
||||
let nodes = if paths::segments(&files.path).map(|s| s.len()) == Ok(1)
|
||||
&& paths::is_real_dir(&base)
|
||||
{
|
||||
count_tree(&base)?
|
||||
} else {
|
||||
0
|
||||
};
|
||||
summary
|
||||
.per_type
|
||||
.push(("filenode", found(nodes + files.links.len())));
|
||||
}
|
||||
if let Some(identities) = &manifest.identities {
|
||||
summary.per_type.push(("identity", found(identities.len())));
|
||||
}
|
||||
if let Some(identities) = &manifest.participant_identities {
|
||||
summary
|
||||
.per_type
|
||||
.push(("participantidentity", found(identities.len())));
|
||||
}
|
||||
Ok(summary)
|
||||
}
|
||||
|
||||
fn count_tree(dir: &Path) -> Result<usize, Error> {
|
||||
let mut n = 0;
|
||||
for entry in fs::read_dir(dir)? {
|
||||
let entry = entry?;
|
||||
let file_type = entry.file_type()?;
|
||||
if file_type.is_dir() {
|
||||
n += 1 + count_tree(&entry.path())?;
|
||||
} else if file_type.is_file() {
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
/// Rows a run found already in the archive, as they were before it changed
|
||||
/// them, so the end of the run can count which it actually changed.
|
||||
#[derive(Default)]
|
||||
pub(crate) struct Rerun {
|
||||
before: HashMap<i64, Vec<rusqlite::types::Value>>,
|
||||
}
|
||||
|
||||
impl Rerun {
|
||||
pub(crate) fn before(
|
||||
&mut self,
|
||||
conn: &Connection,
|
||||
table: &str,
|
||||
columns: &str,
|
||||
id: i64,
|
||||
) -> Result<(), Error> {
|
||||
if self.before.contains_key(&id) {
|
||||
return Ok(());
|
||||
}
|
||||
if let Some(row) = read_row(conn, table, columns, id)? {
|
||||
self.before.insert(id, row);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn tally(
|
||||
&self,
|
||||
conn: &Connection,
|
||||
table: &str,
|
||||
columns: &str,
|
||||
counts: &mut TypeCounts,
|
||||
) -> Result<(), Error> {
|
||||
for (id, was) in &self.before {
|
||||
match read_row(conn, table, columns, *id)? {
|
||||
Some(now) if now == *was => counts.skipped += 1,
|
||||
_ => counts.updated += 1,
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn read_row(
|
||||
conn: &Connection,
|
||||
table: &str,
|
||||
columns: &str,
|
||||
id: i64,
|
||||
) -> Result<Option<Vec<rusqlite::types::Value>>, Error> {
|
||||
let mut stmt = conn.prepare(&format!("SELECT {columns} FROM {table} WHERE id = ?1"))?;
|
||||
let n = stmt.column_count();
|
||||
Ok(stmt
|
||||
.query_row(params![id], |r| {
|
||||
(0..n)
|
||||
.map(|i| r.get::<_, rusqlite::types::Value>(i))
|
||||
.collect()
|
||||
})
|
||||
.optional()?)
|
||||
}
|
||||
|
||||
/// A file's modification time as RFC 3339, if it has a usable one.
|
||||
pub(crate) fn mtime_rfc3339(meta: &fs::Metadata) -> Option<String> {
|
||||
let secs = meta
|
||||
.modified()
|
||||
.ok()?
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.ok()?
|
||||
.as_secs();
|
||||
if secs == 0 {
|
||||
return None;
|
||||
}
|
||||
OffsetDateTime::from_unix_timestamp(i64::try_from(secs).ok()?)
|
||||
.ok()?
|
||||
.format(&Rfc3339)
|
||||
.ok()
|
||||
}
|
||||
|
||||
pub(crate) fn now_rfc3339() -> String {
|
||||
OffsetDateTime::now_utc()
|
||||
.format(&Rfc3339)
|
||||
.unwrap_or_else(|_| "1970-01-01T00:00:00Z".to_owned())
|
||||
}
|
||||
@@ -0,0 +1,441 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! Address books from `contacts/*.vcf` and calendars from
|
||||
//! `calendars/*.ics`, each named and set up as the manifest says: its real
|
||||
//! name, description, color, time zone and whether it was the default. A
|
||||
//! card or event in two files is one row in both collections.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
use serde_json::Value;
|
||||
|
||||
use crate::db::{cards, defaults, files_ids};
|
||||
use crate::error::Error;
|
||||
use crate::logging::Logger;
|
||||
use crate::sync::TypeCounts;
|
||||
use crate::sync::import_dav::calcard;
|
||||
use crate::sync::import_takeout::contacts::split_vcards;
|
||||
use crate::types::ObjectType;
|
||||
|
||||
use super::mail::row_exists;
|
||||
use super::manifest::Collection;
|
||||
use super::{Rerun, paths};
|
||||
|
||||
pub struct Outcome {
|
||||
pub collection: TypeCounts,
|
||||
pub item: TypeCounts,
|
||||
}
|
||||
|
||||
const BOOK_COLUMNS: &str = "name, description, sort_order, is_default";
|
||||
const CARD_COLUMNS: &str = "address_book_ids, data";
|
||||
const CALENDAR_COLUMNS: &str = "name, description, color, time_zone, sort_order, is_default";
|
||||
const EVENT_COLUMNS: &str = "calendar_ids, is_draft, use_default_alerts, data, data_type";
|
||||
|
||||
pub fn import_contacts(
|
||||
conn: &mut Connection,
|
||||
root: &Path,
|
||||
source_id: i64,
|
||||
books: &[Collection],
|
||||
logger: Logger,
|
||||
) -> Result<Outcome, Error> {
|
||||
let mut out = Outcome {
|
||||
collection: TypeCounts::default(),
|
||||
item: TypeCounts::default(),
|
||||
};
|
||||
let mut collections = Rerun::default();
|
||||
let mut items = Rerun::default();
|
||||
let mut seen: HashMap<String, i64> = HashMap::new();
|
||||
for (order, book) in books.iter().enumerate() {
|
||||
let tx = conn.transaction()?;
|
||||
let book_id = ensure_collection(
|
||||
&tx,
|
||||
source_id,
|
||||
Kind::Contacts,
|
||||
book,
|
||||
order,
|
||||
&mut collections,
|
||||
&mut out.collection,
|
||||
logger,
|
||||
)?;
|
||||
let Some(text) = read_part_file(root, &book.file, "contacts", logger) else {
|
||||
out.item.failed += 1;
|
||||
tx.commit()?;
|
||||
continue;
|
||||
};
|
||||
let chunks = split_vcards(&text);
|
||||
for (ordinal, chunk) in chunks.iter().enumerate() {
|
||||
let card = match calcard::vcard_to_jscontact(chunk, &format!("{}#{ordinal}", book.file))
|
||||
{
|
||||
Ok(card) => card,
|
||||
Err(e) => {
|
||||
logger.warn(&format!("{} card {ordinal}: {e}", book.file));
|
||||
out.item.failed += 1;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let data = card.data.to_string();
|
||||
if let Some(&id) = seen.get(&card.uid) {
|
||||
// The same card in another address book.
|
||||
add_to_collection(&tx, "contact_cards", "address_book_ids", id, book_id)?;
|
||||
continue;
|
||||
}
|
||||
let ours =
|
||||
match files_ids::local_for(&tx, source_id, files_ids::CONTACT_CARD, &card.uid)? {
|
||||
Some(id) if row_exists(&tx, "contact_cards", id)? => Some(id),
|
||||
_ => None,
|
||||
};
|
||||
// The uid is the archive's key for a card, whoever put it there.
|
||||
let theirs = match ours {
|
||||
Some(_) => None,
|
||||
None => tx
|
||||
.query_row(
|
||||
"SELECT id, address_book_ids FROM contact_cards WHERE uid = ?1",
|
||||
params![card.uid],
|
||||
|r| Ok((r.get::<_, i64>(0)?, r.get::<_, String>(1)?)),
|
||||
)
|
||||
.optional()?,
|
||||
};
|
||||
let id = match (ours, theirs) {
|
||||
(Some(id), _) => {
|
||||
// Read on an earlier run: now in the books it is in now.
|
||||
items.before(&tx, "contact_cards", CARD_COLUMNS, id)?;
|
||||
tx.execute(
|
||||
"UPDATE contact_cards SET address_book_ids = ?1, data = ?2 WHERE id = ?3",
|
||||
params![format!("[{book_id}]"), data, id],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
(None, Some((id, books))) => {
|
||||
// Another import's card: it keeps the books it was in.
|
||||
items.before(&tx, "contact_cards", CARD_COLUMNS, id)?;
|
||||
tx.execute(
|
||||
"UPDATE contact_cards SET address_book_ids = ?1, data = ?2 WHERE id = ?3",
|
||||
params![cards::with_book(&books, book_id), data, id],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
(None, None) => {
|
||||
tx.execute(
|
||||
"INSERT INTO contact_cards (uid, address_book_ids, data) VALUES (?1, ?2, ?3)",
|
||||
params![card.uid, format!("[{book_id}]"), data],
|
||||
)?;
|
||||
out.item.created += 1;
|
||||
out.item.fetched += 1;
|
||||
tx.last_insert_rowid()
|
||||
}
|
||||
};
|
||||
files_ids::record(&tx, source_id, files_ids::CONTACT_CARD, &card.uid, id)?;
|
||||
seen.insert(card.uid, id);
|
||||
}
|
||||
tx.commit()?;
|
||||
listed_and_found(logger, &book.file, "cards", book.cards, chunks.len());
|
||||
}
|
||||
let tx = conn.transaction()?;
|
||||
collections.tally(&tx, "address_books", BOOK_COLUMNS, &mut out.collection)?;
|
||||
items.tally(&tx, "contact_cards", CARD_COLUMNS, &mut out.item)?;
|
||||
tx.commit()?;
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
pub fn import_calendars(
|
||||
conn: &mut Connection,
|
||||
root: &Path,
|
||||
source_id: i64,
|
||||
calendars: &[Collection],
|
||||
logger: Logger,
|
||||
) -> Result<Outcome, Error> {
|
||||
let mut out = Outcome {
|
||||
collection: TypeCounts::default(),
|
||||
item: TypeCounts::default(),
|
||||
};
|
||||
let mut collections = Rerun::default();
|
||||
let mut items = Rerun::default();
|
||||
let mut seen: HashMap<String, i64> = HashMap::new();
|
||||
for (order, calendar) in calendars.iter().enumerate() {
|
||||
let tx = conn.transaction()?;
|
||||
let calendar_id = ensure_collection(
|
||||
&tx,
|
||||
source_id,
|
||||
Kind::Calendars,
|
||||
calendar,
|
||||
order,
|
||||
&mut collections,
|
||||
&mut out.collection,
|
||||
logger,
|
||||
)?;
|
||||
let Some(text) = read_part_file(root, &calendar.file, "calendars", logger) else {
|
||||
out.item.failed += 1;
|
||||
tx.commit()?;
|
||||
continue;
|
||||
};
|
||||
let entries = match calcard::ical_to_jscalendar_entries(&text) {
|
||||
Ok(entries) => entries,
|
||||
// A calendar with no events is written as an empty VCALENDAR.
|
||||
Err(calcard::CalcardError::NoEntries) => Vec::new(),
|
||||
Err(e) => {
|
||||
logger.warn(&format!("{}: {e}", calendar.file));
|
||||
out.item.failed += 1;
|
||||
tx.commit()?;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let found = entries.len();
|
||||
for (ordinal, mut entry) in entries.into_iter().enumerate() {
|
||||
let (is_draft, use_default_alerts, uid) =
|
||||
calcard::strip_extracted_fields_from_event(&mut entry.data);
|
||||
let uid = uid
|
||||
.map(|u| u.trim().to_owned())
|
||||
.filter(|u| !u.is_empty())
|
||||
.unwrap_or_else(|| {
|
||||
calcard::synthesise_uid(&format!("{}#{ordinal}", calendar.file))
|
||||
});
|
||||
// An occurrence sent on its own shares its series' uid.
|
||||
let key = match entry.data.get("recurrenceId").and_then(Value::as_str) {
|
||||
Some(recurrence) => serde_json::to_string(&[&uid, recurrence]).unwrap_or_default(),
|
||||
None => uid,
|
||||
};
|
||||
if let Some(&id) = seen.get(&key) {
|
||||
add_to_collection(&tx, "calendar_events", "calendar_ids", id, calendar_id)?;
|
||||
continue;
|
||||
}
|
||||
let ids = format!("[{calendar_id}]");
|
||||
let data = entry.data.to_string();
|
||||
let data_type = entry.data_type.as_column();
|
||||
let id = match files_ids::local_for(&tx, source_id, files_ids::CALENDAR_EVENT, &key)? {
|
||||
Some(id) if row_exists(&tx, "calendar_events", id)? => {
|
||||
items.before(&tx, "calendar_events", EVENT_COLUMNS, id)?;
|
||||
tx.execute(
|
||||
"UPDATE calendar_events SET calendar_ids = ?1, is_draft = ?2,
|
||||
use_default_alerts = ?3, data = ?4, data_type = ?5
|
||||
WHERE id = ?6",
|
||||
params![
|
||||
ids,
|
||||
is_draft as i64,
|
||||
use_default_alerts as i64,
|
||||
data,
|
||||
data_type,
|
||||
id
|
||||
],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
_ => {
|
||||
tx.execute(
|
||||
"INSERT INTO calendar_events
|
||||
(calendar_ids, is_draft, use_default_alerts, data, data_type)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5)",
|
||||
params![
|
||||
ids,
|
||||
is_draft as i64,
|
||||
use_default_alerts as i64,
|
||||
data,
|
||||
data_type
|
||||
],
|
||||
)?;
|
||||
out.item.created += 1;
|
||||
out.item.fetched += 1;
|
||||
tx.last_insert_rowid()
|
||||
}
|
||||
};
|
||||
files_ids::record(&tx, source_id, files_ids::CALENDAR_EVENT, &key, id)?;
|
||||
seen.insert(key, id);
|
||||
}
|
||||
tx.commit()?;
|
||||
listed_and_found(logger, &calendar.file, "events", calendar.events, found);
|
||||
}
|
||||
let tx = conn.transaction()?;
|
||||
collections.tally(&tx, "calendars", CALENDAR_COLUMNS, &mut out.collection)?;
|
||||
items.tally(&tx, "calendar_events", EVENT_COLUMNS, &mut out.item)?;
|
||||
tx.commit()?;
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
enum Kind {
|
||||
Contacts,
|
||||
Calendars,
|
||||
}
|
||||
|
||||
impl Kind {
|
||||
fn table(self) -> &'static str {
|
||||
match self {
|
||||
Kind::Contacts => "address_books",
|
||||
Kind::Calendars => "calendars",
|
||||
}
|
||||
}
|
||||
|
||||
fn type_name(self) -> &'static str {
|
||||
match self {
|
||||
Kind::Contacts => files_ids::ADDRESS_BOOK,
|
||||
Kind::Calendars => files_ids::CALENDAR,
|
||||
}
|
||||
}
|
||||
|
||||
fn object(self) -> ObjectType {
|
||||
match self {
|
||||
Kind::Contacts => ObjectType::AddressBook,
|
||||
Kind::Calendars => ObjectType::Calendar,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The address book or calendar for one manifest entry, keyed by its file.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn ensure_collection(
|
||||
conn: &Connection,
|
||||
source_id: i64,
|
||||
kind: Kind,
|
||||
entry: &Collection,
|
||||
order: usize,
|
||||
rerun: &mut Rerun,
|
||||
counts: &mut TypeCounts,
|
||||
logger: Logger,
|
||||
) -> Result<i64, Error> {
|
||||
let table = kind.table();
|
||||
let existing = match files_ids::local_for(conn, source_id, kind.type_name(), &entry.file)? {
|
||||
Some(id) if row_exists(conn, table, id)? => Some(id),
|
||||
_ => None,
|
||||
};
|
||||
let is_default = defaults::unique_default(conn, kind.object(), entry.default, existing)?;
|
||||
if entry.default && !is_default {
|
||||
logger.warn(&format!(
|
||||
"{}: another {} is already the default, so this one is not",
|
||||
entry.file,
|
||||
match kind {
|
||||
Kind::Contacts => "address book",
|
||||
Kind::Calendars => "calendar",
|
||||
}
|
||||
));
|
||||
}
|
||||
let blank = |v: &Option<String>| v.clone().filter(|s| !s.is_empty());
|
||||
let id = match (kind, existing) {
|
||||
(Kind::Contacts, Some(id)) => {
|
||||
rerun.before(conn, table, BOOK_COLUMNS, id)?;
|
||||
conn.execute(
|
||||
"UPDATE address_books SET name = ?1, description = ?2, sort_order = ?3,
|
||||
is_default = ?4
|
||||
WHERE id = ?5",
|
||||
params![
|
||||
entry.name,
|
||||
blank(&entry.description),
|
||||
order as i64,
|
||||
is_default as i64,
|
||||
id
|
||||
],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
(Kind::Contacts, None) => {
|
||||
conn.execute(
|
||||
"INSERT INTO address_books (name, description, sort_order, is_default, is_subscribed)
|
||||
VALUES (?1, ?2, ?3, ?4, 1)",
|
||||
params![
|
||||
entry.name,
|
||||
blank(&entry.description),
|
||||
order as i64,
|
||||
is_default as i64
|
||||
],
|
||||
)?;
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
conn.last_insert_rowid()
|
||||
}
|
||||
(Kind::Calendars, Some(id)) => {
|
||||
rerun.before(conn, table, CALENDAR_COLUMNS, id)?;
|
||||
conn.execute(
|
||||
"UPDATE calendars SET name = ?1, description = ?2, color = ?3, time_zone = ?4,
|
||||
sort_order = ?5, is_default = ?6
|
||||
WHERE id = ?7",
|
||||
params![
|
||||
entry.name,
|
||||
blank(&entry.description),
|
||||
blank(&entry.color),
|
||||
blank(&entry.time_zone),
|
||||
order as i64,
|
||||
is_default as i64,
|
||||
id
|
||||
],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
(Kind::Calendars, None) => {
|
||||
conn.execute(
|
||||
"INSERT INTO calendars
|
||||
(name, description, color, time_zone, sort_order, is_subscribed,
|
||||
is_visible, is_default)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, 1, 1, ?6)",
|
||||
params![
|
||||
entry.name,
|
||||
blank(&entry.description),
|
||||
blank(&entry.color),
|
||||
blank(&entry.time_zone),
|
||||
order as i64,
|
||||
is_default as i64
|
||||
],
|
||||
)?;
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
conn.last_insert_rowid()
|
||||
}
|
||||
};
|
||||
files_ids::record(conn, source_id, kind.type_name(), &entry.file, id)?;
|
||||
Ok(id)
|
||||
}
|
||||
|
||||
fn add_to_collection(
|
||||
conn: &Connection,
|
||||
table: &str,
|
||||
column: &str,
|
||||
id: i64,
|
||||
collection: i64,
|
||||
) -> Result<(), Error> {
|
||||
let ids: String = conn.query_row(
|
||||
&format!("SELECT {column} FROM {table} WHERE id = ?1"),
|
||||
params![id],
|
||||
|r| r.get(0),
|
||||
)?;
|
||||
conn.execute(
|
||||
&format!("UPDATE {table} SET {column} = ?1 WHERE id = ?2"),
|
||||
params![cards::with_book(&ids, collection), id],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A text file the manifest names, if it is safely inside `part`.
|
||||
pub(super) fn read_part_file(root: &Path, rel: &str, part: &str, logger: Logger) -> Option<String> {
|
||||
let path = match paths::resolve(root, rel, part) {
|
||||
Ok(path) if paths::is_real_file(&path) => path,
|
||||
Ok(_) => {
|
||||
logger.warn(&format!("{rel}: not a file"));
|
||||
return None;
|
||||
}
|
||||
Err(reason) => {
|
||||
logger.warn(&reason);
|
||||
return None;
|
||||
}
|
||||
};
|
||||
match fs::read_to_string(&path) {
|
||||
Ok(text) => Some(text),
|
||||
Err(e) => {
|
||||
logger.warn(&format!("{rel}: {e}"));
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn listed_and_found(logger: Logger, file: &str, what: &str, listed: Option<u64>, found: usize) {
|
||||
if let Some(listed) = listed
|
||||
&& listed != found as u64
|
||||
{
|
||||
logger.warn(&format!(
|
||||
"{file}: the manifest lists {listed} {what} and {found} were found"
|
||||
));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,453 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! The mail part: a Maildir++ tree, read with the manifest's real folder
|
||||
//! names, hierarchy, roles and subscriptions in place of what the directory
|
||||
//! names could hold. A message kept in several folders is written to each;
|
||||
//! copies with the same content become one message in several mailboxes,
|
||||
//! as the archive holds them. File names are only read for their Maildir
|
||||
//! flags: the specification says nothing else in them can be relied on.
|
||||
|
||||
use std::collections::{BTreeSet, HashMap};
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
|
||||
use crate::db::{blobs, cards, files_ids, roles};
|
||||
use crate::error::Error;
|
||||
use crate::logging::{LEVEL_PROGRESS, Logger};
|
||||
use crate::sync::TypeCounts;
|
||||
use crate::sync::emailmeta::email_meta_from_blob;
|
||||
use crate::sync::import_maildir::keywords::{
|
||||
custom_keywords_from_filename, flags_from_filename, read_keyword_names, translate_flags,
|
||||
with_custom,
|
||||
};
|
||||
use crate::sync::keys::index_to_json;
|
||||
|
||||
use super::manifest::{Folder, Mail};
|
||||
use super::{Rerun, mtime_rfc3339, now_rfc3339, paths};
|
||||
|
||||
pub struct Outcome {
|
||||
pub mailbox: TypeCounts,
|
||||
pub email: TypeCounts,
|
||||
}
|
||||
|
||||
pub fn import(
|
||||
conn: &mut Connection,
|
||||
root: &Path,
|
||||
source_id: i64,
|
||||
mail: &Mail,
|
||||
logger: Logger,
|
||||
) -> Result<Outcome, Error> {
|
||||
let mut out = Outcome {
|
||||
mailbox: TypeCounts::default(),
|
||||
email: TypeCounts::default(),
|
||||
};
|
||||
if paths::segments(&mail.path).map(|s| s.len()) != Ok(1) {
|
||||
return Err(Error::Usage(format!(
|
||||
"manifest.json: mail path {:?} is not a directory at the export's top",
|
||||
mail.path
|
||||
)));
|
||||
}
|
||||
|
||||
// Parents before children, so each folder finds its parent made.
|
||||
let mut folders: Vec<&Folder> = mail.folders.iter().collect();
|
||||
folders.sort_by_key(|f| real_path(f).len());
|
||||
let mut planned: Vec<(&Folder, i64)> = Vec::new();
|
||||
{
|
||||
let tx = conn.transaction()?;
|
||||
let mut tree = Tree {
|
||||
conn: &tx,
|
||||
source_id,
|
||||
made: HashMap::new(),
|
||||
rerun: Rerun::default(),
|
||||
logger,
|
||||
};
|
||||
for folder in folders {
|
||||
match tree.ensure(&real_path(folder), Some(folder), &mut out.mailbox) {
|
||||
Ok(id) => planned.push((folder, id)),
|
||||
Err(e) if e.aborts_run() => return Err(e),
|
||||
Err(e) => {
|
||||
logger.warn(&format!("folder {:?}: {e}", folder.name));
|
||||
out.mailbox.failed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
tree.rerun
|
||||
.tally(&tx, "mailboxes", MAILBOX_COLUMNS, &mut out.mailbox)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
|
||||
let mut messages = Messages {
|
||||
source_id,
|
||||
seen: HashMap::new(),
|
||||
rerun: Rerun::default(),
|
||||
copies: 0,
|
||||
logger,
|
||||
};
|
||||
for (folder, mailbox) in &planned {
|
||||
let dir = match paths::resolve(root, &folder.dir, &mail.path) {
|
||||
Ok(dir) if paths::is_real_dir(&dir) => dir,
|
||||
Ok(_) => {
|
||||
logger.warn(&format!(
|
||||
"folder {:?}: {} is not a directory",
|
||||
folder.name, folder.dir
|
||||
));
|
||||
out.email.failed += 1;
|
||||
continue;
|
||||
}
|
||||
Err(reason) => {
|
||||
logger.warn(&format!("folder {:?}: {reason}", folder.name));
|
||||
out.email.failed += 1;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let tx = conn.transaction()?;
|
||||
let found = match messages.folder(&tx, &dir, *mailbox, &folder.name, &mut out.email) {
|
||||
Ok(found) => found,
|
||||
Err(e) if e.aborts_run() => return Err(e),
|
||||
Err(e) => {
|
||||
logger.warn(&format!("folder {:?}: {e}", folder.name));
|
||||
out.email.failed += 1;
|
||||
0
|
||||
}
|
||||
};
|
||||
tx.commit()?;
|
||||
if let Some(listed) = folder.messages
|
||||
&& listed != found
|
||||
{
|
||||
logger.warn(&format!(
|
||||
"folder {:?}: the manifest lists {listed} messages and {found} were found",
|
||||
folder.name
|
||||
));
|
||||
}
|
||||
if let Some(dropped) = folder.keywords_dropped.filter(|n| *n > 0) {
|
||||
logger.warn(&format!(
|
||||
"folder {:?}: the export had no room for {dropped} keywords on its messages \
|
||||
(Maildir holds 26 per folder), so those are not on them",
|
||||
folder.name
|
||||
));
|
||||
}
|
||||
}
|
||||
{
|
||||
let tx = conn.transaction()?;
|
||||
messages
|
||||
.rerun
|
||||
.tally(&tx, "emails", EMAIL_COLUMNS, &mut out.email)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
if messages.copies > 0 && logger.enabled(LEVEL_PROGRESS) {
|
||||
eprintln!(
|
||||
"mail: {} copies of messages kept in more than one folder were read as one \
|
||||
message in each of those folders",
|
||||
messages.copies
|
||||
);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
const MAILBOX_COLUMNS: &str = "name, parent_id, role, is_subscribed";
|
||||
const EMAIL_COLUMNS: &str = "mailbox_ids, keywords";
|
||||
|
||||
/// The folder's real names from the top down; the manifest may leave the
|
||||
/// path out of a top-level folder.
|
||||
fn real_path(folder: &Folder) -> Vec<String> {
|
||||
if folder.path.is_empty() {
|
||||
vec![folder.name.clone()]
|
||||
} else {
|
||||
folder.path.clone()
|
||||
}
|
||||
}
|
||||
|
||||
struct Tree<'a> {
|
||||
conn: &'a Connection,
|
||||
source_id: i64,
|
||||
made: HashMap<Vec<String>, i64>,
|
||||
rerun: Rerun,
|
||||
logger: Logger,
|
||||
}
|
||||
|
||||
impl Tree<'_> {
|
||||
/// The mailbox for `path`, made or brought up to date. A parent the
|
||||
/// manifest doesn't list is made too, so its children have one.
|
||||
fn ensure(
|
||||
&mut self,
|
||||
path: &[String],
|
||||
folder: Option<&Folder>,
|
||||
counts: &mut TypeCounts,
|
||||
) -> Result<i64, Error> {
|
||||
if folder.is_none()
|
||||
&& let Some(id) = self.made.get(path)
|
||||
{
|
||||
return Ok(*id);
|
||||
}
|
||||
let Some((name, parents)) = path.split_last() else {
|
||||
return Err(Error::Partial("a folder with no name".to_owned()));
|
||||
};
|
||||
let parent = if parents.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(self.ensure(parents, None, counts)?)
|
||||
};
|
||||
let key = serde_json::to_string(path).unwrap_or_default();
|
||||
let existing =
|
||||
match files_ids::local_for(self.conn, self.source_id, files_ids::MAILBOX, &key)? {
|
||||
Some(id) if row_exists(self.conn, "mailboxes", id)? => Some(id),
|
||||
_ => self
|
||||
.conn
|
||||
.query_row(
|
||||
"SELECT id FROM mailboxes WHERE parent_id IS ?1 AND name = ?2",
|
||||
params![parent, name],
|
||||
|r| r.get::<_, i64>(0),
|
||||
)
|
||||
.optional()?,
|
||||
};
|
||||
let Some(folder) = folder else {
|
||||
// A parent only: one the manifest doesn't list.
|
||||
if let Some(id) = existing {
|
||||
self.made.insert(path.to_vec(), id);
|
||||
files_ids::record(self.conn, self.source_id, files_ids::MAILBOX, &key, id)?;
|
||||
return Ok(id);
|
||||
}
|
||||
if self.logger.enabled(LEVEL_PROGRESS) {
|
||||
eprintln!(
|
||||
"folder {:?}: not in the manifest; made so its folders have a parent",
|
||||
path.join("/")
|
||||
);
|
||||
}
|
||||
self.conn.execute(
|
||||
"INSERT INTO mailboxes (name, parent_id, role, sort_order, is_subscribed)
|
||||
VALUES (?1, ?2, NULL, 0, 1)",
|
||||
params![name, parent],
|
||||
)?;
|
||||
let id = self.conn.last_insert_rowid();
|
||||
files_ids::record(self.conn, self.source_id, files_ids::MAILBOX, &key, id)?;
|
||||
self.made.insert(path.to_vec(), id);
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
return Ok(id);
|
||||
};
|
||||
|
||||
let wanted_role = folder
|
||||
.role
|
||||
.as_deref()
|
||||
.map(str::trim)
|
||||
.filter(|r| !r.is_empty())
|
||||
.map(str::to_ascii_lowercase);
|
||||
let role = roles::unique_role(self.conn, wanted_role.as_deref(), existing)?;
|
||||
if let (Some(wanted), None) = (&wanted_role, &role) {
|
||||
self.logger.warn(&format!(
|
||||
"folder {:?}: another folder already has the role {wanted:?}, so this one \
|
||||
is left without it",
|
||||
path.join("/")
|
||||
));
|
||||
}
|
||||
let subscribed = folder.subscribed as i64;
|
||||
let id = match existing {
|
||||
Some(id) => {
|
||||
self.rerun
|
||||
.before(self.conn, "mailboxes", MAILBOX_COLUMNS, id)?;
|
||||
self.conn.execute(
|
||||
"UPDATE mailboxes SET name = ?1, parent_id = ?2, role = ?3, is_subscribed = ?4
|
||||
WHERE id = ?5",
|
||||
params![name, parent, role, subscribed, id],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
None => {
|
||||
self.conn.execute(
|
||||
"INSERT INTO mailboxes (name, parent_id, role, sort_order, is_subscribed)
|
||||
VALUES (?1, ?2, ?3, 0, ?4)",
|
||||
params![name, parent, role, subscribed],
|
||||
)?;
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
self.conn.last_insert_rowid()
|
||||
}
|
||||
};
|
||||
files_ids::record(self.conn, self.source_id, files_ids::MAILBOX, &key, id)?;
|
||||
self.made.insert(path.to_vec(), id);
|
||||
Ok(id)
|
||||
}
|
||||
}
|
||||
|
||||
struct Messages {
|
||||
source_id: i64,
|
||||
/// Each message read this run, by content hash.
|
||||
seen: HashMap<String, i64>,
|
||||
rerun: Rerun,
|
||||
copies: u64,
|
||||
logger: Logger,
|
||||
}
|
||||
|
||||
impl Messages {
|
||||
/// Every message in one folder's `cur/` and `new/`; returns how many
|
||||
/// were found.
|
||||
fn folder(
|
||||
&mut self,
|
||||
conn: &Connection,
|
||||
dir: &Path,
|
||||
mailbox: i64,
|
||||
folder_name: &str,
|
||||
counts: &mut TypeCounts,
|
||||
) -> Result<u64, Error> {
|
||||
let keyword_file = dir.join("dovecot-keywords");
|
||||
let names = if paths::is_real_file(&keyword_file) {
|
||||
read_keyword_names(dir)
|
||||
} else {
|
||||
if keyword_file.symlink_metadata().is_ok() {
|
||||
self.logger.warn(&format!(
|
||||
"folder {folder_name:?}: its dovecot-keywords is not a plain file and \
|
||||
is not read"
|
||||
));
|
||||
}
|
||||
vec![None; 26]
|
||||
};
|
||||
let mut found = 0;
|
||||
for sub in ["cur", "new"] {
|
||||
let sub_dir = dir.join(sub);
|
||||
if !paths::is_real_dir(&sub_dir) {
|
||||
if sub_dir.symlink_metadata().is_ok() {
|
||||
self.logger.warn(&format!(
|
||||
"folder {folder_name:?}: {sub}/ is not a directory and is not read"
|
||||
));
|
||||
counts.failed += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let mut entries: Vec<fs::DirEntry> = fs::read_dir(&sub_dir)
|
||||
.and_then(|d| d.collect::<Result<_, _>>())
|
||||
.map_err(|e| Error::Partial(format!("reading {sub}/: {e}")))?;
|
||||
entries.sort_by_key(fs::DirEntry::file_name);
|
||||
for entry in entries {
|
||||
let file_name = entry.file_name().to_string_lossy().into_owned();
|
||||
if file_name.starts_with('.') {
|
||||
continue;
|
||||
}
|
||||
let file_type = entry
|
||||
.file_type()
|
||||
.map_err(|e| Error::Partial(format!("{sub}/{file_name}: {e}")))?;
|
||||
if file_type.is_symlink() {
|
||||
self.logger.warn(&format!(
|
||||
"folder {folder_name:?}: {sub}/{file_name} is a symbolic link and is \
|
||||
not followed"
|
||||
));
|
||||
counts.skipped += 1;
|
||||
continue;
|
||||
}
|
||||
if !file_type.is_file() {
|
||||
continue;
|
||||
}
|
||||
found += 1;
|
||||
if let Err(e) = self.message(conn, &entry, &file_name, &names, mailbox, counts) {
|
||||
if e.aborts_run() {
|
||||
return Err(e);
|
||||
}
|
||||
self.logger
|
||||
.warn(&format!("folder {folder_name:?}: {sub}/{file_name}: {e}"));
|
||||
counts.failed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(found)
|
||||
}
|
||||
|
||||
fn message(
|
||||
&mut self,
|
||||
conn: &Connection,
|
||||
entry: &fs::DirEntry,
|
||||
file_name: &str,
|
||||
keyword_names: &[Option<String>],
|
||||
mailbox: i64,
|
||||
counts: &mut TypeCounts,
|
||||
) -> Result<(), Error> {
|
||||
let bytes = fs::read(entry.path()).map_err(|e| Error::Partial(e.to_string()))?;
|
||||
let hash = blake3::hash(&bytes).to_hex().to_string();
|
||||
let keywords: BTreeSet<String> = with_custom(
|
||||
translate_flags(&flags_from_filename(file_name), true),
|
||||
&custom_keywords_from_filename(file_name, keyword_names),
|
||||
)
|
||||
.keywords
|
||||
.into_iter()
|
||||
.collect();
|
||||
|
||||
if let Some(&id) = self.seen.get(&hash) {
|
||||
// Another copy of a message already read this run.
|
||||
let (mailbox_ids, current): (String, String) = conn.query_row(
|
||||
"SELECT mailbox_ids, keywords FROM emails WHERE id = ?1",
|
||||
params![id],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)?;
|
||||
let mut merged: BTreeSet<String> = serde_json::from_str::<Vec<String>>(¤t)
|
||||
.unwrap_or_default()
|
||||
.into_iter()
|
||||
.collect();
|
||||
merged.extend(keywords);
|
||||
conn.execute(
|
||||
"UPDATE emails SET mailbox_ids = ?1, keywords = ?2 WHERE id = ?3",
|
||||
params![
|
||||
cards::with_book(&mailbox_ids, mailbox),
|
||||
keywords_json(&merged),
|
||||
id
|
||||
],
|
||||
)?;
|
||||
self.copies += 1;
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let mailbox_ids = format!("[{mailbox}]");
|
||||
if let Some(id) = files_ids::local_for(conn, self.source_id, files_ids::EMAIL, &hash)?
|
||||
&& row_exists(conn, "emails", id)?
|
||||
{
|
||||
// Read on an earlier run: now in the folders it is in now.
|
||||
self.rerun.before(conn, "emails", EMAIL_COLUMNS, id)?;
|
||||
conn.execute(
|
||||
"UPDATE emails SET mailbox_ids = ?1, keywords = ?2 WHERE id = ?3",
|
||||
params![mailbox_ids, keywords_json(&keywords), id],
|
||||
)?;
|
||||
self.seen.insert(hash, id);
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let (index, date_header) = email_meta_from_blob(&bytes);
|
||||
// The file's modification time is when the message arrived.
|
||||
let received_at = entry
|
||||
.metadata()
|
||||
.ok()
|
||||
.and_then(|m| mtime_rfc3339(&m))
|
||||
.or(date_header)
|
||||
.unwrap_or_else(now_rfc3339);
|
||||
let blob = blobs::intern_blob(conn, &bytes)?;
|
||||
conn.execute(
|
||||
"INSERT INTO emails (blob_id, received_at, mailbox_ids, keywords, message_match)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5)",
|
||||
params![
|
||||
blob,
|
||||
received_at,
|
||||
mailbox_ids,
|
||||
keywords_json(&keywords),
|
||||
index_to_json(&index)
|
||||
],
|
||||
)?;
|
||||
let id = conn.last_insert_rowid();
|
||||
files_ids::record(conn, self.source_id, files_ids::EMAIL, &hash, id)?;
|
||||
self.seen.insert(hash, id);
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn keywords_json(keywords: &BTreeSet<String>) -> String {
|
||||
serde_json::to_string(keywords).unwrap_or_else(|_| "[]".to_owned())
|
||||
}
|
||||
|
||||
pub(super) fn row_exists(conn: &Connection, table: &str, id: i64) -> Result<bool, Error> {
|
||||
Ok(conn
|
||||
.prepare(&format!("SELECT 1 FROM {table} WHERE id = ?1"))?
|
||||
.exists(params![id])?)
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! `manifest.json`, as <https://coffeylabs.org/specs/account-export/>
|
||||
//! defines it. A part that is missing wasn't exported; a part that is there
|
||||
//! and empty says the account had none. Members this release doesn't know
|
||||
//! are ignored, so a later writer can add them.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
use serde::Deserialize;
|
||||
use serde_json::Value;
|
||||
|
||||
use crate::error::Error;
|
||||
use crate::export_files::{FORMAT, FORMAT_VERSION};
|
||||
|
||||
/// What inbuxa-migrate 27.1.0-rc.2 wrote as the format before it was named
|
||||
/// by the specification's address. The specification lets readers accept it.
|
||||
pub const LEGACY_FORMAT: &str = "inbuxa-migrate-files";
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Manifest {
|
||||
#[serde(default)]
|
||||
pub created_by: Option<String>,
|
||||
#[serde(default)]
|
||||
pub created: Option<String>,
|
||||
#[serde(default)]
|
||||
pub sources: Vec<Source>,
|
||||
#[serde(default)]
|
||||
pub mail: Option<Mail>,
|
||||
#[serde(default)]
|
||||
pub contacts: Option<Vec<Collection>>,
|
||||
#[serde(default)]
|
||||
pub calendars: Option<Vec<Collection>>,
|
||||
#[serde(default)]
|
||||
pub sieve: Option<Vec<Script>>,
|
||||
#[serde(default)]
|
||||
pub files: Option<Files>,
|
||||
#[serde(default)]
|
||||
pub identities: Option<Vec<Identity>>,
|
||||
#[serde(default)]
|
||||
pub participant_identities: Option<Vec<ParticipantIdentity>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Source {
|
||||
#[serde(default)]
|
||||
pub kind: Option<String>,
|
||||
#[serde(default)]
|
||||
pub url: Option<String>,
|
||||
#[serde(default)]
|
||||
pub account_name: Option<String>,
|
||||
#[serde(default)]
|
||||
pub username: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Mail {
|
||||
pub layout: String,
|
||||
pub path: String,
|
||||
#[serde(default)]
|
||||
pub folders: Vec<Folder>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Folder {
|
||||
pub name: String,
|
||||
#[serde(default)]
|
||||
pub path: Vec<String>,
|
||||
pub dir: String,
|
||||
#[serde(default)]
|
||||
pub role: Option<String>,
|
||||
#[serde(default = "yes")]
|
||||
pub subscribed: bool,
|
||||
#[serde(default)]
|
||||
pub messages: Option<u64>,
|
||||
#[serde(default)]
|
||||
pub keywords_dropped: Option<u64>,
|
||||
}
|
||||
|
||||
/// An address book or a calendar: the members both have, and those only a
|
||||
/// calendar has.
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Collection {
|
||||
pub name: String,
|
||||
#[serde(default)]
|
||||
pub description: Option<String>,
|
||||
pub file: String,
|
||||
#[serde(default)]
|
||||
pub default: bool,
|
||||
#[serde(default)]
|
||||
pub color: Option<String>,
|
||||
#[serde(default)]
|
||||
pub time_zone: Option<String>,
|
||||
#[serde(default)]
|
||||
pub cards: Option<u64>,
|
||||
#[serde(default)]
|
||||
pub events: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct Script {
|
||||
pub name: String,
|
||||
pub file: String,
|
||||
#[serde(default)]
|
||||
pub active: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Files {
|
||||
pub path: String,
|
||||
#[serde(default)]
|
||||
pub media_types: BTreeMap<String, String>,
|
||||
#[serde(default)]
|
||||
pub links: Vec<Link>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct Link {
|
||||
pub path: String,
|
||||
#[serde(default)]
|
||||
pub target: Value,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Identity {
|
||||
#[serde(default)]
|
||||
pub name: String,
|
||||
pub email: String,
|
||||
#[serde(default)]
|
||||
pub reply_to: Value,
|
||||
#[serde(default)]
|
||||
pub bcc: Value,
|
||||
#[serde(default)]
|
||||
pub text_signature: String,
|
||||
#[serde(default)]
|
||||
pub html_signature: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ParticipantIdentity {
|
||||
#[serde(default)]
|
||||
pub name: String,
|
||||
pub calendar_address: String,
|
||||
#[serde(default)]
|
||||
pub default: bool,
|
||||
}
|
||||
|
||||
fn yes() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
/// The manifest of the export at `root`. A tree without one is unfinished
|
||||
/// and is refused, and so is a format or version this release doesn't read.
|
||||
pub fn read(root: &Path) -> Result<Manifest, Error> {
|
||||
let path = root.join("manifest.json");
|
||||
match fs::symlink_metadata(&path) {
|
||||
Ok(meta) if meta.file_type().is_symlink() => {
|
||||
return Err(Error::Usage(format!(
|
||||
"{}: manifest.json is a symbolic link; an export's files are never links",
|
||||
root.display()
|
||||
)));
|
||||
}
|
||||
Ok(meta) if !meta.is_file() => {
|
||||
return Err(Error::Usage(format!(
|
||||
"{}: manifest.json is not a file",
|
||||
root.display()
|
||||
)));
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
|
||||
return Err(Error::Usage(format!(
|
||||
"{} has no manifest.json: either the export was interrupted (a writer \
|
||||
adds the manifest last) or this is not an account export",
|
||||
root.display()
|
||||
)));
|
||||
}
|
||||
Err(e) => return Err(Error::Io(e)),
|
||||
}
|
||||
let text = fs::read_to_string(&path)?;
|
||||
parse(&text)
|
||||
}
|
||||
|
||||
pub fn parse(text: &str) -> Result<Manifest, Error> {
|
||||
let value: Value = serde_json::from_str(text)
|
||||
.map_err(|e| Error::Usage(format!("manifest.json is not valid JSON: {e}")))?;
|
||||
let Value::Object(map) = &value else {
|
||||
return Err(Error::Usage(
|
||||
"manifest.json is not a JSON object".to_owned(),
|
||||
));
|
||||
};
|
||||
match map.get("format").and_then(Value::as_str) {
|
||||
Some(f) if f == FORMAT || f == LEGACY_FORMAT => {}
|
||||
Some(other) => {
|
||||
return Err(Error::Usage(format!(
|
||||
"manifest.json names the format {other:?}, not the account export layout \
|
||||
({FORMAT})"
|
||||
)));
|
||||
}
|
||||
None => {
|
||||
return Err(Error::Usage(
|
||||
"manifest.json has no format; this is not an account export".to_owned(),
|
||||
));
|
||||
}
|
||||
}
|
||||
match map.get("formatVersion").and_then(Value::as_u64) {
|
||||
Some(v) if v == u64::from(FORMAT_VERSION) => {}
|
||||
Some(v) if v > u64::from(FORMAT_VERSION) => {
|
||||
return Err(Error::Usage(format!(
|
||||
"this export follows version {v} of the layout, and this release reads \
|
||||
version {FORMAT_VERSION}; use a newer inbuxa-migrate"
|
||||
)));
|
||||
}
|
||||
_ => {
|
||||
return Err(Error::Usage(
|
||||
"manifest.json has no usable formatVersion".to_owned(),
|
||||
));
|
||||
}
|
||||
}
|
||||
let manifest: Manifest =
|
||||
serde_json::from_value(value).map_err(|e| Error::Usage(format!("manifest.json: {e}")))?;
|
||||
if let Some(mail) = &manifest.mail
|
||||
&& !mail.layout.eq_ignore_ascii_case("maildir++")
|
||||
{
|
||||
return Err(Error::Usage(format!(
|
||||
"manifest.json: mail layout {:?} is not one this release reads (maildir++)",
|
||||
mail.layout
|
||||
)));
|
||||
}
|
||||
Ok(manifest)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn err(text: &str) -> String {
|
||||
parse(text).unwrap_err().to_string()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_format_is_the_spec_address_or_the_old_name() {
|
||||
let m = parse(&format!(r#"{{"format":"{FORMAT}","formatVersion":1}}"#)).unwrap();
|
||||
assert!(m.mail.is_none() && m.contacts.is_none(), "nothing exported");
|
||||
parse(r#"{"format":"inbuxa-migrate-files","formatVersion":1}"#).unwrap();
|
||||
assert!(
|
||||
err(r#"{"format":"something-else","formatVersion":1}"#).contains("not the account")
|
||||
);
|
||||
assert!(err(r#"{"formatVersion":1}"#).contains("no format"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_later_version_is_refused_rather_than_guessed_at() {
|
||||
let e = err(&format!(r#"{{"format":"{FORMAT}","formatVersion":2}}"#));
|
||||
assert!(e.contains("version 2") && e.contains("newer"), "{e}");
|
||||
assert!(
|
||||
err(&format!(r#"{{"format":"{FORMAT}","formatVersion":"1"}}"#))
|
||||
.contains("formatVersion")
|
||||
);
|
||||
assert!(err(&format!(r#"{{"format":"{FORMAT}"}}"#)).contains("formatVersion"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_members_are_ignored_and_empty_parts_kept_apart_from_missing_ones() {
|
||||
let m = parse(&format!(
|
||||
r#"{{"format":"{FORMAT}","formatVersion":1,"futureThing":{{"x":1}},
|
||||
"contacts":[],"mail":{{"layout":"maildir++","path":"mail","folders":[
|
||||
{{"name":"Inbox","path":["Inbox"],"dir":"mail","role":"inbox",
|
||||
"subscribed":true,"messages":3,"newMember":true}}]}}}}"#
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(m.contacts.as_deref().map(<[Collection]>::len), Some(0));
|
||||
assert!(m.calendars.is_none());
|
||||
let mail = m.mail.unwrap();
|
||||
assert_eq!(mail.folders[0].role.as_deref(), Some("inbox"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn another_mail_layout_is_refused() {
|
||||
let e = err(&format!(
|
||||
r#"{{"format":"{FORMAT}","formatVersion":1,"mail":{{"layout":"mbox","path":"mail"}}}}"#
|
||||
));
|
||||
assert!(e.contains("mbox"), "{e}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_or_linked_manifest_is_refused() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let e = read(td.path()).unwrap_err().to_string();
|
||||
assert!(e.contains("interrupted"), "{e}");
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let real = td.path().join("real.json");
|
||||
fs::write(
|
||||
&real,
|
||||
format!(r#"{{"format":"{FORMAT}","formatVersion":1}}"#),
|
||||
)
|
||||
.unwrap();
|
||||
std::os::unix::fs::symlink(&real, td.path().join("manifest.json")).unwrap();
|
||||
let e = read(td.path()).unwrap_err().to_string();
|
||||
assert!(e.contains("symbolic link"), "{e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,532 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! Sieve scripts with the one that was active, the file tree with its
|
||||
//! media types and links, and the identities, which only the manifest
|
||||
//! holds.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
use serde_json::Value;
|
||||
|
||||
use crate::db::{blobs, defaults, files_ids};
|
||||
use crate::error::Error;
|
||||
use crate::logging::Logger;
|
||||
use crate::sync::TypeCounts;
|
||||
use crate::sync::keys::{identity_key, participant_identity_key};
|
||||
use crate::types::ObjectType;
|
||||
|
||||
use super::mail::row_exists;
|
||||
use super::manifest::{Files, Identity, ParticipantIdentity, Script};
|
||||
use super::{Rerun, mtime_rfc3339, paths};
|
||||
|
||||
const SCRIPT_COLUMNS: &str = "blob_id, is_active";
|
||||
const NODE_COLUMNS: &str = "parent_id, node_type, blob_id, target, name, media_type, modified";
|
||||
const IDENTITY_COLUMNS: &str = "name, email, reply_to, bcc, text_signature, html_signature";
|
||||
const PARTICIPANT_COLUMNS: &str = "name, calendar_address, is_default";
|
||||
|
||||
pub fn import_sieve(
|
||||
conn: &mut Connection,
|
||||
root: &Path,
|
||||
source_id: i64,
|
||||
scripts: &[Script],
|
||||
logger: Logger,
|
||||
) -> Result<TypeCounts, Error> {
|
||||
let mut counts = TypeCounts::default();
|
||||
let mut rerun = Rerun::default();
|
||||
let tx = conn.transaction()?;
|
||||
let mut active: Option<&str> = None;
|
||||
for script in scripts {
|
||||
let path = match paths::resolve(root, &script.file, "sieve") {
|
||||
Ok(path) if paths::is_real_file(&path) => path,
|
||||
Ok(_) => {
|
||||
logger.warn(&format!("{}: not a file", script.file));
|
||||
counts.failed += 1;
|
||||
continue;
|
||||
}
|
||||
Err(reason) => {
|
||||
logger.warn(&format!("Sieve script {:?}: {reason}", script.name));
|
||||
counts.failed += 1;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let bytes = match fs::read(&path) {
|
||||
Ok(bytes) => bytes,
|
||||
Err(e) => {
|
||||
logger.warn(&format!("{}: {e}", script.file));
|
||||
counts.failed += 1;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let is_active = script.active && active.is_none();
|
||||
if script.active && !is_active {
|
||||
logger.warn(&format!(
|
||||
"Sieve script {:?}: the manifest marks more than one script active; only \
|
||||
{:?} is",
|
||||
script.name,
|
||||
active.unwrap_or_default()
|
||||
));
|
||||
}
|
||||
if is_active {
|
||||
active = Some(&script.name);
|
||||
}
|
||||
let blob = blobs::intern_blob(&tx, &bytes)?;
|
||||
let existing =
|
||||
match files_ids::local_for(&tx, source_id, files_ids::SIEVE_SCRIPT, &script.name)? {
|
||||
Some(id) if row_exists(&tx, "sieve_scripts", id)? => Some(id),
|
||||
// Names are unique in the archive.
|
||||
_ => tx
|
||||
.query_row(
|
||||
"SELECT id FROM sieve_scripts WHERE name = ?1",
|
||||
params![script.name],
|
||||
|r| r.get::<_, i64>(0),
|
||||
)
|
||||
.optional()?,
|
||||
};
|
||||
if let Some(id) = existing {
|
||||
rerun.before(&tx, "sieve_scripts", SCRIPT_COLUMNS, id)?;
|
||||
}
|
||||
if is_active {
|
||||
// One script runs at a time.
|
||||
tx.execute(
|
||||
"UPDATE sieve_scripts SET is_active = 0 WHERE is_active = 1 AND id IS NOT ?1",
|
||||
params![existing],
|
||||
)?;
|
||||
}
|
||||
let id = match existing {
|
||||
Some(id) => {
|
||||
tx.execute(
|
||||
"UPDATE sieve_scripts SET blob_id = ?1, is_active = ?2 WHERE id = ?3",
|
||||
params![blob, is_active as i64, id],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
None => {
|
||||
tx.execute(
|
||||
"INSERT INTO sieve_scripts (name, is_active, blob_id) VALUES (?1, ?2, ?3)",
|
||||
params![script.name, is_active as i64, blob],
|
||||
)?;
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
tx.last_insert_rowid()
|
||||
}
|
||||
};
|
||||
files_ids::record(&tx, source_id, files_ids::SIEVE_SCRIPT, &script.name, id)?;
|
||||
}
|
||||
rerun.tally(&tx, "sieve_scripts", SCRIPT_COLUMNS, &mut counts)?;
|
||||
tx.commit()?;
|
||||
Ok(counts)
|
||||
}
|
||||
|
||||
/// The file tree, walked without following links, then the links the
|
||||
/// manifest lists, made as archive nodes and never on disk.
|
||||
pub fn import_files(
|
||||
conn: &mut Connection,
|
||||
root: &Path,
|
||||
source_id: i64,
|
||||
files: &Files,
|
||||
created: &str,
|
||||
logger: Logger,
|
||||
) -> Result<TypeCounts, Error> {
|
||||
let mut counts = TypeCounts::default();
|
||||
let part = match paths::segments(&files.path).as_deref() {
|
||||
Ok([part]) => (*part).to_owned(),
|
||||
_ => {
|
||||
return Err(Error::Usage(format!(
|
||||
"manifest.json: files path {:?} is not a directory at the export's top",
|
||||
files.path
|
||||
)));
|
||||
}
|
||||
};
|
||||
let tx = conn.transaction()?;
|
||||
let mut tree = FileTree {
|
||||
conn: &tx,
|
||||
source_id,
|
||||
files,
|
||||
dirs: HashMap::new(),
|
||||
rerun: Rerun::default(),
|
||||
created,
|
||||
logger,
|
||||
};
|
||||
// A writer with no files to write may not make the directory at all.
|
||||
let base = root.join(&part);
|
||||
match fs::symlink_metadata(&base) {
|
||||
Ok(meta) if meta.is_dir() => tree.walk(&base, &part, None, &mut counts)?,
|
||||
Ok(_) => {
|
||||
logger.warn(&format!("{part}/ is not a directory and is not read"));
|
||||
counts.failed += 1;
|
||||
}
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
|
||||
Err(e) => return Err(Error::Io(e)),
|
||||
}
|
||||
for link in &files.links {
|
||||
if let Err(reason) = tree.link(&part, &link.path, &link.target, &mut counts) {
|
||||
logger.warn(&format!("link {:?}: {reason}", link.path));
|
||||
counts.failed += 1;
|
||||
}
|
||||
}
|
||||
let rerun = std::mem::take(&mut tree.rerun);
|
||||
rerun.tally(&tx, "file_nodes", NODE_COLUMNS, &mut counts)?;
|
||||
tx.commit()?;
|
||||
Ok(counts)
|
||||
}
|
||||
|
||||
struct FileTree<'a> {
|
||||
conn: &'a Connection,
|
||||
source_id: i64,
|
||||
files: &'a Files,
|
||||
/// Each directory made, by its path in the export.
|
||||
dirs: HashMap<String, i64>,
|
||||
rerun: Rerun,
|
||||
/// When the export was written: the time for what has none of its own.
|
||||
created: &'a str,
|
||||
logger: Logger,
|
||||
}
|
||||
|
||||
struct Node<'a> {
|
||||
parent: Option<i64>,
|
||||
node_type: &'a str,
|
||||
blob: Option<i64>,
|
||||
target: Option<String>,
|
||||
name: &'a str,
|
||||
media_type: Option<&'a str>,
|
||||
created: String,
|
||||
modified: Option<String>,
|
||||
}
|
||||
|
||||
impl FileTree<'_> {
|
||||
fn walk(
|
||||
&mut self,
|
||||
dir: &Path,
|
||||
rel: &str,
|
||||
parent: Option<i64>,
|
||||
counts: &mut TypeCounts,
|
||||
) -> Result<(), Error> {
|
||||
let mut entries: Vec<fs::DirEntry> =
|
||||
match fs::read_dir(dir).and_then(|d| d.collect::<Result<_, _>>()) {
|
||||
Ok(entries) => entries,
|
||||
Err(e) => {
|
||||
self.logger.warn(&format!("{rel}/: {e}"));
|
||||
counts.failed += 1;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
entries.sort_by_key(fs::DirEntry::file_name);
|
||||
for entry in entries {
|
||||
let Some(name) = entry.file_name().to_str().map(str::to_owned) else {
|
||||
self.logger.warn(&format!(
|
||||
"{rel}/{}: the name is not UTF-8 and is skipped",
|
||||
entry.file_name().to_string_lossy()
|
||||
));
|
||||
counts.failed += 1;
|
||||
continue;
|
||||
};
|
||||
let rel_path = format!("{rel}/{name}");
|
||||
let Ok(file_type) = entry.file_type() else {
|
||||
self.logger
|
||||
.warn(&format!("{rel_path}: its type can't be read"));
|
||||
counts.failed += 1;
|
||||
continue;
|
||||
};
|
||||
if file_type.is_symlink() {
|
||||
self.logger.warn(&format!(
|
||||
"{rel_path} is a symbolic link and is not followed; an export's links \
|
||||
are listed in its manifest"
|
||||
));
|
||||
counts.skipped += 1;
|
||||
continue;
|
||||
}
|
||||
if file_type.is_dir() {
|
||||
// A writer sets the times of files, not of directories, so a
|
||||
// directory's time is only when the export was written.
|
||||
let id = self.node(
|
||||
&rel_path,
|
||||
Node {
|
||||
parent,
|
||||
node_type: "directory",
|
||||
blob: None,
|
||||
target: None,
|
||||
name: &name,
|
||||
media_type: None,
|
||||
created: self.created.to_owned(),
|
||||
modified: None,
|
||||
},
|
||||
counts,
|
||||
)?;
|
||||
self.dirs.insert(rel_path.clone(), id);
|
||||
self.walk(&entry.path(), &rel_path, Some(id), counts)?;
|
||||
} else if file_type.is_file() {
|
||||
let modified = entry.metadata().ok().and_then(|m| mtime_rfc3339(&m));
|
||||
let created = modified.clone().unwrap_or_else(|| self.created.to_owned());
|
||||
let bytes = match fs::read(entry.path()) {
|
||||
Ok(bytes) => bytes,
|
||||
Err(e) => {
|
||||
self.logger.warn(&format!("{rel_path}: {e}"));
|
||||
counts.failed += 1;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let blob = blobs::intern_blob(self.conn, &bytes)?;
|
||||
let media_type = self.files.media_types.get(&rel_path).map(String::as_str);
|
||||
self.node(
|
||||
&rel_path,
|
||||
Node {
|
||||
parent,
|
||||
node_type: "file",
|
||||
blob: Some(blob),
|
||||
target: None,
|
||||
name: &name,
|
||||
media_type,
|
||||
created,
|
||||
modified,
|
||||
},
|
||||
counts,
|
||||
)?;
|
||||
} else {
|
||||
self.logger
|
||||
.warn(&format!("{rel_path} is not a file or directory; skipped"));
|
||||
counts.skipped += 1;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn link(
|
||||
&mut self,
|
||||
part: &str,
|
||||
rel: &str,
|
||||
target: &Value,
|
||||
counts: &mut TypeCounts,
|
||||
) -> Result<(), String> {
|
||||
let segments = paths::segments(rel)?;
|
||||
let [first, parents @ .., name] = segments.as_slice() else {
|
||||
return Err("names no file".to_owned());
|
||||
};
|
||||
if *first != part {
|
||||
return Err(format!("is outside {part}/"));
|
||||
}
|
||||
let parent = if parents.is_empty() {
|
||||
None
|
||||
} else {
|
||||
let parent_rel = format!("{part}/{}", parents.join("/"));
|
||||
match self.dirs.get(&parent_rel) {
|
||||
Some(id) => Some(*id),
|
||||
None => return Err(format!("its directory {parent_rel} is not in the export")),
|
||||
}
|
||||
};
|
||||
let target = (!target.is_null()).then(|| target.to_string());
|
||||
self.node(
|
||||
rel,
|
||||
Node {
|
||||
parent,
|
||||
node_type: "symlink",
|
||||
blob: None,
|
||||
target,
|
||||
name,
|
||||
media_type: None,
|
||||
created: self.created.to_owned(),
|
||||
modified: None,
|
||||
},
|
||||
counts,
|
||||
)
|
||||
.map(|_| ())
|
||||
.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
fn node(&mut self, key: &str, node: Node<'_>, counts: &mut TypeCounts) -> Result<i64, Error> {
|
||||
let existing =
|
||||
match files_ids::local_for(self.conn, self.source_id, files_ids::FILE_NODE, key)? {
|
||||
Some(id) if row_exists(self.conn, "file_nodes", id)? => Some(id),
|
||||
// Names are unique within a directory.
|
||||
_ => self
|
||||
.conn
|
||||
.query_row(
|
||||
"SELECT id FROM file_nodes WHERE parent_id IS ?1 AND name = ?2",
|
||||
params![node.parent, node.name],
|
||||
|r| r.get::<_, i64>(0),
|
||||
)
|
||||
.optional()?,
|
||||
};
|
||||
let id = match existing {
|
||||
Some(id) => {
|
||||
self.rerun
|
||||
.before(self.conn, "file_nodes", NODE_COLUMNS, id)?;
|
||||
self.conn.execute(
|
||||
"UPDATE file_nodes SET parent_id = ?1, node_type = ?2, blob_id = ?3,
|
||||
target = ?4, name = ?5, media_type = ?6, modified = ?7
|
||||
WHERE id = ?8",
|
||||
params![
|
||||
node.parent,
|
||||
node.node_type,
|
||||
node.blob,
|
||||
node.target,
|
||||
node.name,
|
||||
node.media_type,
|
||||
node.modified,
|
||||
id
|
||||
],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
None => {
|
||||
self.conn.execute(
|
||||
"INSERT INTO file_nodes
|
||||
(parent_id, node_type, blob_id, target, name, media_type, created, modified)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
|
||||
params![
|
||||
node.parent,
|
||||
node.node_type,
|
||||
node.blob,
|
||||
node.target,
|
||||
node.name,
|
||||
node.media_type,
|
||||
node.created,
|
||||
node.modified
|
||||
],
|
||||
)?;
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
self.conn.last_insert_rowid()
|
||||
}
|
||||
};
|
||||
files_ids::record(self.conn, self.source_id, files_ids::FILE_NODE, key, id)?;
|
||||
Ok(id)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn import_identities(
|
||||
conn: &mut Connection,
|
||||
source_id: i64,
|
||||
identities: &[Identity],
|
||||
) -> Result<TypeCounts, Error> {
|
||||
let mut counts = TypeCounts::default();
|
||||
let mut rerun = Rerun::default();
|
||||
let tx = conn.transaction()?;
|
||||
for identity in identities {
|
||||
let key = hex(&identity_key(&identity.name, &identity.email));
|
||||
let addresses = |v: &Value| (!v.is_null()).then(|| v.to_string());
|
||||
let (reply_to, bcc) = (addresses(&identity.reply_to), addresses(&identity.bcc));
|
||||
let id = match files_ids::local_for(&tx, source_id, files_ids::IDENTITY, &key)? {
|
||||
Some(id) if row_exists(&tx, "identities", id)? => {
|
||||
rerun.before(&tx, "identities", IDENTITY_COLUMNS, id)?;
|
||||
tx.execute(
|
||||
"UPDATE identities SET name = ?1, email = ?2, reply_to = ?3, bcc = ?4,
|
||||
text_signature = ?5, html_signature = ?6
|
||||
WHERE id = ?7",
|
||||
params![
|
||||
identity.name,
|
||||
identity.email,
|
||||
reply_to,
|
||||
bcc,
|
||||
identity.text_signature,
|
||||
identity.html_signature,
|
||||
id
|
||||
],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
_ => {
|
||||
tx.execute(
|
||||
"INSERT INTO identities
|
||||
(name, email, reply_to, bcc, text_signature, html_signature)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
|
||||
params![
|
||||
identity.name,
|
||||
identity.email,
|
||||
reply_to,
|
||||
bcc,
|
||||
identity.text_signature,
|
||||
identity.html_signature
|
||||
],
|
||||
)?;
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
tx.last_insert_rowid()
|
||||
}
|
||||
};
|
||||
files_ids::record(&tx, source_id, files_ids::IDENTITY, &key, id)?;
|
||||
}
|
||||
rerun.tally(&tx, "identities", IDENTITY_COLUMNS, &mut counts)?;
|
||||
tx.commit()?;
|
||||
Ok(counts)
|
||||
}
|
||||
|
||||
pub fn import_participant_identities(
|
||||
conn: &mut Connection,
|
||||
source_id: i64,
|
||||
identities: &[ParticipantIdentity],
|
||||
logger: Logger,
|
||||
) -> Result<TypeCounts, Error> {
|
||||
let mut counts = TypeCounts::default();
|
||||
let mut rerun = Rerun::default();
|
||||
let tx = conn.transaction()?;
|
||||
for identity in identities {
|
||||
let key = hex(&participant_identity_key(
|
||||
&identity.calendar_address,
|
||||
&identity.name,
|
||||
));
|
||||
let existing =
|
||||
match files_ids::local_for(&tx, source_id, files_ids::PARTICIPANT_IDENTITY, &key)? {
|
||||
Some(id) if row_exists(&tx, "participant_identities", id)? => Some(id),
|
||||
_ => None,
|
||||
};
|
||||
let is_default = defaults::unique_default(
|
||||
&tx,
|
||||
ObjectType::ParticipantIdentity,
|
||||
identity.default,
|
||||
existing,
|
||||
)?;
|
||||
if identity.default && !is_default {
|
||||
logger.warn(&format!(
|
||||
"calendar identity {:?}: another is already the default, so this one is not",
|
||||
identity.calendar_address
|
||||
));
|
||||
}
|
||||
let id = match existing {
|
||||
Some(id) => {
|
||||
rerun.before(&tx, "participant_identities", PARTICIPANT_COLUMNS, id)?;
|
||||
tx.execute(
|
||||
"UPDATE participant_identities SET name = ?1, calendar_address = ?2,
|
||||
is_default = ?3
|
||||
WHERE id = ?4",
|
||||
params![
|
||||
identity.name,
|
||||
identity.calendar_address,
|
||||
is_default as i64,
|
||||
id
|
||||
],
|
||||
)?;
|
||||
id
|
||||
}
|
||||
None => {
|
||||
tx.execute(
|
||||
"INSERT INTO participant_identities (name, calendar_address, is_default)
|
||||
VALUES (?1, ?2, ?3)",
|
||||
params![identity.name, identity.calendar_address, is_default as i64],
|
||||
)?;
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
tx.last_insert_rowid()
|
||||
}
|
||||
};
|
||||
files_ids::record(&tx, source_id, files_ids::PARTICIPANT_IDENTITY, &key, id)?;
|
||||
}
|
||||
rerun.tally(
|
||||
&tx,
|
||||
"participant_identities",
|
||||
PARTICIPANT_COLUMNS,
|
||||
&mut counts,
|
||||
)?;
|
||||
tx.commit()?;
|
||||
Ok(counts)
|
||||
}
|
||||
|
||||
fn hex(bytes: &[u8]) -> String {
|
||||
bytes.iter().map(|b| format!("{b:02x}")).collect()
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! Paths from the manifest are data: whoever wrote the export chose them.
|
||||
//! Each is checked to stay inside its part of the tree, and no part of it
|
||||
//! may be a symbolic link, before anything is opened.
|
||||
|
||||
use std::fs;
|
||||
use std::path::{Component, Path, PathBuf};
|
||||
|
||||
/// `rel`, a `/`-separated path from the manifest, under `root`. It must
|
||||
/// start with `part` (`mail`, `contacts` and so on), climb nowhere, and
|
||||
/// pass through no symbolic link. What it names must exist.
|
||||
pub fn resolve(root: &Path, rel: &str, part: &str) -> Result<PathBuf, String> {
|
||||
let segments = segments(rel)?;
|
||||
if segments.first() != Some(&part) {
|
||||
return Err(format!("{rel:?} is outside {part}/"));
|
||||
}
|
||||
let mut path = root.to_path_buf();
|
||||
for segment in &segments {
|
||||
path.push(segment);
|
||||
match fs::symlink_metadata(&path) {
|
||||
Ok(meta) if meta.file_type().is_symlink() => {
|
||||
return Err(format!(
|
||||
"{rel:?} passes through a symbolic link, which is never followed"
|
||||
));
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
|
||||
return Err(format!("{rel:?} is not in the export"));
|
||||
}
|
||||
Err(e) => return Err(format!("{rel:?}: {e}")),
|
||||
}
|
||||
}
|
||||
Ok(path)
|
||||
}
|
||||
|
||||
/// The segments of a manifest path, refused if any could climb out or
|
||||
/// mean something other than a plain name.
|
||||
pub fn segments(rel: &str) -> Result<Vec<&str>, String> {
|
||||
if rel.is_empty() {
|
||||
return Err("an empty path".to_owned());
|
||||
}
|
||||
let mut out = Vec::new();
|
||||
for segment in rel.split('/') {
|
||||
let plain = !segment.is_empty()
|
||||
&& segment != "."
|
||||
&& segment != ".."
|
||||
&& !segment.contains(['\\', '\0'])
|
||||
&& matches!(
|
||||
Path::new(segment)
|
||||
.components()
|
||||
.collect::<Vec<_>>()
|
||||
.as_slice(),
|
||||
[Component::Normal(_)]
|
||||
);
|
||||
if !plain {
|
||||
return Err(format!("{rel:?} is not a plain relative path"));
|
||||
}
|
||||
out.push(segment);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// A directory, and not a link to one.
|
||||
pub fn is_real_dir(path: &Path) -> bool {
|
||||
fs::symlink_metadata(path).is_ok_and(|m| m.is_dir())
|
||||
}
|
||||
|
||||
/// A regular file, and not a link to one.
|
||||
pub fn is_real_file(path: &Path) -> bool {
|
||||
fs::symlink_metadata(path).is_ok_and(|m| m.is_file())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn paths_that_climb_or_jump_are_refused() {
|
||||
for bad in [
|
||||
"",
|
||||
"/etc/passwd",
|
||||
"mail/../../etc",
|
||||
"mail/./x",
|
||||
"mail//x",
|
||||
"mail/a\\..\\b",
|
||||
"..",
|
||||
] {
|
||||
assert!(segments(bad).is_err(), "{bad:?}");
|
||||
}
|
||||
assert_eq!(segments("mail/.INBOX.x").unwrap(), ["mail", ".INBOX.x"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_path_must_stay_in_its_part_and_exist() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
fs::create_dir_all(td.path().join("contacts")).unwrap();
|
||||
fs::write(td.path().join("contacts/a.vcf"), "").unwrap();
|
||||
fs::create_dir_all(td.path().join("calendars")).unwrap();
|
||||
fs::write(td.path().join("calendars/b.ics"), "").unwrap();
|
||||
assert!(resolve(td.path(), "contacts/a.vcf", "contacts").is_ok());
|
||||
let e = resolve(td.path(), "calendars/b.ics", "contacts").unwrap_err();
|
||||
assert!(e.contains("outside contacts/"), "{e}");
|
||||
let e = resolve(td.path(), "contacts/missing.vcf", "contacts").unwrap_err();
|
||||
assert!(e.contains("not in the export"), "{e}");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn a_link_anywhere_on_the_way_is_refused() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let outside = tempfile::tempdir().unwrap();
|
||||
fs::write(outside.path().join("secret"), "x").unwrap();
|
||||
fs::create_dir_all(td.path().join("sieve")).unwrap();
|
||||
std::os::unix::fs::symlink(
|
||||
outside.path().join("secret"),
|
||||
td.path().join("sieve/a.sieve"),
|
||||
)
|
||||
.unwrap();
|
||||
std::os::unix::fs::symlink(outside.path(), td.path().join("contacts")).unwrap();
|
||||
let e = resolve(td.path(), "sieve/a.sieve", "sieve").unwrap_err();
|
||||
assert!(e.contains("symbolic link"), "{e}");
|
||||
let e = resolve(td.path(), "contacts/secret", "contacts").unwrap_err();
|
||||
assert!(e.contains("symbolic link"), "{e}");
|
||||
assert!(!is_real_file(&td.path().join("sieve/a.sieve")));
|
||||
assert!(!is_real_dir(&td.path().join("contacts")));
|
||||
}
|
||||
}
|
||||
@@ -160,7 +160,8 @@ fn insert_card(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn split_vcards(text: &str) -> Vec<String> {
|
||||
/// The vCards in one file, each as its own text.
|
||||
pub fn split_vcards(text: &str) -> Vec<String> {
|
||||
let mut chunks: Vec<String> = Vec::new();
|
||||
let mut current: Option<String> = None;
|
||||
for raw_line in text.lines() {
|
||||
|
||||
@@ -14,6 +14,7 @@ pub mod export_managesieve;
|
||||
pub mod import_dav;
|
||||
pub mod import_exchange_ews;
|
||||
pub mod import_exchange_graph;
|
||||
pub mod import_files;
|
||||
pub mod import_imap;
|
||||
pub mod import_jmap;
|
||||
pub mod import_maildir;
|
||||
|
||||
@@ -0,0 +1,789 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! `import files`: an account export read back, manifest and all, so what
|
||||
//! the open formats can't hold comes back too. An archive exported and
|
||||
//! imported again holds the same account, and exporting that gives the
|
||||
//! same tree.
|
||||
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use inbuxa_migrate::db::{blobs, init};
|
||||
use inbuxa_migrate::export_files::{self, ExportFilesConfig};
|
||||
use inbuxa_migrate::logging::Logger;
|
||||
use inbuxa_migrate::sync::import_dav::calcard;
|
||||
use inbuxa_migrate::sync::import_files::{self, FilesImportConfig};
|
||||
use inbuxa_migrate::sync::{CommonConfig, Summary, TypeCounts};
|
||||
use rusqlite::{Connection, params};
|
||||
use serde_json::{Value, json};
|
||||
use tempfile::TempDir;
|
||||
|
||||
const VCARD_ALICE: &str = "BEGIN:VCARD\r\nVERSION:4.0\r\nUID:card-alice\r\nFN:Alice Smith\r\n\
|
||||
EMAIL:[email protected]\r\nEND:VCARD\r\n";
|
||||
const VCARD_BOB: &str = "BEGIN:VCARD\r\nVERSION:4.0\r\nUID:card-bob\r\nFN:Bob Jones\r\n\
|
||||
TEL:+1 555 0100\r\nEND:VCARD\r\n";
|
||||
const STANDUP: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:x\r\nBEGIN:VEVENT\r\nUID:ev-standup\r\n\
|
||||
DTSTAMP:20240101T000000Z\r\nDTSTART;TZID=Europe/Berlin:20240301T090000\r\nDURATION:PT15M\r\n\
|
||||
RRULE:FREQ=WEEKLY;COUNT=4\r\nSUMMARY:Standup\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n";
|
||||
const OFFSITE: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:x\r\nBEGIN:VEVENT\r\nUID:ev-offsite\r\n\
|
||||
DTSTAMP:20240101T000000Z\r\nDTSTART;VALUE=DATE:20240510\r\nSUMMARY:Offsite\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n";
|
||||
|
||||
fn message(subject: &str) -> Vec<u8> {
|
||||
format!(
|
||||
"From: [email protected]\r\nTo: [email protected]\r\nSubject: {subject}\r\n\
|
||||
Message-ID: <{}@example.net>\r\nDate: Fri, 01 Mar 2024 12:00:00 +0000\r\n\r\n{subject}\r\n",
|
||||
subject.replace(' ', "-")
|
||||
)
|
||||
.into_bytes()
|
||||
}
|
||||
|
||||
/// An account with everything the manifest exists for: folder names Maildir
|
||||
/// can't hold, roles, an unsubscribed folder, a message in two folders,
|
||||
/// custom keywords, default address book and calendar, a card and an event
|
||||
/// in two collections, colors and time zones, an active Sieve script,
|
||||
/// media types, a link, identities with signatures, a calendar identity.
|
||||
fn seed(path: &Path) {
|
||||
let conn = init::open(path).unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO sources (id, kind, session_url, account_id, account_name, username)
|
||||
VALUES (1, 'jmap', 'https://mail.example.org/jmap/session', 'a1', '[email protected]', 'alice')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
for (id, name, parent, role, subscribed) in [
|
||||
(1, "Inbox", None, Some("inbox"), 1),
|
||||
(2, "Clients", Some(1), None, 1),
|
||||
(3, "Archive", None, Some("archive"), 1),
|
||||
(4, "2024", Some(3), None, 1),
|
||||
(5, "Équipe & co", None, None, 1),
|
||||
(6, "v1.0 notes", None, None, 0),
|
||||
(7, "Sent Items", None, Some("sent"), 1),
|
||||
] {
|
||||
conn.execute(
|
||||
"INSERT INTO mailboxes (id, name, parent_id, role, is_subscribed)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5)",
|
||||
params![id, name, parent, role, subscribed],
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
for (subject, mailboxes, keywords) in [
|
||||
("Seen and flagged", "[1]", r#"["$flagged","$seen"]"#),
|
||||
("Junk in two folders", "[1,3]", r#"["$Junk"]"#),
|
||||
("Client note", "[2]", r#"["$answered","project-x"]"#),
|
||||
("Old report", "[4]", "[]"),
|
||||
("Bonjour", "[5]", r#"["$seen"]"#),
|
||||
("Release notes", "[6]", "[]"),
|
||||
("Sent reply", "[7]", r#"["$seen"]"#),
|
||||
] {
|
||||
let blob = blobs::intern_blob(&conn, &message(subject)).unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO emails (blob_id, received_at, mailbox_ids, keywords)
|
||||
VALUES (?1, '2024-03-01T12:00:00Z', ?2, ?3)",
|
||||
params![blob, mailboxes, keywords],
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
conn.execute(
|
||||
"INSERT INTO address_books (id, name, description, is_default)
|
||||
VALUES (1, 'Personal', 'Friends and family', 0), (2, 'Work', NULL, 1)",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
for (vcard, books) in [(VCARD_ALICE, "[1,2]"), (VCARD_BOB, "[2]")] {
|
||||
let card = calcard::vcard_to_jscontact(vcard, "/x").unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO contact_cards (uid, address_book_ids, data) VALUES (?1, ?2, ?3)",
|
||||
params![card.uid, books, card.data.to_string()],
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
conn.execute(
|
||||
"INSERT INTO calendars (id, name, description, color, time_zone, is_default)
|
||||
VALUES (1, 'Team, Berlin', 'Stand-ups', '#3366ff', 'Europe/Berlin', 1),
|
||||
(2, 'Company', NULL, '#ff9900', 'America/Phoenix', 0)",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
for (ics, calendars) in [(STANDUP, "[1]"), (OFFSITE, "[1,2]")] {
|
||||
let mut entry = calcard::ical_to_jscalendar_entries(ics).unwrap().remove(0);
|
||||
calcard::strip_extracted_fields_from_event(&mut entry.data);
|
||||
conn.execute(
|
||||
"INSERT INTO calendar_events (calendar_ids, data) VALUES (?1, ?2)",
|
||||
params![calendars, entry.data.to_string()],
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
for (name, active, script) in [
|
||||
(
|
||||
"filters",
|
||||
1,
|
||||
"require \"fileinto\";\r\nif header :contains \"subject\" \"report\" { fileinto \"Archive\"; }\r\n",
|
||||
),
|
||||
("old", 0, "keep;\r\n"),
|
||||
] {
|
||||
let blob = blobs::intern_blob(&conn, script.as_bytes()).unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO sieve_scripts (name, is_active, blob_id) VALUES (?1, ?2, ?3)",
|
||||
params![name, active, blob],
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let text = blobs::intern_blob(&conn, b"hello\n").unwrap();
|
||||
let pdf = blobs::intern_blob(&conn, b"%PDF-1.7 not really\n").unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO file_nodes (id, parent_id, node_type, blob_id, target, name, media_type, created, modified)
|
||||
VALUES (1, NULL, 'directory', NULL, NULL, 'Docs', NULL, '2024-01-01T00:00:00Z', NULL),
|
||||
(2, 1, 'file', ?1, NULL, 'a.txt', 'text/plain', '2024-01-01T00:00:00Z', '2024-02-01T10:00:00Z'),
|
||||
(3, 1, 'directory', NULL, NULL, 'Taxes', NULL, '2024-01-01T00:00:00Z', NULL),
|
||||
(4, 3, 'file', ?2, NULL, '2025.pdf', 'application/pdf', '2024-01-01T00:00:00Z', '2024-04-15T09:30:00Z'),
|
||||
(5, 1, 'symlink', NULL, '{\"path\":\"Taxes/2025.pdf\"}', 'latest', NULL, '2024-01-01T00:00:00Z', NULL)",
|
||||
params![text, pdf],
|
||||
)
|
||||
.unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO identities (name, email, reply_to, bcc, text_signature, html_signature)
|
||||
VALUES ('Alice Smith', '[email protected]', '[{\"name\":\"Alice\",\"email\":\"[email protected]\"}]',
|
||||
NULL, 'Alice\n-- \nExample Org', '<p>Alice</p>'),
|
||||
('Alice (support)', '[email protected]', NULL, '[{\"name\":null,\"email\":\"[email protected]\"}]', '', '')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO participant_identities (name, calendar_address, is_default)
|
||||
VALUES ('Alice Smith', 'mailto:[email protected]', 1),
|
||||
('Alice (support)', 'mailto:[email protected]', 0)",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
struct Fixture {
|
||||
_td: TempDir,
|
||||
root: PathBuf,
|
||||
archive: PathBuf,
|
||||
out: PathBuf,
|
||||
}
|
||||
|
||||
fn exported() -> Fixture {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let root = td.path().to_path_buf();
|
||||
let archive = root.join("archive.sqlite");
|
||||
let out = root.join("out");
|
||||
seed(&archive);
|
||||
export(&archive, &out);
|
||||
Fixture {
|
||||
_td: td,
|
||||
root,
|
||||
archive,
|
||||
out,
|
||||
}
|
||||
}
|
||||
|
||||
fn export(archive: &Path, out: &Path) {
|
||||
let summary = export_files::run(ExportFilesConfig {
|
||||
archive: archive.to_path_buf(),
|
||||
dir: out.to_path_buf(),
|
||||
objects: None,
|
||||
dry_run: false,
|
||||
logger: Logger::from_flags(true, 0),
|
||||
})
|
||||
.unwrap();
|
||||
assert!(!summary.any_failed(), "{:?}", summary.per_type);
|
||||
}
|
||||
|
||||
fn common(archive: &Path, dry_run: bool) -> CommonConfig {
|
||||
CommonConfig {
|
||||
archive: archive.to_path_buf(),
|
||||
threads: 1,
|
||||
dry_run,
|
||||
max_retries: 0,
|
||||
allow_invalid_certs: false,
|
||||
logger: Logger::from_flags(true, 0),
|
||||
}
|
||||
}
|
||||
|
||||
fn import(dir: &Path, archive: &Path) -> Result<Summary, inbuxa_migrate::error::Error> {
|
||||
import_files::run(
|
||||
common(archive, false),
|
||||
FilesImportConfig {
|
||||
dir: dir.to_path_buf(),
|
||||
allow_source_change: false,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
fn counts<'a>(summary: &'a Summary, type_name: &str) -> &'a TypeCounts {
|
||||
&summary
|
||||
.per_type
|
||||
.iter()
|
||||
.find(|(t, _)| *t == type_name)
|
||||
.unwrap_or_else(|| panic!("no {type_name} in {:?}", summary.per_type))
|
||||
.1
|
||||
}
|
||||
|
||||
fn rows(conn: &Connection, sql: &str) -> Vec<Vec<Value>> {
|
||||
let mut stmt = conn.prepare(sql).unwrap();
|
||||
let n = stmt.column_count();
|
||||
stmt.query_map([], |r| {
|
||||
(0..n)
|
||||
.map(|i| {
|
||||
Ok(match r.get::<_, rusqlite::types::Value>(i)? {
|
||||
rusqlite::types::Value::Null => Value::Null,
|
||||
rusqlite::types::Value::Integer(v) => json!(v),
|
||||
rusqlite::types::Value::Real(v) => json!(v),
|
||||
rusqlite::types::Value::Text(v) => json!(v),
|
||||
rusqlite::types::Value::Blob(v) => json!(blake3::hash(&v).to_hex().to_string()),
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
.unwrap()
|
||||
.map(Result::unwrap)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// An archive's account in terms no row id appears in: every collection by
|
||||
/// its names, every item by what it is and where it is.
|
||||
fn account(path: &Path) -> BTreeMap<&'static str, BTreeSet<String>> {
|
||||
let conn = Connection::open(path).unwrap();
|
||||
let mut out: BTreeMap<&'static str, BTreeSet<String>> = BTreeMap::new();
|
||||
|
||||
let boxes: BTreeMap<i64, (String, Option<i64>)> =
|
||||
rows(&conn, "SELECT id, name, parent_id FROM mailboxes")
|
||||
.into_iter()
|
||||
.map(|r| {
|
||||
(
|
||||
r[0].as_i64().unwrap(),
|
||||
(r[1].as_str().unwrap().to_owned(), r[2].as_i64()),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
let path_of = |mut id: i64| {
|
||||
let mut parts = Vec::new();
|
||||
loop {
|
||||
let (name, parent) = &boxes[&id];
|
||||
parts.insert(0, name.clone());
|
||||
match parent {
|
||||
Some(p) => id = *p,
|
||||
None => break parts.join("/"),
|
||||
}
|
||||
}
|
||||
};
|
||||
for r in rows(&conn, "SELECT id, role, is_subscribed FROM mailboxes") {
|
||||
out.entry("mailboxes").or_default().insert(format!(
|
||||
"{} role={} subscribed={}",
|
||||
path_of(r[0].as_i64().unwrap()),
|
||||
r[1],
|
||||
r[2]
|
||||
));
|
||||
}
|
||||
let ids = |v: &Value| -> Vec<i64> { serde_json::from_str(v.as_str().unwrap()).unwrap() };
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT b.data, e.mailbox_ids, e.keywords, e.received_at FROM emails e JOIN blobs b ON b.id = e.blob_id",
|
||||
) {
|
||||
let mut folders: Vec<String> = ids(&r[1]).into_iter().map(path_of).collect();
|
||||
folders.sort();
|
||||
let mut keywords: Vec<String> = serde_json::from_str(r[2].as_str().unwrap()).unwrap();
|
||||
keywords.sort();
|
||||
out.entry("emails").or_default().insert(format!(
|
||||
"{} in {folders:?} keywords={keywords:?} received={}",
|
||||
r[0], r[3]
|
||||
));
|
||||
}
|
||||
|
||||
let names = |sql: &str| -> BTreeMap<i64, String> {
|
||||
rows(&conn, sql)
|
||||
.into_iter()
|
||||
.map(|r| (r[0].as_i64().unwrap(), r[1].as_str().unwrap().to_owned()))
|
||||
.collect()
|
||||
};
|
||||
let books = names("SELECT id, name FROM address_books");
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT name, description, is_default FROM address_books",
|
||||
) {
|
||||
out.entry("address_books")
|
||||
.or_default()
|
||||
.insert(format!("{r:?}"));
|
||||
}
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT uid, address_book_ids, data FROM contact_cards",
|
||||
) {
|
||||
let mut in_books: Vec<&str> = ids(&r[1]).iter().map(|b| books[b].as_str()).collect();
|
||||
in_books.sort();
|
||||
out.entry("cards")
|
||||
.or_default()
|
||||
.insert(format!("{} in {in_books:?}", r[0]));
|
||||
}
|
||||
let calendars = names("SELECT id, name FROM calendars");
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT name, description, color, time_zone, is_default FROM calendars",
|
||||
) {
|
||||
out.entry("calendars").or_default().insert(format!("{r:?}"));
|
||||
}
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT calendar_ids, data, data_type FROM calendar_events",
|
||||
) {
|
||||
let data: Value = serde_json::from_str(r[1].as_str().unwrap()).unwrap();
|
||||
let mut in_calendars: Vec<&str> =
|
||||
ids(&r[0]).iter().map(|c| calendars[c].as_str()).collect();
|
||||
in_calendars.sort();
|
||||
out.entry("events").or_default().insert(format!(
|
||||
"{} {} {} in {in_calendars:?}",
|
||||
data["uid"], data["title"], r[2]
|
||||
));
|
||||
}
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT s.name, s.is_active, b.data FROM sieve_scripts s JOIN blobs b ON b.id = s.blob_id",
|
||||
) {
|
||||
out.entry("sieve").or_default().insert(format!("{r:?}"));
|
||||
}
|
||||
let nodes = names("SELECT id, name FROM file_nodes");
|
||||
let parents: BTreeMap<i64, Option<i64>> = rows(&conn, "SELECT id, parent_id FROM file_nodes")
|
||||
.into_iter()
|
||||
.map(|r| (r[0].as_i64().unwrap(), r[1].as_i64()))
|
||||
.collect();
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT n.id, n.node_type, n.media_type, n.target, b.data, n.modified
|
||||
FROM file_nodes n LEFT JOIN blobs b ON b.id = n.blob_id",
|
||||
) {
|
||||
let mut id = r[0].as_i64().unwrap();
|
||||
let mut path = nodes[&id].clone();
|
||||
while let Some(Some(p)) = parents.get(&id) {
|
||||
path = format!("{}/{path}", nodes[p]);
|
||||
id = *p;
|
||||
}
|
||||
// A link has no file whose time the export could set.
|
||||
let modified = if r[1] == "symlink" {
|
||||
Value::Null
|
||||
} else {
|
||||
r[5].clone()
|
||||
};
|
||||
out.entry("files").or_default().insert(format!(
|
||||
"{path} {} {} {} {} {modified}",
|
||||
r[1], r[2], r[3], r[4]
|
||||
));
|
||||
}
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT name, email, reply_to, bcc, text_signature, html_signature FROM identities",
|
||||
) {
|
||||
// Addresses are JSON; the same addresses may be written in any key order.
|
||||
let mut r = r;
|
||||
for v in &mut r[2..4] {
|
||||
if let Some(text) = v.as_str() {
|
||||
*v = serde_json::from_str(text).unwrap();
|
||||
}
|
||||
}
|
||||
out.entry("identities")
|
||||
.or_default()
|
||||
.insert(format!("{r:?}"));
|
||||
}
|
||||
for r in rows(
|
||||
&conn,
|
||||
"SELECT name, calendar_address, is_default FROM participant_identities",
|
||||
) {
|
||||
out.entry("participant_identities")
|
||||
.or_default()
|
||||
.insert(format!("{r:?}"));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_exported_account_comes_back_whole() {
|
||||
let f = exported();
|
||||
let back = f.root.join("back.sqlite");
|
||||
let summary = import(&f.out, &back).unwrap();
|
||||
assert!(!summary.any_failed(), "{:?}", summary.per_type);
|
||||
|
||||
let original = account(&f.archive);
|
||||
let read_back = account(&back);
|
||||
for part in original.keys() {
|
||||
assert_eq!(read_back.get(part), original.get(part), "{part}");
|
||||
}
|
||||
assert_eq!(read_back, original);
|
||||
|
||||
// One message in two folders came back as one message, not two.
|
||||
assert_eq!(counts(&summary, "email").created, 7);
|
||||
// Folders Maildir can't name kept their real names.
|
||||
let mailboxes = &read_back["mailboxes"];
|
||||
assert!(
|
||||
mailboxes.contains("v1.0 notes role=null subscribed=0"),
|
||||
"{mailboxes:?}"
|
||||
);
|
||||
assert!(
|
||||
mailboxes.contains("Équipe & co role=null subscribed=1"),
|
||||
"{mailboxes:?}"
|
||||
);
|
||||
|
||||
let conn = Connection::open(&back).unwrap();
|
||||
let source: (String, String) = conn
|
||||
.query_row("SELECT kind, account_name FROM sources", [], |r| {
|
||||
Ok((r.get(0)?, r.get(1)?))
|
||||
})
|
||||
.unwrap();
|
||||
assert_eq!(source, ("files".to_owned(), "[email protected]".to_owned()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exporting_what_was_imported_gives_the_same_tree() {
|
||||
let f = exported();
|
||||
let back = f.root.join("back.sqlite");
|
||||
import(&f.out, &back).unwrap();
|
||||
let again = f.root.join("again");
|
||||
export(&back, &again);
|
||||
|
||||
let mut first = manifest(&f.out);
|
||||
let mut second = manifest(&again);
|
||||
for m in [&mut first, &mut second] {
|
||||
let m = m.as_object_mut().unwrap();
|
||||
m.remove("created");
|
||||
m.remove("sources");
|
||||
}
|
||||
assert_eq!(second, first);
|
||||
assert_eq!(tree(&again), tree(&f.out));
|
||||
}
|
||||
|
||||
/// Every file in an export by path, with its content; messages by folder
|
||||
/// and content, since their names carry archive ids.
|
||||
fn tree(root: &Path) -> BTreeMap<String, String> {
|
||||
let mut out = BTreeMap::new();
|
||||
let mut stack = vec![root.to_path_buf()];
|
||||
while let Some(dir) = stack.pop() {
|
||||
for entry in fs::read_dir(&dir).unwrap() {
|
||||
let path = entry.unwrap().path();
|
||||
if path.is_dir() {
|
||||
stack.push(path);
|
||||
continue;
|
||||
}
|
||||
let rel = path
|
||||
.strip_prefix(root)
|
||||
.unwrap()
|
||||
.to_string_lossy()
|
||||
.into_owned();
|
||||
if rel == "manifest.json" {
|
||||
continue;
|
||||
}
|
||||
let bytes = fs::read(&path).unwrap();
|
||||
// iCalendar gives the order of a calendar's events no meaning.
|
||||
let bytes = if rel.ends_with(".ics") {
|
||||
components_sorted(&String::from_utf8(bytes).unwrap()).into_bytes()
|
||||
} else {
|
||||
bytes
|
||||
};
|
||||
let content = blake3::hash(&bytes).to_hex().to_string();
|
||||
let key = match rel.split_once("/cur/") {
|
||||
Some((folder, name)) => {
|
||||
let flags = name.rsplit_once(":2,").map(|(_, f)| f).unwrap_or("");
|
||||
format!("{folder}/cur/{content}:2,{flags}")
|
||||
}
|
||||
None => rel,
|
||||
};
|
||||
out.insert(key, content);
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// An iCalendar object with its components in sorted order, each kept as
|
||||
/// written.
|
||||
fn components_sorted(ics: &str) -> String {
|
||||
let mut head = String::new();
|
||||
let mut components: Vec<String> = Vec::new();
|
||||
let mut current: Option<String> = None;
|
||||
for line in ics.split_inclusive('\n') {
|
||||
let bare = line.trim_end();
|
||||
match &mut current {
|
||||
None if bare.starts_with("BEGIN:") && bare != "BEGIN:VCALENDAR" => {
|
||||
current = Some(line.to_owned());
|
||||
}
|
||||
None => head.push_str(line),
|
||||
Some(text) => {
|
||||
text.push_str(line);
|
||||
if bare.starts_with("END:") && text.starts_with(&format!("BEGIN:{}", &bare[4..])) {
|
||||
components.push(current.take().unwrap());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
components.sort();
|
||||
head + &components.concat()
|
||||
}
|
||||
|
||||
fn manifest(out: &Path) -> Value {
|
||||
serde_json::from_str(&fs::read_to_string(out.join("manifest.json")).unwrap()).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_run_finds_everything_in_place() {
|
||||
let f = exported();
|
||||
let back = f.root.join("back.sqlite");
|
||||
import(&f.out, &back).unwrap();
|
||||
let before = account(&back);
|
||||
let summary = import(&f.out, &back).unwrap();
|
||||
for (type_name, c) in &summary.per_type {
|
||||
assert_eq!(
|
||||
(c.created, c.updated, c.failed),
|
||||
(0, 0, 0),
|
||||
"{type_name}: {c:?}"
|
||||
);
|
||||
}
|
||||
assert_eq!(counts(&summary, "email").skipped, 7);
|
||||
assert_eq!(account(&back), before);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_changed_export_updates_the_archive_and_deletes_nothing() {
|
||||
let f = exported();
|
||||
let back = f.root.join("back.sqlite");
|
||||
import(&f.out, &back).unwrap();
|
||||
|
||||
// Read in Dovecot since: one message in the archive folder is now seen.
|
||||
let archive_cur = f.out.join("mail/.Archive/cur");
|
||||
let name = fs::read_dir(&archive_cur)
|
||||
.unwrap()
|
||||
.next()
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
.file_name()
|
||||
.into_string()
|
||||
.unwrap();
|
||||
fs::rename(
|
||||
archive_cur.join(&name),
|
||||
archive_cur.join(format!("{name}S")),
|
||||
)
|
||||
.unwrap();
|
||||
// And the Sieve script that runs is now the other one.
|
||||
let mut m = manifest(&f.out);
|
||||
for script in m["sieve"].as_array_mut().unwrap() {
|
||||
let active = script["name"] == "old";
|
||||
script["active"] = json!(active);
|
||||
}
|
||||
fs::write(f.out.join("manifest.json"), m.to_string()).unwrap();
|
||||
|
||||
let summary = import(&f.out, &back).unwrap();
|
||||
assert_eq!(
|
||||
counts(&summary, "email").updated,
|
||||
1,
|
||||
"{:?}",
|
||||
summary.per_type
|
||||
);
|
||||
assert_eq!(counts(&summary, "sievescript").updated, 2);
|
||||
let conn = Connection::open(&back).unwrap();
|
||||
let active: String = conn
|
||||
.query_row(
|
||||
"SELECT name FROM sieve_scripts WHERE is_active = 1",
|
||||
[],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(active, "old");
|
||||
let junk: String = conn
|
||||
.query_row(
|
||||
"SELECT keywords FROM emails WHERE json_array_length(mailbox_ids) = 2",
|
||||
[],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
// The copy in the inbox wasn't marked; the one in Archive was, and a
|
||||
// message has one set of keywords.
|
||||
assert_eq!(junk, r#"["$Junk","$seen"]"#);
|
||||
let emails: i64 = conn
|
||||
.query_row("SELECT count(*) FROM emails", [], |r| r.get(0))
|
||||
.unwrap();
|
||||
assert_eq!(emails, 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unfinished_export_or_a_later_version_is_refused() {
|
||||
let f = exported();
|
||||
let back = f.root.join("back.sqlite");
|
||||
let text = fs::read_to_string(f.out.join("manifest.json")).unwrap();
|
||||
|
||||
fs::remove_file(f.out.join("manifest.json")).unwrap();
|
||||
let e = import(&f.out, &back).unwrap_err();
|
||||
assert_eq!(e.exit_code(), 1);
|
||||
assert!(e.to_string().contains("interrupted"), "{e}");
|
||||
assert!(
|
||||
!back.exists(),
|
||||
"nothing is created for an export that can't be read"
|
||||
);
|
||||
|
||||
let mut m: Value = serde_json::from_str(&text).unwrap();
|
||||
m["formatVersion"] = json!(2);
|
||||
fs::write(f.out.join("manifest.json"), m.to_string()).unwrap();
|
||||
let e = import(&f.out, &back).unwrap_err();
|
||||
assert!(e.to_string().contains("version 2"), "{e}");
|
||||
|
||||
// What 27.1.0-rc.2 wrote is the same format under its old name.
|
||||
m["formatVersion"] = json!(1);
|
||||
m["format"] = json!("inbuxa-migrate-files");
|
||||
fs::write(f.out.join("manifest.json"), m.to_string()).unwrap();
|
||||
import(&f.out, &back).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_dry_run_reads_the_export_and_writes_nothing() {
|
||||
let f = exported();
|
||||
let back = f.root.join("back.sqlite");
|
||||
let summary = import_files::run(
|
||||
common(&back, true),
|
||||
FilesImportConfig {
|
||||
dir: f.out.clone(),
|
||||
allow_source_change: false,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
assert!(!back.exists());
|
||||
assert_eq!(counts(&summary, "mailbox").fetched, 7);
|
||||
// The message in two folders is a file in each.
|
||||
assert_eq!(counts(&summary, "email").fetched, 8);
|
||||
assert_eq!(counts(&summary, "contactcard").fetched, 3);
|
||||
assert_eq!(counts(&summary, "calendarevent").fetched, 3);
|
||||
assert_eq!(counts(&summary, "filenode").fetched, 5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn another_export_needs_allow_source_change() {
|
||||
let f = exported();
|
||||
let back = f.root.join("back.sqlite");
|
||||
import(&f.out, &back).unwrap();
|
||||
let copy = f.root.join("copy");
|
||||
export(&back, ©);
|
||||
let e = import(©, &back).unwrap_err();
|
||||
assert_eq!(e.exit_code(), 4, "{e}");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn nothing_outside_the_export_is_read() {
|
||||
use std::os::unix::fs::symlink;
|
||||
|
||||
let f = exported();
|
||||
let outside = f.root.join("outside");
|
||||
fs::create_dir_all(outside.join("cur")).unwrap();
|
||||
fs::create_dir_all(outside.join("new")).unwrap();
|
||||
fs::create_dir_all(outside.join("tmp")).unwrap();
|
||||
let secret = b"Subject: not yours\r\n\r\nsecret\r\n";
|
||||
fs::write(outside.join("cur/1.x:2,S"), secret).unwrap();
|
||||
fs::write(
|
||||
outside.join("secret.vcf"),
|
||||
VCARD_BOB.replace("card-bob", "card-secret"),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let mut m = manifest(&f.out);
|
||||
// Paths that climb out, by name and through links.
|
||||
m["mail"]["folders"].as_array_mut().unwrap().push(json!({
|
||||
"name": "Escape", "path": ["Escape"], "dir": "mail/../outside",
|
||||
"role": null, "subscribed": true, "messages": 1
|
||||
}));
|
||||
symlink(&outside, f.out.join("mail/.Linked")).unwrap();
|
||||
m["mail"]["folders"].as_array_mut().unwrap().push(json!({
|
||||
"name": "Linked", "path": ["Linked"], "dir": "mail/.Linked",
|
||||
"role": null, "subscribed": true, "messages": 1
|
||||
}));
|
||||
symlink(outside.join("cur/1.x:2,S"), f.out.join("mail/cur/2.x:2,S")).unwrap();
|
||||
m["contacts"].as_array_mut().unwrap().push(json!({
|
||||
"name": "Stolen", "description": null, "file": "/etc/passwd", "default": false, "cards": 1
|
||||
}));
|
||||
symlink(
|
||||
outside.join("secret.vcf"),
|
||||
f.out.join("contacts/Linked.vcf"),
|
||||
)
|
||||
.unwrap();
|
||||
m["contacts"].as_array_mut().unwrap().push(json!({
|
||||
"name": "Linked", "description": null, "file": "contacts/Linked.vcf", "default": false, "cards": 1
|
||||
}));
|
||||
symlink(&outside, f.out.join("files/Docs/elsewhere")).unwrap();
|
||||
m["files"]["links"].as_array_mut().unwrap().push(json!({
|
||||
"path": "files/../../x", "target": {"path": "/"}
|
||||
}));
|
||||
fs::write(f.out.join("manifest.json"), m.to_string()).unwrap();
|
||||
|
||||
let back = f.root.join("back.sqlite");
|
||||
let summary = import(&f.out, &back).unwrap();
|
||||
assert!(
|
||||
counts(&summary, "email").failed >= 2,
|
||||
"{:?}",
|
||||
summary.per_type
|
||||
);
|
||||
assert!(counts(&summary, "contactcard").failed >= 2);
|
||||
assert!(counts(&summary, "filenode").failed >= 1);
|
||||
|
||||
let conn = Connection::open(&back).unwrap();
|
||||
let leaked: i64 = conn
|
||||
.query_row(
|
||||
"SELECT count(*) FROM blobs WHERE instr(data, 'secret') > 0",
|
||||
[],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(leaked, 0);
|
||||
let cards: i64 = conn
|
||||
.query_row(
|
||||
"SELECT count(*) FROM contact_cards WHERE uid = 'card-secret'",
|
||||
[],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(cards, 0);
|
||||
let elsewhere: i64 = conn
|
||||
.query_row(
|
||||
"SELECT count(*) FROM file_nodes WHERE name = 'elsewhere'",
|
||||
[],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(elsewhere, 0);
|
||||
// Everything that was safely inside still came in.
|
||||
assert_eq!(counts(&summary, "email").created, 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_card_another_import_put_in_the_archive_keeps_its_books() {
|
||||
let f = exported();
|
||||
let back = f.root.join("back.sqlite");
|
||||
{
|
||||
let conn = init::open(&back).unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO sources (id, kind, session_url, account_id, account_name, username)
|
||||
VALUES (1, 'carddav', 'https://dav.example.org/', 'alice', 'alice', 'alice')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO address_books (id, name) VALUES (1, 'Phone')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
let card = calcard::vcard_to_jscontact(VCARD_ALICE, "/x").unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO contact_cards (uid, address_book_ids, data) VALUES (?1, '[1]', ?2)",
|
||||
params![card.uid, card.data.to_string()],
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
import(&f.out, &back).unwrap();
|
||||
let cards = &account(&back)["cards"];
|
||||
assert!(
|
||||
cards.contains(r#""card-alice" in ["Personal", "Phone", "Work"]"#),
|
||||
"{cards:?}"
|
||||
);
|
||||
}
|
||||
Reference in new issue
Block a user