Add export-files: an archive as Maildir, vCard, iCalendar and Sieve files #26

Merged
jcoffey-dev merged 1 commits from feat/export-files into main 2026-10-30 05:42:15 +00:00
15 changed files with 2256 additions and 15 deletions

No files matched your search

+16
View File
@@ -5,6 +5,22 @@ inbuxa suite: a series (`27.1`), its patches (`27.1.3`) and release
candidates before a series ships (`27.1-rc.1`). The tag is the version with a
`v`. Releases before 27.1 were dates (`v2026.9.30`).
## [Unreleased]
### Added
- `export-files` writes an archive out as plain files in open formats: mail
as a Maildir++ tree, each address book as a `.vcf`, each calendar as an
`.ics`, Sieve scripts, the file tree, and a `manifest.json` with what those
formats don't hold. It needs no network, writes only into a new or empty
directory, and keeps the directory private to its owner.
### Changed
- `import maildir` reads folder names in modified UTF-7, as Dovecot and
Courier write them, and the custom keywords in each folder's
`dovecot-keywords` file (Junk, labels).
- `import takeout` unfolds and unescapes a calendar's `X-WR-CALNAME` and
`X-WR-CALDESC`, so a long name or one with a comma reads back whole.
## [27.1-rc.1]
The first release candidate of the 27.1 series, with the rest of the suite.
+6 -2
View File
@@ -17,6 +17,7 @@ contacts, identities, Sieve scripts and files.
inbuxa-migrate import imap … alice.sqlite an account into an archive
inbuxa-migrate inspect alice.sqlite what landed
inbuxa-migrate export … alice.sqlite the archive into inbuxa
inbuxa-migrate export-files alice.sqlite out the archive as plain files
Between the two sides sits an archive: one SQLite file holding one account.
`import` fills it from the old server; `export` writes it into the new one.
@@ -33,8 +34,11 @@ restore it later with `export`. An archive remembers which account filled it
and refuses a different one unless told otherwise.
Export speaks JMAP only. It writes into inbuxa, or into any JMAP server that
advertises the types being written. The full command reference is in
[docs/usage.md](docs/usage.md).
advertises the types being written. `export-files` is the way out to
everything else: it writes the archive as a Maildir++ tree, vCard,
iCalendar and Sieve files that other servers and mail, calendar and
contacts apps open, with no network and no store format in the way. The full
command reference is in [docs/usage.md](docs/usage.md).
## Sources
+56 -2
View File
@@ -13,7 +13,7 @@ and how to install it.
## Commands and global flags
```
inbuxa-migrate <import|export|inspect> [args...]
inbuxa-migrate <import|export|export-files|inspect> [args...]
```
Every command takes a few global flags -- verbosity, worker pool size, retry
@@ -259,7 +259,8 @@ type ends with a line of what was created, updated, left unchanged and
failed.
`--prune` also deletes what is on the target and not in the archive. It asks
first; `--yes` answers for it, for scripts. Export speaks JMAP only.
first; `--yes` answers for it, for scripts. Export speaks JMAP only; for
anything else, see [Export to files](#export-to-files).
With `--dry-run`, export reads the target and writes nothing to the account.
It prints the plan in plain words: for each type, how much would be
@@ -280,6 +281,59 @@ The plan lists each predicted failure and its reason. When anything would
fail, the dry run exits 5, as the real run would, so a script can stop
before it starts.
## Export to files
```
inbuxa-migrate export-files [--objects <list>] [--dry-run] <ARCHIVE> <DIR>
```
Writes `ARCHIVE` out as plain files in open formats, for a server or app
that doesn't speak JMAP, or to keep a copy that needs no tool to read. It
never opens a network connection. `DIR` must be new or empty, and is made
readable by its owner only, like the archive.
```
DIR/
mail/ Maildir++: the inbox here, other folders as .Parent.Child
contacts/*.vcf one file per address book
calendars/*.ics one file per calendar
sieve/*.sieve one file per script
files/ the file tree
manifest.json what the formats above don't hold
```
- **Mail** is laid out as Dovecot and Courier keep it, so either can serve
it as it is, and `import maildir` reads it back. Folder names are in
modified UTF-7. A `.` or `/` inside a folder's name becomes `_` on disk,
since Maildir++ uses the dot between levels. A message kept in several
folders is written to each. Read, answered, flagged, draft, forwarded and
deleted are the standard Maildir flags; other keywords (Junk, labels) are
the letters `a` to `z`, named in each folder's `dovecot-keywords`, up to
26 per folder. Each file is dated when the message arrived.
- **Contacts** are vCard 4.0, and a card in two address books is in both
files. **Calendars** are iCalendar, one VCALENDAR each, named by
`X-WR-CALNAME`, with each time zone once. Photos and attachments the
archive holds go inside as `data:` URIs, as `export` sends them.
`import takeout` reads both back.
- **Sieve scripts** are written as they are; the manifest says which is
active.
- **Files** keep their names and dates. Links are listed in the manifest
and never made, since the source chose where they point.
Every name comes from the source, so none is used as a path as it stands:
separators and names Windows refuses become `_`, and names that clash, even
only by case, are numbered.
`manifest.json` records the source account, every folder's real name,
path, role and message count, each address book and calendar with its
file, color and time zone, which Sieve script is active, the identities,
and the links. It is written last: a tree without one was interrupted, and
the way to finish it is to run again into an empty directory.
`--objects` takes the same types as `export`; naming a folder or a message
type writes all of mail, and the same for contacts and calendars. With
`--dry-run` it counts what it would write and creates nothing.
## Inspect
```
+44
View File
@@ -14,6 +14,7 @@ use crate::error::Error;
use crate::exchange_ews::oauth::OAuthFlow;
use crate::exchange_ews::types::MailboxKind;
use crate::exchange_graph::types::{EventBodyFormat, MailboxKind as GraphMailboxKind, Surfaces};
use crate::export_files::ExportFilesConfig;
use crate::inspect::InspectConfig;
use crate::jmap::account::AccountSelector;
use crate::jmap::http::Auth;
@@ -49,6 +50,11 @@ enum Command {
},
#[command(about = "Push a local archive into a target JMAP server")]
Export(Box<ExportArgs>),
#[command(
about = "Write a local archive out as plain files: Maildir, vCard, iCalendar, Sieve",
name = "export-files"
)]
ExportFiles(Box<ExportFilesArgs>),
#[command(about = "Show the contents of a local archive (read-only)")]
Inspect(Box<InspectArgs>),
}
@@ -359,6 +365,31 @@ struct ExportArgs {
archive: PathBuf,
}
#[derive(Args)]
struct ExportFilesArgs {
#[arg(
long,
value_name = "LIST",
help = "Types to write (default: everything in the archive)"
)]
objects: Option<String>,
#[arg(long, help = "Count what would be written; write nothing")]
dry_run: bool,
#[arg(short = 'v', long, action = ArgAction::Count, conflicts_with = "quiet", help = "Increase log verbosity")]
verbose: u8,
#[arg(short = 'q', long, help = "Quiet: warnings and errors only")]
quiet: bool,
#[arg(value_name = "ARCHIVE", help = "Local SQLite archive")]
archive: PathBuf,
#[arg(value_name = "DIR", help = "Directory to write, new or empty")]
dir: PathBuf,
}
#[derive(Args)]
struct InspectArgs {
#[arg(value_name = "ARCHIVE", help = "Local SQLite archive")]
@@ -396,6 +427,7 @@ pub enum Action {
ImportExchangeEws(CommonConfig, EwsImportConfig),
ImportExchangeGraph(CommonConfig, GraphImportConfig),
Export(CommonConfig, ExportConfig),
ExportFiles(ExportFilesConfig),
Inspect(InspectConfig),
}
@@ -521,6 +553,7 @@ impl Cli {
match self.command {
Command::Import { source } => resolve_import(source),
Command::Export(args) => resolve_export(*args),
Command::ExportFiles(args) => resolve_export_files(*args),
Command::Inspect(args) => resolve_inspect(*args),
}
}
@@ -899,6 +932,17 @@ fn resolve_export(args: ExportArgs) -> Result<Action, Error> {
))
}
fn resolve_export_files(args: ExportFilesArgs) -> Result<Action, Error> {
let objects = args.objects.as_deref().map(parse_object_list).transpose()?;
Ok(Action::ExportFiles(ExportFilesConfig {
archive: args.archive,
dir: args.dir,
objects,
dry_run: args.dry_run,
logger: Logger::from_flags(args.quiet, args.verbose),
}))
}
fn resolve_inspect(args: InspectArgs) -> Result<Action, Error> {
let target = match args.target.as_deref() {
Some(token) => Some(ObjectType::parse(token)?),
+479
View File
@@ -0,0 +1,479 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
//! Address books as `.vcf` files and calendars as `.ics` files, one per
//! collection, converted back from the JSContact and JSCalendar the archive
//! holds.
use std::collections::{HashMap, HashSet};
use std::fs;
use std::path::Path;
use calcard::jscalendar::JSCalendar;
use calcard::jscontact::JSContact;
use rusqlite::Connection;
use serde_json::{Value, json};
use crate::db::blobs;
use crate::error::Error;
use crate::jmap::blob::{BlobWalkError, InlineShape, inline_blob_data_uris};
use crate::jmap::error::JmapError;
use crate::logging::Logger;
use crate::sync::TypeCounts;
use super::paths::{Names, safe_component};
pub struct Outcome {
pub collection: TypeCounts,
pub item: TypeCounts,
pub manifest: Value,
}
struct Collection {
id: i64,
name: String,
description: Option<String>,
color: Option<String>,
time_zone: Option<String>,
is_default: bool,
}
pub fn write_contacts(
conn: &Connection,
root: &Path,
dry_run: bool,
logger: Logger,
) -> Result<Outcome, Error> {
let books = read_collections(
conn,
"SELECT id, name, description, NULL, NULL, is_default FROM address_books ORDER BY sort_order, id",
)?;
let mut per_book: HashMap<i64, Vec<String>> = HashMap::new();
let mut item = TypeCounts::default();
let mut stmt =
conn.prepare("SELECT id, uid, address_book_ids, data FROM contact_cards ORDER BY id")?;
let rows = stmt.query_map([], |row| {
Ok((
row.get::<_, i64>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
))
})?;
for row in rows {
let (id, uid, book_ids, data) = row?;
let text = match vcard_text(conn, &uid, &data) {
Ok(text) => text,
Err(reason) => {
logger.warn(&format!(
"contact {id} ({uid}): not written as a vCard: {reason}"
));
item.failed += 1;
continue;
}
};
let book_ids: Vec<i64> = serde_json::from_str(&book_ids).unwrap_or_default();
for book in book_ids
.iter()
.filter(|b| books.iter().any(|c| c.id == **b))
{
per_book.entry(*book).or_default().push(text.clone());
}
item.created += 1;
}
let mut names = Names::default();
let mut manifest = Vec::new();
for book in &books {
let file = names.unique(&safe_component(&book.name), ".vcf");
let cards = per_book.remove(&book.id).unwrap_or_default();
if !dry_run {
fs::create_dir_all(root)?;
fs::write(root.join(&file), cards.concat())?;
}
manifest.push(json!({
"name": book.name,
"description": book.description,
"file": format!("contacts/{file}"),
"default": book.is_default,
"cards": cards.len(),
}));
}
Ok(Outcome {
collection: TypeCounts {
created: books.len() as u64,
..Default::default()
},
item,
manifest: Value::Array(manifest),
})
}
pub fn write_calendars(
conn: &Connection,
root: &Path,
dry_run: bool,
logger: Logger,
) -> Result<Outcome, Error> {
let calendars = read_collections(
conn,
"SELECT id, name, description, color, time_zone, is_default FROM calendars ORDER BY sort_order, id",
)?;
let mut per_calendar: HashMap<i64, Vec<String>> = HashMap::new();
let mut item = TypeCounts::default();
let mut stmt =
conn.prepare("SELECT id, calendar_ids, data, data_type FROM calendar_events ORDER BY id")?;
let rows = stmt.query_map([], |row| {
Ok((
row.get::<_, i64>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
))
})?;
for row in rows {
let (id, calendar_ids, data, data_type) = row?;
let text = match icalendar_text(conn, &data, &data_type) {
Ok(text) => text,
Err(reason) => {
logger.warn(&format!("event {id}: not written as iCalendar: {reason}"));
item.failed += 1;
continue;
}
};
let calendar_ids: Vec<i64> = serde_json::from_str(&calendar_ids).unwrap_or_default();
for calendar in calendar_ids
.iter()
.filter(|c| calendars.iter().any(|k| k.id == **c))
{
per_calendar
.entry(*calendar)
.or_default()
.push(text.clone());
}
item.created += 1;
}
let mut names = Names::default();
let mut manifest = Vec::new();
for calendar in &calendars {
let file = names.unique(&safe_component(&calendar.name), ".ics");
let events = per_calendar.remove(&calendar.id).unwrap_or_default();
if !dry_run {
fs::create_dir_all(root)?;
fs::write(root.join(&file), merge_calendar(calendar, &events))?;
}
manifest.push(json!({
"name": calendar.name,
"description": calendar.description,
"color": calendar.color,
"timeZone": calendar.time_zone,
"file": format!("calendars/{file}"),
"default": calendar.is_default,
"events": events.len(),
}));
}
Ok(Outcome {
collection: TypeCounts {
created: calendars.len() as u64,
..Default::default()
},
item,
manifest: Value::Array(manifest),
})
}
fn read_collections(conn: &Connection, sql: &str) -> Result<Vec<Collection>, Error> {
let mut stmt = conn.prepare(sql)?;
let rows = stmt.query_map([], |row| {
Ok(Collection {
id: row.get(0)?,
name: row.get(1)?,
description: row.get(2)?,
color: row.get(3)?,
time_zone: row.get(4)?,
is_default: row.get::<_, i64>(5)? != 0,
})
})?;
Ok(rows.collect::<Result<_, _>>()?)
}
/// The archive keeps a card's uid beside its JSContact, and photos and
/// other media as blobs; the vCard needs both back inside.
fn vcard_text(conn: &Connection, uid: &str, data: &str) -> Result<String, String> {
let mut value: Value = serde_json::from_str(data).map_err(|e| e.to_string())?;
let map = value.as_object_mut().ok_or("not a JSON object")?;
map.insert("uid".to_owned(), Value::String(uid.to_owned()));
map.entry("@type").or_insert_with(|| json!("Card"));
inline_blobs(conn, &mut value, InlineShape::JsContactResource)?;
let json = value.to_string();
let card = JSContact::<String, String>::parse(&json)?;
Ok(card.into_vcard().ok_or("no vCard came of it")?.to_string())
}
/// calcard writes a VCALENDAR from a Group, so each event goes in one, with
/// its attachments inlined from the archive's blobs.
fn icalendar_text(conn: &Connection, data: &str, data_type: &str) -> Result<String, String> {
let mut value: Value = serde_json::from_str(data).map_err(|e| e.to_string())?;
value
.as_object_mut()
.ok_or("not a JSON object")?
.entry("@type")
.or_insert_with(|| json!(data_type));
inline_blobs(conn, &mut value, InlineShape::JsCalendarLink)?;
let group = if data_type == "Group" {
value
} else {
json!({ "@type": "Group", "entries": [value] })
};
let json = group.to_string();
let event = JSCalendar::<String, String>::parse(&json)?;
Ok(event
.into_icalendar()
.ok_or("no iCalendar came of it")?
.to_string())
}
/// Blobs the archive refers to, as `data:` URIs, the way `export` sends them.
fn inline_blobs(conn: &Connection, value: &mut Value, shape: InlineShape) -> Result<(), String> {
inline_blob_data_uris(value, shape, |local_id| {
match blobs::blob_bytes(conn, local_id) {
Ok(Some(bytes)) => Ok(bytes),
Ok(None) => Err(BlobWalkError::resolver(JmapError::Transport(format!(
"blob {local_id} is missing from the archive"
)))),
Err(e) => Err(BlobWalkError::resolver(JmapError::Transport(e.to_string()))),
}
})
.map_err(|e| e.to_string())
}
/// One VCALENDAR for a whole calendar: its name and details as the
/// `X-WR-*` properties calendar apps read, then every event's components,
/// each time zone once.
fn merge_calendar(calendar: &Collection, events: &[String]) -> String {
let mut out = String::new();
out.push_str("BEGIN:VCALENDAR\r\nVERSION:2.0\r\n");
out.push_str("PRODID:-//Coffey Labs//inbuxa-migrate//EN\r\n");
push_property(&mut out, "X-WR-CALNAME", &calendar.name);
if let Some(description) = calendar.description.as_deref().filter(|d| !d.is_empty()) {
push_property(&mut out, "X-WR-CALDESC", description);
}
if let Some(tz) = calendar.time_zone.as_deref().filter(|t| !t.is_empty()) {
push_property(&mut out, "X-WR-TIMEZONE", tz);
}
if let Some(color) = calendar.color.as_deref().filter(|c| !c.is_empty()) {
push_property(&mut out, "X-APPLE-CALENDAR-COLOR", color);
}
let mut zones: HashSet<String> = HashSet::new();
for event in events {
for component in components(event) {
if let Some(tzid) = component.tzid
&& !zones.insert(tzid)
{
continue;
}
out.push_str(&component.text);
}
}
out.push_str("END:VCALENDAR\r\n");
out
}
struct Component {
tzid: Option<String>,
text: String,
}
/// The components directly inside an iCalendar object's VCALENDAR, each
/// with its lines as they were (folded lines stay folded).
fn components(ics: &str) -> Vec<Component> {
let mut out = Vec::new();
let mut depth = 0usize;
let mut current: Option<(String, Component)> = None;
for line in ics.split_inclusive('\n') {
let bare = line.trim_end_matches(['\r', '\n']);
let upper = bare.to_ascii_uppercase();
let continuation = bare.starts_with([' ', '\t']);
if !continuation && upper.starts_with("BEGIN:") {
depth += 1;
if depth == 2 {
current = Some((
upper[6..].to_owned(),
Component {
tzid: None,
text: String::new(),
},
));
}
}
if depth >= 2
&& let Some((name, component)) = current.as_mut()
{
component.text.push_str(bare);
component.text.push_str("\r\n");
if depth == 2 && name == "VTIMEZONE" && !continuation && upper.starts_with("TZID") {
component.tzid = bare.split_once(':').map(|(_, v)| v.to_owned());
}
}
if !continuation && upper.starts_with("END:") {
if depth == 2
&& let Some((_, component)) = current.take()
{
out.push(component);
}
depth = depth.saturating_sub(1);
}
}
out
}
fn push_property(out: &mut String, name: &str, value: &str) {
let mut escaped = String::with_capacity(value.len());
for c in value.chars() {
match c {
'\\' => escaped.push_str("\\\\"),
';' => escaped.push_str("\\;"),
',' => escaped.push_str("\\,"),
'\n' => escaped.push_str("\\n"),
'\r' => {}
c => escaped.push(c),
}
}
out.push_str(&fold(&format!("{name}:{escaped}")));
}
/// RFC 5545 folding: lines of at most 75 octets, never splitting a
/// character.
fn fold(line: &str) -> String {
let mut out = String::with_capacity(line.len() + 8);
let mut width = 0;
for c in line.chars() {
let len = c.len_utf8();
if width + len > 75 {
out.push_str("\r\n ");
width = 1;
}
out.push(c);
width += len;
}
out.push_str("\r\n");
out
}
#[cfg(test)]
mod tests {
use super::*;
const EVENT_A: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:x\r\n\
BEGIN:VTIMEZONE\r\nTZID:Europe/Berlin\r\nBEGIN:STANDARD\r\nDTSTART:19701025T030000\r\n\
TZOFFSETFROM:+0200\r\nTZOFFSETTO:+0100\r\nEND:STANDARD\r\nEND:VTIMEZONE\r\n\
BEGIN:VEVENT\r\nUID:a\r\nSUMMARY:A long\r\n folded summary\r\nBEGIN:VALARM\r\nACTION:DISPLAY\r\n\
END:VALARM\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n";
const EVENT_B: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\n\
BEGIN:VTIMEZONE\r\nTZID:Europe/Berlin\r\nEND:VTIMEZONE\r\n\
BEGIN:VEVENT\r\nUID:b\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n";
fn archive() -> Connection {
let conn = Connection::open_in_memory().unwrap();
crate::db::init::apply_schema(&conn).unwrap();
conn
}
#[test]
fn a_photo_kept_as_a_blob_goes_into_the_vcard() {
let conn = archive();
let blob = blobs::intern_blob(&conn, b"\x89PNG fake").unwrap();
let data = json!({
"@type": "Card",
"name": { "full": "Carol" },
"media": { "p": { "kind": "photo", "@blob": blob, "mediaType": "image/png" } }
});
let text = vcard_text(&conn, "card-carol", &data.to_string()).unwrap();
assert!(text.contains("PHOTO"), "{text}");
// calcard escapes the comma inside the URI, as it does for CardDAV.
assert!(text.contains("data:image/png;base64"), "{text}");
assert!(!text.contains("@blob"), "{text}");
}
#[test]
fn a_missing_blob_fails_the_item_with_a_reason() {
let data = json!({ "@type": "Card", "media": { "p": { "kind": "photo", "@blob": 99 } } });
let err = vcard_text(&archive(), "x", &data.to_string()).unwrap_err();
assert!(err.contains("blob 99"), "{err}");
}
fn calendar() -> Collection {
Collection {
id: 1,
name: "Team, Berlin; ops".to_owned(),
description: Some("Line one\nline two".to_owned()),
color: Some("#ff0000".to_owned()),
time_zone: Some("Europe/Berlin".to_owned()),
is_default: true,
}
}
#[test]
fn events_merge_into_one_calendar_with_each_zone_once() {
let ics = merge_calendar(&calendar(), &[EVENT_A.to_owned(), EVENT_B.to_owned()]);
assert!(ics.starts_with("BEGIN:VCALENDAR\r\nVERSION:2.0\r\n"));
assert!(ics.ends_with("END:VCALENDAR\r\n"));
assert_eq!(ics.matches("BEGIN:VCALENDAR").count(), 1);
assert_eq!(ics.matches("BEGIN:VTIMEZONE").count(), 1);
assert_eq!(ics.matches("BEGIN:VEVENT").count(), 2);
assert_eq!(ics.matches("PRODID:").count(), 1);
assert!(ics.contains("SUMMARY:A long\r\n folded summary\r\n"));
assert!(ics.contains("BEGIN:VALARM\r\nACTION:DISPLAY\r\nEND:VALARM\r\n"));
assert!(ics.contains("X-WR-CALNAME:Team\\, Berlin\\; ops\r\n"));
assert!(ics.contains("X-WR-CALDESC:Line one\\nline two\r\n"));
assert!(ics.contains("X-WR-TIMEZONE:Europe/Berlin\r\n"));
}
#[test]
fn long_lines_fold_on_character_boundaries() {
let folded = fold(&format!("X-WR-CALNAME:{}", "é".repeat(60)));
for line in folded.split("\r\n").filter(|l| !l.is_empty()) {
assert!(line.len() <= 75, "{} octets", line.len());
}
let unfolded = folded.replace("\r\n ", "");
assert_eq!(unfolded, format!("X-WR-CALNAME:{}\r\n", "é".repeat(60)));
}
#[test]
fn a_card_round_trips_through_jscontact() {
let card = crate::sync::import_dav::calcard::vcard_to_jscontact(
"BEGIN:VCARD\r\nVERSION:4.0\r\nUID:abc-123\r\nFN:Alice Smith\r\n\
EMAIL:[email protected]\r\nEND:VCARD\r\n",
"/x",
)
.unwrap();
let text = vcard_text(&archive(), &card.uid, &card.data.to_string()).unwrap();
assert!(text.contains("VERSION:4.0"), "{text}");
assert!(text.contains("UID:abc-123"), "{text}");
assert!(text.contains("FN:Alice Smith"), "{text}");
assert!(text.contains("[email protected]"), "{text}");
}
#[test]
fn an_event_round_trips_through_jscalendar() {
let entries = crate::sync::import_dav::calcard::ical_to_jscalendar_entries(
"BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:x\r\nBEGIN:VEVENT\r\nUID:ev-1\r\n\
DTSTAMP:20240101T000000Z\r\nDTSTART;TZID=Europe/Berlin:20240301T090000\r\n\
DURATION:PT1H\r\nRRULE:FREQ=WEEKLY;COUNT=4\r\nSUMMARY:Standup\r\nEND:VEVENT\r\n\
END:VCALENDAR\r\n",
)
.unwrap();
let mut data = entries[0].data.clone();
crate::sync::import_dav::calcard::strip_extracted_fields_from_event(&mut data);
let text = icalendar_text(&archive(), &data.to_string(), "Event").unwrap();
assert!(text.contains("UID:ev-1"), "{text}");
assert!(text.contains("SUMMARY:Standup"), "{text}");
assert!(text.contains("FREQ=WEEKLY"), "{text}");
assert!(text.contains("Europe/Berlin"), "{text}");
}
}
+485
View File
@@ -0,0 +1,485 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
//! Mail as a Maildir++ tree, the layout Dovecot and Courier serve and
//! `import maildir` reads: the inbox at the root, every other folder a
//! `.Parent.Child` directory beside it, names in modified UTF-7, flags in
//! the file name and other keywords in each folder's `dovecot-keywords`.
use std::collections::HashMap;
use std::fs;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use encodify::utf7;
use rusqlite::Connection;
use serde_json::{Value, json};
use time::OffsetDateTime;
use time::format_description::well_known::{Rfc2822, Rfc3339};
use crate::db::blobs;
use crate::error::Error;
use crate::logging::Logger;
use crate::sync::TypeCounts;
use super::paths::Names;
/// Keywords with a Maildir flag letter of their own.
const FLAG_LETTERS: [(&str, char); 6] = [
("$draft", 'D'),
("$flagged", 'F'),
("$forwarded", 'P'),
("$answered", 'R'),
("$seen", 'S'),
("$deleted", 'T'),
];
/// Dovecot gives each folder 26 keyword letters, `a` to `z`.
const MAX_CUSTOM_KEYWORDS: usize = 26;
struct Mailbox {
id: i64,
name: String,
parent_id: Option<i64>,
role: Option<String>,
is_subscribed: bool,
}
struct Folder {
/// The Maildir++ name, `Parent.Child`; empty for the inbox.
maildir_name: String,
/// The folder's real name and those of its parents.
display_path: Vec<String>,
dir: PathBuf,
role: Option<String>,
is_subscribed: bool,
messages: u64,
custom_keywords: Vec<String>,
keywords_dropped: u64,
}
pub struct MailOutcome {
pub mailbox: TypeCounts,
pub email: TypeCounts,
pub manifest: Value,
}
pub fn write(
conn: &Connection,
root: &Path,
dry_run: bool,
logger: Logger,
) -> Result<MailOutcome, Error> {
let mailboxes = read_mailboxes(conn)?;
let mut folders = plan_folders(&mailboxes, root);
let mut mailbox_counts = TypeCounts::default();
if !dry_run {
for folder in folders.values() {
make_maildir(&folder.dir, !folder.maildir_name.is_empty())?;
}
}
mailbox_counts.created = mailboxes.len() as u64;
let root_folder = folders
.iter()
.find(|(_, f)| f.maildir_name.is_empty())
.map(|(id, _)| *id)
.expect("plan_folders always makes a root");
let mut email_counts = TypeCounts::default();
let mut stmt = conn.prepare(
"SELECT id, blob_id, received_at, mailbox_ids, keywords FROM emails ORDER BY id",
)?;
let rows = stmt.query_map([], |row| {
Ok((
row.get::<_, i64>(0)?,
row.get::<_, i64>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
row.get::<_, String>(4)?,
))
})?;
for row in rows {
let (id, blob_id, received_at, mailbox_ids, keywords) = row?;
let Some(bytes) = blobs::blob_bytes(conn, blob_id)? else {
logger.warn(&format!(
"email {id}: its message is missing from the archive"
));
email_counts.failed += 1;
continue;
};
let mut targets: Vec<i64> = serde_json::from_str::<Vec<i64>>(&mailbox_ids)
.unwrap_or_default()
.into_iter()
.filter(|m| folders.contains_key(m))
.collect();
if targets.is_empty() {
logger.warn(&format!(
"email {id}: in no folder that was written; put in the inbox"
));
targets.push(root_folder);
}
let keywords: Vec<String> = serde_json::from_str(&keywords).unwrap_or_default();
let secs = delivery_secs(&received_at, &bytes);
for mailbox_id in targets {
let folder = folders.get_mut(&mailbox_id).expect("filtered above");
let flags = flags_for(&keywords, folder, logger);
let name = format!("{secs}.M{id}.inbuxa-migrate,S={}:2,{flags}", bytes.len());
if !dry_run {
write_message(&folder.dir.join("cur").join(name), &bytes, secs)?;
}
folder.messages += 1;
}
email_counts.created += 1;
}
let mut entries: Vec<&Folder> = folders.values().collect();
entries.sort_by(|a, b| a.maildir_name.cmp(&b.maildir_name));
if !dry_run {
for folder in &entries {
write_keywords_file(folder)?;
}
}
let manifest = json!({
"layout": "maildir++",
"path": "mail",
"folders": entries.iter().map(|f| folder_manifest(f, root)).collect::<Vec<_>>(),
});
Ok(MailOutcome {
mailbox: mailbox_counts,
email: email_counts,
manifest,
})
}
fn read_mailboxes(conn: &Connection) -> Result<Vec<Mailbox>, Error> {
let mut stmt = conn.prepare(
"SELECT id, name, parent_id, role, is_subscribed FROM mailboxes ORDER BY sort_order, id",
)?;
let rows = stmt.query_map([], |row| {
Ok(Mailbox {
id: row.get(0)?,
name: row.get(1)?,
parent_id: row.get(2)?,
role: row.get(3)?,
is_subscribed: row.get::<_, i64>(4)? != 0,
})
})?;
Ok(rows.collect::<Result<_, _>>()?)
}
/// Every mailbox's directory, parents before children so a renamed parent
/// takes its children with it.
fn plan_folders(mailboxes: &[Mailbox], root: &Path) -> HashMap<i64, Folder> {
let by_id: HashMap<i64, &Mailbox> = mailboxes.iter().map(|m| (m.id, m)).collect();
let inbox = mailboxes
.iter()
.find(|m| m.role.as_deref() == Some("inbox"))
.map(|m| m.id);
let mut names = Names::default();
names.reserve("INBOX");
let mut folders: HashMap<i64, Folder> = HashMap::new();
let mut order: Vec<&Mailbox> = mailboxes.iter().collect();
order.sort_by_key(|m| depth(m, &by_id));
for mailbox in order {
let parent = mailbox
.parent_id
.filter(|p| by_id.contains_key(p) && *p != mailbox.id);
let (maildir_name, display_path) = if Some(mailbox.id) == inbox {
(String::new(), vec![mailbox.name.clone()])
} else {
let (prefix, mut display) = match parent.and_then(|p| folders.get(&p)) {
Some(p) if p.maildir_name.is_empty() => {
("INBOX.".to_owned(), p.display_path.clone())
}
Some(p) => (format!("{}.", p.maildir_name), p.display_path.clone()),
None => (String::new(), Vec::new()),
};
display.push(mailbox.name.clone());
let wanted = format!("{prefix}{}", encode_segment(&mailbox.name));
(names.unique(&wanted, ""), display)
};
let dir = if maildir_name.is_empty() {
root.to_path_buf()
} else {
root.join(format!(".{maildir_name}"))
};
folders.insert(
mailbox.id,
Folder {
maildir_name,
display_path,
dir,
role: mailbox.role.clone(),
is_subscribed: mailbox.is_subscribed,
messages: 0,
custom_keywords: Vec::new(),
keywords_dropped: 0,
},
);
}
if !folders.values().any(|f| f.maildir_name.is_empty()) {
// Maildir++ needs its root to be a maildir, even with no inbox.
folders.insert(
i64::MIN,
Folder {
maildir_name: String::new(),
display_path: vec!["INBOX".to_owned()],
dir: root.to_path_buf(),
role: Some("inbox".to_owned()),
is_subscribed: true,
messages: 0,
custom_keywords: Vec::new(),
keywords_dropped: 0,
},
);
}
folders
}
fn depth(mailbox: &Mailbox, by_id: &HashMap<i64, &Mailbox>) -> usize {
let mut depth = 0;
let mut current = mailbox.parent_id;
while let Some(id) = current {
depth += 1;
if depth > by_id.len() {
break;
}
current = by_id.get(&id).and_then(|m| m.parent_id);
}
depth
}
/// One level of a Maildir++ name: modified UTF-7, with the hierarchy
/// separator and anything a directory name can't hold turned into `_`. The
/// manifest keeps the real name.
fn encode_segment(name: &str) -> String {
let encoded: String = utf7::IMAP
.encode(name)
.chars()
.map(|c| match c {
'.' | '/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|' => '_',
c => c,
})
.collect();
let trimmed = encoded.trim_end_matches(' ');
if trimmed.is_empty() {
"_".to_owned()
} else {
trimmed.to_owned()
}
}
fn flags_for(keywords: &[String], folder: &mut Folder, logger: Logger) -> String {
let mut letters: Vec<char> = Vec::new();
for keyword in keywords {
let lower = keyword.to_ascii_lowercase();
if let Some((_, letter)) = FLAG_LETTERS.iter().find(|(k, _)| *k == lower) {
letters.push(*letter);
continue;
}
let index = match folder
.custom_keywords
.iter()
.position(|k| k.eq_ignore_ascii_case(keyword))
{
Some(i) => i,
None if folder.custom_keywords.len() < MAX_CUSTOM_KEYWORDS => {
folder.custom_keywords.push(keyword.clone());
folder.custom_keywords.len() - 1
}
None => {
if folder.keywords_dropped == 0 {
logger.warn(&format!(
"folder {:?}: more than {MAX_CUSTOM_KEYWORDS} keywords; \
the rest are left off its messages",
folder.display_path.join("/")
));
}
folder.keywords_dropped += 1;
continue;
}
};
letters.push((b'a' + index as u8) as char);
}
letters.sort_unstable();
letters.dedup();
letters.into_iter().collect()
}
/// When the message arrived, as Unix seconds: the archive's received time,
/// else its Date header, else now.
fn delivery_secs(received_at: &str, message: &[u8]) -> i64 {
let parsed = OffsetDateTime::parse(received_at, &Rfc3339)
.or_else(|_| OffsetDateTime::parse(received_at, &Rfc2822))
.map(|t| t.unix_timestamp())
.ok()
.or_else(|| {
mail_parser::MessageParser::default()
.parse_headers(message)
.and_then(|m| m.date().map(|d| d.to_timestamp()))
});
parsed
.filter(|s| *s > 0)
.unwrap_or_else(|| OffsetDateTime::now_utc().unix_timestamp())
}
fn make_maildir(dir: &Path, is_subfolder: bool) -> Result<(), Error> {
for sub in ["cur", "new", "tmp"] {
fs::create_dir_all(dir.join(sub))?;
}
if is_subfolder {
fs::File::create(dir.join("maildirfolder"))?;
}
Ok(())
}
fn write_message(path: &Path, bytes: &[u8], secs: i64) -> Result<(), Error> {
let mut file = fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(path)?;
file.write_all(bytes)?;
if secs > 0 {
file.set_modified(UNIX_EPOCH + Duration::from_secs(secs as u64))
.or_else(|_| file.set_modified(SystemTime::now()))?;
}
Ok(())
}
fn write_keywords_file(folder: &Folder) -> Result<(), Error> {
if folder.custom_keywords.is_empty() {
return Ok(());
}
let mut text = String::new();
for (i, keyword) in folder.custom_keywords.iter().enumerate() {
text.push_str(&format!("{i} {keyword}\n"));
}
fs::write(folder.dir.join("dovecot-keywords"), text)?;
Ok(())
}
fn folder_manifest(folder: &Folder, root: &Path) -> Value {
let dir = match folder.dir.strip_prefix(root) {
Ok(rel) if !rel.as_os_str().is_empty() => Path::new("mail").join(rel),
_ => PathBuf::from("mail"),
};
let mut entry = json!({
"name": folder.display_path.last().cloned().unwrap_or_default(),
"path": folder.display_path,
"dir": dir.to_string_lossy().replace('\\', "/"),
"role": folder.role,
"subscribed": folder.is_subscribed,
"messages": folder.messages,
});
if folder.keywords_dropped > 0 {
entry["keywordsDropped"] = json!(folder.keywords_dropped);
}
entry
}
#[cfg(test)]
mod tests {
use super::*;
fn mailbox(id: i64, name: &str, parent_id: Option<i64>, role: Option<&str>) -> Mailbox {
Mailbox {
id,
name: name.to_owned(),
parent_id,
role: role.map(str::to_owned),
is_subscribed: true,
}
}
#[test]
fn folders_follow_maildir_plus_plus() {
let root = Path::new("/x");
let folders = plan_folders(
&[
mailbox(1, "Inbox", None, Some("inbox")),
mailbox(2, "Clients", Some(1), None),
mailbox(3, "Archive", None, Some("archive")),
mailbox(4, "2024", Some(3), None),
mailbox(5, "v1.0 notes", None, None),
mailbox(6, "Équipe & co", None, None),
mailbox(7, "INBOX", None, None),
],
root,
);
let name = |id| folders[&id].maildir_name.as_str();
assert_eq!(name(1), "");
assert_eq!(folders[&1].dir, root);
assert_eq!(name(2), "INBOX.Clients");
assert_eq!(name(3), "Archive");
assert_eq!(name(4), "Archive.2024");
assert_eq!(folders[&4].dir, root.join(".Archive.2024"));
assert_eq!(name(5), "v1_0 notes");
assert_eq!(folders[&5].display_path, vec!["v1.0 notes"]);
assert_eq!(name(6), "&AMk-quipe &- co");
assert_eq!(name(7), "INBOX (2)");
}
#[test]
fn a_store_with_no_inbox_still_gets_a_root() {
let folders = plan_folders(&[mailbox(1, "Sent", None, Some("sent"))], Path::new("/x"));
assert!(folders.values().any(|f| f.maildir_name.is_empty()));
assert_eq!(folders[&1].maildir_name, "Sent");
}
#[test]
fn keywords_become_letters() {
let mut folder = Folder {
maildir_name: "Work".to_owned(),
display_path: vec!["Work".to_owned()],
dir: PathBuf::from("/x/.Work"),
role: None,
is_subscribed: true,
messages: 0,
custom_keywords: Vec::new(),
keywords_dropped: 0,
};
let kw = |list: &[&str]| list.iter().map(|s| s.to_string()).collect::<Vec<_>>();
let logger = Logger::new(0);
assert_eq!(
flags_for(
&kw(&["$seen", "$flagged", "$answered"]),
&mut folder,
logger
),
"FRS"
);
assert_eq!(
flags_for(&kw(&["$Junk", "$seen"]), &mut folder, logger),
"Sa"
);
assert_eq!(
flags_for(&kw(&["project-x", "$junk"]), &mut folder, logger),
"ab"
);
assert_eq!(folder.custom_keywords, vec!["$Junk", "project-x"]);
let many: Vec<String> = (0..30).map(|i| format!("k{i}")).collect();
let flags = flags_for(&many, &mut folder, logger);
// Two letters were already taken above.
assert_eq!(flags.len(), MAX_CUSTOM_KEYWORDS - 2);
assert_eq!(folder.keywords_dropped, 6);
}
#[test]
fn delivery_time_prefers_the_archive_then_the_date_header() {
assert_eq!(delivery_secs("2024-03-01T12:00:00Z", b""), 1_709_294_400);
assert_eq!(
delivery_secs("Fri, 01 Mar 2024 12:00:00 +0000", b""),
1_709_294_400
);
assert_eq!(
delivery_secs("", b"Date: Fri, 01 Mar 2024 12:00:00 +0000\r\n\r\nbody"),
1_709_294_400
);
assert!(delivery_secs("", b"\r\n") > 1_709_294_400);
}
}
+419
View File
@@ -0,0 +1,419 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
//! `export-files`: an archive written out as plain files in open formats,
//! so an account can leave for any server or app without this tool or any
//! vendor's store format. Mail is a Maildir++ tree, contacts are vCard,
//! calendars are iCalendar, filters are Sieve, files are files, and
//! `manifest.json` records what doesn't fit those formats. It never opens a
//! network connection.
use std::collections::HashMap;
use std::fs;
use std::path::{Path, PathBuf};
use std::time::{Duration, UNIX_EPOCH};
use rusqlite::Connection;
use serde_json::{Value, json};
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
use crate::db::{blobs, init};
use crate::error::Error;
use crate::logging::Logger;
use crate::sync::{Summary, TypeCounts};
use crate::types::ObjectType;
pub mod cards;
pub mod mail;
pub mod paths;
use paths::{Names, safe_component};
/// The layout this release writes; raised when `manifest.json` or the tree
/// changes in a way a reader has to know about.
pub const FORMAT_VERSION: u32 = 1;
pub struct ExportFilesConfig {
pub archive: PathBuf,
pub dir: PathBuf,
pub objects: Option<Vec<ObjectType>>,
pub dry_run: bool,
pub logger: Logger,
}
pub fn run(config: ExportFilesConfig) -> Result<Summary, Error> {
if !config.archive.is_file() {
return Err(Error::Usage(format!(
"no archive at {}",
config.archive.display()
)));
}
check_target(&config.dir)?;
let conn = init::open(&config.archive)?;
let wanted = |types: &[ObjectType]| match &config.objects {
None => true,
Some(list) => types.iter().any(|t| list.contains(t)),
};
let dry_run = config.dry_run;
let logger = config.logger;
let dir = config.dir.as_path();
if !dry_run {
make_private_dir(dir)?;
}
let mut summary = Summary::default();
let mut manifest = json!({
"format": "inbuxa-migrate-files",
"formatVersion": FORMAT_VERSION,
"createdBy": format!("inbuxa-migrate {}", crate::version::shown()),
"created": OffsetDateTime::now_utc().format(&Rfc3339).unwrap_or_default(),
"sources": read_sources(&conn)?,
});
if wanted(&[ObjectType::Mailbox, ObjectType::Email]) {
let out = mail::write(&conn, &dir.join("mail"), dry_run, logger)?;
summary.per_type.push(("mailbox", out.mailbox));
summary.per_type.push(("email", out.email));
manifest["mail"] = out.manifest;
}
if wanted(&[ObjectType::AddressBook, ObjectType::ContactCard]) {
let out = cards::write_contacts(&conn, &dir.join("contacts"), dry_run, logger)?;
summary.per_type.push(("addressbook", out.collection));
summary.per_type.push(("contactcard", out.item));
manifest["contacts"] = out.manifest;
}
if wanted(&[ObjectType::Calendar, ObjectType::CalendarEvent]) {
let out = cards::write_calendars(&conn, &dir.join("calendars"), dry_run, logger)?;
summary.per_type.push(("calendar", out.collection));
summary.per_type.push(("calendarevent", out.item));
manifest["calendars"] = out.manifest;
}
if wanted(&[ObjectType::SieveScript]) {
let (counts, entries) = write_sieve(&conn, &dir.join("sieve"), dry_run, logger)?;
summary.per_type.push(("sievescript", counts));
manifest["sieve"] = entries;
}
if wanted(&[ObjectType::FileNode]) {
let (counts, entries) = write_files(&conn, &dir.join("files"), dry_run, logger)?;
summary.per_type.push(("filenode", counts));
manifest["files"] = entries;
}
if wanted(&[ObjectType::Identity]) {
let (counts, entries) = read_identities(&conn)?;
summary.per_type.push(("identity", counts));
manifest["identities"] = entries;
}
if wanted(&[ObjectType::ParticipantIdentity]) {
let (counts, entries) = read_participant_identities(&conn)?;
summary.per_type.push(("participantidentity", counts));
manifest["participantIdentities"] = entries;
}
if dry_run {
logger.warn(&format!(
"dry run: nothing was written; {} would hold what is counted below",
dir.display()
));
} else {
// Written last, so a tree with a manifest is a finished one.
let text = serde_json::to_string_pretty(&manifest).map_err(std::io::Error::other)?;
fs::write(dir.join("manifest.json"), text + "\n")?;
}
Ok(summary)
}
/// Only a new or empty directory: what is written must be exactly the
/// archive, with nothing of an earlier run mixed in.
fn check_target(dir: &Path) -> Result<(), Error> {
match fs::read_dir(dir) {
Ok(mut entries) => {
if entries.next().is_some() {
return Err(Error::Usage(format!(
"{} is not empty; give a new or empty directory",
dir.display()
)));
}
Ok(())
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(e) if e.kind() == std::io::ErrorKind::NotADirectory => Err(Error::Usage(format!(
"{} is not a directory",
dir.display()
))),
Err(e) => Err(Error::Io(e)),
}
}
/// Mail, contacts and calendars are as private as the archive they came
/// from.
fn make_private_dir(dir: &Path) -> Result<(), Error> {
fs::create_dir_all(dir)?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(dir, fs::Permissions::from_mode(0o700))?;
}
Ok(())
}
fn read_sources(conn: &Connection) -> Result<Value, Error> {
let mut stmt = conn.prepare(
"SELECT kind, session_url, account_id, account_name, username FROM sources ORDER BY id",
)?;
let rows = stmt.query_map([], |r| {
Ok(json!({
"kind": r.get::<_, String>(0)?,
"url": r.get::<_, String>(1)?,
"accountId": r.get::<_, String>(2)?,
"accountName": r.get::<_, Option<String>>(3)?,
"username": r.get::<_, String>(4)?,
}))
})?;
Ok(Value::Array(rows.collect::<Result<_, _>>()?))
}
fn write_sieve(
conn: &Connection,
root: &Path,
dry_run: bool,
logger: Logger,
) -> Result<(TypeCounts, Value), Error> {
let mut counts = TypeCounts::default();
let mut names = Names::default();
let mut entries = Vec::new();
let mut stmt =
conn.prepare("SELECT id, name, is_active, blob_id FROM sieve_scripts ORDER BY name, id")?;
let rows = stmt.query_map([], |r| {
Ok((
r.get::<_, i64>(0)?,
r.get::<_, Option<String>>(1)?,
r.get::<_, i64>(2)? != 0,
r.get::<_, i64>(3)?,
))
})?;
for row in rows {
let (id, name, is_active, blob_id) = row?;
let name = name.unwrap_or_else(|| format!("script-{id}"));
let Some(bytes) = blobs::blob_bytes(conn, blob_id)? else {
logger.warn(&format!(
"Sieve script {name:?}: its content is missing from the archive"
));
counts.failed += 1;
continue;
};
let file = names.unique(&safe_component(&name), ".sieve");
if !dry_run {
fs::create_dir_all(root)?;
fs::write(root.join(&file), &bytes)?;
}
entries.push(json!({
"name": name,
"file": format!("sieve/{file}"),
"active": is_active,
}));
counts.created += 1;
}
Ok((counts, Value::Array(entries)))
}
struct FileNode {
id: i64,
parent_id: Option<i64>,
node_type: String,
blob_id: Option<i64>,
target: Option<String>,
name: String,
media_type: Option<String>,
modified: Option<String>,
created: String,
}
/// The file tree as directories and files. Links are listed in the manifest
/// and never made: the source chose their targets, and one could point
/// anywhere on this machine.
fn write_files(
conn: &Connection,
root: &Path,
dry_run: bool,
logger: Logger,
) -> Result<(TypeCounts, Value), Error> {
let mut stmt = conn.prepare(
"SELECT id, parent_id, node_type, blob_id, target, name, media_type, modified, created
FROM file_nodes ORDER BY name, id",
)?;
let nodes: Vec<FileNode> = stmt
.query_map([], |r| {
Ok(FileNode {
id: r.get(0)?,
parent_id: r.get(1)?,
node_type: r.get(2)?,
blob_id: r.get(3)?,
target: r.get(4)?,
name: r.get(5)?,
media_type: r.get(6)?,
modified: r.get(7)?,
created: r.get(8)?,
})
})?
.collect::<Result<_, _>>()?;
let mut children: HashMap<Option<i64>, Vec<&FileNode>> = HashMap::new();
let known: std::collections::HashSet<i64> = nodes.iter().map(|n| n.id).collect();
for node in &nodes {
let parent = node
.parent_id
.filter(|p| known.contains(p) && *p != node.id);
children.entry(parent).or_default().push(node);
}
let mut state = FilesState {
conn,
children: &children,
dry_run,
logger,
counts: TypeCounts::default(),
links: Vec::new(),
media_types: serde_json::Map::new(),
visited: std::collections::HashSet::new(),
};
state.write_dir(None, root, "files")?;
let manifest = json!({
"path": "files",
"mediaTypes": state.media_types,
"links": state.links,
});
Ok((state.counts, manifest))
}
struct FilesState<'a> {
conn: &'a Connection,
children: &'a HashMap<Option<i64>, Vec<&'a FileNode>>,
dry_run: bool,
logger: Logger,
counts: TypeCounts,
links: Vec<Value>,
media_types: serde_json::Map<String, Value>,
visited: std::collections::HashSet<i64>,
}
impl FilesState<'_> {
fn write_dir(&mut self, parent: Option<i64>, dir: &Path, rel: &str) -> Result<(), Error> {
let Some(nodes) = self.children.get(&parent) else {
return Ok(());
};
let mut names = Names::default();
for node in nodes.iter().copied() {
if !self.visited.insert(node.id) {
continue;
}
let name = names.unique(&safe_component(&node.name), "");
let path = dir.join(&name);
let rel_path = format!("{rel}/{name}");
match node.node_type.as_str() {
"directory" => {
if !self.dry_run {
fs::create_dir_all(&path)?;
}
self.counts.created += 1;
self.write_dir(Some(node.id), &path, &rel_path)?;
}
"file" => {
let bytes = match node.blob_id {
Some(blob) => blobs::blob_bytes(self.conn, blob)?,
None => Some(Vec::new()),
};
let Some(bytes) = bytes else {
self.logger.warn(&format!(
"{rel_path}: its content is missing from the archive"
));
self.counts.failed += 1;
continue;
};
if !self.dry_run {
fs::create_dir_all(dir)?;
fs::write(&path, &bytes)?;
if let Some(when) = node
.modified
.as_deref()
.or(Some(node.created.as_str()))
.and_then(|t| OffsetDateTime::parse(t, &Rfc3339).ok())
.filter(|t| t.unix_timestamp() > 0)
{
let _ = fs::File::options().write(true).open(&path).and_then(|f| {
f.set_modified(
UNIX_EPOCH + Duration::from_secs(when.unix_timestamp() as u64),
)
});
}
}
if let Some(media_type) = &node.media_type {
self.media_types
.insert(rel_path.clone(), Value::String(media_type.clone()));
}
self.counts.created += 1;
}
_ => {
let target = node
.target
.as_deref()
.and_then(|t| serde_json::from_str::<Value>(t).ok());
self.links
.push(json!({ "path": rel_path, "target": target }));
self.counts.skipped += 1;
}
}
}
Ok(())
}
}
fn read_identities(conn: &Connection) -> Result<(TypeCounts, Value), Error> {
let mut stmt = conn.prepare(
"SELECT name, email, reply_to, bcc, text_signature, html_signature
FROM identities ORDER BY id",
)?;
let rows: Vec<Value> = stmt
.query_map([], |r| {
let json_column = |i: usize| -> rusqlite::Result<Value> {
Ok(r.get::<_, Option<String>>(i)?
.and_then(|s| serde_json::from_str(&s).ok())
.unwrap_or(Value::Null))
};
Ok(json!({
"name": r.get::<_, String>(0)?,
"email": r.get::<_, String>(1)?,
"replyTo": json_column(2)?,
"bcc": json_column(3)?,
"textSignature": r.get::<_, String>(4)?,
"htmlSignature": r.get::<_, String>(5)?,
}))
})?
.collect::<Result<_, _>>()?;
let counts = TypeCounts {
created: rows.len() as u64,
..Default::default()
};
Ok((counts, Value::Array(rows)))
}
fn read_participant_identities(conn: &Connection) -> Result<(TypeCounts, Value), Error> {
let mut stmt = conn.prepare(
"SELECT name, calendar_address, is_default FROM participant_identities ORDER BY id",
)?;
let rows: Vec<Value> = stmt
.query_map([], |r| {
Ok(json!({
"name": r.get::<_, String>(0)?,
"calendarAddress": r.get::<_, String>(1)?,
"default": r.get::<_, i64>(2)? != 0,
}))
})?
.collect::<Result<_, _>>()?;
let counts = TypeCounts {
created: rows.len() as u64,
..Default::default()
};
Ok((counts, Value::Array(rows)))
}
+111
View File
@@ -0,0 +1,111 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
//! File names for what the archive holds. Every name comes from the source
//! server, so none is trusted as a path: each becomes one safe component,
//! and two that end up the same are told apart.
use std::collections::HashSet;
/// One path component made from a name the source chose: no separators, no
/// `.` or `..`, nothing Windows refuses, never empty.
pub fn safe_component(name: &str) -> String {
let mut out: String = name
.chars()
.map(|c| match c {
'/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|' => '_',
c if c.is_control() => '_',
c => c,
})
.collect();
while out.ends_with(['.', ' ']) {
out.pop();
}
let trimmed = out.trim_start();
if trimmed.len() != out.len() {
out = trimmed.to_owned();
}
if out.is_empty() || out.starts_with('.') {
out.insert(0, '_');
}
if is_windows_reserved(&out) {
out.push('_');
}
out
}
fn is_windows_reserved(name: &str) -> bool {
let stem = name.split('.').next().unwrap_or(name).to_ascii_uppercase();
matches!(stem.as_str(), "CON" | "PRN" | "AUX" | "NUL")
|| (stem.len() == 4
&& (stem.starts_with("COM") || stem.starts_with("LPT"))
&& stem.as_bytes()[3].is_ascii_digit())
}
/// Names already used in one directory, compared without case so the tree
/// unpacks the same on a case-insensitive file system.
#[derive(Default)]
pub struct Names {
used: HashSet<String>,
}
impl Names {
pub fn reserve(&mut self, name: &str) {
self.used.insert(name.to_lowercase());
}
/// `stem` + `ext`, or `stem (2)` + `ext` and up when that is taken.
pub fn unique(&mut self, stem: &str, ext: &str) -> String {
let mut candidate = format!("{stem}{ext}");
let mut n = 2;
while !self.used.insert(candidate.to_lowercase()) {
candidate = format!("{stem} ({n}){ext}");
n += 1;
}
candidate
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn separators_and_traversal_never_survive() {
assert_eq!(safe_component("../../etc/passwd"), "_.._.._etc_passwd");
assert_eq!(safe_component(".."), "_");
assert_eq!(safe_component("."), "_");
assert_eq!(safe_component(""), "_");
assert_eq!(safe_component("a\\b"), "a_b");
assert_eq!(safe_component("a\0b"), "a_b");
assert_eq!(safe_component(".hidden"), "_.hidden");
}
#[test]
fn ordinary_names_are_kept() {
assert_eq!(safe_component("Family & Friends"), "Family & Friends");
assert_eq!(safe_component("Équipe"), "Équipe");
assert_eq!(safe_component("v1.0"), "v1.0");
}
#[test]
fn windows_trouble_is_avoided() {
assert_eq!(safe_component("CON"), "CON_");
assert_eq!(safe_component("com1.txt"), "com1.txt_");
assert_eq!(safe_component("Notes. "), "Notes");
assert_eq!(safe_component("a:b?"), "a_b_");
}
#[test]
fn clashing_names_are_numbered_case_blind() {
let mut names = Names::default();
assert_eq!(names.unique("Work", ".ics"), "Work.ics");
assert_eq!(names.unique("work", ".ics"), "work (2).ics");
assert_eq!(names.unique("Work", ".ics"), "Work (3).ics");
names.reserve("INBOX");
assert_eq!(names.unique("inbox", ""), "inbox (2)");
}
}
+1
View File
@@ -12,6 +12,7 @@ pub mod error;
pub mod exchange;
pub mod exchange_ews;
pub mod exchange_graph;
pub mod export_files;
pub mod imap;
pub mod inspect;
pub mod jmap;
+5 -1
View File
@@ -9,8 +9,8 @@ use clap::Parser;
use inbuxa_migrate::cli::{Action, Cli};
use inbuxa_migrate::error::Error;
use inbuxa_migrate::inspect;
use inbuxa_migrate::sync::{self, RunOutcome, Summary};
use inbuxa_migrate::{export_files, inspect};
fn main() {
let code = run();
@@ -83,6 +83,10 @@ fn run() -> i32 {
logger,
)
}
Action::ExportFiles(config) => {
let logger = config.logger;
(RunOutcome::from_result(export_files::run(config)), logger)
}
Action::Inspect(config) => {
return match inspect::run(config) {
Ok(()) => 0,
+76
View File
@@ -1,9 +1,12 @@
/*
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
use std::path::Path;
use mail_parser::mailbox::maildir::Flag;
#[derive(Debug, Clone, PartialEq, Eq, Default)]
@@ -37,6 +40,53 @@ pub fn translate_flags(flags: &[Flag], include_deleted: bool) -> Translation {
}
}
/// The translation with a message's own keywords added: those Dovecot
/// keeps as the letters `a` to `z`.
pub fn with_custom(mut translation: Translation, custom: &[String]) -> Translation {
for k in custom {
push_unique(&mut translation.keywords, k);
}
translation.keywords.sort();
translation
}
/// What the keyword letters `a` to `z` stand for in one folder, from its
/// `dovecot-keywords` file: a line `0 $Junk` names `a`, `1 work` names `b`.
pub fn read_keyword_names(folder_path: &Path) -> Vec<Option<String>> {
let mut names = vec![None; 26];
let Ok(text) = std::fs::read_to_string(folder_path.join("dovecot-keywords")) else {
return names;
};
for line in text.lines() {
let Some((index, name)) = line.split_once(' ') else {
continue;
};
let name = name.trim();
if let Ok(i) = index.parse::<usize>()
&& i < names.len()
&& !name.is_empty()
{
names[i] = Some(name.to_owned());
}
}
names
}
pub fn custom_keywords_from_filename(filename: &str, names: &[Option<String>]) -> Vec<String> {
let Some(part) = filename.rsplit_once("2,").map(|(_, s)| s) else {
return Vec::new();
};
let mut out = Vec::new();
for ch in part.bytes().take_while(u8::is_ascii_alphanumeric) {
if ch.is_ascii_lowercase()
&& let Some(Some(name)) = names.get((ch - b'a') as usize)
{
push_unique(&mut out, name);
}
}
out
}
fn push_unique(out: &mut Vec<String>, k: &str) {
if !out.iter().any(|x| x == k) {
out.push(k.to_owned());
@@ -77,6 +127,32 @@ pub fn unique_id_from_filename(filename: &str) -> &str {
mod tests {
use super::*;
#[test]
fn dovecot_keyword_letters_name_keywords() {
let td = tempfile::tempdir().unwrap();
std::fs::write(
td.path().join("dovecot-keywords"),
"0 $Junk\n1 project-x\nbad\n30 x\n",
)
.unwrap();
let names = read_keyword_names(td.path());
assert_eq!(names[0].as_deref(), Some("$Junk"));
assert_eq!(names[1].as_deref(), Some("project-x"));
assert_eq!(
custom_keywords_from_filename("1.M1.h,S=10:2,FSab", &names),
vec!["$Junk", "project-x"]
);
assert!(custom_keywords_from_filename("1.M1.h:2,Sc", &names).is_empty());
let t = with_custom(translate_flags(&[Flag::Seen], false), &["$Junk".to_owned()]);
assert_eq!(t.keywords, vec!["$Junk", "$seen"]);
}
#[test]
fn a_folder_without_dovecot_keywords_names_none() {
let td = tempfile::tempdir().unwrap();
assert!(read_keyword_names(td.path()).iter().all(Option::is_none));
}
#[test]
fn empty_flags_yields_empty_keywords() {
let t = translate_flags(&[], false);
+20 -3
View File
@@ -20,7 +20,10 @@ use crate::sync::TypeCounts;
use crate::sync::emailmeta::email_meta_from_blob;
use crate::sync::keys::index_to_json;
use super::keywords::{Translation, flags_from_filename, translate_flags, unique_id_from_filename};
use super::keywords::{
Translation, custom_keywords_from_filename, flags_from_filename, read_keyword_names,
translate_flags, unique_id_from_filename, with_custom,
};
#[derive(Debug, Clone)]
pub struct DiskEntry {
@@ -28,6 +31,9 @@ pub struct DiskEntry {
pub filename: String,
pub path: PathBuf,
pub flags: Vec<Flag>,
/// Keywords beyond the standard flags, from the folder's
/// `dovecot-keywords`.
pub keywords: Vec<String>,
pub mtime_unix: u64,
}
@@ -40,6 +46,7 @@ pub struct DiskListing {
pub fn list_folder(folder_path: &Path) -> std::io::Result<DiskListing> {
let mut out = DiskListing::default();
let mut seen: std::collections::HashMap<String, usize> = std::collections::HashMap::new();
let keyword_names = read_keyword_names(folder_path);
for sub in ["cur", "new"] {
let dir = folder_path.join(sub);
if !dir.is_dir() {
@@ -80,6 +87,7 @@ pub fn list_folder(folder_path: &Path) -> std::io::Result<DiskListing> {
continue;
}
let flags = flags_from_filename(&filename);
let keywords = custom_keywords_from_filename(&filename, &keyword_names);
let mtime_unix = match entry.metadata().and_then(|m| m.modified()).and_then(|t| {
t.duration_since(UNIX_EPOCH).map_err(|e| {
std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string())
@@ -94,6 +102,7 @@ pub fn list_folder(folder_path: &Path) -> std::io::Result<DiskListing> {
filename,
path,
flags,
keywords,
mtime_unix,
});
}
@@ -146,7 +155,10 @@ pub fn insert_new(
entry: &DiskEntry,
) -> Result<Option<i64>, InsertError> {
let bytes = std::fs::read(&entry.path).map_err(InsertError::Io)?;
let translation = translate_flags(&entry.flags, ctx.include_deleted);
let translation = with_custom(
translate_flags(&entry.flags, ctx.include_deleted),
&entry.keywords,
);
if translation.has_trashed_flag && !ctx.include_deleted {
return Ok(None);
}
@@ -219,7 +231,10 @@ pub fn refresh_flags(
stored_keywords: &str,
include_deleted: bool,
) -> Result<PresentOutcome, rusqlite::Error> {
let translation = translate_flags(&entry.flags, include_deleted);
let translation = with_custom(
translate_flags(&entry.flags, include_deleted),
&entry.keywords,
);
if translation.has_trashed_flag && !include_deleted {
return Ok(PresentOutcome::DeleteRequested);
}
@@ -554,6 +569,7 @@ mod tests {
filename: "a".into(),
path: PathBuf::new(),
flags: vec![],
keywords: vec![],
mtime_unix: 0,
},
DiskEntry {
@@ -561,6 +577,7 @@ mod tests {
filename: "b".into(),
path: PathBuf::new(),
flags: vec![],
keywords: vec![],
mtime_unix: 0,
},
];
+14 -3
View File
@@ -1,5 +1,6 @@
/*
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
@@ -104,7 +105,7 @@ pub fn discover(root: &Path, automap_enabled: bool) -> Result<Vec<ResolvedFolder
};
folders.push(ResolvedFolder {
name: canonical.clone(),
leaf: leaf.to_owned(),
leaf: display_leaf(leaf),
parent_path,
path,
role,
@@ -125,7 +126,7 @@ fn ensure_ephemeral_parents(
if known.iter().any(|n| n == &ancestor) {
continue;
}
let leaf = leaf_of(&ancestor).to_owned();
let leaf = display_leaf(leaf_of(&ancestor));
let parent_path = split_parent(&ancestor);
folders.push(ResolvedFolder {
name: ancestor.clone(),
@@ -156,6 +157,13 @@ fn split_parent(name: &str) -> Option<String> {
name.rfind('.').map(|i| name[..i].to_owned())
}
/// A folder's name as shown: Maildir++ keeps names in modified UTF-7, as
/// Dovecot and Courier write them. A name that isn't valid modified UTF-7 is
/// shown as it is on disk.
fn display_leaf(raw: &str) -> String {
crate::imap::name::decode_mailbox_name(raw).unwrap_or_else(|_| raw.to_owned())
}
fn leaf_of(name: &str) -> &str {
match name.rfind('.') {
Some(i) => &name[i + 1..],
@@ -179,7 +187,7 @@ pub fn restore_ephemeral_parents(folders: &mut Vec<ResolvedFolder>, root: &Path)
if known.contains(&ancestor) || to_add.iter().any(|x| x.name == ancestor) {
continue;
}
let leaf = leaf_of(&ancestor).to_owned();
let leaf = display_leaf(leaf_of(&ancestor));
let parent_path = split_parent(&ancestor);
to_add.push(ResolvedFolder {
name: ancestor,
@@ -378,6 +386,9 @@ mod tests {
fn leaf_and_split_handle_root_and_nested() {
assert_eq!(leaf_of("INBOX"), "INBOX");
assert_eq!(leaf_of("Archive.2025"), "2025");
assert_eq!(display_leaf("&AMk-quipe &- co"), "Équipe & co");
assert_eq!(display_leaf("Plain"), "Plain");
assert_eq!(display_leaf("Équipe"), "Équipe");
assert_eq!(split_parent("INBOX"), None);
assert_eq!(split_parent("Archive.2025"), Some("Archive".to_owned()));
}
+46 -4
View File
@@ -1,5 +1,6 @@
/*
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
@@ -233,8 +234,8 @@ fn merge_container_id(
fn parse_calendar_metadata(text: &str) -> CalendarMetadata {
let mut out = CalendarMetadata::default();
let mut inside_vevent = false;
for raw_line in text.lines() {
let line = raw_line.trim_end_matches('\r');
for line in unfolded_lines(text) {
let line = line.as_str();
if line.starts_with("BEGIN:VEVENT")
|| line.starts_with("BEGIN:VTODO")
|| line.starts_with("BEGIN:VJOURNAL")
@@ -253,9 +254,9 @@ fn parse_calendar_metadata(text: &str) -> CalendarMetadata {
continue;
}
if let Some(rest) = line.strip_prefix("X-WR-CALNAME:") {
out.x_wr_calname = Some(rest.trim().to_owned());
out.x_wr_calname = Some(unescape_text(rest.trim()));
} else if let Some(rest) = line.strip_prefix("X-WR-CALDESC:") {
out.x_wr_caldesc = Some(rest.trim().to_owned());
out.x_wr_caldesc = Some(unescape_text(rest.trim()));
} else if let Some(rest) = line.strip_prefix("X-WR-TIMEZONE:") {
out.x_wr_timezone = Some(rest.trim().to_owned());
}
@@ -263,10 +264,51 @@ fn parse_calendar_metadata(text: &str) -> CalendarMetadata {
out
}
/// Content lines with RFC 5545 folding undone.
fn unfolded_lines(text: &str) -> Vec<String> {
let mut out: Vec<String> = Vec::new();
for raw in text.lines() {
let line = raw.trim_end_matches('\r');
match (line.strip_prefix([' ', '\t']), out.last_mut()) {
(Some(rest), Some(last)) => last.push_str(rest),
_ => out.push(line.to_owned()),
}
}
out
}
/// An iCalendar TEXT value with its `\\`, `\;`, `\,` and `\n` escapes undone.
fn unescape_text(value: &str) -> String {
let mut out = String::with_capacity(value.len());
let mut chars = value.chars();
while let Some(c) = chars.next() {
if c != '\\' {
out.push(c);
continue;
}
match chars.next() {
Some('n' | 'N') => out.push('\n'),
Some(other) => out.push(other),
None => out.push('\\'),
}
}
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_metadata_unfolds_and_unescapes() {
let meta = parse_calendar_metadata(
"BEGIN:VCALENDAR\r\nX-WR-CALNAME:Team\\, Berlin\\; o\r\n ps\r\n\
X-WR-CALDESC:one\\ntwo\\\\three\r\nEND:VCALENDAR\r\n",
);
assert_eq!(meta.x_wr_calname.as_deref(), Some("Team, Berlin; ops"));
assert_eq!(meta.x_wr_caldesc.as_deref(), Some("one\ntwo\\three"));
}
#[test]
fn parse_metadata_finds_xwr_fields() {
let ical = "BEGIN:VCALENDAR\r\n\
+478
View File
@@ -0,0 +1,478 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
//! `export-files`: an archive written out as open formats, and those files
//! read back by `import maildir` and `import takeout` with nothing lost
//! that the formats can hold.
use std::collections::{BTreeMap, BTreeSet};
use std::fs;
use std::path::{Path, PathBuf};
use inbuxa_migrate::db::{blobs, init};
use inbuxa_migrate::export_files::{ExportFilesConfig, run};
use inbuxa_migrate::logging::Logger;
use inbuxa_migrate::sync::CommonConfig;
use inbuxa_migrate::sync::import_dav::calcard;
use inbuxa_migrate::sync::import_maildir::{self, MaildirImportConfig};
use inbuxa_migrate::sync::import_takeout::{self, TakeoutImportConfig};
use rusqlite::{Connection, params};
use serde_json::Value;
use tempfile::TempDir;
const VCARD_ALICE: &str = "BEGIN:VCARD\r\nVERSION:4.0\r\nUID:card-alice\r\nFN:Alice Smith\r\n\
EMAIL:[email protected]\r\nEND:VCARD\r\n";
const VCARD_BOB: &str = "BEGIN:VCARD\r\nVERSION:4.0\r\nUID:card-bob\r\nFN:Bob Jones\r\n\
TEL:+1 555 0100\r\nEND:VCARD\r\n";
const EVENT: &str = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:x\r\nBEGIN:VEVENT\r\nUID:ev-standup\r\n\
DTSTAMP:20240101T000000Z\r\nDTSTART;TZID=Europe/Berlin:20240301T090000\r\nDURATION:PT15M\r\n\
RRULE:FREQ=WEEKLY;COUNT=4\r\nSUMMARY:Standup\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n";
fn message(subject: &str) -> Vec<u8> {
format!(
"From: [email protected]\r\nTo: [email protected]\r\nSubject: {subject}\r\n\
Message-ID: <{}@example.net>\r\nDate: Fri, 01 Mar 2024 12:00:00 +0000\r\n\r\n{subject}\r\n",
subject.replace(' ', "-")
)
.into_bytes()
}
/// An archive as an import would leave it: mail in nested and non-ASCII
/// folders, a message in two folders, standard and custom keywords, two
/// address books sharing a card, a calendar, Sieve scripts, files and a link.
fn seed(path: &Path) {
let conn = init::open(path).unwrap();
conn.execute(
"INSERT INTO sources (id, kind, session_url, account_id, account_name, username)
VALUES (1, 'jmap', 'https://mail.example.org/jmap/session', 'a1', 'alice', 'alice')",
[],
)
.unwrap();
for (id, name, parent, role) in [
(1, "Inbox", None, Some("inbox")),
(2, "Clients", Some(1), None),
(3, "Archive", None, Some("archive")),
(4, "2024", Some(3), None),
(5, "Équipe & co", None, None),
(6, "v1.0 notes", None, None),
] {
conn.execute(
"INSERT INTO mailboxes (id, name, parent_id, role) VALUES (?1, ?2, ?3, ?4)",
params![id, name, parent, role],
)
.unwrap();
}
for (subject, mailboxes, keywords) in [
("Seen and flagged", "[1]", r#"["$seen","$flagged"]"#),
("Junk in two folders", "[1,3]", r#"["$Junk"]"#),
("Client note", "[2]", r#"["$answered","project-x"]"#),
("Old report", "[4]", "[]"),
("Bonjour", "[5]", r#"["$seen"]"#),
("Release notes", "[6]", "[]"),
] {
let blob = blobs::intern_blob(&conn, &message(subject)).unwrap();
conn.execute(
"INSERT INTO emails (blob_id, received_at, mailbox_ids, keywords)
VALUES (?1, '2024-03-01T12:00:00Z', ?2, ?3)",
params![blob, mailboxes, keywords],
)
.unwrap();
}
conn.execute(
"INSERT INTO address_books (id, name, is_default) VALUES (1, 'Personal', 1), (2, 'Work', 0)",
[],
)
.unwrap();
for (vcard, books) in [(VCARD_ALICE, "[1,2]"), (VCARD_BOB, "[2]")] {
let card = calcard::vcard_to_jscontact(vcard, "/x").unwrap();
conn.execute(
"INSERT INTO contact_cards (uid, address_book_ids, data) VALUES (?1, ?2, ?3)",
params![card.uid, books, card.data.to_string()],
)
.unwrap();
}
conn.execute(
"INSERT INTO calendars (id, name, description, color, time_zone, is_default)
VALUES (1, 'Team, Berlin', 'Stand-ups', '#3366ff', 'Europe/Berlin', 1)",
[],
)
.unwrap();
let mut entry = calcard::ical_to_jscalendar_entries(EVENT)
.unwrap()
.remove(0);
calcard::strip_extracted_fields_from_event(&mut entry.data);
conn.execute(
"INSERT INTO calendar_events (calendar_ids, data) VALUES ('[1]', ?1)",
params![entry.data.to_string()],
)
.unwrap();
for (name, active, script) in [
(
"filters",
1,
"require \"fileinto\";\r\nif header :contains \"subject\" \"report\" { fileinto \"Archive\"; }\r\n",
),
("old", 0, "keep;\r\n"),
] {
let blob = blobs::intern_blob(&conn, script.as_bytes()).unwrap();
conn.execute(
"INSERT INTO sieve_scripts (name, is_active, blob_id) VALUES (?1, ?2, ?3)",
params![name, active, blob],
)
.unwrap();
}
let file_blob = blobs::intern_blob(&conn, b"hello\n").unwrap();
conn.execute(
"INSERT INTO file_nodes (id, parent_id, node_type, blob_id, target, name, media_type, created)
VALUES (1, NULL, 'directory', NULL, NULL, 'Docs', NULL, '2024-01-01T00:00:00Z'),
(2, 1, 'file', ?1, NULL, 'a.txt', 'text/plain', '2024-01-01T00:00:00Z'),
(3, NULL, 'symlink', NULL, '{\"path\":\"/etc/passwd\"}', 'link', NULL, '2024-01-01T00:00:00Z'),
(4, NULL, 'file', ?1, NULL, '../escape', NULL, '2024-01-01T00:00:00Z')",
params![file_blob],
)
.unwrap();
conn.execute(
"INSERT INTO identities (name, email) VALUES ('Alice Smith', '[email protected]')",
[],
)
.unwrap();
}
struct Fixture {
_td: TempDir,
root: PathBuf,
archive: PathBuf,
out: PathBuf,
}
fn exported() -> Fixture {
let td = tempfile::tempdir().unwrap();
let root = td.path().to_path_buf();
let archive = root.join("archive.sqlite");
let out = root.join("out");
seed(&archive);
let summary = run(ExportFilesConfig {
archive: archive.clone(),
dir: out.clone(),
objects: None,
dry_run: false,
logger: Logger::from_flags(true, 0),
})
.unwrap();
assert!(!summary.any_failed(), "{:?}", summary.per_type);
Fixture {
_td: td,
root,
archive,
out,
}
}
fn common(archive: &Path) -> CommonConfig {
CommonConfig {
archive: archive.to_path_buf(),
threads: 1,
dry_run: false,
max_retries: 0,
allow_invalid_certs: false,
logger: Logger::from_flags(true, 0),
}
}
fn manifest(out: &Path) -> Value {
serde_json::from_str(&fs::read_to_string(out.join("manifest.json")).unwrap()).unwrap()
}
#[test]
fn writes_open_formats_and_a_manifest() {
let f = exported();
let mail = f.out.join("mail");
assert_eq!(fs::read_dir(mail.join("cur")).unwrap().count(), 2);
for dir in [
".INBOX.Clients",
".Archive",
".Archive.2024",
".&AMk-quipe &- co",
".v1_0 notes",
] {
assert!(mail.join(dir).join("cur").is_dir(), "{dir}");
assert!(mail.join(dir).join("maildirfolder").is_file(), "{dir}");
}
assert_eq!(
fs::read_to_string(mail.join("dovecot-keywords")).unwrap(),
"0 $Junk\n"
);
let personal = fs::read_to_string(f.out.join("contacts/Personal.vcf")).unwrap();
assert!(personal.contains("UID:card-alice") && !personal.contains("card-bob"));
let work = fs::read_to_string(f.out.join("contacts/Work.vcf")).unwrap();
assert_eq!(work.matches("BEGIN:VCARD").count(), 2);
let ics = fs::read_to_string(f.out.join("calendars/Team, Berlin.ics")).unwrap();
assert!(ics.contains("X-WR-CALNAME:Team\\, Berlin\r\n"), "{ics}");
assert!(ics.contains("UID:ev-standup"), "{ics}");
assert_eq!(ics.matches("BEGIN:VCALENDAR").count(), 1);
assert!(f.out.join("sieve/filters.sieve").is_file());
assert_eq!(
fs::read(f.out.join("files/Docs/a.txt")).unwrap(),
b"hello\n"
);
assert!(f.out.join("files/_.._escape").is_file());
assert!(!f.out.join("files/link").exists());
assert!(!f.root.join("escape").exists());
let m = manifest(&f.out);
assert_eq!(m["format"], "inbuxa-migrate-files");
assert_eq!(m["formatVersion"], 1);
assert_eq!(m["sources"][0]["kind"], "jmap");
let folders = m["mail"]["folders"].as_array().unwrap();
let inbox = folders.iter().find(|x| x["role"] == "inbox").unwrap();
assert_eq!(inbox["dir"], "mail");
assert_eq!(inbox["name"], "Inbox");
let equipe = folders
.iter()
.find(|x| x["dir"] == "mail/.&AMk-quipe &- co")
.unwrap();
assert_eq!(equipe["name"], "Équipe & co");
let dotted = folders
.iter()
.find(|x| x["dir"] == "mail/.v1_0 notes")
.unwrap();
assert_eq!(dotted["name"], "v1.0 notes");
let sieve = m["sieve"].as_array().unwrap();
assert!(
sieve
.iter()
.any(|s| s["name"] == "filters" && s["active"] == true)
);
assert_eq!(m["files"]["links"][0]["path"], "files/link");
assert_eq!(m["identities"][0]["email"], "[email protected]");
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let mode = fs::metadata(&f.out).unwrap().permissions().mode();
assert_eq!(mode & 0o077, 0, "export readable by others: {mode:o}");
}
}
/// Each message as (folder path, subject) → keywords, from an archive.
fn mail_by_folder(conn: &Connection) -> BTreeMap<(String, String), BTreeSet<String>> {
let mut paths: BTreeMap<i64, String> = BTreeMap::new();
let mut stmt = conn
.prepare("SELECT id, name, parent_id, role FROM mailboxes")
.unwrap();
let rows: Vec<(i64, String, Option<i64>, Option<String>)> = stmt
.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?)))
.unwrap()
.map(Result::unwrap)
.collect();
let name_of = |id: i64| {
let (_, name, _, role) = rows.iter().find(|r| r.0 == id).unwrap();
if role.as_deref() == Some("inbox") {
"INBOX".to_owned()
} else {
name.clone()
}
};
for (id, _, _, _) in &rows {
let mut parts = vec![name_of(*id)];
let mut parent = rows.iter().find(|r| r.0 == *id).unwrap().2;
while let Some(p) = parent {
parts.insert(0, name_of(p));
parent = rows.iter().find(|r| r.0 == p).unwrap().2;
}
paths.insert(*id, parts.join("/"));
}
let mut out = BTreeMap::new();
let mut stmt = conn
.prepare("SELECT b.data, e.mailbox_ids, e.keywords FROM emails e JOIN blobs b ON b.id = e.blob_id")
.unwrap();
let rows = stmt
.query_map([], |r| {
Ok((
r.get::<_, Vec<u8>>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
))
})
.unwrap();
for row in rows {
let (data, mailboxes, keywords) = row.unwrap();
let text = String::from_utf8(data).unwrap();
let subject = text
.lines()
.find_map(|l| l.strip_prefix("Subject: "))
.unwrap()
.to_owned();
let keywords: BTreeSet<String> = serde_json::from_str::<Vec<String>>(&keywords)
.unwrap()
.into_iter()
.map(|k| k.to_lowercase())
.collect();
for mailbox in serde_json::from_str::<Vec<i64>>(&mailboxes).unwrap() {
out.insert((paths[&mailbox].clone(), subject.clone()), keywords.clone());
}
}
out
}
#[test]
fn mail_reads_back_with_import_maildir() {
let f = exported();
let back = f.root.join("back.sqlite");
import_maildir::run(
common(&back),
MaildirImportConfig {
maildir: f.out.join("mail"),
include: Vec::new(),
exclude: Vec::new(),
folder: Vec::new(),
automap: true,
include_deleted: false,
allow_source_change: false,
},
)
.unwrap();
let original = mail_by_folder(&Connection::open(&f.archive).unwrap());
let read_back = mail_by_folder(&Connection::open(&back).unwrap());
// Maildir++ has no `.` inside a folder name; the manifest keeps it.
let expected: BTreeMap<_, _> = original
.into_iter()
.map(|((path, subject), keywords)| ((path.replace("v1.0", "v1_0"), subject), keywords))
.collect();
assert_eq!(read_back, expected);
let conn = Connection::open(&back).unwrap();
let received: String = conn
.query_row("SELECT received_at FROM emails LIMIT 1", [], |r| r.get(0))
.unwrap();
assert_eq!(received, "2024-03-01T12:00:00Z");
}
#[test]
fn contacts_and_calendars_read_back_with_import_takeout() {
let f = exported();
let back = f.root.join("back.sqlite");
import_takeout::run(
common(&back),
TakeoutImportConfig {
takeout_root: f.out.clone(),
allow_source_change: false,
automap: true,
},
)
.unwrap();
let conn = Connection::open(&back).unwrap();
let uids: BTreeSet<String> = conn
.prepare("SELECT uid FROM contact_cards")
.unwrap()
.query_map([], |r| r.get(0))
.unwrap()
.map(Result::unwrap)
.collect();
assert_eq!(
uids,
BTreeSet::from(["card-alice".to_owned(), "card-bob".to_owned()])
);
let calendar: String = conn
.query_row("SELECT name FROM calendars", [], |r| r.get(0))
.unwrap();
assert_eq!(calendar, "Team, Berlin");
let event: String = conn
.query_row("SELECT data FROM calendar_events", [], |r| r.get(0))
.unwrap();
let event: Value = serde_json::from_str(&event).unwrap();
assert_eq!(event["uid"], "ev-standup");
assert_eq!(event["title"], "Standup");
assert_eq!(event["timeZone"], "Europe/Berlin");
assert_eq!(event["start"], "2024-03-01T09:00:00");
assert_eq!(event["recurrenceRule"]["frequency"], "weekly");
}
#[test]
fn refuses_a_directory_that_is_not_empty() {
let td = tempfile::tempdir().unwrap();
let archive = td.path().join("archive.sqlite");
seed(&archive);
let out = td.path().join("out");
fs::create_dir_all(&out).unwrap();
fs::write(out.join("keep.txt"), "mine").unwrap();
let err = run(ExportFilesConfig {
archive,
dir: out.clone(),
objects: None,
dry_run: false,
logger: Logger::from_flags(true, 0),
})
.unwrap_err();
assert_eq!(err.exit_code(), 1, "{err}");
assert_eq!(fs::read_dir(&out).unwrap().count(), 1);
}
#[test]
fn a_dry_run_counts_and_writes_nothing() {
let td = tempfile::tempdir().unwrap();
let archive = td.path().join("archive.sqlite");
seed(&archive);
let out = td.path().join("out");
let summary = run(ExportFilesConfig {
archive,
dir: out.clone(),
objects: None,
dry_run: true,
logger: Logger::from_flags(true, 0),
})
.unwrap();
assert!(!out.exists());
let email = summary
.per_type
.iter()
.find(|(t, _)| *t == "email")
.unwrap();
assert_eq!(email.1.created, 6);
}
#[test]
fn objects_limits_what_is_written() {
let td = tempfile::tempdir().unwrap();
let archive = td.path().join("archive.sqlite");
seed(&archive);
let out = td.path().join("out");
run(ExportFilesConfig {
archive,
dir: out.clone(),
objects: Some(vec![inbuxa_migrate::types::ObjectType::SieveScript]),
dry_run: false,
logger: Logger::from_flags(true, 0),
})
.unwrap();
assert!(out.join("sieve").is_dir());
assert!(!out.join("mail").exists());
assert!(manifest(&out).get("mail").is_none());
}
#[test]
fn a_missing_archive_is_a_usage_error_and_creates_nothing() {
let td = tempfile::tempdir().unwrap();
let archive = td.path().join("nope.sqlite");
let err = run(ExportFilesConfig {
archive: archive.clone(),
dir: td.path().join("out"),
objects: None,
dry_run: false,
logger: Logger::from_flags(true, 0),
})
.unwrap_err();
assert_eq!(err.exit_code(), 1);
assert!(!archive.exists());
}