Add a POP3 import source #82
No files matched your search
@@ -8,6 +8,13 @@ candidates before a series ships (`27.1-rc.1`). The tag is the version with a
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- `import pop3` imports from a POP3 server: the whole maildrop into one INBOX,
|
||||
keyed by each message's UIDL so a later run adds only what arrived since.
|
||||
`pop3s://` is TLS, `pop3://` upgrades with STLS (or sends cleartext with
|
||||
`--allow-cleartext`). It is additive -- a message gone from the server is
|
||||
kept, since POP3 mailboxes are routinely emptied after download -- and it
|
||||
deletes nothing on the server. `--estimate` reports counts and sizes from
|
||||
STAT/UIDL/LIST before any body is fetched.
|
||||
- `import fastmail` imports from a Fastmail account over JMAP. It is `import
|
||||
jmap` with Fastmail's server and bearer-token sign-in filled in, so it needs
|
||||
only an API token; with no `--account-*` it imports the account Fastmail
|
||||
|
||||
@@ -39,12 +39,12 @@ later `import` into the same archive adds what arrived since. Every command
|
||||
takes `--dry-run` and then only reports the plan. An `import` also takes
|
||||
`--estimate`, which connects to the source and reports how much is new to
|
||||
fetch — counts, bytes and a rough time — before committing, and writes
|
||||
nothing. It reports sizes for `imap`, `maildir`, `caldav`, `carddav`, Exchange
|
||||
(`ews`) and a `takeout` (whose files it reads to count the mail, events and
|
||||
contacts that are new). For `jmap` and Microsoft Graph it reports counts only:
|
||||
the enumeration gives the new-object count for free, but a size there would
|
||||
cost a body fetch per object. The remaining sources fall back to a plain
|
||||
dry-run. `inspect` reads an archive and changes nothing, and `reconcile`
|
||||
nothing. It reports sizes for `imap`, `pop3`, `maildir`, `caldav`, `carddav`,
|
||||
Exchange (`ews`) and a `takeout` (whose files it reads to count the mail,
|
||||
events and contacts that are new). For `jmap` and Microsoft Graph it reports
|
||||
counts only: the enumeration gives the new-object count for free, but a size
|
||||
there would cost a body fetch per object. The remaining sources fall back to a
|
||||
plain dry-run. `inspect` reads an archive and changes nothing, and `reconcile`
|
||||
reads it folder by folder: a count and a content checksum for each, a checksum
|
||||
for the whole, and a check that every message resolves to stored content and a
|
||||
real folder. Compare the per-folder counts against the old mailbox to prove the
|
||||
@@ -83,6 +83,9 @@ command reference is in [docs/usage.md](docs/usage.md).
|
||||
token. Mail always; calendars and contacts where the account advertises
|
||||
them.
|
||||
- **IMAP** -- mail only. Folders are chosen by name or by pattern.
|
||||
- **POP3** -- mail only, the whole maildrop into one INBOX, keyed by UIDL. It
|
||||
deletes nothing on the server and keeps a message even once the server no
|
||||
longer has it.
|
||||
- **CalDAV, CardDAV** -- calendars and events, address books and contacts.
|
||||
- **WebDAV** -- a plain file collection, kept as a file tree.
|
||||
- **ManageSieve** -- Sieve scripts only, with the active one recorded.
|
||||
|
||||
@@ -179,6 +179,42 @@ the variable that holds it as `"secrets": {"googleKey": "..."}`.
|
||||
Each connection signs in with a token for that user, and takes a new one
|
||||
when the last is close to running out.
|
||||
|
||||
### POP3
|
||||
|
||||
```
|
||||
inbuxa-migrate import pop3 \
|
||||
--url pop3s://host[:port] | pop3://host[:port] \
|
||||
--user <USER> [--password <PASS>] \
|
||||
[--allow-cleartext] [--allow-source-change] \
|
||||
<ARCHIVE>
|
||||
```
|
||||
|
||||
Imports mail, and only mail, from a POP3 server. POP3 has no folders and no
|
||||
flags, so the whole maildrop lands in one INBOX. Each message is keyed by its
|
||||
UIDL -- the stable unique-id POP3 gives it -- so a later run into the same
|
||||
archive adds only what arrived since.
|
||||
|
||||
`pop3s://` is TLS from the first byte (port 995 by default). `pop3://` (port
|
||||
110) upgrades with STLS before the password is sent; `--allow-cleartext`
|
||||
permits a plain `pop3://` connection with no STLS, sending the password in the
|
||||
clear, for a trusted local network only. The password comes from `--password`,
|
||||
`$INBUXA_MIGRATE_PASSWORD`, or the prompt (prefer the last two; a password on
|
||||
the command line is visible in the shell history and the process list).
|
||||
|
||||
Two things set POP3 apart from the other sources:
|
||||
|
||||
- **It is additive.** A message that is in the archive but no longer on the
|
||||
server is kept, not deleted. POP3 mailboxes are routinely emptied after
|
||||
download -- that is the protocol's own model -- so a message leaving the
|
||||
server says nothing about whether the archive should still hold it. This is
|
||||
the one importer that does not mirror deletions.
|
||||
- **It never changes the source.** There is no `DELE`: a run only reads, so it
|
||||
is safe to repeat and leaves the server mailbox exactly as it found it.
|
||||
|
||||
OAuth (SASL XOAUTH2) is not offered here; use `--user`/password over TLS. A
|
||||
provider that requires an app-specific password (for an account with
|
||||
two-factor sign-in) takes that password the same way.
|
||||
|
||||
### CalDAV
|
||||
|
||||
```
|
||||
|
||||
+76
-1
@@ -36,6 +36,7 @@ use crate::sync::import_files::FilesImportConfig;
|
||||
use crate::sync::import_imap::{ImapAuth, ImapImportConfig};
|
||||
use crate::sync::import_maildir::MaildirImportConfig;
|
||||
use crate::sync::import_managesieve::{ManageSieveAuth, ManageSieveImportConfig};
|
||||
use crate::sync::import_pop3::Pop3ImportConfig;
|
||||
use crate::sync::import_takeout::TakeoutImportConfig;
|
||||
use crate::sync::{CommonConfig, ConnectConfig, ExportConfig, ImportConfig};
|
||||
use crate::types::{ObjectType, parse_object_list};
|
||||
@@ -144,6 +145,14 @@ enum Source {
|
||||
Fastmail(Box<FastmailImportArgs>),
|
||||
#[command(about = "IMAP server")]
|
||||
Imap(Box<ImapImportArgs>),
|
||||
#[command(
|
||||
about = "POP3 server (mail only)",
|
||||
long_about = "POP3: the whole maildrop into one INBOX, keyed by each message's UIDL so a \
|
||||
later run adds only what arrived since. POP3 has no folders and no flags. This never \
|
||||
deletes anything on the server, and it keeps a message already in the archive even \
|
||||
once the server no longer has it."
|
||||
)]
|
||||
Pop3(Box<Pop3ImportArgs>),
|
||||
#[command(about = "CalDAV server")]
|
||||
Caldav(Box<DavImportArgs>),
|
||||
#[command(about = "CardDAV server")]
|
||||
@@ -185,7 +194,7 @@ struct GlobalArgs {
|
||||
help = "Preview an import: counts, sizes and a rough time, before any writes",
|
||||
long_help = "Preview an import before committing: for each type, how much is new since \
|
||||
the archive was last filled, how many bytes that is, and a rough time.\n \
|
||||
Sizes for imap, maildir, caldav, carddav, exchange (EWS) and takeout (mail, events \
|
||||
Sizes for imap, pop3, maildir, caldav, carddav, exchange (EWS) and takeout (mail, events \
|
||||
and contacts); counts only for jmap and Graph (a size there needs a body fetch per \
|
||||
object). Other sources fall back to --dry-run with a note. It implies --dry-run, so \
|
||||
it never writes. On an export it is just --dry-run."
|
||||
@@ -996,6 +1005,7 @@ struct InspectArgs {
|
||||
pub enum Action {
|
||||
Import(CommonConfig, ImportConfig),
|
||||
ImportImap(CommonConfig, ImapImportConfig),
|
||||
ImportPop3(CommonConfig, Pop3ImportConfig),
|
||||
ImportDav(CommonConfig, DavImportConfig),
|
||||
ImportManageSieve(CommonConfig, ManageSieveImportConfig),
|
||||
ImportMaildir(CommonConfig, MaildirImportConfig),
|
||||
@@ -1026,6 +1036,7 @@ impl Action {
|
||||
match self {
|
||||
Action::Import(c, _)
|
||||
| Action::ImportImap(c, _)
|
||||
| Action::ImportPop3(c, _)
|
||||
| Action::ImportDav(c, _)
|
||||
| Action::ImportManageSieve(c, _)
|
||||
| Action::ImportMaildir(c, _)
|
||||
@@ -1044,6 +1055,9 @@ impl Action {
|
||||
pub fn estimate_available(&self) -> bool {
|
||||
match self {
|
||||
Action::ImportImap(c, _) | Action::ImportMaildir(c, _) => c.estimate,
|
||||
// POP3 LIST sizes every new message, paired with UIDL, before any
|
||||
// body is fetched.
|
||||
Action::ImportPop3(c, _) => c.estimate,
|
||||
// CalDAV and CardDAV size their items from the getcontentlength
|
||||
// the enumeration PROPFIND already returns. WebDAV files go
|
||||
// through the same command but report differently, so they stay
|
||||
@@ -1064,6 +1078,66 @@ impl Action {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Args)]
|
||||
pub struct Pop3ImportArgs {
|
||||
#[arg(
|
||||
long,
|
||||
value_name = "URL",
|
||||
help = "pop3s://host[:port] (TLS) or pop3://host[:port] (STLS, or cleartext with --allow-cleartext)"
|
||||
)]
|
||||
url: String,
|
||||
|
||||
#[arg(long, value_name = "USER", help = "The POP3 account to sign in as")]
|
||||
user: String,
|
||||
|
||||
#[arg(
|
||||
long,
|
||||
value_name = "PASS",
|
||||
help = "Password (prefer $INBUXA_MIGRATE_PASSWORD or the prompt)"
|
||||
)]
|
||||
password: Option<String>,
|
||||
|
||||
#[arg(
|
||||
long,
|
||||
help = "Permit a password on pop3:// without STLS (cleartext). Ignored for pop3s://"
|
||||
)]
|
||||
allow_cleartext: bool,
|
||||
|
||||
#[arg(
|
||||
long,
|
||||
help = "Permit importing when the archive records a different POP3 source"
|
||||
)]
|
||||
allow_source_change: bool,
|
||||
|
||||
#[command(flatten)]
|
||||
global: GlobalArgs,
|
||||
|
||||
#[arg(
|
||||
value_name = "ARCHIVE",
|
||||
help = "Local SQLite archive (created if absent)"
|
||||
)]
|
||||
archive: PathBuf,
|
||||
}
|
||||
|
||||
fn resolve_pop3_import(args: Pop3ImportArgs) -> Result<Action, Error> {
|
||||
let common = common_config(&args.global, args.archive);
|
||||
let password = secret::resolve(
|
||||
args.password.as_deref(),
|
||||
"INBUXA_MIGRATE_PASSWORD",
|
||||
"password",
|
||||
)?;
|
||||
Ok(Action::ImportPop3(
|
||||
common,
|
||||
Pop3ImportConfig {
|
||||
url: args.url,
|
||||
user: args.user,
|
||||
password,
|
||||
allow_cleartext: args.allow_cleartext,
|
||||
allow_source_change: args.allow_source_change,
|
||||
},
|
||||
))
|
||||
}
|
||||
|
||||
#[derive(Args)]
|
||||
#[command(group(clap::ArgGroup::new("auth").required(true).args(["auth_basic", "auth_bearer"])))]
|
||||
pub struct ManageSieveImportArgs {
|
||||
@@ -1225,6 +1299,7 @@ fn resolve_import(source: Source) -> Result<Action, Error> {
|
||||
Source::Jmap(args) => resolve_jmap_import(*args),
|
||||
Source::Fastmail(args) => resolve_fastmail_import(*args),
|
||||
Source::Imap(args) => resolve_imap_import(*args),
|
||||
Source::Pop3(args) => resolve_pop3_import(*args),
|
||||
Source::Caldav(args) => resolve_dav_import(*args, DavKindArg::Caldav),
|
||||
Source::Carddav(args) => resolve_dav_import(*args, DavKindArg::Carddav),
|
||||
Source::Webdav(args) => resolve_dav_import(*args, DavKindArg::Webdav),
|
||||
|
||||
+17
-12
@@ -128,7 +128,7 @@ fn schema_tables() -> impl Iterator<Item = &'static str> {
|
||||
pub fn apply_schema(conn: &Connection) -> Result<(), OpenError> {
|
||||
// Before the transaction: it turns foreign keys off, which SQLite only
|
||||
// allows outside one.
|
||||
ensure_sources_accept_files(conn)?;
|
||||
ensure_sources_kind_current(conn)?;
|
||||
let tx = conn.unchecked_transaction()?;
|
||||
tx.execute_batch(SCHEMA_SQL)?;
|
||||
ensure_calendar_events_data_type(&tx)?;
|
||||
@@ -253,7 +253,7 @@ fn ensure_dav_ids_share_local_rows(conn: &Connection) -> Result<(), OpenError> {
|
||||
/// in place, so an older archive's table is rebuilt. Foreign keys are off
|
||||
/// while it is: with them on, dropping the old table would cascade into
|
||||
/// every row that came from a source.
|
||||
fn ensure_sources_accept_files(conn: &Connection) -> Result<(), OpenError> {
|
||||
fn ensure_sources_kind_current(conn: &Connection) -> Result<(), OpenError> {
|
||||
let sql: Option<String> = conn
|
||||
.query_row(
|
||||
"SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'sources'",
|
||||
@@ -261,10 +261,14 @@ fn ensure_sources_accept_files(conn: &Connection) -> Result<(), OpenError> {
|
||||
|row| row.get(0),
|
||||
)
|
||||
.optional()?;
|
||||
// Only the table as earlier releases made it: its kind list ended at
|
||||
// `takeout`.
|
||||
// A table from an earlier release whose kind CHECK predates a source we
|
||||
// now write (most recently `pop3`, before that `files`). Rebuilding with
|
||||
// the current list is always safe: the new list is a superset of every
|
||||
// kind we have ever shipped, so copying the rows can violate no CHECK.
|
||||
let Some(sql) = sql else { return Ok(()) };
|
||||
if sql.contains("'files'") || !sql.contains("'takeout')") {
|
||||
// Already current, or not a kind CHECK we recognize (every shipped list has
|
||||
// held 'takeout'): leave it. Otherwise rebuild with the current list.
|
||||
if sql.contains("'pop3'") || !sql.contains("'takeout'") {
|
||||
return Ok(());
|
||||
}
|
||||
let foreign_keys: i64 = conn.query_row("PRAGMA foreign_keys", [], |row| row.get(0))?;
|
||||
@@ -281,7 +285,7 @@ fn rebuild_sources(conn: &Connection) -> Result<(), OpenError> {
|
||||
tx.execute_batch(
|
||||
"CREATE TABLE sources_new (
|
||||
id INTEGER PRIMARY KEY,
|
||||
kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout','files')),
|
||||
kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','pop3','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout','files')),
|
||||
session_url TEXT NOT NULL,
|
||||
account_id TEXT NOT NULL,
|
||||
account_name TEXT,
|
||||
@@ -556,10 +560,11 @@ mod migration_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn an_older_sources_table_is_rebuilt_to_accept_files_and_keeps_every_row() {
|
||||
fn an_older_sources_table_is_rebuilt_to_accept_pop3_and_keeps_every_row() {
|
||||
let conn = Connection::open_in_memory().unwrap();
|
||||
// The table as releases before `import files` made it.
|
||||
let old = SCHEMA_SQL.replace("'takeout','files')", "'takeout')");
|
||||
// The table as releases before `import pop3` made it: every other kind,
|
||||
// but not pop3.
|
||||
let old = SCHEMA_SQL.replace("'imap','pop3',", "'imap',");
|
||||
assert_ne!(old, SCHEMA_SQL);
|
||||
conn.execute_batch(&old).unwrap();
|
||||
conn.pragma_update(None, "foreign_keys", "ON").unwrap();
|
||||
@@ -572,7 +577,7 @@ mod migration_tests {
|
||||
assert!(
|
||||
conn.execute(
|
||||
"INSERT INTO sources (kind, session_url, account_id, username)
|
||||
VALUES ('files', 'file:///e', '/e', '')",
|
||||
VALUES ('pop3', 'pop3s://h:995', 'a', 'a')",
|
||||
[],
|
||||
)
|
||||
.is_err(),
|
||||
@@ -583,10 +588,10 @@ mod migration_tests {
|
||||
|
||||
conn.execute(
|
||||
"INSERT INTO sources (kind, session_url, account_id, username)
|
||||
VALUES ('files', 'file:///e', '/e', '')",
|
||||
VALUES ('pop3', 'pop3s://h:995', 'a', 'a')",
|
||||
[],
|
||||
)
|
||||
.expect("files is a source kind now");
|
||||
.expect("pop3 is a source kind now");
|
||||
let kept: (String, i64) = conn
|
||||
.query_row(
|
||||
"SELECT s.kind, m.local_id FROM sync_id_maildir m JOIN sources s ON s.id = m.source_id",
|
||||
|
||||
@@ -17,6 +17,7 @@ pub mod imap_state;
|
||||
pub mod init;
|
||||
pub mod maildir_ids;
|
||||
pub mod managesieve_ids;
|
||||
pub mod pop3_ids;
|
||||
pub mod roles;
|
||||
pub mod sources;
|
||||
pub mod sync_state_jmap;
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! The id map for a POP3 source: which archive row each UIDL became, and the
|
||||
//! single INBOX mailbox. POP3 is flat, so there is no folder column -- unlike
|
||||
//! the maildir map it mirrors.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
|
||||
const MAILBOX: &str = "mailbox";
|
||||
const EMAIL: &str = "email";
|
||||
|
||||
pub fn insert_mailbox(
|
||||
conn: &Connection,
|
||||
source_id: i64,
|
||||
local_id: i64,
|
||||
) -> Result<(), rusqlite::Error> {
|
||||
conn.execute(
|
||||
"INSERT INTO sync_id_pop3 (source_id, type_name, unique_id, local_id)
|
||||
VALUES (?1, ?2, '', ?3)",
|
||||
params![source_id, MAILBOX, local_id],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn local_for_mailbox(
|
||||
conn: &Connection,
|
||||
source_id: i64,
|
||||
) -> Result<Option<i64>, rusqlite::Error> {
|
||||
conn.query_row(
|
||||
"SELECT local_id FROM sync_id_pop3
|
||||
WHERE source_id = ?1 AND type_name = ?2 AND unique_id = ''",
|
||||
params![source_id, MAILBOX],
|
||||
|row| row.get(0),
|
||||
)
|
||||
.optional()
|
||||
}
|
||||
|
||||
pub fn insert_email(
|
||||
conn: &Connection,
|
||||
source_id: i64,
|
||||
unique_id: &str,
|
||||
local_id: i64,
|
||||
) -> Result<(), rusqlite::Error> {
|
||||
conn.execute(
|
||||
"INSERT INTO sync_id_pop3 (source_id, type_name, unique_id, local_id)
|
||||
VALUES (?1, ?2, ?3, ?4)",
|
||||
params![source_id, EMAIL, unique_id, local_id],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every UIDL already stored for this source, mapped to its archive row, so a
|
||||
/// later run can tell what is new.
|
||||
pub fn email_ids(
|
||||
conn: &Connection,
|
||||
source_id: i64,
|
||||
) -> Result<HashMap<String, i64>, rusqlite::Error> {
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT unique_id, local_id FROM sync_id_pop3
|
||||
WHERE source_id = ?1 AND type_name = ?2",
|
||||
)?;
|
||||
let rows = stmt.query_map(params![source_id, EMAIL], |row| {
|
||||
Ok((row.get::<_, String>(0)?, row.get::<_, i64>(1)?))
|
||||
})?;
|
||||
let mut map = HashMap::new();
|
||||
for r in rows {
|
||||
let (u, l) = r?;
|
||||
map.insert(u, l);
|
||||
}
|
||||
Ok(map)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::db::init;
|
||||
use crate::db::sources::{SourceKey, upsert_source};
|
||||
|
||||
fn archive() -> (Connection, i64) {
|
||||
let c = Connection::open_in_memory().unwrap();
|
||||
init::apply_schema(&c).unwrap();
|
||||
let sid = upsert_source(
|
||||
&c,
|
||||
&SourceKey {
|
||||
kind: "pop3".to_owned(),
|
||||
session_url: "pop3s://mail.example.org:995".to_owned(),
|
||||
account_id: "alice".to_owned(),
|
||||
},
|
||||
Some("alice"),
|
||||
"alice",
|
||||
)
|
||||
.unwrap();
|
||||
(c, sid)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mailbox_round_trips() {
|
||||
let (c, sid) = archive();
|
||||
assert_eq!(local_for_mailbox(&c, sid).unwrap(), None);
|
||||
insert_mailbox(&c, sid, 42).unwrap();
|
||||
assert_eq!(local_for_mailbox(&c, sid).unwrap(), Some(42));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn emails_map_by_uidl() {
|
||||
let (c, sid) = archive();
|
||||
insert_email(&c, sid, "uid-a", 1).unwrap();
|
||||
insert_email(&c, sid, "uid-b", 2).unwrap();
|
||||
let map = email_ids(&c, sid).unwrap();
|
||||
assert_eq!(map.get("uid-a"), Some(&1));
|
||||
assert_eq!(map.get("uid-b"), Some(&2));
|
||||
assert_eq!(map.len(), 2);
|
||||
}
|
||||
}
|
||||
+12
-1
@@ -1,6 +1,6 @@
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id INTEGER PRIMARY KEY,
|
||||
kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout','files')),
|
||||
kind TEXT NOT NULL CHECK (kind IN ('jmap','imap','pop3','caldav','carddav','webdav','managesieve','maildir','exchange_ews','exchange_graph','takeout','files')),
|
||||
session_url TEXT NOT NULL,
|
||||
account_id TEXT NOT NULL,
|
||||
account_name TEXT,
|
||||
@@ -94,6 +94,17 @@ CREATE TABLE IF NOT EXISTS sync_id_maildir (
|
||||
UNIQUE (source_id, type_name, local_id)
|
||||
);
|
||||
|
||||
-- POP3 has no folders: one INBOX per source, messages keyed by their stable
|
||||
-- UIDL. The mailbox row uses type_name 'mailbox' with unique_id ''.
|
||||
CREATE TABLE IF NOT EXISTS sync_id_pop3 (
|
||||
source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
|
||||
type_name TEXT NOT NULL CHECK (type_name IN ('mailbox','email')),
|
||||
unique_id TEXT NOT NULL,
|
||||
local_id INTEGER NOT NULL,
|
||||
PRIMARY KEY (source_id, type_name, unique_id),
|
||||
UNIQUE (source_id, type_name, local_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS sync_id_maildir_folder_idx
|
||||
ON sync_id_maildir (source_id, type_name, folder);
|
||||
|
||||
|
||||
@@ -24,6 +24,7 @@ pub mod logging;
|
||||
pub mod managesieve;
|
||||
pub mod migrate;
|
||||
pub mod net;
|
||||
pub mod pop3;
|
||||
pub mod profile;
|
||||
pub mod reconcile;
|
||||
pub mod secret;
|
||||
|
||||
@@ -69,6 +69,10 @@ fn run() -> i32 {
|
||||
let logger = common.logger;
|
||||
(sync::import_imap::run_reporting(common, config), logger)
|
||||
}
|
||||
Action::ImportPop3(common, config) => {
|
||||
let logger = common.logger;
|
||||
(sync::import_pop3::run_reporting(common, config), logger)
|
||||
}
|
||||
Action::ImportDav(common, config) => {
|
||||
let logger = common.logger;
|
||||
(sync::import_dav::run_reporting(common, config), logger)
|
||||
|
||||
@@ -0,0 +1,496 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! A small POP3 client (RFC 1939, STLS from RFC 2595), enough to read a
|
||||
//! mailbox once for migration: sign in, list the messages with their stable
|
||||
//! unique-ids and sizes, fetch each, and quit. It does not delete anything on
|
||||
//! the server. The TLS transport is shared with the IMAP client -- the same
|
||||
//! `Connector` and stream -- so STLS is an `upgrade_tls` on the stream.
|
||||
|
||||
use std::io::Write;
|
||||
|
||||
use crate::imap::transport::{Connector, ImapStream};
|
||||
|
||||
use super::error::Pop3Error;
|
||||
|
||||
/// How to reach the server: a TLS port from the first byte, a plain port that
|
||||
/// upgrades with STLS, or (only when the caller allows it) plain throughout.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum Pop3Mode {
|
||||
Plain,
|
||||
StartTls,
|
||||
ImplicitTls,
|
||||
}
|
||||
|
||||
/// One message as the server lists it: its number for this session, its stable
|
||||
/// unique-id (UIDL), and its size in octets.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct MessageListing {
|
||||
pub number: u32,
|
||||
pub uid: String,
|
||||
pub octets: u64,
|
||||
}
|
||||
|
||||
const READ_CHUNK: usize = 16 * 1024;
|
||||
|
||||
pub struct Pop3Client {
|
||||
stream: Box<dyn ImapStream>,
|
||||
rx: Vec<u8>,
|
||||
rx_pos: usize,
|
||||
}
|
||||
|
||||
impl Pop3Client {
|
||||
/// Connect, read the greeting, and (for `StartTls`) upgrade before anything
|
||||
/// secret is sent. The returned client is connected but not signed in.
|
||||
pub fn connect(
|
||||
connector: &Connector,
|
||||
host: &str,
|
||||
port: u16,
|
||||
mode: Pop3Mode,
|
||||
) -> Result<Pop3Client, Pop3Error> {
|
||||
let stream = match mode {
|
||||
Pop3Mode::ImplicitTls => connector.connect_tls(host, port)?,
|
||||
Pop3Mode::Plain | Pop3Mode::StartTls => connector.connect_plain(host, port)?,
|
||||
};
|
||||
let mut client = Pop3Client {
|
||||
stream,
|
||||
rx: Vec::new(),
|
||||
rx_pos: 0,
|
||||
};
|
||||
// The greeting is one status line.
|
||||
client.read_status()?;
|
||||
if mode == Pop3Mode::StartTls {
|
||||
client.start_tls(connector, host)?;
|
||||
}
|
||||
Ok(client)
|
||||
}
|
||||
|
||||
/// Build a client over a stream that is already connected and past its
|
||||
/// greeting -- used by the tests to drive the protocol without a socket.
|
||||
#[cfg(test)]
|
||||
pub fn from_stream(stream: Box<dyn ImapStream>) -> Pop3Client {
|
||||
Pop3Client {
|
||||
stream,
|
||||
rx: Vec::new(),
|
||||
rx_pos: 0,
|
||||
}
|
||||
}
|
||||
|
||||
fn start_tls(&mut self, connector: &Connector, host: &str) -> Result<(), Pop3Error> {
|
||||
self.send("STLS")?;
|
||||
self.read_status()?;
|
||||
// The buffer must be empty before the handshake: a conformant server
|
||||
// sends nothing between the "+OK" and the TLS ClientHello, so anything
|
||||
// buffered would be a prefix of the handshake we must not swallow.
|
||||
if self.rx_pos < self.rx.len() {
|
||||
return Err(Pop3Error::Protocol(
|
||||
"server sent data after STLS +OK but before the TLS handshake".to_owned(),
|
||||
));
|
||||
}
|
||||
self.rx.clear();
|
||||
self.rx_pos = 0;
|
||||
let stream = std::mem::replace(&mut self.stream, Box::new(Dead));
|
||||
self.stream = stream.upgrade_tls(connector, host)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Sign in with a username and password (USER/PASS, RFC 1939). Over a plain
|
||||
/// connection this sends the password in the clear, which the caller gates.
|
||||
pub fn login(&mut self, user: &str, password: &str) -> Result<(), Pop3Error> {
|
||||
self.send(&format!("USER {user}"))?;
|
||||
self.read_status()
|
||||
.map_err(|e| Pop3Error::Auth(format!("USER refused: {e}")))?;
|
||||
self.send(&format!("PASS {password}"))?;
|
||||
self.read_status()
|
||||
.map_err(|e| Pop3Error::Auth(format!("PASS refused: {e}")))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `STAT`: the message count and total size of the maildrop.
|
||||
pub fn stat(&mut self) -> Result<(u64, u64), Pop3Error> {
|
||||
self.send("STAT")?;
|
||||
let line = self.read_status()?;
|
||||
let mut parts = line.split_whitespace();
|
||||
let count = parts
|
||||
.next()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.ok_or_else(|| Pop3Error::Protocol(format!("STAT: no count in {line:?}")))?;
|
||||
let octets = parts
|
||||
.next()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.ok_or_else(|| Pop3Error::Protocol(format!("STAT: no size in {line:?}")))?;
|
||||
Ok((count, octets))
|
||||
}
|
||||
|
||||
/// Every message's number, UIDL and size, by pairing `UIDL` and `LIST`.
|
||||
/// A message `UIDL` lists but `LIST` does not (or the reverse) is dropped
|
||||
/// with no size rather than guessed at.
|
||||
pub fn listing(&mut self) -> Result<Vec<MessageListing>, Pop3Error> {
|
||||
let uids = self.uidl()?;
|
||||
let sizes = self.list()?;
|
||||
let mut out = Vec::with_capacity(uids.len());
|
||||
for (number, uid) in uids {
|
||||
if let Some(&octets) = sizes.get(&number) {
|
||||
out.push(MessageListing {
|
||||
number,
|
||||
uid,
|
||||
octets,
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// `UIDL`: each message's number and its unique-id, stable across sessions.
|
||||
fn uidl(&mut self) -> Result<Vec<(u32, String)>, Pop3Error> {
|
||||
self.send("UIDL")?;
|
||||
self.read_status()?;
|
||||
let lines = self.read_multiline()?;
|
||||
let mut out = Vec::with_capacity(lines.len());
|
||||
for line in lines {
|
||||
let text = String::from_utf8_lossy(&line);
|
||||
let mut parts = text.split_whitespace();
|
||||
let number = parts
|
||||
.next()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.ok_or_else(|| Pop3Error::Protocol(format!("UIDL: no number in {text:?}")))?;
|
||||
let uid = parts
|
||||
.next()
|
||||
.ok_or_else(|| Pop3Error::Protocol(format!("UIDL: no unique-id in {text:?}")))?;
|
||||
out.push((number, uid.to_owned()));
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// `LIST`: each message's number and size in octets.
|
||||
fn list(&mut self) -> Result<std::collections::HashMap<u32, u64>, Pop3Error> {
|
||||
self.send("LIST")?;
|
||||
self.read_status()?;
|
||||
let lines = self.read_multiline()?;
|
||||
let mut out = std::collections::HashMap::with_capacity(lines.len());
|
||||
for line in lines {
|
||||
let text = String::from_utf8_lossy(&line);
|
||||
let mut parts = text.split_whitespace();
|
||||
let number = parts
|
||||
.next()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.ok_or_else(|| Pop3Error::Protocol(format!("LIST: no number in {text:?}")))?;
|
||||
let octets = parts
|
||||
.next()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.ok_or_else(|| Pop3Error::Protocol(format!("LIST: no size in {text:?}")))?;
|
||||
out.insert(number, octets);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// `RETR`: the whole message, as RFC 822 bytes with CRLF line endings,
|
||||
/// dot-unstuffed and with the terminating dot removed.
|
||||
pub fn retr(&mut self, number: u32) -> Result<Vec<u8>, Pop3Error> {
|
||||
self.send(&format!("RETR {number}"))?;
|
||||
self.read_status()?;
|
||||
let lines = self.read_multiline()?;
|
||||
let mut body = Vec::new();
|
||||
for line in lines {
|
||||
body.extend_from_slice(&line);
|
||||
body.extend_from_slice(b"\r\n");
|
||||
}
|
||||
Ok(body)
|
||||
}
|
||||
|
||||
/// `QUIT`, best effort: the caller is done, so a failure to read the final
|
||||
/// `+OK` does not matter.
|
||||
pub fn quit(&mut self) {
|
||||
if self.send("QUIT").is_ok() {
|
||||
let _ = self.read_status();
|
||||
}
|
||||
}
|
||||
|
||||
fn send(&mut self, command: &str) -> Result<(), Pop3Error> {
|
||||
self.stream.write_all(command.as_bytes())?;
|
||||
self.stream.write_all(b"\r\n")?;
|
||||
self.stream.flush()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Read one status line and return the text after `+OK`, or map `-ERR` to
|
||||
/// an error carrying the server's message.
|
||||
fn read_status(&mut self) -> Result<String, Pop3Error> {
|
||||
let line = self.next_line()?;
|
||||
let text = String::from_utf8_lossy(&line);
|
||||
if let Some(rest) = text.strip_prefix("+OK") {
|
||||
Ok(rest.trim().to_owned())
|
||||
} else if let Some(rest) = text.strip_prefix("-ERR") {
|
||||
Err(Pop3Error::Err(rest.trim().to_owned()))
|
||||
} else {
|
||||
Err(Pop3Error::Protocol(format!(
|
||||
"expected +OK or -ERR, got {text:?}"
|
||||
)))
|
||||
}
|
||||
}
|
||||
|
||||
/// Read a dot-terminated multiline response: the data lines, each with any
|
||||
/// leading dot unstuffed and the trailing CRLF removed, up to the lone `.`.
|
||||
fn read_multiline(&mut self) -> Result<Vec<Vec<u8>>, Pop3Error> {
|
||||
let mut lines = Vec::new();
|
||||
loop {
|
||||
let mut line = self.next_line()?;
|
||||
if line == b"." {
|
||||
return Ok(lines);
|
||||
}
|
||||
// Dot-stuffing: a data line that begins with '.' had one prepended.
|
||||
if line.first() == Some(&b'.') {
|
||||
line.remove(0);
|
||||
}
|
||||
lines.push(line);
|
||||
}
|
||||
}
|
||||
|
||||
/// One line, without its CRLF. A `\n` ends it; a bare `\r` does not. EOF
|
||||
/// before a newline is a protocol error -- a line, or the terminating dot,
|
||||
/// never arrived.
|
||||
fn next_line(&mut self) -> Result<Vec<u8>, Pop3Error> {
|
||||
loop {
|
||||
if let Some(rel) = self.rx[self.rx_pos..].iter().position(|&b| b == b'\n') {
|
||||
let end = self.rx_pos + rel;
|
||||
let mut line = self.rx[self.rx_pos..end].to_vec();
|
||||
if line.last() == Some(&b'\r') {
|
||||
line.pop();
|
||||
}
|
||||
self.rx_pos = end + 1;
|
||||
return Ok(line);
|
||||
}
|
||||
if !self.fill()? {
|
||||
return Err(Pop3Error::Protocol("connection closed mid-line".to_owned()));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Pull more bytes into the buffer. Returns false at end of stream. Drops
|
||||
/// the already-consumed prefix first, so the buffer does not grow without
|
||||
/// bound over a large RETR.
|
||||
fn fill(&mut self) -> Result<bool, Pop3Error> {
|
||||
if self.rx_pos > 0 {
|
||||
self.rx.drain(..self.rx_pos);
|
||||
self.rx_pos = 0;
|
||||
}
|
||||
let start = self.rx.len();
|
||||
self.rx.resize(start + READ_CHUNK, 0);
|
||||
let n = self.stream.read(&mut self.rx[start..])?;
|
||||
self.rx.truncate(start + n);
|
||||
Ok(n > 0)
|
||||
}
|
||||
}
|
||||
|
||||
/// A placeholder stream, held for the instant between taking the real stream
|
||||
/// out to upgrade it and putting the TLS stream back. It is never used.
|
||||
struct Dead;
|
||||
|
||||
impl std::io::Read for Dead {
|
||||
fn read(&mut self, _: &mut [u8]) -> std::io::Result<usize> {
|
||||
Err(std::io::Error::other("stream taken for TLS upgrade"))
|
||||
}
|
||||
}
|
||||
impl Write for Dead {
|
||||
fn write(&mut self, _: &[u8]) -> std::io::Result<usize> {
|
||||
Err(std::io::Error::other("stream taken for TLS upgrade"))
|
||||
}
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
impl ImapStream for Dead {
|
||||
fn upgrade_tls(
|
||||
self: Box<Self>,
|
||||
_: &Connector,
|
||||
_: &str,
|
||||
) -> Result<Box<dyn ImapStream>, crate::imap::error::ImapError> {
|
||||
Err(crate::imap::error::ImapError::Protocol(
|
||||
"dead stream".into(),
|
||||
))
|
||||
}
|
||||
fn is_tls(&self) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::io::{self, Read};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
/// A scripted in-memory peer: it serves `inbound` bytes to reads and
|
||||
/// records everything written, so a test can assert the commands sent.
|
||||
#[derive(Clone)]
|
||||
struct Script {
|
||||
inbound: Arc<Mutex<Vec<u8>>>,
|
||||
inbound_pos: Arc<Mutex<usize>>,
|
||||
outbound: Arc<Mutex<Vec<u8>>>,
|
||||
}
|
||||
|
||||
impl Script {
|
||||
fn new(inbound: &[u8]) -> Self {
|
||||
Script {
|
||||
inbound: Arc::new(Mutex::new(inbound.to_vec())),
|
||||
inbound_pos: Arc::new(Mutex::new(0)),
|
||||
outbound: Arc::new(Mutex::new(Vec::new())),
|
||||
}
|
||||
}
|
||||
fn sent(&self) -> String {
|
||||
String::from_utf8_lossy(&self.outbound.lock().unwrap()).into_owned()
|
||||
}
|
||||
}
|
||||
|
||||
impl Read for Script {
|
||||
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
|
||||
let inbound = self.inbound.lock().unwrap();
|
||||
let mut pos = self.inbound_pos.lock().unwrap();
|
||||
if *pos >= inbound.len() {
|
||||
return Ok(0);
|
||||
}
|
||||
let avail = &inbound[*pos..];
|
||||
let n = avail.len().min(buf.len());
|
||||
buf[..n].copy_from_slice(&avail[..n]);
|
||||
*pos += n;
|
||||
Ok(n)
|
||||
}
|
||||
}
|
||||
impl Write for Script {
|
||||
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
|
||||
self.outbound.lock().unwrap().extend_from_slice(buf);
|
||||
Ok(buf.len())
|
||||
}
|
||||
fn flush(&mut self) -> io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
impl ImapStream for Script {
|
||||
fn upgrade_tls(
|
||||
self: Box<Self>,
|
||||
_: &Connector,
|
||||
_: &str,
|
||||
) -> Result<Box<dyn ImapStream>, crate::imap::error::ImapError> {
|
||||
Ok(self)
|
||||
}
|
||||
fn is_tls(&self) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
fn client_for(inbound: &[u8]) -> (Pop3Client, Script) {
|
||||
let script = Script::new(inbound);
|
||||
let client = Pop3Client::from_stream(Box::new(script.clone()));
|
||||
(client, script)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn login_sends_user_then_pass() {
|
||||
let (mut c, script) = client_for(b"+OK user ok\r\n+OK signed in\r\n");
|
||||
c.login("alice", "secret").unwrap();
|
||||
assert_eq!(script.sent(), "USER alice\r\nPASS secret\r\n");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn login_maps_err_to_auth_failure() {
|
||||
let (mut c, _) = client_for(b"+OK user ok\r\n-ERR bad password\r\n");
|
||||
let err = c.login("alice", "wrong").unwrap_err();
|
||||
assert!(matches!(err, Pop3Error::Auth(_)), "{err}");
|
||||
assert!(err.to_string().contains("bad password"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stat_parses_count_and_size() {
|
||||
let (mut c, _) = client_for(b"+OK 3 10240\r\n");
|
||||
assert_eq!(c.stat().unwrap(), (3, 10240));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listing_pairs_uidl_with_sizes() {
|
||||
// UIDL response, then LIST response.
|
||||
let (mut c, script) =
|
||||
client_for(b"+OK\r\n1 abc\r\n2 def\r\n.\r\n+OK\r\n1 100\r\n2 250\r\n.\r\n");
|
||||
let got = c.listing().unwrap();
|
||||
assert_eq!(
|
||||
got,
|
||||
vec![
|
||||
MessageListing {
|
||||
number: 1,
|
||||
uid: "abc".into(),
|
||||
octets: 100
|
||||
},
|
||||
MessageListing {
|
||||
number: 2,
|
||||
uid: "def".into(),
|
||||
octets: 250
|
||||
},
|
||||
]
|
||||
);
|
||||
assert_eq!(script.sent(), "UIDL\r\nLIST\r\n");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listing_drops_a_message_with_a_uid_but_no_size() {
|
||||
let (mut c, _) = client_for(b"+OK\r\n1 abc\r\n2 def\r\n.\r\n+OK\r\n1 100\r\n.\r\n");
|
||||
let got = c.listing().unwrap();
|
||||
assert_eq!(got.len(), 1);
|
||||
assert_eq!(got[0].uid, "abc");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retr_reconstructs_crlf_and_unstuffs_dots() {
|
||||
// Body has a line starting with a dot (stuffed to "..") and a line that
|
||||
// is just "." in the content would itself be stuffed, so ".." decodes
|
||||
// to "." and "..foo" to ".foo".
|
||||
let (mut c, script) = client_for(
|
||||
b"+OK 20 octets\r\nFrom: a@x\r\nSubject: hi\r\n\r\n..stuffed\r\nplain\r\n.\r\n",
|
||||
);
|
||||
let body = c.retr(7).unwrap();
|
||||
assert_eq!(
|
||||
body,
|
||||
b"From: a@x\r\nSubject: hi\r\n\r\n.stuffed\r\nplain\r\n".to_vec()
|
||||
);
|
||||
assert_eq!(script.sent(), "RETR 7\r\n");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retr_handles_an_empty_body() {
|
||||
let (mut c, _) = client_for(b"+OK 0\r\n.\r\n");
|
||||
assert_eq!(c.retr(1).unwrap(), Vec::<u8>::new());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_connection_that_closes_mid_line_is_a_protocol_error() {
|
||||
let (mut c, _) = client_for(b"+OK no newline");
|
||||
let err = c.stat().unwrap_err();
|
||||
assert!(matches!(err, Pop3Error::Protocol(_)), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn eight_bit_bytes_in_a_message_survive_retr() {
|
||||
let mut inbound = b"+OK\r\n".to_vec();
|
||||
inbound.extend_from_slice(b"Subject: \xc3\xa9\r\nbody \xff\xfe\r\n.\r\n");
|
||||
let (mut c, _) = client_for(&inbound);
|
||||
let body = c.retr(1).unwrap();
|
||||
assert_eq!(body, b"Subject: \xc3\xa9\r\nbody \xff\xfe\r\n".to_vec());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_long_body_spanning_many_read_chunks_is_read_whole() {
|
||||
// More than READ_CHUNK bytes, to exercise fill() and prefix-draining.
|
||||
let mut inbound = b"+OK\r\n".to_vec();
|
||||
let line = vec![b'x'; 1000];
|
||||
let count = (READ_CHUNK / 1002) + 20;
|
||||
for _ in 0..count {
|
||||
inbound.extend_from_slice(&line);
|
||||
inbound.extend_from_slice(b"\r\n");
|
||||
}
|
||||
inbound.extend_from_slice(b".\r\n");
|
||||
let (mut c, _) = client_for(&inbound);
|
||||
let body = c.retr(1).unwrap();
|
||||
assert_eq!(body.len(), count * 1002);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
use crate::imap::error::ImapError;
|
||||
|
||||
/// What can go wrong talking POP3. The transport is shared with the IMAP
|
||||
/// client (the same TLS `Connector` and stream), so a connect or TLS failure
|
||||
/// arrives as an `ImapError` and is wrapped here.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum Pop3Error {
|
||||
#[error("io: {0}")]
|
||||
Io(#[from] std::io::Error),
|
||||
|
||||
#[error("transport: {0}")]
|
||||
Transport(#[from] ImapError),
|
||||
|
||||
/// The server answered `-ERR`, with its message.
|
||||
#[error("server refused: {0}")]
|
||||
Err(String),
|
||||
|
||||
/// The server's answer did not fit the protocol (no `+OK`/`-ERR`, a
|
||||
/// malformed list line, a response that ended early).
|
||||
#[error("protocol: {0}")]
|
||||
Protocol(String),
|
||||
|
||||
#[error("sign-in failed: {0}")]
|
||||
Auth(String),
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
pub mod client;
|
||||
pub mod error;
|
||||
|
||||
pub use client::{MessageListing, Pop3Client, Pop3Mode};
|
||||
pub use error::Pop3Error;
|
||||
@@ -0,0 +1,409 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*/
|
||||
|
||||
//! Read a POP3 mailbox into an archive.
|
||||
//!
|
||||
//! POP3 is the simplest source: one flat maildrop, no folders, no flags. Every
|
||||
//! message lands in a single INBOX. Messages are keyed by their UIDL -- the
|
||||
//! stable unique-id POP3 gives each -- so a later run adds only what arrived
|
||||
//! since, exactly like the other sources.
|
||||
//!
|
||||
//! It is **additive**: a message that is in the archive but no longer on the
|
||||
//! server is kept, not deleted. POP3 mailboxes are routinely emptied after
|
||||
//! download (that is the protocol's own model), so a message vanishing from
|
||||
//! the server is expected and says nothing about whether the archive should
|
||||
//! still hold it. This is the one place an importer does not mirror deletions.
|
||||
//!
|
||||
//! It never deletes anything on the server: no `DELE`, so a run is safe to
|
||||
//! repeat and leaves the source mailbox untouched.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use rusqlite::{Connection, params};
|
||||
use serde_json::Value;
|
||||
use time::OffsetDateTime;
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
|
||||
use crate::db;
|
||||
use crate::db::sources::SourceKey;
|
||||
use crate::error::Error;
|
||||
use crate::imap::transport::Connector;
|
||||
use crate::logging::{LEVEL_DEFAULT, LEVEL_PROGRESS, Logger};
|
||||
use crate::pop3::{MessageListing, Pop3Client, Pop3Mode};
|
||||
use crate::sync::emailmeta::email_meta_from_blob;
|
||||
use crate::sync::keys::index_to_json;
|
||||
use crate::sync::{CommonConfig, RunOutcome, Summary, TypeCounts};
|
||||
|
||||
/// New messages are committed in groups of this many, so an interrupted run of
|
||||
/// a large mailbox keeps what it already fetched.
|
||||
const COMMIT_EVERY: u64 = 200;
|
||||
const PROGRESS_TICK: u64 = 500;
|
||||
|
||||
pub struct Pop3ImportConfig {
|
||||
/// `pop3://host[:port]`, `pop3s://host[:port]`, or a bare `host[:port]`
|
||||
/// (treated as POP3S).
|
||||
pub url: String,
|
||||
pub user: String,
|
||||
pub password: String,
|
||||
/// Permit a plain `pop3://` connection with no STLS, sending the password
|
||||
/// in the clear. Ignored for `pop3s://`.
|
||||
pub allow_cleartext: bool,
|
||||
pub allow_source_change: bool,
|
||||
}
|
||||
|
||||
pub fn run(common: CommonConfig, config: Pop3ImportConfig) -> Result<Summary, Error> {
|
||||
run_reporting(common, config).into_result()
|
||||
}
|
||||
|
||||
pub fn run_reporting(common: CommonConfig, config: Pop3ImportConfig) -> RunOutcome {
|
||||
let mut summary = Summary::default();
|
||||
let error = run_into(common, config, &mut summary).err();
|
||||
RunOutcome { summary, error }
|
||||
}
|
||||
|
||||
struct Endpoint {
|
||||
host: String,
|
||||
port: u16,
|
||||
implicit_tls: bool,
|
||||
}
|
||||
|
||||
fn parse_endpoint(url: &str) -> Result<Endpoint, Error> {
|
||||
let (implicit_tls, rest) = if let Some(rest) = url.strip_prefix("pop3s://") {
|
||||
(true, rest)
|
||||
} else if let Some(rest) = url.strip_prefix("pop3://") {
|
||||
(false, rest)
|
||||
} else if url.contains("://") {
|
||||
return Err(Error::Usage(format!(
|
||||
"unsupported scheme in {url:?}: use pop3:// or pop3s://"
|
||||
)));
|
||||
} else {
|
||||
// No scheme: POP3S is the safe reading.
|
||||
(true, url)
|
||||
};
|
||||
let rest = rest.trim_end_matches('/');
|
||||
if rest.is_empty() {
|
||||
return Err(Error::Usage(format!("no host in {url:?}")));
|
||||
}
|
||||
let (host, port) = match rest.rsplit_once(':') {
|
||||
Some((h, p)) => {
|
||||
let port = p
|
||||
.parse::<u16>()
|
||||
.map_err(|_| Error::Usage(format!("invalid port in {url:?}")))?;
|
||||
(h, port)
|
||||
}
|
||||
None => (rest, if implicit_tls { 995 } else { 110 }),
|
||||
};
|
||||
if host.is_empty() {
|
||||
return Err(Error::Usage(format!("no host in {url:?}")));
|
||||
}
|
||||
Ok(Endpoint {
|
||||
host: host.to_owned(),
|
||||
port,
|
||||
implicit_tls,
|
||||
})
|
||||
}
|
||||
|
||||
fn run_into(
|
||||
common: CommonConfig,
|
||||
config: Pop3ImportConfig,
|
||||
summary: &mut Summary,
|
||||
) -> Result<(), Error> {
|
||||
let logger = common.logger;
|
||||
if common.threads > 1 {
|
||||
log_at(
|
||||
logger,
|
||||
LEVEL_PROGRESS,
|
||||
"pop3 importer is single-threaded; --threads value will be ignored",
|
||||
);
|
||||
}
|
||||
|
||||
let endpoint = parse_endpoint(&config.url)?;
|
||||
let scheme = if endpoint.implicit_tls {
|
||||
"pop3s"
|
||||
} else {
|
||||
"pop3"
|
||||
};
|
||||
let session_url = format!("{scheme}://{}:{}", endpoint.host, endpoint.port);
|
||||
let mode = if endpoint.implicit_tls {
|
||||
Pop3Mode::ImplicitTls
|
||||
} else if config.allow_cleartext {
|
||||
Pop3Mode::Plain
|
||||
} else {
|
||||
Pop3Mode::StartTls
|
||||
};
|
||||
|
||||
let mut conn = db::init::open(&common.archive)?;
|
||||
|
||||
let source_key = SourceKey {
|
||||
kind: "pop3".to_owned(),
|
||||
session_url: session_url.clone(),
|
||||
account_id: config.user.clone(),
|
||||
};
|
||||
if let Some((existing_url, existing_account)) =
|
||||
db::sources::conflicting_source(&conn, "pop3", &session_url, &config.user)?
|
||||
&& !config.allow_source_change
|
||||
{
|
||||
return Err(Error::SourceChange(format!(
|
||||
"archive already records pop3 source {existing_url} / {existing_account}; \
|
||||
re-run with --allow-source-change or use a fresh archive"
|
||||
)));
|
||||
}
|
||||
|
||||
let connector =
|
||||
Connector::new(common.allow_invalid_certs).map_err(|e| Error::Connection(e.to_string()))?;
|
||||
let mut client = Pop3Client::connect(&connector, &endpoint.host, endpoint.port, mode)
|
||||
.map_err(|e| Error::Connection(format!("pop3 connect {session_url}: {e}")))?;
|
||||
client
|
||||
.login(&config.user, &config.password)
|
||||
.map_err(|e| Error::Connection(format!("pop3 sign-in as {}: {e}", config.user)))?;
|
||||
log_at(
|
||||
logger,
|
||||
LEVEL_PROGRESS,
|
||||
&format!("signed in as {} on {session_url}", config.user),
|
||||
);
|
||||
|
||||
let listing = client
|
||||
.listing()
|
||||
.map_err(|e| Error::Connection(format!("pop3 listing: {e}")))?;
|
||||
|
||||
let source_id = if common.dry_run {
|
||||
db::sources::find_source(&conn, &source_key)?.unwrap_or(-1)
|
||||
} else {
|
||||
db::sources::upsert_source(&conn, &source_key, Some(&config.user), &config.user)?
|
||||
};
|
||||
|
||||
let local: HashMap<String, i64> = if source_id < 0 {
|
||||
HashMap::new()
|
||||
} else {
|
||||
db::pop3_ids::email_ids(&conn, source_id)?
|
||||
};
|
||||
let mailbox_exists =
|
||||
source_id >= 0 && db::pop3_ids::local_for_mailbox(&conn, source_id)?.is_some();
|
||||
|
||||
let new: Vec<&MessageListing> = listing
|
||||
.iter()
|
||||
.filter(|m| !local.contains_key(&m.uid))
|
||||
.collect();
|
||||
let present = listing.len() - new.len();
|
||||
|
||||
if common.dry_run {
|
||||
client.quit();
|
||||
*summary = dry_run_summary(mailbox_exists, &new, present, common.estimate);
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let mut mailbox_counts = TypeCounts::default();
|
||||
let mut email_counts = TypeCounts::default();
|
||||
email_counts.skipped += present as u64;
|
||||
|
||||
let mailbox_local = ensure_inbox(&mut conn, source_id, mailbox_exists, &mut mailbox_counts)?;
|
||||
|
||||
let total_new = new.len() as u64;
|
||||
let mut inserted: u64 = 0;
|
||||
let mut tx = conn.transaction()?;
|
||||
for msg in &new {
|
||||
let sp = tx.savepoint()?;
|
||||
match fetch_and_store(&sp, &mut client, source_id, mailbox_local, msg) {
|
||||
Ok(()) => {
|
||||
sp.commit()?;
|
||||
email_counts.created += 1;
|
||||
email_counts.fetched += 1;
|
||||
inserted += 1;
|
||||
if inserted.is_multiple_of(COMMIT_EVERY) {
|
||||
tx.commit()?;
|
||||
tx = conn.transaction()?;
|
||||
}
|
||||
if inserted.is_multiple_of(PROGRESS_TICK) && logger.enabled(LEVEL_PROGRESS) {
|
||||
eprintln!("pop3: fetched {inserted}/{total_new}");
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
drop(sp);
|
||||
logger.warn(&format!("pop3 message {} ({}): {e}", msg.number, msg.uid));
|
||||
email_counts.failed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
tx.commit()?;
|
||||
client.quit();
|
||||
|
||||
if email_counts.failed == 0 && mailbox_counts.failed == 0 {
|
||||
let tx = conn.unchecked_transaction()?;
|
||||
db::blobs::gc_orphan_blobs(&tx)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
|
||||
log_at(
|
||||
logger,
|
||||
LEVEL_DEFAULT,
|
||||
&format!(
|
||||
"pop3: mailbox created={}; email created={} skipped={} failed={}",
|
||||
mailbox_counts.created, email_counts.created, email_counts.skipped, email_counts.failed
|
||||
),
|
||||
);
|
||||
|
||||
*summary = Summary {
|
||||
per_type: vec![("mailbox", mailbox_counts), ("email", email_counts)],
|
||||
retries_observed: 0,
|
||||
retry_after_sleeps: 0,
|
||||
};
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The INBOX for this source, created and mapped on the first run.
|
||||
fn ensure_inbox(
|
||||
conn: &mut Connection,
|
||||
source_id: i64,
|
||||
mailbox_exists: bool,
|
||||
counts: &mut TypeCounts,
|
||||
) -> Result<i64, Error> {
|
||||
if mailbox_exists && let Some(id) = db::pop3_ids::local_for_mailbox(conn, source_id)? {
|
||||
return Ok(id);
|
||||
}
|
||||
let tx = conn.transaction()?;
|
||||
tx.execute(
|
||||
"INSERT INTO mailboxes (name, parent_id, role, sort_order, is_subscribed)
|
||||
VALUES ('INBOX', NULL, 'inbox', 0, 1)",
|
||||
[],
|
||||
)?;
|
||||
let id = tx.last_insert_rowid();
|
||||
db::pop3_ids::insert_mailbox(&tx, source_id, id)?;
|
||||
tx.commit()?;
|
||||
counts.created += 1;
|
||||
counts.fetched += 1;
|
||||
Ok(id)
|
||||
}
|
||||
|
||||
fn fetch_and_store(
|
||||
tx: &Connection,
|
||||
client: &mut Pop3Client,
|
||||
source_id: i64,
|
||||
mailbox_local: i64,
|
||||
msg: &MessageListing,
|
||||
) -> Result<(), Error> {
|
||||
let bytes = client
|
||||
.retr(msg.number)
|
||||
.map_err(|e| Error::Partial(format!("RETR {}: {e}", msg.number)))?;
|
||||
let (index, date_header) = email_meta_from_blob(&bytes);
|
||||
let received_at = date_header.unwrap_or_else(now_rfc3339);
|
||||
let blob_id = db::blobs::intern_blob(tx, &bytes)?;
|
||||
let message_match = index_to_json(&index);
|
||||
let mailbox_ids = Value::Array(vec![Value::from(mailbox_local)]).to_string();
|
||||
tx.execute(
|
||||
"INSERT INTO emails (blob_id, received_at, mailbox_ids, keywords, message_match)
|
||||
VALUES (?1, ?2, ?3, '[]', ?4)",
|
||||
params![blob_id, received_at, mailbox_ids, message_match],
|
||||
)?;
|
||||
let local_id = tx.last_insert_rowid();
|
||||
db::pop3_ids::insert_email(tx, source_id, &msg.uid, local_id)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn dry_run_summary(
|
||||
mailbox_exists: bool,
|
||||
new: &[&MessageListing],
|
||||
present: usize,
|
||||
want_sizes: bool,
|
||||
) -> Summary {
|
||||
let mut mailbox = TypeCounts::default();
|
||||
let mut email = TypeCounts::default();
|
||||
if !mailbox_exists {
|
||||
mailbox.created += 1;
|
||||
mailbox.fetched += 1;
|
||||
}
|
||||
email.created += new.len() as u64;
|
||||
email.fetched += new.len() as u64;
|
||||
email.skipped += present as u64;
|
||||
if want_sizes {
|
||||
// POP3 LIST gives every new message's size for free -- no body fetch.
|
||||
email.bytes += new.iter().map(|m| m.octets).sum::<u64>();
|
||||
}
|
||||
Summary {
|
||||
per_type: vec![("mailbox", mailbox), ("email", email)],
|
||||
retries_observed: 0,
|
||||
retry_after_sleeps: 0,
|
||||
}
|
||||
}
|
||||
|
||||
fn now_rfc3339() -> String {
|
||||
OffsetDateTime::now_utc()
|
||||
.format(&Rfc3339)
|
||||
.unwrap_or_else(|_| "1970-01-01T00:00:00Z".to_owned())
|
||||
}
|
||||
|
||||
fn log_at(logger: Logger, level: u8, msg: &str) {
|
||||
if logger.enabled(level) {
|
||||
println!("{msg}");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn listing(entries: &[(&str, u64)]) -> Vec<MessageListing> {
|
||||
entries
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, (uid, octets))| MessageListing {
|
||||
number: i as u32 + 1,
|
||||
uid: (*uid).to_owned(),
|
||||
octets: *octets,
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_endpoint_defaults_ports_and_tls() {
|
||||
let e = parse_endpoint("pop3s://mail.example.org").unwrap();
|
||||
assert_eq!(
|
||||
(e.host.as_str(), e.port, e.implicit_tls),
|
||||
("mail.example.org", 995, true)
|
||||
);
|
||||
let e = parse_endpoint("pop3://mail.example.org").unwrap();
|
||||
assert_eq!(
|
||||
(e.host.as_str(), e.port, e.implicit_tls),
|
||||
("mail.example.org", 110, false)
|
||||
);
|
||||
// Bare host: POP3S.
|
||||
let e = parse_endpoint("mail.example.org").unwrap();
|
||||
assert_eq!((e.port, e.implicit_tls), (995, true));
|
||||
// Explicit port wins.
|
||||
let e = parse_endpoint("pop3://host:1100").unwrap();
|
||||
assert_eq!((e.port, e.implicit_tls), (1100, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_endpoint_rejects_junk() {
|
||||
assert!(parse_endpoint("imap://host").is_err());
|
||||
assert!(parse_endpoint("pop3://").is_err());
|
||||
assert!(parse_endpoint("pop3://host:notaport").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dry_run_counts_new_and_present_and_the_first_mailbox() {
|
||||
let msgs = listing(&[("a", 100), ("b", 200), ("c", 50)]);
|
||||
let new: Vec<&MessageListing> = msgs.iter().collect();
|
||||
let s = dry_run_summary(false, &new, 0, false);
|
||||
let mailbox = &s.per_type[0].1;
|
||||
let email = &s.per_type[1].1;
|
||||
assert_eq!(mailbox.created, 1, "first run creates INBOX");
|
||||
assert_eq!((email.created, email.fetched), (3, 3));
|
||||
assert_eq!(email.bytes, 0, "no sizes without --estimate");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn estimate_sums_new_message_sizes_and_skips_the_mailbox_when_it_exists() {
|
||||
let msgs = listing(&[("a", 100), ("b", 200)]);
|
||||
let new: Vec<&MessageListing> = msgs.iter().collect();
|
||||
let s = dry_run_summary(true, &new, 4, true);
|
||||
let mailbox = &s.per_type[0].1;
|
||||
let email = &s.per_type[1].1;
|
||||
assert_eq!(mailbox.created, 0, "mailbox already exists on a later run");
|
||||
assert_eq!(email.bytes, 300);
|
||||
assert_eq!(email.skipped, 4);
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,7 @@ pub mod import_imap;
|
||||
pub mod import_jmap;
|
||||
pub mod import_maildir;
|
||||
pub mod import_managesieve;
|
||||
pub mod import_pop3;
|
||||
pub mod import_takeout;
|
||||
pub mod keys;
|
||||
pub mod progress;
|
||||
|
||||
Reference in new issue
Block a user