Export/import: keep archived items, spam samples and the spam model

--export skipped three things, so a move from one database to another
(RocksDB to PostgreSQL, say) lost them without a word:

- archived items (subspace j), the records behind undelete;
- spam training samples (subspace w);
- the trained spam classifier and its trainer state, blobs stored under
  fixed names that no blob link points at, so the walk over links never
  reached them.

j and w now travel with the registry family, where their indexes and id
counters already were, so EXPORT_TYPES=registry keeps them consistent.
The two named blobs travel with the blob family. The file format is
unchanged and import reads any subspace it is given, so an export made
by an older binary still imports.

The full-text index (subspace z) stays out, on purpose. It belongs to one
search backend: PostgreSQL and MySQL index into their own tables and have
no z table at all, and external engines keep the index themselves. So
--import now returns the subspaces it wrote, and boot queues the
reindexAccounts and reindexTelemetry store maintenance tasks, the same
ones an administrator can queue by hand, to rebuild the index for
whichever search store the server runs with once it starts.

The round trip also turned up a loss in import itself: the SQL stores
add a negative amount with an UPDATE, which does nothing to a row that
isn't there yet, so every negative counter or quota vanished on import
into PostgreSQL, MySQL or SQLite. Import now creates the row first.

The in-memory subspaces (m, y) stay out: rate limits, locks, greylisting,
ACME challenge tokens and OAuth codes, all short-lived. Issued
certificates are registry objects and travel.

The store test now writes archived items, spam samples, directory
entries, the fork's own subspace and the named blobs, checks they come
back in place, then imports the same export into a fresh store of the
other local backend (RocksDB to SQLite, or SQLite to RocksDB), compares
it key for key and counter for counter, and checks the queued reindex.
It fails on the old export code ("Subspace j was not exported").
--help now says what an export holds.

crates/common/src/manager/backup.rs

@@ -23,6 +23,13 @@ use utils::{UnwrapFailure, codec::leb128::Leb128_};
 
 pub(super) const MAGIC_MARKER: u8 = 123;
 
+// inbuxa: blobs kept under a fixed name instead of a content hash. Nothing
+// links to them, so the export names them outright.
+const NAMED_BLOBS: &[&[u8]] = &[
+    crate::manager::SPAM_CLASSIFIER_KEY,
+    crate::manager::SPAM_TRAINER_KEY,
+];
+
 #[derive(Debug, Clone, Copy, Hash, PartialEq, Eq)]
 pub(super) enum Family {
     Data = 0,
@@ -143,15 +150,21 @@ impl Core {
                     .await
                     .failed("Failed to iterate over data store");
 
-                for hash in blobs {
+                // inbuxa: the trained spam classifier and its trainer state are
+                // blobs stored under fixed names with no blob link, so the walk
+                // over links above never reaches them.
+                let named = NAMED_BLOBS.iter().map(|key| key.to_vec());
+                for key in blobs
+                    .into_iter()
+                    .map(|hash| hash.as_slice().to_vec())
+                    .chain(named)
+                {
                     if let Some(blob) = blob_store
-                        .get_blob(hash.as_slice(), 0..usize::MAX)
+                        .get_blob(&key, 0..usize::MAX)
                         .await
                         .failed("Failed to get blob")
                     {
-                        writer
-                            .send((hash.as_slice().to_vec(), blob))
-                            .failed("Failed to send key");
+                        writer.send((key, blob)).failed("Failed to send key");
                     }
                 }
             }),
@@ -323,7 +336,13 @@ impl Family {
                 SUBSPACE_REGISTRY_IDX,
                 SUBSPACE_REGISTRY_PK,
                 SUBSPACE_DIRECTORY,
-                store::SUBSPACE_INBUXA, // inbuxa: masked email
+                // inbuxa: registry objects the upstream list left out, so an
+                // export dropped them: archived items (undelete) and spam
+                // training samples. Their indexes and id counters already
+                // travel in this family and in `data`, so they ride along.
+                SUBSPACE_DELETED_ITEMS,
+                SUBSPACE_SPAM_SAMPLES,
+                store::SUBSPACE_INBUXA, // inbuxa: the fork's own data (masked email, undelete, policies)
             ],
             Family::Changelog => &[SUBSPACE_LOGS],
             Family::Queue => &[SUBSPACE_QUEUE_MESSAGE, SUBSPACE_QUEUE_EVENT],

crates/common/src/manager/boot.rs

@@ -54,6 +54,13 @@ Options:
   -o, --console                    Open the store console
   -h, --help                       Print help
   -V, --version                    Print version
+
+An export holds everything in the data and blob stores except short-lived
+in-memory state (rate limits, locks, greylisting) and the full-text search
+index, which belongs to one search backend. An import into an empty store
+queues the index to be rebuilt when the server next starts. EXPORT_TYPES
+limits an export to some of: data, registry, blob, changelog, queue, report,
+telemetry, tasks.
 "#
 );
 
@@ -256,10 +263,10 @@ impl BootManager {
                 telemetry.enable();
 
                 // Parse settings and restore
-                Box::pin(Core::parse(&mut bootstrap, storage))
-                    .await
-                    .restore(path)
-                    .await;
+                let core = Box::pin(Core::parse(&mut bootstrap, storage)).await;
+                let imported = core.restore(path).await;
+                // inbuxa: the search index isn't exported; rebuild it
+                core.queue_reindex(&imported).await;
                 std::process::exit(0);
             }
             StoreOp::Console => {

crates/common/src/manager/restore.rs

@@ -9,15 +9,22 @@
 use super::backup::MAGIC_MARKER;
 use crate::{Core, DATABASE_SCHEMA_VERSION};
 use lz4_flex::frame::FrameDecoder;
-use registry::schema::enums::CompressionAlgo;
+use registry::{
+    schema::{
+        enums::{CompressionAlgo, TaskStoreMaintenanceType},
+        structs::{Task, TaskStatus, TaskStoreMaintenance},
+    },
+    types::EnumImpl,
+};
 use std::{
     fs::File,
     io::{BufReader, ErrorKind, Read},
     path::{Path, PathBuf},
 };
 use store::{
-    BlobStore, IterateParams, SUBSPACE_BLOBS, SUBSPACE_COUNTER, SUBSPACE_INDEXES, SUBSPACE_QUOTA,
-    SUBSPACE_REGISTRY_PK, Store, U32_LEN,
+    BlobStore, IterateParams, SUBSPACE_BLOBS, SUBSPACE_COUNTER, SUBSPACE_INDEXES,
+    SUBSPACE_PROPERTY, SUBSPACE_QUOTA, SUBSPACE_REGISTRY_PK, SUBSPACE_TELEMETRY_SPAN, Store,
+    U32_LEN,
     write::{
         AnyClass, AnyKey, BatchBuilder, ValueClass,
         key::{DeserializeBigEndian, is_node_id_key},
@@ -27,7 +34,9 @@ use types::{collection::Collection, field::Field};
 use utils::{UnwrapFailure, failed};
 
 impl Core {
-    pub async fn restore(&self, src: PathBuf) {
+    /// Imports an export into an empty store and returns the subspaces it
+    /// wrote. inbuxa: the caller hands them to [`Core::queue_reindex`].
+    pub async fn restore(&self, src: PathBuf) -> Vec<u8> {
         // Backup the core
         let paths = if src.is_dir() {
             let mut paths = Vec::new();
@@ -64,6 +73,13 @@ impl Core {
             std::process::exit(1);
         }
 
+        let mut imported = paths
+            .iter()
+            .map(|path| KeyValueReader::new(path).subspace)
+            .collect::<Vec<_>>();
+        imported.sort_unstable();
+        imported.dedup();
+
         let mut tasks = Vec::new();
         for path in paths {
             let storage = self.storage.clone();
@@ -76,6 +92,54 @@ impl Core {
         for task in tasks {
             task.await.failed("Failed to wait for task");
         }
+
+        imported
+    }
+
+    /// inbuxa: an export never carries the full-text index. It is built by
+    /// and for one search backend (the SQL stores index into their own
+    /// tables, the key-value stores into a subspace, external engines keep it
+    /// themselves), so it would be wrong or unreadable after a move to
+    /// another one. Instead, an import queues the same reindex tasks an
+    /// administrator can queue by hand (`reindexAccounts` and
+    /// `reindexTelemetry` store maintenance), and the server rebuilds the
+    /// index for whatever search store it is configured with once it starts.
+    pub async fn queue_reindex(&self, imported: &[u8]) -> Vec<TaskStoreMaintenanceType> {
+        let mut queued = Vec::new();
+        if imported.contains(&SUBSPACE_PROPERTY) {
+            queued.push(TaskStoreMaintenanceType::ReindexAccounts);
+        }
+        if imported.contains(&SUBSPACE_TELEMETRY_SPAN) {
+            queued.push(TaskStoreMaintenanceType::ReindexTelemetry);
+        }
+        if queued.is_empty() {
+            return queued;
+        }
+
+        let mut batch = BatchBuilder::new();
+        for maintenance_type in &queued {
+            batch.schedule_task(Task::StoreMaintenance(TaskStoreMaintenance {
+                maintenance_type: *maintenance_type,
+                status: TaskStatus::now(),
+                shard_index: None,
+            }));
+        }
+        self.storage
+            .data
+            .write(batch.build_all())
+            .await
+            .failed("Failed to queue the reindex tasks");
+
+        println!(
+            "Queued {} to rebuild the search index; it runs when the server starts.",
+            queued
+                .iter()
+                .map(|t| t.as_str())
+                .collect::<Vec<_>>()
+                .join(" and ")
+        );
+
+        queued
     }
 }
 
@@ -125,17 +189,22 @@ async fn restore_file(store: Store, blob_store: BlobStore, path: &Path) {
         }
         SUBSPACE_COUNTER | SUBSPACE_QUOTA => {
             while let Some((key, value)) = reader.next() {
-                batch.add(
-                    ValueClass::Any(AnyClass {
-                        subspace: reader.subspace,
-                        key,
-                    }),
-                    u64::from_le_bytes(
-                        value
-                            .try_into()
-                            .expect("Failed to deserialize counter/quota"),
-                    ) as i64,
-                );
+                let class = ValueClass::Any(AnyClass {
+                    subspace: reader.subspace,
+                    key,
+                });
+                let value = u64::from_le_bytes(
+                    value
+                        .try_into()
+                        .expect("Failed to deserialize counter/quota"),
+                ) as i64;
+                // inbuxa: the SQL stores add a negative amount with an UPDATE,
+                // which does nothing to a row that isn't there yet, so a
+                // negative counter vanished on import. Create the row first.
+                if value < 0 {
+                    batch.add(class.clone(), 0);
+                }
+                batch.add(class, value);
                 if batch.is_large_batch() {
                     store
                         .write(batch.build_all())