Export/import: keep archived items, spam samples and the spam model #31

Merged

jcoffey-dev merged 1 commits from fix/export-all-subspaces into main

2026-09-23 08:50:16 +00:00

jcoffey-dev commented

2026-09-23 08:41:56 +00:00

Owner

--export / --import lost data on a move between databases (RocksDB to PostgreSQL, say). This makes the round trip keep everything that isn't short-lived or rebuildable.

What was missing

Data	Where	Now
Archived items (undelete records)	subspace `j`	exported with the `registry` family
Spam training samples	subspace `w`	exported with the `registry` family
Trained spam classifier + trainer state	blobs under fixed names, with no blob link, so the link walk never reached them	exported with the `blob` family
Negative counters and quotas	SQL targets add a negative amount with an `UPDATE`, which does nothing when the row doesn't exist yet	import creates the row first

j and w ride with registry because their indexes (b) and id counters (n) already travel in registry and data. That keeps EXPORT_TYPES=registry consistent, and the list of type names stays the same.

Backward compatible: the file format hasn't changed, and import reads whatever subspace files it finds, so exports made by an older binary still import. They just lack the three items above.

The search index (`z`) stays out, and gets rebuilt

It isn't portable. PostgreSQL and MySQL index into their own tables and have no z table at all; ElasticSearch and Meilisearch keep the index themselves; only the key-value stores use z. Exporting it would be dead weight at best, and a failed import into SQL at worst.

So restore() now returns the subspaces it wrote. --import then queues the reindexAccounts and reindexTelemetry store maintenance tasks, the same ones an administrator can queue by hand, and prints that it did. The index is rebuilt for whichever search store the new server is configured with once it starts. --help now says what an export holds.

Left out on purpose

m / y (in-memory values and counters): rate limits, locks, greylisting, ACME challenge tokens (1 h), OAuth codes, vacation-reply dedupe. All short-lived. Issued certificates are registry objects and do travel.

Tests

The store test now writes archived items, spam samples, directory entries, the fork's own subspace (_) and the named blobs, then:

checks they come back in place, as before;
imports the same export into a fresh store of the other local backend (RocksDB to SQLite, or SQLite to RocksDB), and compares it key for key, counter for counter (through the counter API, since the encodings differ) and blob for blob;
checks that the two reindex tasks were queued.

STORE=RocksDb RUST_MIN_STACK=16777216 unshare -rn cargo test -p tests store::store_tests   # ok
STORE=Sqlite  RUST_MIN_STACK=16777216 unshare -rn cargo test -p tests store::store_tests   # ok

With the old backup.rs the same test fails: Subspace j was not exported. Before the counter fix, the cross-backend step lost 20 of 50 random counters (the negative ones).

Still not covered by export

The tracing and metrics stores are exported only when they are the data store (the default). A separately configured tracing store isn't read.
An external search store (ElasticSearch, Meilisearch) isn't exported. The queued reindex rebuilds into it.

`--export` / `--import` lost data on a move between databases (RocksDB to PostgreSQL, say). This makes the round trip keep everything that isn't short-lived or rebuildable. ## What was missing | Data | Where | Now | |---|---|---| | Archived items (undelete records) | subspace `j` | exported with the `registry` family | | Spam training samples | subspace `w` | exported with the `registry` family | | Trained spam classifier + trainer state | blobs under fixed names, with no blob link, so the link walk never reached them | exported with the `blob` family | | Negative counters and quotas | SQL targets add a negative amount with an `UPDATE`, which does nothing when the row doesn't exist yet | import creates the row first | `j` and `w` ride with `registry` because their indexes (`b`) and id counters (`n`) already travel in `registry` and `data`. That keeps `EXPORT_TYPES=registry` consistent, and the list of type names stays the same. **Backward compatible:** the file format hasn't changed, and import reads whatever subspace files it finds, so exports made by an older binary still import. They just lack the three items above. ## The search index (`z`) stays out, and gets rebuilt It isn't portable. PostgreSQL and MySQL index into their own tables and have no `z` table at all; ElasticSearch and Meilisearch keep the index themselves; only the key-value stores use `z`. Exporting it would be dead weight at best, and a failed import into SQL at worst. So `restore()` now returns the subspaces it wrote. `--import` then queues the `reindexAccounts` and `reindexTelemetry` store maintenance tasks, the same ones an administrator can queue by hand, and prints that it did. The index is rebuilt for whichever search store the new server is configured with once it starts. `--help` now says what an export holds. ## Left out on purpose - `m` / `y` (in-memory values and counters): rate limits, locks, greylisting, ACME challenge tokens (1 h), OAuth codes, vacation-reply dedupe. All short-lived. Issued certificates are registry objects and do travel. ## Tests The store test now writes archived items, spam samples, directory entries, the fork's own subspace (`_`) and the named blobs, then: 1. checks they come back in place, as before; 2. imports the same export into a fresh store of the *other* local backend (RocksDB to SQLite, or SQLite to RocksDB), and compares it key for key, counter for counter (through the counter API, since the encodings differ) and blob for blob; 3. checks that the two reindex tasks were queued. ``` STORE=RocksDb RUST_MIN_STACK=16777216 unshare -rn cargo test -p tests store::store_tests # ok STORE=Sqlite RUST_MIN_STACK=16777216 unshare -rn cargo test -p tests store::store_tests # ok ``` With the old `backup.rs` the same test fails: `Subspace j was not exported`. Before the counter fix, the cross-backend step lost 20 of 50 random counters (the negative ones). ## Still not covered by export - The tracing and metrics stores are exported only when they are the data store (the default). A separately configured tracing store isn't read. - An external search store (ElasticSearch, Meilisearch) isn't exported. The queued reindex rebuilds into it.

jcoffey-dev added 1 commit 2026-09-23 08:41:56 +00:00

Export/import: keep archived items, spam samples and the spam model

ci / fork-checks (pull_request) Successful in 32s

Details

ci / build (pull_request) Successful in 7m57s

Details

212cd77cd3

--export skipped three things, so a move from one database to another
(RocksDB to PostgreSQL, say) lost them without a word:

- archived items (subspace j), the records behind undelete;
- spam training samples (subspace w);
- the trained spam classifier and its trainer state, blobs stored under
  fixed names that no blob link points at, so the walk over links never
  reached them.

j and w now travel with the registry family, where their indexes and id
counters already were, so EXPORT_TYPES=registry keeps them consistent.
The two named blobs travel with the blob family. The file format is
unchanged and import reads any subspace it is given, so an export made
by an older binary still imports.

The full-text index (subspace z) stays out, on purpose. It belongs to one
search backend: PostgreSQL and MySQL index into their own tables and have
no z table at all, and external engines keep the index themselves. So
--import now returns the subspaces it wrote, and boot queues the
reindexAccounts and reindexTelemetry store maintenance tasks, the same
ones an administrator can queue by hand, to rebuild the index for
whichever search store the server runs with once it starts.

The round trip also turned up a loss in import itself: the SQL stores
add a negative amount with an UPDATE, which does nothing to a row that
isn't there yet, so every negative counter or quota vanished on import
into PostgreSQL, MySQL or SQLite. Import now creates the row first.

The in-memory subspaces (m, y) stay out: rate limits, locks, greylisting,
ACME challenge tokens and OAuth codes, all short-lived. Issued
certificates are registry objects and travel.

The store test now writes archived items, spam samples, directory
entries, the fork's own subspace and the named blobs, checks they come
back in place, then imports the same export into a fresh store of the
other local backend (RocksDB to SQLite, or SQLite to RocksDB), compares
it key for key and counter for counter, and checks the queued reindex.
It fails on the old export code ("Subspace j was not exported").
--help now says what an export holds.

jcoffey-dev merged commit 499e4d7810 into main

2026-09-23 08:50:16 +00:00

jcoffey-dev deleted branch fix/export-all-subspaces

2026-09-23 08:50:16 +00:00

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#31

Export/import: keep archived items, spam samples and the spam model #31

What was missing

The search index (z) stays out, and gets rebuilt

Left out on purpose

Tests

Still not covered by export

The search index (`z`) stays out, and gets rebuilt