Classify the unmigrated settings instead of just counting them

A production rehearsal reported 12,182 settings not carried over by
migrate_v016.py. As a bare number that reads as an impossible amount of
manual reconstruction, and it is misleading. Snapshotting a v0.16.14 store
migrated from a real v0.15.5 showed what those settings actually are:

    8547  regenerates         server.blocked-ip
    3337  shipped with v0.16  lookup.url-redirectors, lookup.trusted-domains,
                              spam-filter.list, spam-filter.rule,
                              spam-filter.dnsbl, lookup.surbl-hashbl
     224  already carried     server.listener, signature.* (DKIM)
     293  NEED YOUR REVIEW    queue.schedule, config.local-keys,
                              server.auto-ban, spam-filter.llm, queue.tls, ...

server.blocked-ip is auto-ban state that repopulates from live traffic. The
stock groups are data v0.16 provides itself - 2,084 MemoryLookupKey, 66
SpamRule and 18 SpamDnsblServer objects were already present in the migrated
store. DKIM came across as DkimSignature objects with private keys intact,
verified on that instance. So the real worklist is ~293 keys, not 12,182.

backup.UnmigratedReport.Classify encodes this and the rehearsal now reports
the categorised view. Rules match longest-prefix-first, because
server.blocked-ip is runtime state while server.auto-ban beside it is
configuration, and an unrecognized prefix defaults to "needs review" -
assuming an unknown setting is safe to ignore is the wrong default.

This also retired the lookup and spam-filter generators that were the
planned next step. v0.15's rules are stwt_rbl_senderscore_ip; v0.16's are
STWT_RBL_SENDERSCORE_IP - the same stock set, already installed. Generating
them from v0.15 would duplicate every rule and revert upstream updates, so
they were deliberately not written. The targets worth generating are the
small site-specific groups instead: queue.schedule, queue.tls,
session.auth, server.auto-ban.

No production data in this commit: the test fixture uses the real group
names and counts with example.com standing in for customer domains.
This commit is contained in:
2026-08-23 22:07:29 -07:00
parent a69f0bbff1
commit 0d83283caa
5 changed files with 332 additions and 7 deletions
+12 -4
View File
@@ -124,10 +124,18 @@ stalwart-cli apply --file <state-dir>/runs/<run-id>/supplement.json \
--url https://mail.example.com
```
Expect the worklist to be long. Measured against a real production instance,
`migrate_v016.py` carried **219 of 12,401 settings — 1.8%**. The rest,
including `server.listener`, has to be rebuilt by hand; until it is, a
migrated instance answers on no ports at all. Both outputs are preserved
The worklist is long but mostly not work, and `rehearse` says which is
which. Measured against a real production instance, `migrate_v016.py`
carried **219 of 12,401 settings**. Of the 12,182 it left:
- **8,547** are runtime auto-ban state that repopulates itself
- **3,337** are stock spam-filter and lookup data v0.16 ships its own copies
of — restoring v0.15's would revert a year of upstream updates
- **~224** were already carried another way, DKIM signatures included
- **~293** genuinely need your eyes
`server.listener` is in that third group only because this tool regenerates
it for you; without that a migrated instance answers on no ports at all. Both outputs are preserved
under `<state-dir>/runs/<run-id>/` (`export.json` and `unmigrated.txt`) even
though the rest of the scratch directory is cleaned up, because they are the
conclusions.