jcoffey-dev
a8dde68800
Phase 2 of the personal-data catalog spec. resources/privacy/catalog.toml classifies every object in the schema (316) and inbuxa's own JMAP objects (12): each property that can hold personal data, with its categories, and for objects that hold any, whose data it is, where it lives, its scope and what bounds its retention (a named setting where there is one). Twenty sources that are no object -- the log file, exporters, webhooks, spam lookups, the Explain cache, relays and hooks, push, legacy-use records -- carry the same facts plus the settings that turn them on, whether the data leaves the host, and the code that writes it. Classifications of objects that hold data about people are from the spec's source map; the rest are typed from the schema alone (address, IP, secret). tools/fork/privacy-check.py fails CI when an object or inbuxa object has no entry, when a property the schema types as an address, IP or secret is left to its object's default, when an entry names an object, property, setting or code path that is gone, or when it uses a word outside the catalog's vocabulary. --unlisted prints starting entries. strip.py's report gains "Unclassified in the privacy catalog": objects and fields new in an import and not classified, informational like the Enterprise flags. Tested: 13 unit tests (tools/fork/tests): the check passes on this tree; fails on an unclassified object, an address hidden behind a default, a secret in a set or object reference, stale properties, objects, settings and code paths, an unlisted inbuxa object and a word outside the vocabulary; --unlisted's entries; and the strip report on a synthetic import. The check and the tests run in the fork-checks job.
111 lines
4.5 KiB
Markdown
111 lines
4.5 KiB
Markdown
# Fork tooling
|
|
|
|
## strip.py
|
|
|
|
Makes an Enterprise-free snapshot of an upstream release. See the docstring
|
|
and docs/spec/SPEC.md §2.2 for what it does and why.
|
|
|
|
```bash
|
|
git clone https://github.com/stalwartlabs/stalwart.git ~/src/stalwart-upstream # outside this repo
|
|
git -C ~/src/stalwart-upstream fetch --tags
|
|
tools/fork/strip.py --upstream ~/src/stalwart-upstream --ref v0.16.22 --out /tmp/strip-v0.16.22
|
|
```
|
|
|
|
It writes `OUT/tree` (the stripped source) and `OUT/STRIP-REPORT.md` and
|
|
`.json`. Exit 0 means verified clean. Exit 1 means malformed markers, or
|
|
something Enterprise-only survived. Read the report's Problems section.
|
|
|
|
The report's Third-party code section lists upstream code under other
|
|
licenses. Files marked **new** need their notice added to `THIRD-PARTY.md`
|
|
at the repository root before the import is merged.
|
|
|
|
It needs Python 3.12+ (for `tarfile`'s `data` filter), git and cargo.
|
|
|
|
Two passes run after the strip:
|
|
|
|
- **Renames.** The upstream name is replaced where it's an identifier
|
|
clients, users or operators meet: wire-protocol names, the web interface's
|
|
client id, store keys, configuration defaults and the served schema, as
|
|
`renames.py` lists them. `main` was renamed with the same module. A
|
|
re-import arrives purged, so those lines never conflict. Copyright notices
|
|
and prose are left alone. The report lists every substitution by file.
|
|
- **Build check.** The stripped tree is compiled (`cargo check --workspace
|
|
--all-targets`) into `target/strip-check`, which stays warm between
|
|
imports. A file that survived the strip but calls code that didn't fails
|
|
the run; the report names it. Handle it in the merge into `main`, never on
|
|
`upstream`: `upstream` holds the strip's output and nothing else. Imports
|
|
the strip left unused are listed without failing. `--no-build-check`
|
|
skips the pass.
|
|
|
|
## name-check.py
|
|
|
|
Fails when the upstream project's name appears in a Rust string literal that
|
|
`name-allowlist.txt` doesn't list. CI runs it on every push and pull request,
|
|
so an upstream merge can't bring the name back into what users and operators
|
|
see. Comments, copyright headers and test directories aren't checked.
|
|
|
|
```bash
|
|
tools/fork/name-check.py # exit 1 on anything new
|
|
tools/fork/name-check.py --list # every finding, in allowlist format
|
|
```
|
|
|
|
Rename what it reports. If a string has to stay, such as a key-derivation
|
|
context or a wire-protocol identifier, add its `--list` line to the allowlist
|
|
under the reason it stays.
|
|
|
|
## notice-check.py
|
|
|
|
Fails when an upstream file the fork changed doesn't carry the AGPL 5(a)
|
|
notice, `Modified by Coffey Labs in <year> for INBUXA.`, under upstream's
|
|
license line. "Changed" means it differs from the `upstream` branch, so the
|
|
list comes from the diff, not from memory. CI runs it beside the name check.
|
|
|
|
```bash
|
|
tools/fork/notice-check.py # exit 1 on a missing notice
|
|
tools/fork/notice-check.py --fix # add it where it's missing
|
|
```
|
|
|
|
Run `--fix` after resolving an upstream merge: a conflict resolved by taking
|
|
upstream's side can drop a notice the file had.
|
|
|
|
## privacy-check.py
|
|
|
|
Fails when the personal-data catalog (`resources/privacy/catalog.toml`) and
|
|
the code disagree: an object in the schema or one of inbuxa's own JMAP
|
|
objects with no entry, a property the schema types as an address, an IP or
|
|
a secret left to its object's default, or an entry naming an object,
|
|
property, setting or code path that no longer exists. CI runs it beside the
|
|
name check, with its tests (`python3 -m unittest discover -s tools/fork/tests`).
|
|
See `docs/spec/features/personal-data-catalog.md`.
|
|
|
|
```bash
|
|
tools/fork/privacy-check.py # exit 1 on any finding
|
|
tools/fork/privacy-check.py --unlisted # starting entries for what's missing
|
|
```
|
|
|
|
After an upstream import, the strip report lists what is new and unclassified
|
|
under "Unclassified in the privacy catalog". `--unlisted` types each from the
|
|
schema alone; read the field's description before trusting it.
|
|
|
|
## record-compat.py
|
|
|
|
Records what the `*_compat` tests compare against, from the Enterprise
|
|
server, while it is still running. Read-only: `/get` and `/query` only.
|
|
See `docs/spec/compat-tests.md`.
|
|
|
|
```bash
|
|
tools/fork/record-compat.py --server https://mail.example.org \
|
|
--admin '[email protected]:PASSWORD' --out ./compat \
|
|
--tenant-admin '[email protected]:PASSWORD'
|
|
```
|
|
|
|
## run-compat.sh
|
|
|
|
Runs the `*_compat` tests against a copy of INBUXA's RocksDB store, making
|
|
a fresh copy for each one. See `docs/spec/compat-tests.md`.
|
|
|
|
```bash
|
|
tools/fork/run-compat.sh --store /srv/inbuxa-copy/rocks.db \
|
|
--admin '[email protected]:PASSWORD' --recordings ~/compat
|
|
```
|