Fork tooling: a build check and a rename pass in the strip, a notice check in CI
ci / fork-checks (pull_request) Successful in 18s
ci / build (pull_request) Successful in 7m11s

strip.py compiles the stripped tree, so a dual-licensed file that only
serves an Enterprise feature fails the import instead of the merge, as
v0.16.23's tests/src/directory/issuer.rs does. Upstream's tests of the
features the fork rebuilt are expected not to compile there and are listed
in build-check-known.txt; an error anywhere else fails the run. Checked
against both imports: v0.16.22 passes with its 16 expected errors, v0.16.23
fails on issuer.rs alone. Imports the strip leaves unused are reported.

It also renames the upstream name where clients, users or operators meet
it as an identifier, from tools/fork/renames.py: wire-protocol names, the
web interface's client id, store keys, configuration defaults and the
served schema. main is renamed with the same module, so a re-import
arrives purged and those lines don't conflict.

notice-check.py fails CI when an upstream file the fork changed, measured
against the upstream branch, lacks its AGPL 5(a) notice; --fix adds it.
It runs beside the name check in a renamed fork-checks job.

Also commits v0.16.23's strip report under docs/fork/strip-reports/, which
the import in #18 left out.
This commit is contained in:
2026-09-22 19:02:34 -07:00
parent c5bf67f1bf
commit 4799d191a0
9 changed files with 1124 additions and 7 deletions
+32 -1
View File
@@ -19,7 +19,23 @@ The report's Third-party code section lists upstream code under other
licenses. Files marked **new** need their notice added to `THIRD-PARTY.md`
at the repository root before the import is merged.
It needs Python 3.12+ (for `tarfile`'s `data` filter) and git.
It needs Python 3.12+ (for `tarfile`'s `data` filter), git and cargo.
Two passes run after the strip:
- **Renames.** The upstream name is replaced where it's an identifier
clients, users or operators meet: wire-protocol names, the web interface's
client id, store keys, configuration defaults and the served schema, as
`renames.py` lists them. `main` was renamed with the same module. A
re-import arrives purged, so those lines never conflict. Copyright notices
and prose are left alone. The report lists every substitution by file.
- **Build check.** The stripped tree is compiled (`cargo check --workspace
--all-targets`) into `target/strip-check`, which stays warm between
imports. A file that survived the strip but calls code that didn't fails
the run; the report names it. Handle it in the merge into `main`, never on
`upstream`: `upstream` holds the strip's output and nothing else. Imports
the strip left unused are listed without failing. `--no-build-check`
skips the pass.
## name-check.py
@@ -37,6 +53,21 @@ Rename what it reports. If a string has to stay, such as a key-derivation
context or a wire-protocol identifier, add its `--list` line to the allowlist
under the reason it stays.
## notice-check.py
Fails when an upstream file the fork changed doesn't carry the AGPL 5(a)
notice, `Modified by Coffey Labs in <year> for INBUXA.`, under upstream's
license line. "Changed" means it differs from the `upstream` branch, so the
list comes from the diff, not from memory. CI runs it beside the name check.
```bash
tools/fork/notice-check.py # exit 1 on a missing notice
tools/fork/notice-check.py --fix # add it where it's missing
```
Run `--fix` after resolving an upstream merge: a conflict resolved by taking
upstream's side can drop a notice the file had.
## record-compat.py
Records what the `*_compat` tests compare against, from the Enterprise
+18
View File
@@ -0,0 +1,18 @@
# Files in a stripped upstream tree that are expected not to compile, read by
# tools/fork/strip.py's build check. Each is upstream's shared test of a
# feature upstream builds only in Enterprise and the fork rebuilt clean-room
# on `main` (docs/spec/SPEC.md §2.2b, §4), so the stripped tree alone lacks
# what they call. Errors here are reported as expected; an error in any other
# file fails the strip. One path per line.
# OIDC directories: `main` has its own tests/src/directory/oidc.rs.
tests/src/directory/mod.rs
# Tenants and archiving.
tests/src/system/mod.rs
# The LLM spam-filter classifier.
tests/src/smtp/inbound/antispam.rs
# Telemetry: alerts, stored metrics and traces, webhooks.
tests/src/telemetry/alerts.rs
tests/src/telemetry/metrics.rs
tests/src/telemetry/tracing.rs
tests/src/telemetry/webhooks.rs
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2026 Coffey Labs
# SPDX-License-Identifier: AGPL-3.0-only
"""
Fail when an upstream file the fork changed doesn't say so (AGPL section 5(a)).
tools/fork/notice-check.py # check; exit 1 on a missing notice
tools/fork/notice-check.py --fix # add the notice where it's missing
The AGPL asks a modified work to carry a prominent notice that it was
modified, with a date. Every upstream file this fork changes carries one
beneath upstream's own notice:
* Modified by Coffey Labs in 2026 for INBUXA.
"Changed" is measured against the `upstream` branch, which holds the stripped
upstream release `main` was last merged with (docs/spec/SPEC.md §2.2a), so
the list is what actually differs rather than a guess. A file counts as
upstream's when its header names Stalwart Labs as a copyright holder; files
the fork wrote carry their own copyright and need nothing. Files with no
comment header at all (README, manifests) are covered by the README's prose.
"""
import argparse
import datetime
import re
import subprocess
import sys
HEADER_LINES = 15
UPSTREAM_HOLDER = re.compile(r'SPDX-FileCopyrightText:.*Stalwart Labs')
NOTICE = re.compile(r'Modified by Coffey Labs in \d{4}')
LICENSE_LINE = re.compile(r'^(\s*(?:\*|//|#)\s*)SPDX-License-Identifier:.*$')
def git(*args):
return subprocess.run(['git', *args], check=True, capture_output=True, text=True).stdout
def snapshot_ref():
for ref in ('origin/upstream', 'upstream'):
if subprocess.run(['git', 'rev-parse', '--verify', '--quiet', f'{ref}^{{commit}}'],
capture_output=True).returncode == 0:
return ref
sys.exit('notice-check: no upstream snapshot branch (origin/upstream or upstream); fetch it first')
def changed_upstream_files(ref):
# Against the working tree, not HEAD, so it also checks work not yet
# committed; in CI the two are the same.
for path in git('diff', '--name-only', '--diff-filter=M', ref).splitlines():
try:
head = open(path, encoding='utf-8').read().split('\n')[:HEADER_LINES]
except (OSError, UnicodeDecodeError):
continue
if any(UPSTREAM_HOLDER.search(line) for line in head):
yield path, head
def add_notice(path):
"""Put the notice under the license line, in that comment's own style."""
lines = open(path, encoding='utf-8').read().split('\n')
for n, line in enumerate(lines[:HEADER_LINES]):
m = LICENSE_LINE.match(line)
if m:
prefix = m.group(1)
blank = prefix.rstrip()
year = datetime.date.today().year
lines[n + 1:n + 1] = [blank, f'{prefix}Modified by Coffey Labs in {year} for INBUXA.']
open(path, 'w', encoding='utf-8').write('\n'.join(lines))
return True
return False
def main():
ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0].strip())
ap.add_argument('--fix', action='store_true', help='add the notice to every file missing it')
args = ap.parse_args()
ref = snapshot_ref()
checked, missing = 0, []
for path, head in changed_upstream_files(ref):
checked += 1
if not any(NOTICE.search(line) for line in head):
missing.append(path)
if args.fix:
unfixable = [p for p in missing if not add_notice(p)]
for p in sorted(set(missing) - set(unfixable)):
print(f'added: {p}')
missing = unfixable
if missing:
print(f'{len(missing)} upstream file(s) changed against {ref} without the modification notice:\n')
for p in missing:
print(f' {p}')
print('\nAdd "Modified by Coffey Labs in <year> for INBUXA." under the license line, '
'or run tools/fork/notice-check.py --fix.')
return 1
print(f'notice check: clean ({checked} changed upstream file(s), all marked; against {ref}).')
return 0
if __name__ == '__main__':
sys.exit(main())
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2026 Coffey Labs
# SPDX-License-Identifier: AGPL-3.0-only
"""
The upstream name where it's an identifier, and its replacement.
tools/fork/renames.py DIR # apply to a tree; prints what changed
Clients, users and operators meet the upstream name in a few places that
aren't notices: wire-protocol names (the JMAP registry capability, WebDAV
state tokens, Sieve extensions), the web interface's OAuth client id, store keys an
operator sees as blob names, and configuration defaults (SQL database and
user, the log file prefix). docs/spec/SPEC.md §2.4 renames them all.
strip.py applies this to every upstream import, so each release arrives
already renamed and those lines never conflict in the merge. `main` was
renamed with it once, on 2026-09-22. Don't run it on `main` again: code
written since that names the old spelling on purpose (the migrations that
retire it) would be renamed too. Those strings are listed in
name-allowlist.txt instead.
Names nobody sees keep upstream's spelling: the OAuth key-derivation contexts
and the hashed application prefix. Renaming them would only destroy state.
Copyright notices and prose (`.md`, `.txt`) are never touched.
"""
import gzip
import hashlib
import base64
import re
import sys
from pathlib import Path
# Plain substrings, in code and data: (old, new, the roots it applies under).
TEXT_RENAMES = [
# Upstream's JMAP capability for its registry (`x:`) objects. Not plain
# `urn:inbuxa:jmap`, which is the fork's own capability (contract C-1);
# listed before the general prefix below so it wins.
('urn:stalwart:jmap', 'urn:inbuxa:jmap:registry', None),
('urn:stalwart:', 'urn:inbuxa:', None),
('vnd.stalwart.', 'vnd.inbuxa.', None),
('(vnd.stalwart)', '(vnd.inbuxa)', None),
('stalwart-webui', 'inbuxa-webui', None),
('STALWART_SPAM_', 'INBUXA_SPAM_', None),
# Configuration defaults, as upstream's generated registry code spells
# them. Server code only: the tests use the same spelling for fixtures
# that must match their containers and identity provider (database users,
# passwords, an OIDC audience), and name their databases explicitly.
('"stalwart".to_string()', '"inbuxa".to_string()', ('crates',)),
]
ROOTS = ('crates', 'tests', 'resources')
SKIP_SUFFIXES = {'.md', '.txt'}
TEXT_SUFFIXES = {'.rs', '.toml', '.py', '.sh', '.json', '.yml', '.yaml', '.js', '.ts', '.html', '.sieve', '.sql'}
COPYRIGHT = re.compile(r'(?i)(?:SPDX-FileCopyrightText:|\bcopyright\b|©)')
# The JSON Schema the server serves to INBUXA Admin, and its checksum.
SCHEMA = Path('resources/schema/schema.json.gz')
SCHEMA_HASH = Path('resources/schema/schema.json.sha256')
SCHEMA_RENAMES = [
('"stalwart"', '"inbuxa"'),
('vnd.stalwart', 'vnd.inbuxa'),
]
def rename_line(line, root):
"""Returns (new line, [(old, new, count)]) for a line of a file under `root`."""
if COPYRIGHT.search(line):
return line, []
done = []
for old, new, roots in TEXT_RENAMES:
if roots is not None and root not in roots:
continue
if old in line:
done.append((old, new, line.count(old)))
line = line.replace(old, new)
return line, done
def schema_bytes(gz):
"""The renamed schema, gzipped deterministically, and its checksum."""
text = gzip.decompress(gz).decode('utf-8')
for old, new in SCHEMA_RENAMES:
text = text.replace(old, new)
out = gzip.compress(text.encode('utf-8'), compresslevel=9, mtime=0)
digest = base64.urlsafe_b64encode(hashlib.sha256(out).digest()).decode().rstrip('=')
return out, digest
def apply(tree):
"""Apply every rename under `tree`; returns {"old → new": {file: count}}."""
tree = Path(tree)
done = {}
def note(old, new, rel, count):
done.setdefault(f'{old}{new}', {})
done[f'{old}{new}'][rel] = done[f'{old}{new}'].get(rel, 0) + count
for root in ROOTS:
base = tree / root
if not base.is_dir():
continue
for path in sorted(base.rglob('*')):
if not path.is_file() or path.suffix in SKIP_SUFFIXES:
continue
if path.suffix not in TEXT_SUFFIXES and not path.name.startswith('Dockerfile'):
continue
try:
lines = path.read_text(encoding='utf-8').split('\n')
except UnicodeDecodeError:
continue
changed = False
rel = str(path.relative_to(tree))
for n, line in enumerate(lines):
new_line, subs = rename_line(line, root)
for old, new, count in subs:
note(old, new, rel, count)
if subs:
lines[n] = new_line
changed = True
if changed:
path.write_text('\n'.join(lines), encoding='utf-8')
schema = tree / SCHEMA
if schema.is_file():
before = schema.read_bytes()
text = gzip.decompress(before).decode('utf-8')
counts = {old: text.count(old) for old, _ in SCHEMA_RENAMES}
out, digest = schema_bytes(before)
if any(counts.values()):
schema.write_bytes(out)
(tree / SCHEMA_HASH).write_text(digest, encoding='utf-8')
for old, new in SCHEMA_RENAMES:
if counts[old]:
note(old, new, str(SCHEMA), counts[old])
return {k: dict(sorted(v.items())) for k, v in sorted(done.items())}
if __name__ == '__main__':
if len(sys.argv) != 2:
sys.exit(__doc__.split('\n\n')[1])
for sub, files in apply(sys.argv[1]).items():
print(f'{sub}: ' + ', '.join(f'{f} ({n})' for f, n in files.items()))
+93 -2
View File
@@ -33,6 +33,15 @@ What it does, in order (docs/spec/SPEC.md §2.2):
comments mark it (another copyright holder or license, or "ported from"
and the like), and names any file THIRD-PARTY.md doesn't cover yet. That's
a report, not a failure: the notice goes in THIRD-PARTY.md with the merge.
8. Renames the upstream name where it's an identifier clients, users or
operators meet (renames.py beside this script), so a re-import arrives
purged and merges without conflicts on those lines. Copyright notices
and prose are never touched.
9. Compiles the result (`cargo check --workspace --all-targets`). A file
that survived the strip but calls code that didn't -- a dual-licensed test
of an Enterprise feature, say -- fails here, on the `upstream` branch,
instead of in the merge. Imports the strip left unused are reported, not
failed. `--no-build-check` skips it.
Only license markers and Cargo manifests are read for meaning. The code inside
an Enterprise file or snippet is never printed, reported or kept, which is what
@@ -52,6 +61,9 @@ import tarfile
import tempfile
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from renames import apply as rename_upstream_names # noqa: E402
SEL = 'LicenseRef-SEL'
AGPL = 'AGPL-3.0-only'
# A license identifier as it appears in a comment header, in any comment style.
@@ -375,6 +387,50 @@ def remaining_hooks(tree):
return dict(sorted(gates.items())), dict(sorted(checks.items()))
BUILD_KNOWN = Path(__file__).resolve().parent / 'build-check-known.txt'
def known_build_failures():
"""Files expected not to compile in a stripped tree (build-check-known.txt)."""
if not BUILD_KNOWN.is_file():
return set()
return {l.strip() for l in BUILD_KNOWN.read_text(encoding='utf-8').split('\n')
if l.strip() and not l.lstrip().startswith('#')}
def build_check(tree, target_dir):
"""
`cargo check` the stripped tree. Returns (errors, unused): each error is
{file, line, message} from a compiler diagnostic, grouped by the file it
points at; unused lists the imports reported unused.
Only compiler diagnostics are read, and they point at the shared code
that failed, never at the removed code.
"""
cmd = ['cargo', 'check', '--workspace', '--all-targets', '--locked', '--message-format=json',
'--target-dir', str(target_dir)]
r = subprocess.run(cmd, cwd=tree, capture_output=True, text=True)
errors, unused = [], []
for line in r.stdout.splitlines():
try:
msg = json.loads(line)
except ValueError:
continue
if msg.get('reason') != 'compiler-message':
continue
d = msg['message']
span = next((s for s in d.get('spans', []) if s.get('is_primary')), None)
where = {'file': span['file_name'], 'line': span['line_start']} if span else {'file': '?', 'line': 0}
if d.get('level') == 'error':
errors.append({**where, 'message': d.get('message', '')})
elif (d.get('code') or {}).get('code') == 'unused_imports':
unused.append({**where, 'message': d.get('message', '')})
if r.returncode != 0 and not errors:
errors.append({'file': '?', 'line': 0, 'message': (r.stderr.strip().splitlines() or ['cargo check failed'])[-1]})
dedup = lambda items: [dict(t) for t in sorted({tuple(sorted(i.items())) for i in items}, key=lambda t: (dict(t)['file'], dict(t)['line']))]
return dedup(errors), dedup(unused)
def write_report(out_dir, report):
(out_dir / 'STRIP-REPORT.json').write_text(json.dumps(report, indent=2) + '\n', encoding='utf-8')
r = report
@@ -390,6 +446,13 @@ def write_report(out_dir, report):
f'in {len(r["feature_gates"])} files; {sum(r["edition_checks"].values())} `is_enterprise_edition()` checks '
f'in {len(r["edition_checks"])} files',
f'- Third-party code: {len(r["third_party"])} files, **{len(r["third_party_unlisted"])}** not in THIRD-PARTY.md',
f'- Renamed identifiers: {sum(sum(f.values()) for f in r["renames"].values())} in '
f'{len({p for f in r["renames"].values() for p in f})} files',
'- Build check: ' + ('skipped' if r['build'] is None else
f'**{"clean" if not r["build"]["errors"] else f"{len(r["build"]["errors"])} errors"}**, '
f'{len(r["build"]["expected"])} expected errors in '
f'{len({e["file"] for e in r["build"]["expected"]})} rebuilt-feature tests, '
f'{len(r["build"]["unused"])} imports left unused'),
]
if r['schema']:
md.append(f'- Upstream schema flags {len(r["schema"]["objects"])} objects and {len(r["schema"]["fields"])} fields as Enterprise')
@@ -406,6 +469,19 @@ def write_report(out_dir, report):
for f, found in r['third_party'].items():
md.append(f'- `{f}`{" **new**" if f in r["third_party_unlisted"] else ""}')
md += [f' - {h["line"]}: {h["text"]}' for h in found]
md += ['', '## Renamed identifiers', '']
for sub, files in r['renames'].items():
md.append(f'- `{sub}`: ' + ', '.join(f'`{f}` ({n})' for f, n in files.items()))
if r['build'] is not None:
md += ['', '## Build check', '',
'Errors mean shared code calls something the strip removed: usually a dual-licensed file that only '
'serves an Enterprise feature. Drop or rework it in the merge into `main`, never on `upstream`.', '']
md += [f'- error `{e["file"]}:{e["line"]}`: {e["message"]}' for e in r['build']['errors']]
md += [f'- unused `{u["file"]}:{u["line"]}`: {u["message"]}' for u in r['build']['unused']]
md += ['', 'Expected: upstream\'s tests of features the fork rebuilt on `main` '
'(tools/fork/build-check-known.txt).', '']
md += [f'- `{e["file"]}:{e["line"]}`: {e["message"]}' for e in r['build']['expected']]
md += [f'- `{f}` now compiles: take it off the known list' for f in r['build']['known_clean']]
if r['problems']:
md += ['', '## Problems', ''] + [f'- {p}' for p in r['problems']]
(out_dir / 'STRIP-REPORT.md').write_text('\n'.join(md) + '\n', encoding='utf-8')
@@ -416,6 +492,10 @@ def main():
ap.add_argument('--upstream', required=True, type=Path, help='a git clone of upstream Stalwart')
ap.add_argument('--ref', required=True, help='tag, branch or commit to snapshot, e.g. v0.16.22')
ap.add_argument('--out', required=True, type=Path, help='new directory; the tree goes in OUT/tree')
ap.add_argument('--target-dir', type=Path, default=Path(__file__).resolve().parents[2] / 'target' / 'strip-check',
help="cargo's target dir for the build check (default: this repo's target/strip-check, "
'which keeps the dependency build warm between imports)')
ap.add_argument('--no-build-check', action='store_true', help='skip compiling the stripped tree')
args = ap.parse_args()
if args.out.exists():
@@ -428,7 +508,7 @@ def main():
write_report(args.out, {'ref': args.ref, 'commit': commit, 'removed_files': [], 'removed_snippets': {},
'cargo_edits': [], 'dangling_mods': [], 'problems': malformed, 'feature_gates': {}, 'edition_checks': {},
'schema': None, 'third_party': {}, 'third_party_unlisted': [],
'ossify_log': ''})
'renames': {}, 'build': None, 'ossify_log': ''})
print('\n'.join(malformed), file=sys.stderr)
fail('malformed snippet markers; nothing stripped', code=1)
@@ -436,10 +516,21 @@ def main():
log = run_ossify(tree, rust_roots(tree))
edits = deactivate_enterprise(tree) + deactivate_enterprise_in_scripts(tree)
dangling = remove_dangling_mods(tree)
renames = rename_upstream_names(tree)
problems = verify(tree)
gates, checks = remaining_hooks(tree)
others = third_party(tree)
new_others = unlisted(others)
build = None
if not args.no_build_check and not problems:
print('strip: compiling the stripped tree (cargo check)...', file=sys.stderr)
errors, unused = build_check(tree, args.target_dir)
known = known_build_failures()
expected = [e for e in errors if e['file'] in known]
errors = [e for e in errors if e['file'] not in known]
build = {'errors': errors, 'expected': expected, 'unused': unused,
'known_clean': sorted(known - {e['file'] for e in expected})}
problems += [f'{e["file"]}:{e["line"]}: does not compile: {e["message"]}' for e in errors]
report = {
'ref': args.ref, 'commit': commit,
@@ -447,7 +538,7 @@ def main():
'cargo_edits': edits, 'dangling_mods': dangling, 'problems': problems,
'feature_gates': gates, 'edition_checks': checks,
'schema': schema_flags(tree), 'third_party': others, 'third_party_unlisted': new_others,
'ossify_log': log,
'renames': renames, 'build': build, 'ossify_log': log,
}
write_report(args.out, report)
print(f'{args.ref} ({commit[:12]}): removed {len(removed_files)} files and '