2026-09-28 13:05:02 +00:00
1 changed files with 632 additions and 0 deletions
@@ -0,0 +1,632 @@
 # Feature spec: personal-data catalog, compliance role, and the Compliance section's first pages
 Status: approved 2026-09-28, with the answers under [Settled](#settled).
 Phase 1 of the GDPR auditor foundation: the investigation and the design. Not a
 rebuild of an upstream feature, so it has no line in SPEC.md §4's table.
 ## Provenance
 Written for the record SPEC.md §3 rule 3 asks for. Sources, and nothing else:
 | Source | License | Used for |
 |---|---|---|
 | This repository at `de275ba` (2026-09-28): the code, `resources/schema/schema.json.gz`, `tools/fork/`, `.gitea/workflows/` | AGPL-3.0-only | Every claim in the source map: each row names the code that writes the data |
 | inbuxa-admin (the console) at `8e281e5` | AGPL-3.0-only | How navigation, hand-built pages and visibility work |
 | docs.inbuxa.org (the inbuxa.org repository at `6b12467`) | Own | What the documentation says, for "Contradictions" |
 | `inbuxa-drafts/specs/audit-hold-lock.md`, `legacy-protocols.md` | Own | The Compliance pages that already exist, and the permission-rollout notes |
 | Regulation (EU) 2016/679 (GDPR), Articles 5, 25, 30 and 32 | Public law | The shape of a record of processing: categories, whose data, where, how long, who receives it |
 No Enterprise-only file or snippet was used. Claims were checked by reading the
 code; where a search found nothing, the row says so rather than asserting a
 negative.
 ## What it is
 Three things, the foundation for a GDPR auditor:
 1. **A personal-data catalog.** Every place the server can store or send
   personal data, with its categories, whose data it is, where it lives, what
   bounds its retention, and the settings that control it. What a particular
   server holds depends on how its operator configured it, so the catalog
   describes *possibilities*; the server evaluates it against live settings to
   say what *this* server does (Phase 3).
 2. **A compliance role** that can see and act on compliance data without being
   able to change server settings.
 3. **Two console pages** under Compliance: **Overview** and **Data
   inventory**.
 The catalog records facts and the auditor reports findings. Neither judges:
 nothing in code, docs, UI text or output claims the server meets a legal
 standard, or that one is guaranteed. Nothing here changes a default; proposed
 changes are listed under [Default profile](#default-profile) for John to
 decide.
 **Out of scope** (their own specs): retention policies, data subject requests,
 the guided questionnaire, jurisdiction packs, mobile and ihasmail screens.
 Legal hold and the audit log are built already (audit-hold-lock spec, phases
 1–4); this spec places them in the Compliance navigation and gives the
 compliance role access to them, nothing more.
 ## 1. Vocabulary
 **Categories** (fixed; a source may have several):
 | Category | Means |
 |---|---|
 | `identifier` | Names, email addresses, account and principal ids, login names |
 | `contact` | Phone numbers, postal addresses, vCard details, contact addresses given for an object (an OAuth client's contacts) |
 | `network` | IP addresses and ports, client hostnames (EHLO, PTR), user agents, device push URLs |
 | `content` | Message bodies, subjects, attachments, events, contacts' cards, files, scripts, free-text descriptions and reasons |
 | `metadata` | Timestamps, sizes, flags, envelopes, delivery status, message-ids, counts |
 | `credential` | Passwords and their hashes, tokens, keys, secrets |
 **Whose** (data subjects): `holder` (the account holder), `correspondent`
 (people who send to or receive from the server, and people in contact cards
 and events), `administrator` (people acting on the server).
 **Where** (location): `data-store`, `blob-store`, `search-store`,
 `in-memory-store` (the configured one; the data store by default),
 `memory` (process RAM, per node), `log-file`, `external` (sent to an
 endpoint). **Leaves the host** is a separate column: a store is local unless
 the operator points it at a remote backend.
 **Scope**: `tenant` (attributable to a tenant through the account or domain),
 `server` (server-wide only).
 ## 2. Source map
 Defaults are those of a new install: the struct defaults, plus what first boot
 inserts (`crates/common/src/manager/defaults.rs`). "Unbounded" means no setting
 or process removes the data on a schedule.
 ### 2.1 Mail, groupware and user content
 | Source | Categories | Whose | Controlled by | Retention bound | Where / leaves host | Scope | Written by | Default |
 |---|---|---|---|---|---|---|---|---|
 | Emails, mailboxes, threads | content, identifier, metadata | holder, correspondent | always on; `x:Email.encryptAtRest`, `x:Email.maxMessages` | unbounded until deleted; Trash emptied after `x:DataRetention.expungeTrashAfter` | data + blob store; leaves when sent | tenant | `email/src/message/ingest.rs` `email_ingest`; `delivery.rs` `deliver_message`; `message/delete.rs` `emails_delete`, `emails_auto_expunge` | collected; Trash 30 d |
 | Email submissions | metadata (envelopes) | holder, correspondent | always on | `x:DataRetention.expungeSubmissionsAfter` | data store | tenant | `message/delete.rs` `purge_email_submissions` (purge) | 3 d |
 | Full-text index: mail | content, identifier, metadata | holder, correspondent | `x:Search.indexEmail`, `indexEmailFields` | follows the item | search store (external if Elasticsearch/Meilisearch) | tenant | `services/src/task_manager/index.rs` `build_email_document` | indexed, all fields |
 | Calendars, events, iTIP | content, identifier, metadata | holder, correspondent | `x:CalendarScheduling.enable`, `httpRsvpEnable`; `x:CalendarAlarm.enable` | unbounded; scheduling inbox `expungeSchedulingInboxAfter`, share notices `expungeShareNotifyAfter` | data + blob store; **iMIP mails event details to external attendees** | tenant | `dav/src/calendar/update.rs`; `jmap/src/calendar_event/set.rs`; `groupware/src/calendar/storage.rs`; `services/src/task_manager/imip.rs` | collected; 30 d / 30 d |
 | Contacts, address books | contact, identifier | correspondent, holder | always on | unbounded | data store | tenant | `dav/src/card/update.rs`; `jmap/src/contact/set.rs`; `groupware/src/contact/storage.rs` | collected |
 | Full-text index: calendars, contacts | content, contact, identifier | correspondent, holder | `x:Search.indexCalendar`/`Fields`, `indexContacts`/`Fields` | follows the item | search store | tenant | `index.rs` `build_calendar_document`, `build_contact_document` | indexed |
 | Files (FileNode, WebDAV) | content, metadata | holder | `x:FileStorage.maxSize`, `maxFiles` | unbounded | data + blob store | tenant | `dav/src/file/update.rs`; `jmap/src/file/set.rs`; `groupware/src/file/storage.rs` | collected |
 | Sieve scripts, vacation responses | content | holder (scripts may name correspondents) | user-set | unbounded | data + blob store | tenant | `jmap/src/sieve/set.rs`; `managesieve/src/op/putscript.rs`; `jmap/src/vacation/set.rs` | when set |
 | Sieve "seen" ids (vacation, duplicate) | identifier (blake3 hash) | correspondent | used only by scripts that call vacation/duplicate | the script's TTL | in-memory store | tenant | `email/src/sieve/ingest.rs` | when used |
 | Identities | identifier, content (signatures) | holder | `x:Email.maxIdentities` | unbounded | data store | tenant | `jmap/src/identity/set.rs` | collected |
 | Push subscriptions | network (push URL), credential (keys); **EmailPush can carry email properties to the push URL** | holder | client opt-in; `x:Jmap.maxSubscriptions` | expire within 7 days | data store → **external push service** | tenant | `jmap/src/push/set.rs`; `services/src/state_manager/http.rs` | client-driven |
 ### 2.2 Records kept by inbuxa's own features
 These live in inbuxa's own key space (`SUBSPACE_INBUXA`) unless noted.
 `Store::danger_destroy_account` doesn't clear that space, so a row's
 "survives account deletion" note matters.
 | Source | Categories | Whose | Controlled by | Retention bound | Where | Scope | Written by | Default |
 |---|---|---|---|---|---|---|---|---|
 | Archived items (undelete) | content, identifier, metadata | holder, correspondent | `x:DataRetention.archiveDeletedItemsFor`; a legal hold forces keeping | each item's `archivedUntil`; **held: year 9999** | registry (data store) + blob store | tenant | `features/src/undelete/records.rs` `insert`, `set_deadline`, `remove_expired`; `undelete/email.rs` `note`; `groupware/src/inbuxa.rs`; `services/.../index.rs` `archive_noted` | **off** |
 | Deleted accounts kept | identifier (name, addresses), **credential (the whole account record, password hashes included)**, metadata | holder | `x:DataRetention.archiveDeletedAccountsFor`; a hold forces keeping | `kept_until`; held: year 9999 | data store | tenant | `jmap/src/inbuxa/deleted_account.rs` `keep`; `features/src/undelete/data.rs` `set_kept_account` | **off** |
 | Legal holds | identifier, content (reference, description, reasons) | administrator, holder | `sysLegalHold*` | **unbounded, by design** | data store | server (scope may name tenants) | `features/src/hold/mod.rs` `create`, `update`, `pin_account`, `keep_moved` | none |
 | Hold exports: records | identifier, content (reason), metadata | administrator | `sysLegalHoldExport` | **unbounded, by design** | data store | server | `hold/mod.rs` `create_export`, `update_export` | none |
 | Hold exports: ZIP | content (everything the hold covers) | holder, correspondent | as above | `x:Jmap.uploadTtl`, then the next blob purge (`blobCleanupSchedule`) | blob store; **leaves when downloaded** | server | `jmap/src/inbuxa/hold_export.rs` `build` → `put_jmap_blob` | 1 h + up to a day |
 | Account locks, delegations | identifier, content (reason), metadata | holder, administrator | `sysAccountLock*` | until the lock is removed; **no removal on account deletion found** | data store | tenant | `features/src/lock/mod.rs` `set`, `remove` | none |
 | Audit log | identifier, **network (sign-in IPs)**, content (setting diffs, secrets redacted; reasons) | administrator, holder (as target), correspondent (e.g. banned IPs) | always on; `inbuxa:AuditSettings.keepForDays` (not a registry setting) | `keepForDays`, minimum 90; records about held accounts kept while held; **kept after account deletion** | data store (hash-chained) | tenant-filterable | `common/src/audit.rs` `audit_append`, `audit_sign_in`, `audit_sign_in_failed`, `audit_foreign_access`, `audit_delegate`; `jmap/src/inbuxa/audit.rs` `recorded`; purge `audit_purge` | **collected, 730 d** |
 | Audit exports | as the audit log | as above | `sysAuditExport` | `x:Jmap.uploadTtl`, then the next blob purge | blob store; leaves when downloaded | server or tenant | `jmap/src/inbuxa/audit_log.rs` `build_export` | 1 h + up to a day |
 | Masked addresses | identifier, content (description, site), metadata (last message) | holder | `x:Email.maxMaskedAddresses` | the registry object goes with the account; **the address tombstone is kept forever by design**; its `Mm`/`Mc` records aren't cleared on deletion (search found no clearing) | registry + data store | tenant | `features/src/masked_email/ops.rs`; `data.rs` `set_address`, `set_record`, `log_change` | when used (5 max) |
 | Legacy-protocol last use | metadata (one timestamp per account and protocol; no IP) | holder | none | **unbounded; not cleared on account deletion** | data store | tenant | `features/src/security/legacy_use.rs` `record` (from `common/src/network/legacy.rs` `admit_legacy_session`) | **collected** |
 ### 2.3 Mail transport and reports
 | Source | Categories | Whose | Controlled by | Retention bound | Where / leaves host | Scope | Written by | Default |
 |---|---|---|---|---|---|---|---|---|
 | Queue (`x:QueuedMessage`) | content, metadata, network (`receivedFromIp`) | holder, correspondent | always on | delivered, or `x:MtaDeliveryExpirationTtl.expire` | data + blob store; delivered out | server | `smtp/src/queue/spool.rs` `queue`, `save_changes`, `remove` | 3 d max |
 | DSNs | content (**original headers**), metadata | correspondent, holder | `x:DsnReportSettings` | sent through the queue | **leaves** | server | `smtp/src/queue/dsn.rs` `send_dsn` | on |
 | Received DMARC, TLS-RPT, ARF reports | network (source IPs), identifier (envelope/header from); ARF: **message and headers**, original recipient, user agent | correspondent | `x:ReportSettings.inboundReportAddresses`, `inboundReportForwarding` | `x:DataRetention.holdMtaReportsFor` | data store; also forwarded to the mailbox | tenant (by domain) | `smtp/src/reporting/analysis.rs` | collected, 30 d |
 | DMARC aggregate reports we send | network (source IPs), identifier (domains) | correspondent | `x:DmarcReportSettings.aggregateSendFrequency`; only to domains publishing `rua` | kept until sent | data store → **external `rua`** | server | `smtp/src/reporting/dmarc.rs` | daily |
 | DMARC/SPF/DKIM failure reports we send | content (**raw headers**), network, identifier | correspondent | `x:DmarcReportSettings.failureSendFrequency`, `x:SpfReportSettings`, `x:DkimReportSettings`; only when requested | through the queue | **external `ruf`** | server | `reporting/dmarc.rs` `send_dmarc_report`; `spf.rs`; `dkim.rs` | on, rate-limited |
 | TLS-RPT we send | network (remote MX hosts) | operators of other servers | `x:TlsReportSettings.sendFrequency` | until sent (deletion path not confirmed) | **external** | server | `smtp/src/reporting/tls.rs` | daily |
 | Relay (smart host) | everything in the message | holder, correspondent | `x:MtaRoute` (Relay) | receiver's | **external** | server | `smtp/src/outbound` | none |
 | Milters, MTA hooks | network (client IP, PTR, HELO), identifier (SASL login), **full message** | holder, correspondent | `x:MtaMilter`, `x:MtaHook` | receiver's | **external** | server | `smtp/src/inbound/milter`, `inbound/hooks/mod.rs` | none |
 ### 2.4 Telemetry and logs
 | Source | Categories | Whose | Controlled by | Retention bound | Where / leaves host | Scope | Written by | Default |
 |---|---|---|---|---|---|---|---|---|
 | Stored traces (`x:Trace`) | network (IP, EHLO), identifier (envelope addresses, account names), metadata (message-ids, results) | correspondent, holder | `x:TracingStore`; `x:EventTracingLevel` | `x:DataRetention.holdTracesFor` (unset = unbounded) | data store, or a remote PostgreSQL/MySQL/FDB | server (records name accounts) | `common/src/telemetry/tracers/store.rs` `spawn_store_tracer`; purge `purge_spans` | **collected** (first boot inserts `TracingStore::Default` although the struct default is off); 30 d |
 | Trace search index | same keywords, lowercased | as above | `x:Search.indexTelemetry`, `indexTracingFields` | purged with the traces | search store | server | `index.rs` `trace_search_document` | indexed |
 | Stored metrics | none personal (aggregates) | — | `x:MetricsStore`, `x:Metrics` | `holdMetricsFor` | data store | server | `telemetry/metrics/store.rs` | collected, 90 d |
 | Log file | network (**client IP on every in-session line**, from the span), identifier (MAIL FROM, RCPT TO, account names), metadata (message-ids). No subject key exists; bodies only in trace-level raw-I/O events | correspondent, holder, administrator | `x:TracerLog` (`level`, `events`, `path`, `rotate`) | **unbounded: rotated daily, never deleted** | log file (`/var/log/inbuxa`); read back as `x:Log` and fed to Explain | server | `common/src/telemetry/tracers/log.rs` | **collected, info level** |
 | Console / journald tracers | as the log file | as above | `x:Tracer` Stdout/Journal | the host's journal policy | stdout / journald (leaves only if the host forwards it) | server | `tracers/stdout.rs`, `journald.rs` | none (recovery mode adds a console tracer) |
 | OpenTelemetry tracer | as the log file, every event and span key | as above | `x:Tracer` OtelHttp/OtelGrpc (`level`, `events`, `endpoint`) | receiver's | **external** | server | `tracers/otel.rs` | none |
 | Webhooks | as the log file; **can include raw SMTP input (message content) and model replies** — see [Findings](#findings) 1 | as above | `x:WebHook` (`events`, `eventsPolicy`, `url`) | receiver's; retried up to `discardAfter` | **external** | server | `telemetry/webhooks/mod.rs` `post_webhook_events` | none |
 | Alerts | contact (the addresses an administrator sets) | administrator | `x:Alert` | as mail | mail; may leave | server | `telemetry/alerts.rs` | none |
 | Failed tasks | identifier (account names, e.g. a failed account destroy), content (iTIP messages) | holder, correspondent | none | **not confirmed: failed tasks are rescheduled at `u64::MAX` and no purge was found** | data store | server | `services/src/task_manager/manager.rs` | when a task fails |
 ### 2.5 Sign-in and network defence
 | Source | Categories | Whose | Controlled by | Retention bound | Where | Scope | Written by | Default |
 |---|---|---|---|---|---|---|---|---|
 | Accounts, groups, lists, domains, tenants (registry) | identifier, contact, metadata | holder | registry | the object's life | data store | tenant | registry writes (`jmap/src/registry/`) | — |
 | Passwords, app passwords, API keys | credential (argon2id hashes), network (`allowedIps`), metadata | holder | `x:Authentication.passwordHashAlgorithm`; `maxAppPasswords`, `maxApiKeys`; `expiresAt` | the account's life or `expiresAt` | data store | tenant | `jmap/src/registry/mapping/account.rs` | — |
 | OAuth clients | identifier, contact (`contacts`), credential (secret hash) | administrator, third-party developer | `x:OidcProvider.requireClientRegistration` | `expiresAt` or unbounded | data store | tenant | `http/src/auth/oauth/registration.rs` | none |
 | OAuth codes and tokens | credential | holder | `x:OidcProvider.*Expiry` | tokens are sealed and stateless (not stored); codes in the in-memory store with TTL | in-memory store | tenant | `http/src/auth/oauth/auth.rs`, `token.rs` | codes 10 min |
 | Rate-limit state | network, identifier (login names) | holder, correspondent | `x:Http.rateLimit*`, `x:Imap.maxRequestRate`, `x:Security.*BanRate` | the rate's period | in-memory store | server | `common/src/auth/rate_limit.rs`, `network/security.rs` | on |
 | Greylist | identifier (sender/recipient pairs, plain) | correspondent, holder | `x:SpamSettings.greylistFor` | that period | in-memory store | server | `smtp/src/inbound/rcpt.rs` | **off** |
 | Automatic IP bans (`x:BlockedIp`) | network | correspondent, holder | `x:Security.authBanRate`, `abuseBanRate`, `loiterBanRate`, `scanBanRate` (on); `*BanPeriod` (**no default**) | **unbounded: a ban with no period never expires, and no purge of expired bans was found**; each ban is also an audit record | data store (registry) | server | `common/src/network/security.rs` `block_ip` | **collected, permanent** |
 | Allowed IPs | network | administrator's choice | manual; `expiresAt` | optional | data store | server | registry | none |
 ### 2.6 Spam filter and AI
 | Source | Categories | Whose | Controlled by | Retention bound | Where / leaves host | Scope | Written by | Default |
 |---|---|---|---|---|---|---|---|---|
 | Training samples (`x:SpamTrainingSample`) | content (**the whole message, pinned**), identifier (`from`), content (`subject`) | correspondent, holder | `x:SpamClassifier.model`, `learnSpamFromTraps`, `learnSpamFromRblHits`, `learnHamFromCard`, `learnHamFromReply`; users' Junk moves | `x:SpamClassifier.holdSamplesFor`; **outlives the user deleting the mail** | data + blob store | tenant for users' samples; **server, unattributed, for SMTP autolearn** | `email/src/message/ingest.rs` `add_spam_sample`; `smtp/src/queue/spool.rs` | **collected, 180 d** |
 | Trainer state (`INBUXA_SPAM_TRAIN_DATA`) | metadata (blob hashes with account ids; hashed tokens that include addresses and domains) | correspondent, holder | as above | **unbounded (overwritten on training, never expired)** | blob store | server | `spam-filter/src/modules/classifier.rs` `spam_train` | after 100 + 100 samples |
 | Trained model | hashed weights, not directly personal | — | as above | unbounded | blob store | server | same | same |
 | AI spam classification | content (subject + up to `inbuxa:AiLimits.maxContentBytes` of text) | correspondent | `x:SpamLlm`, `x:AiModel` (`url`), `inbuxa:AiLimits` | the endpoint's; the verdict is written into the stored message | **external endpoint** (local or hosted; a remote URL only warns) | server | `spam-filter/src/analysis/llm.rs`; `common/src/enterprise/llm.rs` | **off** |
 | Sieve `llm_prompt` | whatever a script sends | any | `x:AiModel`; `interactAi` | endpoint's | **external** | per account | `enterprise/llm.rs` `sieve_prompt` | no model |
 | Explain | identifier (return path, recipients), network (trace IPs), metadata (remote replies, log details); never contents or raw I/O | correspondent, holder | `inbuxa:AiLimits.explainEnabled`, `explainModelId` | answer cache: memory, per node, 24 h, 1000 answers | **external endpoint** + memory | server | `jmap/src/inbuxa/explanation.rs`; `features/src/ai/explain/memory.rs` | no model |
 | DNSBL lookups | network (client IPs), identifier (domains; **SHA-1 of addresses** to msbl.org), metadata (MD5 of URLs) | correspondent | bundled `x:SpamDnsblServer` rules; `x:SpamDnsblSettings` | receiver's | **external (DNS)** | server | spam-filter DNSBL module | **on: 17 of 18 bundled lists** |
 | Pyzor | content (SHA-1 digest of normalized body) | correspondent | `x:SpamPyzor` | receiver's | **external: `public.pyzor.org`** | server | `spam-filter/src/modules/pyzor.rs` | **on** |
 | URL-shortener expansion | network (fetches links, which can carry per-recipient tracking tokens) | correspondent, holder | bundled redirector list | — | **external (HTTP)** | server | `spam-filter/src/analysis/url.rs` | **on** |
 ### 2.7 Stores and directories an operator can point elsewhere
 | Registry object | Receives | Default |
 |---|---|---|
 | `x:DataStore` | everything above that says data store | RocksDB, local |
 | `x:BlobStore` (S3, Azure, FS, SQL, FDB) | message bodies, attachments, files, samples, exports | the data store |
 | `x:SearchStore` (Elasticsearch, Meilisearch, SQL, FDB) | full text of mail, contacts, calendars; trace keywords | the data store |
 | `x:InMemoryStore` (Redis) | rate limits, greylist, OAuth codes, Sieve seen ids | the data store |
 | `x:TracingStore`, `x:MetricsStore` | traces, metrics | the data store |
 | `x:Coordinator` (Kafka, NATS, Zenoh, Redis) | cluster broadcasts (whether personal: not confirmed) | off |
 | `x:Directory` (LDAP, SQL, OIDC) | login names, passwords to verify, address lookups | internal |
 | `x:AcmeProvider`, `x:DnsServer` | hostnames, a role address; no personal data found | Let's Encrypt if chosen; manual DNS |
 ## Findings
 Facts from the investigation that matter beyond the catalog. Each is a finding,
 not a judgment; what to do about each is John's call.
 1. **A webhook ignores levels and, left at its defaults, receives every
   event, message content included.** `config/telemetry.rs` builds a webhook's
   interests with `apply_events` alone: no level check. With the default
   `eventsPolicy` (exclude) and no events listed, that is every event type,
   including trace-level `smtp.raw-input` (the raw SMTP bytes, `DATA`
   included) and `ai.llm-response`. The audit-log docs suggest a webhook to
   pass records to a SIEM. Probably upstream behavior; not yet checked
   against upstream.
 2. **Log files are never deleted.** Daily rotation opens a new file; nothing
   removes old ones, and no logrotate configuration ships. At the default
   level every in-session line carries the client IP.
 3. **Automatic IP bans are permanent.** No `*BanPeriod` has a default, and no
   purge of expired `x:BlockedIp` records was found.
 4. **Some records outlive the account.** Deleting an account doesn't clear
   inbuxa's own key space: legacy-protocol last use, account locks, masked
   address records and audit records stay (audit records by design).
 5. **A kept deleted account keeps its password hashes** for
   `archiveDeletedAccountsFor`, or indefinitely under a legal hold.
 6. **Spam training keeps whole messages for 180 days**, after the user
   deleted them, and SMTP autolearn keeps them with no account attribution, so
   they can't be found by person. The trainer state never expires.
 7. **Traces are on in a new install** although the struct default says off:
   first boot inserts the tracing and metrics stores.
 8. **Data leaves the host by default** through the spam filter (DNSBL queries
   with IPs, domains and hashed addresses to 17 lists; Pyzor body digests;
   shortener URL fetches) and through reports (DMARC aggregate daily; failure
   reports with headers where requested). No AI, webhook, OpenTelemetry,
   milter, hook or relay endpoint is configured by default.
 9. **Export files outlive their stated lifetime by up to a day.** The link
   expires with `uploadTtl`; the bytes stay until the next blob purge
   (04:00 daily).
 10. **Failed tasks may stay forever** (rescheduled at `u64::MAX`; no purge
    found), holding account names or iTIP messages. Not confirmed by a test.
 ## 3. How the schema is produced, and what the catalog works from
 The prompt says not to hand-edit generated files and to work from the source
 of truth. What the investigation found:
 - **`resources/schema/schema.json.gz` and the registry Rust code
  (`crates/registry/src/schema/*.rs`, headed "auto-generated") come from
  upstream's generator, which isn't in this repository or upstream's public
  tree.** Nothing here regenerates them.
 - **The fork edits both by hand.** Since the v0.16.22 import the gz has
  changed in 18 commits: the Compliance layout (Audit Log, Legal Holds, Locked
  Accounts), the Local AI and Legacy Protocols pages, the fork's permissions
  (`sysAudit*`, `sysAccountLock*`, `sysLegalHold*`, `sysAiExplain`), always
  paired with `enums.rs`/`enums_impl.rs` edits marked `// inbuxa:`. Scripted
  edits go through `tools/fork/renames.py` (`schema_bytes`), which writes the gz
  with `gzip.compress(compresslevel=9, mtime=0)` and the `.sha256` sidecar as
  unpadded URL-safe base64 of the gz's SHA-256.
 - **`/api/schema`** (`crates/http/src/api/mod.rs`, `"schema"` arm) serves the
  embedded gz as-is to any authenticated caller, unfiltered; the console
  hides what a person can't use from the permission list `/api/account`
  returns.
 - **The schema carries 150 objects and 2,102 properties** (316 field
  schemas). About 34 objects hold data about people, roughly 150–180
  properties counting nested structs; about 20 configuration objects carry
  about 100 credential or contact properties. inbuxa's own objects
  (`inbuxa:AuditEvent`, `LegalHold`, `HoldExport`, `AccountLock`,
  `DeletedAccount`, `ProtocolPolicy`, `TenantProtocolPolicy`,
  `Explanation`, `AiLimits`, Fastmail `MaskedEmail`) aren't in the schema
  at all: they are defined in `crates/jmap-proto/src/object/inbuxa_*.rs`.
 So the source of truth for *what exists* is the schema (for registry objects)
 and `jmap-proto` (for inbuxa's own). Neither should carry the catalog:
 **Decision (Settled 1):** the catalog is a sidecar, and the schema and generated
 code are read, never annotated, by it. The fork's existing hand edits to the
 schema (layout entries, permissions) continue as today for the Compliance
 pages and the new permissions, because the console's navigation comes from
 the schema and there is no other way in; each is marked in the commit and
 listed in the strip report as now.
 ## 4. The catalog: format and location
 **Location:** `resources/privacy/catalog.toml`, a fork-owned file. The server
 embeds it (`include_str!`) and parses it at start, as it does the schema.
 **Why a sidecar** rather than annotating in place:
 - The schema and registry code are upstream's generated output; annotations
  there are overwritten or conflict on every import (SPEC.md §2.2, §2.3).
 - Many sources aren't registry objects: log files, telemetry exporters,
  inbuxa's key-space records, DNSBL and Pyzor, the in-memory cache. An
  in-place annotation has nowhere to put them.
 - One file is one diff to review when an import brings new fields, and it is
  what the CI check and the strip report read.
 **Shape.** Three kinds of entry:
 ```toml
 # A registry object: a default for its properties, and the ones that differ.
 [object."x:Account"]
 default = "none"                  # properties not listed hold no personal data
 whose = ["holder"]
 where = ["data-store"]
 scope = "tenant"
 retention = "object-life"
 [object."x:Account".properties]
 name = ["identifier"]
 emailAddress = ["identifier"]
 aliases = ["identifier"]
 description = ["content"]
 credentials = ["credential", "network"]   # secrets, allowedIps
 locale = ["metadata"]
 # One of inbuxa's own JMAP objects (not in the schema).
 [object."inbuxa:AuditEvent"]
 default = "none"
 whose = ["administrator", "holder", "correspondent"]
 where = ["data-store"]
 scope = "tenant"
 retention = { setting = "inbuxa:AuditSettings.keepForDays" }
 [object."inbuxa:AuditEvent".properties]
 actor = ["identifier"]
 remoteIp = ["network"]
 changes = ["content"]
 reason = ["content"]
 # A source that is no object: files, exporters, lookups, caches.
 [source."log-file"]
 categories = ["network", "identifier", "metadata"]
 whose = ["correspondent", "holder", "administrator"]
 where = ["log-file"]
 scope = "server"
 enabled_by = ["x:TracerLog.enable"]
 captures = ["x:TracerLog.level", "x:TracerLog.events", "x:EventTracingLevel"]
 retention = "unbounded"
 leaves_host = false
 written_by = ["crates/common/src/telemetry/tracers/log.rs"]
 [source."spam-dnsbl"]
 categories = ["network", "identifier", "metadata"]
 whose = ["correspondent"]
 where = ["external"]
 scope = "server"
 enabled_by = ["x:SpamDnsblServer.enable"]
 retention = "receiver"
 leaves_host = true
 written_by = ["crates/spam-filter/src/modules/dnsbl.rs"]
 ```
 Rules: `retention` is `unbounded`, `object-life`, `receiver` (it left the
 host), or `{ setting = "<object>.<property>" }`; every setting named is a real
 `x:` or `inbuxa:` property. Every object in the schema and every `inbuxa:`
 object has an entry, even if only `default = "none"` (configuration objects
 with nothing personal). The file holds facts only: no wording about legal
 status.
 **Evaluation (Phase 3).** Each entry's `enabled_by`, `retention` and
 `leaves_host` are evaluated against the live registry: a store pointed at a
 remote backend makes everything in it `leaves_host = true`; an unset
 retention setting reads `unbounded`; an external endpoint lists its URL's
 host as a candidate processor.
 ## 5. The CI check
 `tools/fork/privacy-check.py`, modeled on `name-check.py` and
 `notice-check.py` (plain Python, no dependencies, exit 1 on findings). Wired
 into the `fork-checks` job in `.gitea/workflows/ci.yml`, beside them. It fails
 when:
 1. an object in `schema.json.gz` `fields`, or an `inbuxa:` object in
   `crates/jmap-proto/src/object/`, has no catalog entry;
 2. a property whose schema format is `emailAddress`, `ipAddress`,
   `ipNetwork`, `secret`/`secretText`, or whose type is `x:SecretKey*` or
   `x:HttpAuth`, is covered only by an object's `default = "none"` — those
   must be classified explicitly, so a new personal field can't hide behind a
   default;
 3. an entry names an object, property or setting that no longer exists
   (stale);
 4. an entry uses a category, subject or location outside the vocabulary.
 `tools/fork/strip.py` gains `privacy_flags(tree)`, beside `schema_flags`:
 objects and properties new in an import and not in the catalog, reported in
 `STRIP-REPORT.md` under "Unclassified in the privacy catalog", the way
 Enterprise flags are reported. Informational, as the Enterprise list is; CI
 is what fails.
 Tests (Phase 2): the check passes on main; fails on a synthetic unclassified
 field; fails on a synthetic `emailAddress` property covered only by a
 default; fails on a stale entry.
 ## Default profile
 **Today's defaults in a new install** that collect or send personal data:
 | Source | Collected | Retention |
 |---|---|---|
 | Traces + trace index | yes | 30 days |
 | Log file | yes, info level | **unbounded** |
 | Audit log | yes | 730 days |
 | Spam training samples (whole messages) | yes | 180 days |
 | Spam trainer state | yes, after 200 samples | **unbounded** |
 | Automatic IP bans | yes | **unbounded** |
 | Legacy-protocol last use | yes | **unbounded** |
 | Received DMARC/TLS/ARF reports | yes | 30 days |
 | DNSBL lookups (17 lists), Pyzor, URL-shortener fetches | yes | leave the host |
 | Outbound DMARC aggregate and failure reports | yes, where the domain asks | leave the host |
 | Trash | yes | 30 days |
 | Undelete, deleted-account keeping | **off** | — |
 | AI classification, Explain | **off** (no model) | — |
 | Greylisting | **off** | — |
 **Changes for new installs only.** Settled (5): all seven are built, in
 Phase 3; existing servers keep their settings. D7 is also covered by the bug
 fix for finding 1 (Settled 6).
 | # | Change | Trade-off |
 |---|---|---|
 | D1 | A **log retention** setting on `x:TracerLog` (delete rotated files older than N days), default 30 days for new installs | Needs code (a new field, so a schema edit); older logs gone for troubleshooting; operators wanting longer set it |
 | D2 | A default **ban period**, e.g. 30 days, for the four ban rates, and a purge of expired `x:BlockedIp` | A persistent attacker is re-banned after expiry rather than kept out; permanent bans of shared/NAT addresses stop being permanent |
 | D3 | **Spam sample retention** 180 → 90 days | Fewer samples to retrain from; the classifier is retrained regularly, so the effect on accuracy is likely small (not measured) |
 | D4 | **Pyzor off** by default | One fewer spam signal; no body digests leave the host |
 | D5 | **msbl.org EBL off** by default (the list that receives hashed addresses) | One fewer signal on address-based spam; no hashed addresses leave the host |
 | D6 | **Trace retention** 30 → 14 days | Shorter delivery history in **Emails › History** and for support |
 | D7 | Webhooks default to `eventsPolicy = include` with no events, and never receive raw-I/O events unless listed by name | A webhook does nothing until events are chosen; closes finding 1 |
 Separately, **not defaults but gaps** a later spec should close (listed so
 they aren't lost): clearing inbuxa's own records on account deletion
 (finding 4); an expiry for the spam trainer state; a purge for failed tasks
 (finding 10, once confirmed); removing export blobs when their link expires
 (finding 9).
 ## 6. The inventory, over JMAP (Phase 3)
 Under `urn:inbuxa:jmap`, read-only.
 ### `inbuxa:DataInventory/get`
 A singleton, evaluated on request.
 Request: `{ accountId, ids: null | ["singleton"], properties? }`.
 Response `list[0]`:
 ```json
 {
  "id": "singleton",
  "evaluatedAt": "2026-09-28T10:00:00Z",
  "catalogVersion": "2026.9.28.3",
  "sources": [
    {
      "id": "log-file",
      "kind": "source",
      "categories": ["network", "identifier", "metadata"],
      "whose": ["correspondent", "holder", "administrator"],
      "where": ["log-file"],
      "collected": true,
      "retention": { "unbounded": true },
      "leavesHost": false,
      "scope": "server",
      "controlledBy": ["x:TracerLog.enable", "x:TracerLog.level"],
      "endpoints": []
    },
    {
      "id": "x:Trace",
      "kind": "object",
      "categories": ["network", "identifier", "metadata"],
      "collected": true,
      "retention": { "days": 30, "setting": "x:DataRetention.holdTracesFor" },
      "leavesHost": false,
      "scope": "server",
      "endpoints": []
    }
  ],
  "processors": [
    { "host": "public.pyzor.org", "receives": ["content"], "sources": ["spam-pyzor"] }
  ]
 }
 ```
 `collected` is false when the source's switch is off; `retention` reads the
 live setting; `leavesHost` is true when the source's location is a remote
 store or an external endpoint; `processors` lists each external host once,
 with what it receives — candidates, since whether a host is a processor in
 law is the operator's determination.
 **Permission:** `sysComplianceGet`. **Tenant scoping:** inside a tenant,
 `sources` holds only `scope = "tenant"` entries, evaluated with the tenant's
 own settings where it has them (its protocol switches, its domains' stores
 are the server's); server-scope sources and `processors` are left out, because
 they describe the whole server. A tenant principal without the permission gets
 `forbidden`.
 ### `inbuxa:InventorySnapshot/get`, `/query`
 A dated copy of the evaluated inventory, taken when a setting the catalog
 references changes (hooked where the audit log already sees registry writes)
 and at most once a day otherwise. `query` filters by `after`/`before`,
 newest first. Fields: `id`, `takenAt`, `trigger` (`setting-changed` with the
 setting, or `daily`), `summary` (counts: sources collected, unbounded,
 leaving the host, processors), and on `get` the full inventory as above.
 Kept for `inbuxa:AuditSettings.keepForDays`, so history is as long as the
 audit log's. Same permission and tenant scoping as the inventory.
 ## 7. The compliance role
 ### What it holds
 A new built-in role, **Compliance Officer**, at server level:
 | Permission | New? | Gives |
 |---|---|---|
 | `sysComplianceGet` | new | Overview, Data inventory, snapshots |
 | `sysAuditGet`, `sysAuditExport` | existing | read and export the audit log, verify the chain |
 | `sysLegalHoldGet`, `sysLegalHoldCreate`, `sysLegalHoldUpdate`, `sysLegalHoldExport` | existing | place, widen, release and export holds |
 | `sysAccountLockGet` | existing | see locks and delegations |
 | read (`*Get`, `*Query`) on accounts, groups, lists, domains, tenants, roles | existing | know who and what the records refer to |
 It holds **no** `*Create`/`*Update`/`*Destroy` on registry objects, no
 `sysAuditSettingsUpdate`, no `impersonate`, no `fetchAnyBlob`. It can't
 change a server setting.
 **Settled (2):** the officer *places and releases* holds, because that is
 the job; every placing, widening and release is already recorded with its
 reason, under the officer's own identity (AU-12), and a release can't be
 undone silently (LH-10).
 A **Tenant Compliance Officer** (inside a tenant: `sysComplianceGet`,
 `sysAuditGet`, `sysAuditExport`, `sysAccountLockGet`, reads) is built with
 it (Settled 3). It has no hold permissions, since legal holds are
 server-only by the tenant ceiling (LH-13), and it sees the tenant's slice of
 the inventory only (§6).
 ### Reaching existing servers
 New permissions get ids 673 onward and `COUNT` grows (`enums.rs`,
 `enums_impl.rs`, and the schema's `Permission` enum, a marked hand edit as
 before). `DefaultPermissions` places `sysComplianceGet` with superusers;
 `granted_permissions.rs` `ADMIN_GRANTS` adds it once to stored administrator
 roles, as it did for the audit and hold permissions. The **role object**
 itself is seeded only on a fresh install (`defaults.rs`, `count == 0`); for
 existing servers a one-time step modeled on `granted_permissions.rs` creates
 it once, recorded so that deleting it sticks. That step is new code.
 ### Keeping administrators reviewable
 What already holds: every administrator's change to the registry and to
 inbuxa's own objects is recorded before it is allowed, under their identity,
 with a reason where required; sign-ins by anyone holding `sys*` permissions
 are recorded; records can't be deleted over JMAP; the chain is verifiable;
 changing retention is itself recorded.
 What this adds: the compliance officer reads all of it, server administrators'
 actions included, without being an administrator. The Overview (§8) puts on
 top anything that weakens review: audit retention shortened, a tracer or the
 audit export webhook removed, a change to who holds the compliance or
 administrator roles, a hold released — each with who and when.
 **Settled (4):** a server administrator can still shorten audit retention to
 90 days; that is recorded and surfaced on the Overview, not gated on a second
 person, since a server may have only one. Anyone with shell access can edit
 the store directly; that is outside what the server can review.
 ## 8. The Compliance section in the console
 The navigation comes from the schema's `layouts` (`/api/schema`); a
 hand-built page is a `CustomComponent/<Name>` link there, a render branch in
 `MainContent.tsx`, and a visibility rule in `layout.ts` `checkSpecialLink`.
 The section shows when at least one of its pages is visible to the person
 (Phase 4 confirms the container behaves so with every child hidden).
 Planned navigation, in order:
 | Page | This task | Built as | Visible with |
 |---|---|---|---|
 | **Overview** | **builds** | hand-built (`CustomComponent/ComplianceOverview`) | `sysComplianceGet` |
 | **Data inventory** | **builds** | hand-built (`CustomComponent/DataInventory`) | `sysComplianceGet` |
 | Retention | later spec | likely hand-built over `x:DataRetention`, `x:SpamClassifier`, `x:TracerLog` and `inbuxa:AuditSettings`, since they're spread across objects | — |
 | Legal holds | exists | hand-built | `sysLegalHoldGet` |
 | Audit log | exists | hand-built | `sysAuditGet` |
 | Locked accounts | exists | hand-built | `sysAccountLockGet` |
 | Data subject requests | later spec | hand-built | — |
 | Records and documents | later spec | could be schema-driven if it becomes a registry object | — |
 | Jurisdiction packs | later spec | hand-built | — |
 **Decision:** later pages are left out of the navigation until
 built, not shown disabled: a disabled entry reads as a feature that exists.
 **Overview** shows the latest snapshot's findings as facts ("Log files are
 kept with no limit", "Sent to a host outside this network: public.pyzor.org"),
 the review items from §7, and the snapshot history (when and why the inventory
 changed). **Data inventory** lists the evaluated sources, filterable by
 source, category and location, with each external endpoint listed as a
 candidate processor. UI text states facts and never claims a legal standard is met.
 ## Contradictions with the docs and SPEC.md
 1. **docs, Security › "What the AI features do and do not send":** "message
   content is never written to the logs". True at the default level. The
   model's reply to the spam classifier is logged as `ai.llm-response` at
   trace level (up to 1,024 characters, which can restate the message), so a
   trace-level tracer writes it, and any webhook at its defaults receives it,
   along with raw SMTP input (finding 1).
 2. **docs, Audit log:** suggests a webhook to pass records to a SIEM, without
   saying a webhook's `level` is ignored and its default policy sends every
   event.
 3. **docs, Monitoring:** "rotated daily" is right, but doesn't say old files
   are never removed (finding 2). Not strictly a contradiction; an omission
   that matters here.
 4. **SPEC.md §2.3** calls `schema.json.gz` upstream's published schema and
   the checklist for rebuilt features. It doesn't say the fork now hand-edits
   it (layout, permissions); §2.2's strip step would overwrite those edits on
   an import unless they're re-applied. Worth a sentence in SPEC.md.
 5. **The schema's `TracingStore`/`MetricsStore` defaults** say disabled; first
   boot turns both on. The docs describe the real behavior; the schema default
   (what the console shows as the default) doesn't.
 The rest checked out: tracing 30 days, metrics 90 days, the Explain cache (in
 memory, a day), audit retention (two years, minimum 90), undelete off by
 default.
 ## Settled
 John's answers, 2026-09-28, to the questions this spec asked:
 1. **The catalog is a sidecar**, `resources/privacy/catalog.toml`; the
   Compliance pages and the new permission are hand-added to the schema as
   the audit and hold work did.
 2. **The Compliance Officer places and releases legal holds.** It's their
   role.
 3. **The Tenant Compliance Officer is built now**, with the server role
   (Phase 3), not after data subject requests.
 4. **Shortening audit retention is recorded and surfaced**, not gated on a
   second person: a server may have only one.
 5. **All seven defaults, D1–D7, for new installs.** Built in Phase 3.
 6. **Finding 1 is fixed now**, as a bug, separately from this work.
 7. **Snapshots are kept as long as the audit log** (`keepForDays`).
 ## Phases
 1. **This spec.** Stop for approval.
 2. **The catalog and its check:** `resources/privacy/catalog.toml`,
   `tools/fork/privacy-check.py` in `fork-checks`, the strip report section,
   and the tests in §5.
 3. **Server:** `sysComplianceGet`, the Compliance Officer and Tenant
   Compliance Officer roles (with the existing-server step),
   `inbuxa:DataInventory`, `inbuxa:InventorySnapshot`, and defaults D1–D7
   for new installs; tests with several configurations (defaults, a remote
   store, a hosted AI endpoint, telemetry off), tenant scoping, refusal
   without the permission.
 4. **Console:** Overview and Data inventory, the navigation entries, a PR
   linking this spec.