From 2a851ea2303fdba28294a6ff265d809e8917a88e Mon Sep 17 00:00:00 2001 From: John Coffey Date: Mon, 28 Sep 2026 15:57:53 -0700 Subject: [PATCH 1/3] Spec: data loss prevention and mail flow rules Phase 1: one native rule engine at DATA, after the system Sieve script, for both DLP policies and transport rules. DLP checks outgoing mail with counted detectors (payment cards, IBAN, US SSN, word lists, patterns) and blocks, warns with an audited override, or holds for review. Held mail stays in the queue unscheduled, with its own review record, so the queue's stored format is unchanged. Matches go to the audit log without the matched text. Six questions for John at the end. --- docs/spec/features/dlp-and-mail-flow-rules.md | 333 ++++++++++++++++++ 1 file changed, 333 insertions(+) create mode 100644 docs/spec/features/dlp-and-mail-flow-rules.md diff --git a/docs/spec/features/dlp-and-mail-flow-rules.md b/docs/spec/features/dlp-and-mail-flow-rules.md new file mode 100644 index 0000000..4f4221c --- /dev/null +++ b/docs/spec/features/dlp-and-mail-flow-rules.md @@ -0,0 +1,333 @@ +# Feature spec: data loss prevention and mail flow rules + +Status: **draft for approval.** Phase 1 of DLP and the rule builder, specced +together because they need the same conditions, the same place in the mail +path and the same record of what matched. Not a rebuild of an upstream +feature, so it has no line in SPEC.md §4's table. Open questions are under +[For John](#for-john); nothing is built until they're answered. + +## Provenance + +Written for the record SPEC.md §3 rule 3 asks for. Sources, and nothing else: + +| Source | License | Used for | +|---|---|---| +| This repository at `0502eb4` (2026-09-28): `crates/smtp/src/inbound/data.rs`, `crates/smtp/src/queue/`, `crates/jmap/src/submission/set.rs`, `crates/common/src/scripts/`, `vendor/sieve-rs`, `resources/schema/schema.json.gz` | AGPL-3.0-only | Where a check can run, what the queue stores, what a sender sees on a refusal | +| inbuxa-admin at `b82904c` | AGPL-3.0-only | Where the pages go | +| ihasmail-inbuxa (the webmail) at `290bc63` | AGPL-3.0-or-later | How a refused send reaches the person sending | +| `inbuxa-drafts/queue/dlp.md`, `rule-builder.md` | Own | What John asked for and settled | +| The personal-data catalog spec and the audit-hold-lock spec | Own | Roles, the audit log, legal holds, the catalog check | +| RFC 5321, RFC 3463 (enhanced status codes), RFC 8620/8621 (JMAP) | Public | Refusal codes and the submission error shape | + +No Enterprise-only file or snippet was used, and no third-party DLP product +was consulted for design: the detectors are public checksum and format rules +(Luhn, ISO 13616 mod 97, the SSA's published SSN rules). + +## What it is + +1. **Data loss prevention (DLP).** Policies that look at mail as someone + sends it, find what shouldn't leave (card numbers, bank accounts, national + ID numbers, words and patterns an organization names, attachments of a + kind), and then **block** it with a notice, **warn** and let the sender + send anyway with a stated reason, or **hold** it until a reviewer releases + or rejects it. +2. **Mail flow rules.** The same engine, for ordinary transport rules an + administrator writes in a form instead of in Sieve: disclaimers, banners, + headers, copies, redirects, refusals. +3. **One record of what matched**, in the audit log, and a **review queue** + for held mail. + +Settled before this spec (John, 2026-09-27 and 2026-09-28): DLP's first +version checks **outgoing mail only**, content and attachments, as it's sent; +its actions are block with a notice, warn with an override (audited), and hold +for review. A record-only action was not chosen. The rule builder goes +alongside DLP, one design; journaling comes after both. + +**Out of scope** (later specs): inbound DLP, files and calendar sharing, +scanning mail already stored, a machine-learning classifier (the local AI +model could add one later; nothing here depends on it), mobile and ihasmail +screens, journaling. + +Nothing in code, docs, UI text or output claims the product meets a legal +standard or prevents every leak. The pages say what a policy checks and what +it did. + +## 1. What exists today + +Checked by reading the code at `0502eb4`: + +| Need | Today | +|---|---| +| A place in the send path that sees every outgoing message | Yes. SMTP submission and webmail sends both reach `Session::queue_message` (`inbound/data.rs`); JMAP submission builds a local session and runs MAIL, RCPT and DATA (`jmap/src/submission/set.rs` ~L690–760). Nothing reaches the queue around it. | +| Order at DATA | Authentication checks → spam filter → milters → MTA hooks → the DATA system Sieve script → headers, DKIM signing → queue. | +| Rules without code | System Sieve (`x:SieveSystemScript`): one script per stage, chosen by an expression on `x:MtaStageData.script`. Hand-written only; the console has a text field. | +| Refusing a message | A 5xx at DATA. The webmail gets `forbiddenToSend` with the text `Server rejected DATA: ` and nothing structured. | +| Holding a message | **Nowhere.** "Quarantine" in the code is only DMARC's disposition. The queue's recipient status is `Scheduled`, `Completed`, `TemporaryFailure`, `PermanentFailure`, archived with rkyv. | +| Reading attachments | Text and HTML parts only. There's no PDF or Office text extraction: search indexes text parts and file names. | +| Recording what happened | The audit log, with system actors, reasons and outcomes (AU-1…AU-12). | +| Who is allowed | Roles with per-permission grants; server and tenant levels; the compliance roles from the catalog spec. | + +## 2. Design + +### 2.1 One engine, native, after the system script + +Rules are evaluated by a new native engine at DATA, **after** the system Sieve +script and before headers and DKIM signing. Mail flow rules and DLP policies +are two views of the same rule list. + +Why not generate Sieve: holding a message, a warning the sender can override, +counted detectors with checksums, and a per-rule match record are all things +Sieve doesn't have. Adding them means new extensions in `vendor/sieve-rs`, +which widens the fork of a crate we'd otherwise take from upstream, and a +generated script would have to share the single DATA script with whatever an +administrator wrote by hand. A native engine leaves hand-written Sieve exactly +as it is: it still runs, first, and the rules see its result. + +The engine lives in `crates/features/src/mailflow/` (pure evaluation over a +parsed message and an envelope, unit-testable), called from `data.rs` behind +one `// inbuxa:` marked block. + +### 2.2 Rules + +A fork-owned JMAP object, `inbuxa:MailRule`, stored in inbuxa's own subspace +like legal holds (not a registry object, so upstream schema imports never +touch it): + +| Property | | +|---|---| +| `name`, `description` | | +| `kind` | `dlp` or `transport`: which page shows it and which permission edits it | +| `enabled` | | +| `priority` | Order; lower runs first | +| `direction` | `outgoing` (authenticated senders), `incoming`, or `any`. **DLP rules are `outgoing` only** in this version. | +| `conditions` | All must match (list below) | +| `exceptions` | Any matching one skips the rule | +| `actions` | What happens (list below) | +| `stopProcessing` | Later rules don't run for this message | +| `tenantId` | Server-level (none) or one tenant's; see [For John](#for-john) Q3 | +| `createdBy`, `updatedAt` | | + +Every create, update and delete is audited with its before and after, like +any setting, and appears on the compliance Overview's **Changes that affect +review**. + +### 2.3 Conditions + +Shared by both kinds: + +| Condition | Matches when | +|---|---| +| Sender | the sender is one of the chosen accounts, or in a chosen group, domain or tenant | +| Recipient | any recipient is one of the chosen addresses, domains, groups | +| **Recipient outside** | any recipient isn't at a domain this server hosts (or, with a tenant rule, isn't in the tenant) | +| Subject or body contains | any of a list of words or phrases (whole words, case-insensitive) | +| Subject or body matches | a regular expression (the `regex` crate: linear time, no backtracking) | +| Header | a header exists, or its value contains or matches | +| Attachment | its detected type, extension or name matches; its size is over a limit; there are more than N | +| **Can't be inspected** | an attachment is encrypted or password-protected (ZIP, PDF, Office), or bigger than the inspection limit | +| Message size | over a limit | + +DLP adds **detectors**, each with a minimum count (for example "5 or more card +numbers"): + +| Detector | Rule | +|---|---| +| Payment card number | 13–19 digits, spaces or dashes allowed, a known issuer prefix, passes Luhn | +| IBAN | country code, check digits and length for that country, passes ISO 13616 mod 97 | +| US Social Security number | `AAA-GG-SSSS` or nine digits beside words like "SSN"; never area 000, 666 or 9xx, group 00, serial 0000 | +| Word list | a list the organization maintains (project names, "confidential"), counted | +| Pattern | the organization's own regular expression, counted | + +What the detectors read: the subject, every text and HTML part (as text), +and attachments whose detected type is text (`text/*`, CSV, JSON, XML). +Office and PDF files are read only if Q2 says so; until then they count as +**can't be inspected** so a policy can still act on them. Inspection stops at +a limit per message (proposed 10 MB of text), and what's past it counts as +can't be inspected too. + +### 2.4 Actions + +**Transport actions** (both kinds): add a disclaimer (text and HTML, top or +bottom, once per thread), add or remove a header, prefix the subject, add a +recipient (a copy), redirect to other recipients, refuse with a text, send +through a chosen route (an existing `x:MtaVirtualQueue`). + +**DLP actions**, exactly one per DLP rule: + +| Action | Sender sees | Message | +|---|---|---| +| **Block** | SMTP `550 5.7.1` with the rule's notice text; in the webmail, the notice in the send dialog | Not accepted; nothing is stored | +| **Warn** | The notice and, once they give a reason, can send anyway | Sent after an override; the reason is audited | +| **Hold for review** | Accepted with "held for review"; a notice mail from the server if the rule asks | Waits in the queue for a reviewer | + +When several DLP rules match, the strictest wins: block, then hold, then warn. + +### 2.5 Warn and override + +**Webmail (JMAP).** The first submission fails with a new error type, +`inbuxa:dlpWarning`, carrying the matched rules' names and notice texts (never +the matched text). The webmail shows them and asks for a reason; it +resubmits with `inbuxa:dlpOverride: {"reason": "..."}` on the +`EmailSubmission` create (capability `urn:inbuxa:jmap`). The server passes the +reason into the local SMTP session as trusted session data, not as a header, +so it can't be forged from the message. An override covers only the rules +that warned; if a block or hold rule also matches, that still applies. + +**Mail apps (SMTP).** They show whatever text the server returns, so the +refusal says how to override: `550 5.7.1 . To send anyway, start the +subject with [override: your reason]`. On the next attempt the engine strips +the tag before DKIM signing, and records the reason. See Q1. + +A block's refusal uses the same error path with `inbuxa:dlpBlocked` in the +webmail. + +### 2.6 Hold for review + +A held message is queued normally but **not scheduled**: its due time is set +to never, and a review record `inbuxa:HeldMessage` (queue id, sender, +recipients, subject, size, matched rules and counts, held at, expires at) is +written in inbuxa's own subspace. The queue's stored format is untouched, so a +node still on the previous version during a rolling upgrade reads the message +fine and simply never sends it. + +The sender gets `250 2.0.0 Held for review`, and the rule may send them a +notice mail. The message stays in their Sent folder as usual. + +A reviewer, under **Management › Compliance › Held mail**: + +- sees the list, and opens one to read it (each opening is audited, like any + access to someone else's mail); +- **releases** it with a reason: it's scheduled at once and delivered as + normal; +- **rejects** it with a reason: it's removed from the queue and the sender + gets a notice with the reviewer's note, not their name. + +Unreviewed mail is rejected after a set time (Q5), with a notice to the +sender. Emails › Queue shows held mail as held and refuses **Retry** on it, so +nobody can deliver it around the review. The sender can't unsend it either +once it's held (the webmail says so). + +Held messages count against no one's quota. Each held message and each +decision is in the audit log. + +### 2.7 What's recorded + +Every DLP match writes one audit record: actor **DLP** (a system actor), +target the message (queue id, sender, recipient domains), the rules and each +detector's count, the action, and for an override the sender's reason. +**Never the matched text**: the log would otherwise become a second copy of +what the policy was keeping in. A card number isn't written, even masked. + +Transport rules that change a message record the rule and action the same way. +Unmatched mail writes nothing. + +### 2.8 Permissions and who does what + +New permissions (ids from 674): + +| Permission | Gives | +|---|---| +| `sysMailRuleGet` / `Update` | See / change transport rules | +| `sysDlpPolicyGet` / `Update` | See / change DLP rules | +| `sysDlpReviewGet` | See held mail and open it | +| `sysDlpReviewUpdate` | Release or reject held mail | + +Proposed defaults: **Administrator** has all; **Compliance Officer** +(server-level) has `sysDlpPolicyGet`, `sysDlpReviewGet`, `sysDlpReviewUpdate`; +tenant roles per Q3. See Q4 on who edits DLP rules. + +### 2.9 Privacy catalog + +New entries, so the catalog check passes: `inbuxa:MailRule` (administrator +identities), `inbuxa:HeldMessage` (sender, recipients, subject: held until +reviewed or expired, then removed), and the held message's content in the +queue (content, the sender's and correspondents'). The DLP audit records are +covered by the audit log's entry. + +### 2.10 Mixed versions and clusters + +Rules and review records live in the shared data store, so every node sees the +same ones. During a rolling upgrade a node still on the old version doesn't +check mail against rules; the Overview can't tell. The console says so when +nodes report different versions, and the release notes say to enable DLP +rules after every node is upgraded. + +### 2.11 Cost + +Rules are compiled once when they change (regexes, word lists as an +Aho-Corasick automaton) and shared by every session. Detectors only run on +mail that some enabled rule could match (direction, sender, recipient checks +first). The inspection limit caps the worst case. + +## 3. Console + +- **Management › Compliance › Data loss prevention**: DLP rules, a form with + conditions, exceptions, detectors and the action; the notice text; what + matched in the last 30 days (from the audit log). +- **Management › Compliance › Held mail**: the review queue. +- **Settings › Mail flow › Rules** (with the settings reorganization's + approved order): transport rules, same form, ordered, with **Stop + processing**. + +Every form previews the rule in words ("If a recipient is outside and the +message contains 5 or more card numbers, hold it for review"). + +## 4. Webmail (ihasmail-inbuxa) + +- A warning dialog: the notice, a reason field, **Send anyway** and **Edit + message**. +- A block dialog with the notice. +- A held message shows as **Held for review** in Sent, and its undo is gone. + +## 5. Tests + +Unit: each detector against valid and near-miss numbers (Luhn-failing cards, +IBANs with a wrong check, SSN areas 000/666/9xx), word lists, regexes, the +inspection limit, the can't-be-inspected cases, rule order and stop +processing. Integration (`tests/src/smtp/`, `tests/src/jmap/`): block, warn +and override over SMTP and JMAP, hold then release and reject, expiry, +Retry refused on held mail, audit records carrying no matched text, a +message queued by a node without the engine (held message format unchanged). + +## 6. Phases + +1. This spec, approved. +2. Engine, conditions, detectors, transport actions; DLP block and warn over + SMTP and JMAP; audit records; catalog entries. +3. Hold for review: review records, release, reject, expiry, queue guard. +4. Console: DLP rules, held mail, mail flow rules. +5. Webmail dialogs; docs; a row in `inbuxa-drafts/divergence-log.md`. + +Each phase is its own PR with tests; releases as John decides. + +## Known gaps + +- Mail a user's own filter forwards automatically to an outside address isn't + checked in this version (it leaves as generated mail, not a submission). +- What a mail app keeps in its own Sent folder, or sends through another + server, is outside what this server sees. +- Detectors find formats, not meaning: a card number in a harmless test + message matches; a number written in words doesn't. + +## For John + +1. **Override from mail apps.** Recommended: the `[override: reason]` subject + tag, stripped before sending. Alternatives: no override outside the + webmail (mail apps get warn rules as blocks), or a header only a few apps + can set. +2. **Office and PDF attachments.** Recommended: read DOCX, XLSX, PPTX and ODF + in this version (ZIP plus XML, done in-house, no new service), PDF later. + Alternative: all of them count as can't be inspected until a later spec. +3. **Tenants.** Recommended: server-level rules only in this version, with a + tenant condition, so a service provider sets policy for everyone. + Alternative: tenant administrators write rules for their own tenant too + (more to build: a tenant ceiling on actions, a tenant's review queue). +4. **Who edits DLP rules.** Recommended: administrators edit, compliance + officers see rules and review held mail, keeping "officers change no + setting" true. Alternative: officers edit DLP rules too. +5. **Unreviewed held mail.** Recommended: rejected back to the sender after 7 + days, with a notice. Alternatives: a different number, or held until + someone decides. +6. **Detectors in the first version.** Recommended: the five above. Others + (UK NI number, EU national IDs, passport numbers) are a table entry each + later; name any that must be there from the start. -- 2.54.0 From 3fadf8290922ccbde487fea51497de348ef222af Mon Sep 17 00:00:00 2001 From: John Coffey Date: Mon, 28 Sep 2026 16:04:31 -0700 Subject: [PATCH 2/3] Spec: John's answers, and the detector catalog answer 6 asks for All six settled as recommended. Answer 6 ("and any other recognized and protected PII") becomes a catalog of identifiers with published formats and checks, grouped by region, each either checked by its check digit or counted only beside a corroborating word, plus templates named for what they find. Data with no number to find is covered by word lists and not claimed as detection. Office documents are read; PDF counts as can't be inspected. --- docs/spec/features/dlp-and-mail-flow-rules.md | 169 +++++++++++++----- 1 file changed, 122 insertions(+), 47 deletions(-) diff --git a/docs/spec/features/dlp-and-mail-flow-rules.md b/docs/spec/features/dlp-and-mail-flow-rules.md index 4f4221c..de44921 100644 --- a/docs/spec/features/dlp-and-mail-flow-rules.md +++ b/docs/spec/features/dlp-and-mail-flow-rules.md @@ -1,10 +1,11 @@ # Feature spec: data loss prevention and mail flow rules -Status: **draft for approval.** Phase 1 of DLP and the rule builder, specced -together because they need the same conditions, the same place in the mail -path and the same record of what matched. Not a rebuild of an upstream -feature, so it has no line in SPEC.md §4's table. Open questions are under -[For John](#for-john); nothing is built until they're answered. +Status: **questions answered 2026-09-28** (see [Settled](#settled)); the +detector catalog in §2.3 was widened by answer 6 and is up for approval with +the rest. Phase 1 of DLP and the rule builder, specced together because they +need the same conditions, the same place in the mail path and the same record +of what matched. Not a rebuild of an upstream feature, so it has no line in +SPEC.md §4's table. ## Provenance @@ -18,6 +19,7 @@ Written for the record SPEC.md §3 rule 3 asks for. Sources, and nothing else: | `inbuxa-drafts/queue/dlp.md`, `rule-builder.md` | Own | What John asked for and settled | | The personal-data catalog spec and the audit-hold-lock spec | Own | Roles, the audit log, legal holds, the catalog check | | RFC 5321, RFC 3463 (enhanced status codes), RFC 8620/8621 (JMAP) | Public | Refusal codes and the submission error shape | +| The issuing authorities' published formats and check-digit rules for each identifier in §2.3 (ISO 13616, ISO/IEC 7812, ISO 7064, and each national scheme's own publication) | Public | The detector rules; each is implemented from its publication and tested against its published examples | No Enterprise-only file or snippet was used, and no third-party DLP product was consulted for design: the detectors are public checksum and format rules @@ -104,7 +106,7 @@ touch it): | `exceptions` | Any matching one skips the rule | | `actions` | What happens (list below) | | `stopProcessing` | Later rules don't run for this message | -| `tenantId` | Server-level (none) or one tenant's; see [For John](#for-john) Q3 | +| `tenantId` | Always none in this version: rules are server-level (settled answer 3); a **Tenant** condition narrows a rule to tenants | | `createdBy`, `updatedAt` | | Every create, update and delete is audited with its before and after, like @@ -118,8 +120,9 @@ Shared by both kinds: | Condition | Matches when | |---|---| | Sender | the sender is one of the chosen accounts, or in a chosen group, domain or tenant | +| Tenant | the sender is in one of the chosen tenants | | Recipient | any recipient is one of the chosen addresses, domains, groups | -| **Recipient outside** | any recipient isn't at a domain this server hosts (or, with a tenant rule, isn't in the tenant) | +| **Recipient outside** | any recipient isn't at a domain this server hosts | | Subject or body contains | any of a list of words or phrases (whole words, case-insensitive) | | Subject or body matches | a regular expression (the `regex` crate: linear time, no backtracking) | | Header | a header exists, or its value contains or matches | @@ -127,21 +130,94 @@ Shared by both kinds: | **Can't be inspected** | an attachment is encrypted or password-protected (ZIP, PDF, Office), or bigger than the inspection limit | | Message size | over a limit | -DLP adds **detectors**, each with a minimum count (for example "5 or more card -numbers"): +DLP adds **detectors**. Each counts what it finds, and a rule sets a minimum +(for example "5 or more card numbers"). A detector is one of two strengths: -| Detector | Rule | -|---|---| -| Payment card number | 13–19 digits, spaces or dashes allowed, a known issuer prefix, passes Luhn | -| IBAN | country code, check digits and length for that country, passes ISO 13616 mod 97 | -| US Social Security number | `AAA-GG-SSSS` or nine digits beside words like "SSN"; never area 000, 666 or 9xx, group 00, serial 0000 | -| Word list | a list the organization maintains (project names, "confidential"), counted | -| Pattern | the organization's own regular expression, counted | +- **Checked**: the identifier has a published check digit or checksum, so a + random number rarely passes. Found on its own. +- **Needs a word**: the format is too common to trust alone (nine digits, a + date). Counted only with a corroborating word nearby, within 50 characters + either side, in the languages where the identifier is used ("passport", + "Reisepass", "pasaporte"...). + +The catalog (settled answer 6: the recognized, protected identifiers, not a +chosen few). Each row is one table entry and one check function in +`crates/features/src/mailflow/detectors/`: + +| Region | Detector | Strength | Rule | +|---|---|---|---| +| Any | Payment card number | Checked | 13–19 digits, spaces or dashes allowed, a known issuer prefix (ISO/IEC 7812), Luhn | +| Any | IBAN | Checked | country code, length for that country, ISO 13616 mod 97 | +| Any | SWIFT/BIC | Needs a word | 8 or 11 characters, a valid country code in positions 5–6 | +| Any | Email addresses, in bulk | Checked | a count of distinct addresses (a customer list leaving), not one address | +| Any | Phone numbers, in bulk | Needs a word | a count of distinct numbers in international or national form | +| Any | Date of birth | Needs a word | a date beside "born", "DOB", "date of birth" and their translations | +| Any | Passport number | Needs a word | the formats of the issuing countries in this table | +| Any | Private key | Checked | a PEM or OpenSSH private-key block | +| Any | Cloud and service credentials | Checked | the published prefixes and lengths: AWS access key IDs, GitHub tokens, Slack tokens, Stripe live secret keys, Google API keys | +| US | Social Security number | Checked | `AAA-GG-SSSS`, or nine digits with a word; never area 000, 666 or 9xx, group 00, serial 0000 | +| US | ITIN | Checked | 9XX-GG-SSSS with the IRS's group ranges | +| US | EIN | Needs a word | a valid IRS prefix and seven digits | +| US | Bank routing number (ABA) | Checked | nine digits, a valid Federal Reserve prefix, the 3-7-1 checksum | +| US | Driver's license | Needs a word | each state's published format | +| US | Medicare Beneficiary Identifier | Checked | CMS's 11-character pattern and excluded letters | +| US | National Provider Identifier | Checked | ten digits, Luhn over the `80840` prefix | +| US | DEA registration number | Checked | two letters, seven digits, DEA's check digit | +| UK | National Insurance number | Checked | two letters (HMRC's excluded prefixes), six digits, A–D | +| UK | NHS number | Checked | ten digits, mod 11 | +| UK | Unique Taxpayer Reference | Needs a word | ten digits | +| Canada | Social Insurance Number | Checked | nine digits, Luhn | +| Australia | Tax File Number | Checked | weighted mod 11 | +| Australia | Medicare number | Checked | ten digits, weighted check digit | +| EU | Germany: tax ID (Steuer-ID) | Checked | eleven digits, ISO 7064 MOD 11,10 | +| EU | Germany: ID card number | Checked | nine characters, the 7-3-1 check digit | +| EU | France: social security number (NIR) | Checked | fifteen characters, mod 97 key | +| EU | Spain: DNI and NIE | Checked | eight digits and the mod 23 letter | +| EU | Italy: codice fiscale | Checked | sixteen characters, the check letter | +| EU | Netherlands: BSN | Checked | nine digits, the eleven test | +| EU | Belgium: national number | Checked | eleven digits, mod 97 | +| EU | Poland: PESEL | Checked | eleven digits, weighted check digit | +| EU | Sweden: personnummer | Checked | a date, three digits and a Luhn check digit | +| EU | Denmark: CPR number | Needs a word | a valid date and four digits | +| EU | Finland: personal identity code | Checked | a date, a century sign, three digits, the mod 31 character | +| EU | Ireland: PPS number | Checked | seven digits, one or two letters, mod 23 | +| EU | Portugal: NIF | Checked | nine digits, mod 11 | +| EU | Austria: social insurance number | Checked | ten digits, weighted check digit | +| Europe | Norway: national identity number | Checked | eleven digits, two mod 11 check digits | +| Europe | Switzerland: AHV number | Checked | `756`, then ten digits, EAN-13 check | +| Asia | India: Aadhaar | Checked | twelve digits, Verhoeff | +| Asia | India: PAN | Needs a word | five letters, four digits, a letter | +| Asia | China: resident ID | Checked | eighteen characters, ISO 7064 MOD 11-2 | +| Asia | Japan: My Number | Checked | twelve digits, weighted check digit | +| Asia | Singapore: NRIC and FIN | Checked | a letter, seven digits, the check letter | +| Asia | South Korea: resident registration number | Needs a word | thirteen digits with a valid date | +| Americas | Brazil: CPF and CNPJ | Checked | two mod 11 check digits | +| Americas | Mexico: CURP | Checked | eighteen characters, the check digit | +| Africa | South Africa: ID number | Checked | thirteen digits with a valid date, Luhn | +| Any | Word list | — | a list the organization maintains, counted | +| Any | Pattern | — | the organization's own regular expression, counted | + +Some protected data has no number to find: health conditions, religion, +union membership, sexual orientation, criminal records. No detector claims to +recognize those; a **word list** is how an organization covers its own terms +for them, and the console offers editable starting lists (medical terms, +diagnosis codes as ICD-10 patterns) rather than presenting them as detection. + +**Templates**, so a policy doesn't pick forty detectors one at a time. Each +is a named set, editable once added, and named for what it finds, never for a +law: *Payment cards and bank accounts*, *US personal identifiers*, *UK +personal identifiers*, *EU national identifiers*, *Health identifiers* (the NHS number, the US Medicare Beneficiary +Identifier, NPI and DEA numbers, the Australian Medicare number), *Credentials and keys*, *Contact lists*. + +The catalog grows by table entry: a new identifier is one row, one check +function and its published examples as tests. What the detectors read: the subject, every text and HTML part (as text), and attachments whose detected type is text (`text/*`, CSV, JSON, XML). -Office and PDF files are read only if Q2 says so; until then they count as -**can't be inspected** so a policy can still act on them. Inspection stops at +Office documents (DOCX, XLSX, PPTX, ODT, ODS, ODP) are read too (settled +answer 2): they're ZIP files of XML, unpacked and read in-house with limits on +unpacked size and entry count. PDF files count as **can't be inspected** in +this version, so a policy can still act on them. Inspection stops at a limit per message (proposed 10 MB of text), and what's past it counts as can't be inspected too. @@ -176,7 +252,7 @@ that warned; if a block or hold rule also matches, that still applies. **Mail apps (SMTP).** They show whatever text the server returns, so the refusal says how to override: `550 5.7.1 . To send anyway, start the subject with [override: your reason]`. On the next attempt the engine strips -the tag before DKIM signing, and records the reason. See Q1. +the tag before DKIM signing, and records the reason (settled answer 1). A block's refusal uses the same error path with `inbuxa:dlpBlocked` in the webmail. @@ -202,8 +278,8 @@ A reviewer, under **Management › Compliance › Held mail**: - **rejects** it with a reason: it's removed from the queue and the sender gets a notice with the reviewer's note, not their name. -Unreviewed mail is rejected after a set time (Q5), with a notice to the -sender. Emails › Queue shows held mail as held and refuses **Retry** on it, so +Unreviewed mail is rejected back to the sender after **7 days**, with a +notice (settled answer 5); the number is a setting. Emails › Queue shows held mail as held and refuses **Retry** on it, so nobody can deliver it around the review. The sender can't unsend it either once it's held (the webmail says so). @@ -232,9 +308,11 @@ New permissions (ids from 674): | `sysDlpReviewGet` | See held mail and open it | | `sysDlpReviewUpdate` | Release or reject held mail | -Proposed defaults: **Administrator** has all; **Compliance Officer** -(server-level) has `sysDlpPolicyGet`, `sysDlpReviewGet`, `sysDlpReviewUpdate`; -tenant roles per Q3. See Q4 on who edits DLP rules. +**Administrator** has all. **Compliance Officer** (server-level) has +`sysDlpPolicyGet`, `sysDlpReviewGet` and `sysDlpReviewUpdate`: officers see +the rules and review held mail, administrators edit (settled answer 4), so +"officers change no setting" stays true. Tenant roles get none: rules and the +review queue are server-level (settled answer 3). ### 2.9 Privacy catalog @@ -292,8 +370,10 @@ message queued by a node without the engine (held message format unchanged). ## 6. Phases 1. This spec, approved. -2. Engine, conditions, detectors, transport actions; DLP block and warn over - SMTP and JMAP; audit records; catalog entries. +2. Engine, conditions, the detector framework and the catalog in §2.3, + Office text extraction, transport actions; DLP block and warn over SMTP and + JMAP; audit records; catalog entries. The detector catalog may land in + more than one PR (by region), each with its published test vectors. 3. Hold for review: review records, release, reject, expiry, queue guard. 4. Console: DLP rules, held mail, mail flow rules. 5. Webmail dialogs; docs; a row in `inbuxa-drafts/divergence-log.md`. @@ -309,25 +389,20 @@ Each phase is its own PR with tests; releases as John decides. - Detectors find formats, not meaning: a card number in a harmless test message matches; a number written in words doesn't. -## For John +## Settled -1. **Override from mail apps.** Recommended: the `[override: reason]` subject - tag, stripped before sending. Alternatives: no override outside the - webmail (mail apps get warn rules as blocks), or a header only a few apps - can set. -2. **Office and PDF attachments.** Recommended: read DOCX, XLSX, PPTX and ODF - in this version (ZIP plus XML, done in-house, no new service), PDF later. - Alternative: all of them count as can't be inspected until a later spec. -3. **Tenants.** Recommended: server-level rules only in this version, with a - tenant condition, so a service provider sets policy for everyone. - Alternative: tenant administrators write rules for their own tenant too - (more to build: a tenant ceiling on actions, a tenant's review queue). -4. **Who edits DLP rules.** Recommended: administrators edit, compliance - officers see rules and review held mail, keeping "officers change no - setting" true. Alternative: officers edit DLP rules too. -5. **Unreviewed held mail.** Recommended: rejected back to the sender after 7 - days, with a notice. Alternatives: a different number, or held until - someone decides. -6. **Detectors in the first version.** Recommended: the five above. Others - (UK NI number, EU national IDs, passport numbers) are a table entry each - later; name any that must be there from the start. +John, 2026-09-28, all six as recommended, with 6 widened: + +1. **Override from mail apps**: the `[override: reason]` subject tag, stripped + before sending (§2.5). +2. **Office and PDF**: Office documents are read in this version; PDF counts + as can't be inspected (§2.3). +3. **Tenants**: server-level rules only, with a Tenant condition (§2.2, §2.8). +4. **Who edits DLP rules**: administrators; compliance officers see the rules + and review held mail (§2.8). +5. **Unreviewed held mail**: rejected back to the sender after 7 days, with a + notice (§2.6). +6. **Detectors**: the five proposed "and any other recognized and protected + PII", which §2.3 turns into a catalog of identifiers with published formats + and checks, plus templates. Data with no number to find (health, + religion...) is covered by word lists, not claimed as detection. -- 2.54.0 From 2b45a2e4128efb8db9c4fe1b91f581c7aa7f0185 Mon Sep 17 00:00:00 2001 From: John Coffey Date: Mon, 28 Sep 2026 16:41:38 -0700 Subject: [PATCH 3/3] Spec: approved --- docs/spec/features/dlp-and-mail-flow-rules.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/spec/features/dlp-and-mail-flow-rules.md b/docs/spec/features/dlp-and-mail-flow-rules.md index de44921..7a74e38 100644 --- a/docs/spec/features/dlp-and-mail-flow-rules.md +++ b/docs/spec/features/dlp-and-mail-flow-rules.md @@ -1,8 +1,7 @@ # Feature spec: data loss prevention and mail flow rules -Status: **questions answered 2026-09-28** (see [Settled](#settled)); the -detector catalog in §2.3 was widened by answer 6 and is up for approval with -the rest. Phase 1 of DLP and the rule builder, specced together because they +Status: **approved 2026-09-28**, with the answers under [Settled](#settled) +and the detector catalog in §2.3. Phase 1 of DLP and the rule builder, specced together because they need the same conditions, the same place in the mail path and the same record of what matched. Not a rebuild of an upstream feature, so it has no line in SPEC.md §4's table. -- 2.54.0