DLP: the detector framework, the region-free detectors, word lists and attachment text #99

Merged

jcoffey-dev merged 2 commits from feature/dlp-detectors into main

2026-09-29 00:13:21 +00:00

jcoffey-dev commented

2026-09-29 00:00:54 +00:00

Owner

Phase 2a of the DLP and mail flow rules spec (#97). Pure functions in crates/features/src/mailflow; nothing is wired into the mail path yet.

Detectors: distinct values, each either checked by a published check digit or counted only with a corroborating word within 50 characters. This PR: payment cards (ISO/IEC 7812 prefixes + Luhn), IBAN (registry lengths + mod 97), SWIFT/BIC, bulk email addresses and phone numbers, dates of birth, passports, private keys, published service-token formats. Regional identifiers follow, one region per PR.
Word lists and patterns: Aho-Corasick whole-word any-case; regex with a compiled-size limit.
Attachment text: text (UTF-8/UTF-16), HTML, DOCX/XLSX/PPTX, ODT/ODS/ODP, ZIP one level deep, all in-house with the workspace's zip and quick-xml. Encrypted, PDF, legacy Office, nested archives and over-limit files are not inspectable, with the reason.

No new third-party crates. 21 unit tests (published test card numbers, the IBAN registry's examples, built DOCX/XLSX/ODT/ZIP files, an AES-encrypted ZIP).

Phase 2a of the DLP and mail flow rules spec (#97). Pure functions in `crates/features/src/mailflow`; nothing is wired into the mail path yet. - **Detectors**: distinct values, each either *checked* by a published check digit or counted only with a corroborating word within 50 characters. This PR: payment cards (ISO/IEC 7812 prefixes + Luhn), IBAN (registry lengths + mod 97), SWIFT/BIC, bulk email addresses and phone numbers, dates of birth, passports, private keys, published service-token formats. Regional identifiers follow, one region per PR. - **Word lists and patterns**: Aho-Corasick whole-word any-case; regex with a compiled-size limit. - **Attachment text**: text (UTF-8/UTF-16), HTML, DOCX/XLSX/PPTX, ODT/ODS/ODP, ZIP one level deep, all in-house with the workspace's zip and quick-xml. Encrypted, PDF, legacy Office, nested archives and over-limit files are *not inspectable*, with the reason. No new third-party crates. 21 unit tests (published test card numbers, the IBAN registry's examples, built DOCX/XLSX/ODT/ZIP files, an AES-encrypted ZIP).

jcoffey-dev added 1 commit 2026-09-29 00:00:54 +00:00

DLP: the detector framework, the region-free detectors, word lists and attachment text

ci / fork-checks (pull_request) Canceled after 8s

Details

ci / build (pull_request) Canceled after 8s

Details

dc49bf4d14

Phase 2a of the DLP and mail flow rules spec: pure functions in
crates/features/src/mailflow, nothing wired into the mail path yet.

- Detectors report distinct values found, each either checked by its
  published check digit or counted only beside a corroborating word
  within 50 characters. This PR adds the region-free ones: payment
  cards (issuer prefixes, Luhn), IBAN (registry lengths, mod 97),
  SWIFT/BIC, email addresses and phone numbers in bulk, dates of birth,
  passport numbers, private keys and published service-token formats.
  Regional identifiers follow, a region per PR.
- Word lists (Aho-Corasick, whole words, any case) and patterns (regex
  with a compiled-size limit) count occurrences.
- Attachment text: text files with or without a UTF-16 mark, HTML,
  DOCX/XLSX/PPTX, ODT/ODS/ODP and ZIP archives one level deep, read
  with the zip and quick-xml crates the workspace already has.
  Encrypted files, PDF, legacy binary Office files, nested archives
  and anything past the limits come back as not inspectable, with why.

21 unit tests, against the networks' test card numbers and the IBAN
registry's own examples among others.

jcoffey-dev added 1 commit 2026-09-29 00:01:04 +00:00

Cargo.lock: the features crate's new dependencies

ci / fork-checks (pull_request) Successful in 2m23s

Details

ci / build (pull_request) Successful in 11m50s

Details

3eb5a454fd

jcoffey-dev merged commit 01f6b99631 into main

2026-09-29 00:13:21 +00:00

jcoffey-dev deleted branch feature/dlp-detectors

2026-09-29 00:13:21 +00:00

jcoffey-dev referenced this pull request