jcoffey-dev
02ebbc86a1
regex-lite on the agent, full regex at ingest, conformance corpus limited to the syntax both accept. The agent had no regex dependency at all and the full crate is a megabyte-plus against a binary whose pitch is that it is small and static. The rejected alternative, regex ingest-side only, would have given up redacting PII before it leaves the host -- the one capability that most needs to be on the agent, and most of the reason on-host processing exists at all. Checked rather than assumed: every pattern the corpus uses compiles and behaves under regex-lite, including named captures and replace_all replacing every occurrence, which is what mask needs and what a redaction stopping at the first hit would get wrong. Both engines also resolve alternation leftmost-first, confirmed by running the same pattern through regex-lite and Go's regexp and getting the same output. A case now pins that, since it is the kind of semantic two independent implementations can differ on silently. No canary gate. An edit reaches every matching agent at once, and this design had suggested that might have to block the feature. It does not, but the risk is accepted rather than waved away: total evaluation is the real mitigation, a fatal rule set crash-loops rather than strands because overrides are never persisted, apply-then-verify makes that self-healing, and applied_override_version already shows the blast radius. What is genuinely given up is the ability to stop a bad rollout partway through -- everything else shortens the outage without preventing the rule reaching every host first. That raises the stakes on apply-then-verify, which is still open. Without a canary, total evaluation stops being a nice property of a well-designed DSL and becomes the only thing between a bad rule and every host at once, so cutting the backstop from v1 is a harder call than it looked when it was written down. Signed-off-by: John Coffey <[email protected]>
processing
The Phase 8 processing rule language: its specification, and the conformance suite that is that specification.
Status: the suite exists; the implementations do not. Nothing in Phase 8 is built. See
/docs/phase-8-processing-design.mdfor the design and its open questions.
Why this is a top-level directory
Rules run in two places, in two languages: the Rust agent
(/agent) and the Go ingest tier (/ingest). Neither owns the
definition. This is the same shape as /proto — a language-neutral
contract that several modules consume and none of them is the source of
truth for.
Why the cases are the specification
Two hand-written implementations of one language will diverge. Not maybe, eventually. The only thing that reliably prevents it is a shared corpus both must satisfy, written down before either exists.
So the prose below is a summary of conformance/cases/, not the other
way round. When they disagree, the cases win, and the prose is the bug.
Running it
Nothing executes the cases yet, because neither implementation exists. What runs today is a structural check that every case is well-formed — valid JSON, known actions, no unknown keys, declared fields that exist:
python3 processing/conformance/validate.py
That is worth having on its own: it stops the corpus rotting between now and the first implementation, and it fails loudly if someone adds a case using an action the spec does not define.
Each implementation is expected to add a test that walks
conformance/cases/*.json, feeds inputs through rules, and asserts
the emitted records equal expect. Those runners are part of building
each side, not part of this directory.
Case format
One JSON object per file:
{
"name": "drop_discards_the_record",
"description": "A matched drop emits nothing at all.",
"rules": [
{
"match": [{"field": "message", "op": "eq", "value": "noise"}],
"actions": [{"action": "drop"}]
}
],
"inputs": [
{"timestamp_unix_nano": 1000, "host": "h1", "service": "s1",
"severity": "SEVERITY_INFO", "message": "noise", "attributes": {}}
],
"expect": []
}
inputs is always a list, even for a single record, because windowed
actions need a sequence. expect is the records emitted, in order.
Records
Mirrors LogRecord in /proto/sentry/logs/v1/logs.proto, snake_case,
with severity as the enum's name string.
record_id is deliberately absent from every case. The agent never sets
it, and whether an ingest-side rule can see one depends on where in the
ingest path rules run — an open question. No case may depend on it, so
that decision stays free.
Matching
A rule matches when every clause in match holds. An empty match
matches every record.
op |
Meaning |
|---|---|
eq / ne |
exact string equality |
contains |
substring |
prefix / suffix |
string boundary |
regex |
linear-time regex, unanchored |
exists / not_exists |
field present (no value) |
Addressable fields: message, host, service, severity, and
attributes.<key>. A field that does not exist compares as absent, not
as empty string — eq "" does not match a missing attribute.
Actions
Applied in order. A drop ends processing for that record immediately;
no later action or rule runs.
| Action | Parameters |
|---|---|
drop |
— |
drop_fields |
fields |
keep_fields |
fields (attributes only; never removes top-level fields) |
rename |
from, to |
derive |
field, and one of value / from_field |
mask |
field, pattern, replacement |
parse_json |
field (default message), optional prefix |
parse_regex |
field, pattern (named captures become attributes) |
sample |
keep_one_in |
suppress_duplicates |
window_ms, optional key_fields |
Rules are evaluated in the order given. Every matching rule's actions apply, to the record as left by the rule before it.
Regex: only what both engines support
The agent uses regex-lite and ingest uses the full regex crate — a
size decision, since the agent ships as a small static musl binary and
the full crate is a megabyte-plus (see
phase-8-processing-design.md).
Cases may only use syntax regex-lite accepts. A case the agent
cannot run is not a conformance case. In practice that means quantifiers,
character classes, alternation and named captures are fine, while
Unicode-aware classes and the richer Perl classes are not.
Both engines are linear-time automata with no catastrophic backtracking, and both resolve alternation leftmost-first. The second is pinned by a case rather than trusted, because it is exactly the kind of semantic two independent implementations can differ on silently.
Two determinism decisions the suite forces
Conformance testing cannot assert on nondeterminism, so two things that are usually left vague are pinned here:
sample is counter-based, not random. keep_one_in: 3 keeps the
first record and every third thereafter, per rule, per process. Random
sampling is statistically nicer and untestable; a counter is testable and
close enough at volume.
Windows are measured on record timestamps, not wall-clock. A
suppress_duplicates window of 5000ms compares
timestamp_unix_nano values, so replaying the same input always gives
the same output regardless of how fast the test runs. This also means the
behaviour is correct under backfill, which wall-clock windows are not.
aggregate_count has no cases, deliberately
The design lists it as an action but does not answer what it emits, or
what a query that is not expecting a synthetic record sees. Writing cases
now would invent that answer by accident and freeze it. It stays
unspecified until that question is decided —
phase-8-processing-design.md
open question 5.
Adding a case
One behaviour per file, named for the behaviour rather than the action. A case is added for every bug found in either implementation — that is the mechanism that keeps the two from drifting, and it only works if the case lands with the fix.