# Phase 8 processing design: rules, where they run, and how they arrive

> **Status:** Design, drafted 2026-09-05. **Not approved and not
> implemented.** Nothing in Phase 8 is started. This is a proposal to
> argue with — the sections marked **Open** are genuine decisions, not
> rhetorical ones. If implementation shows this is wrong somewhere, fix
> this doc in the same change rather than letting them drift.

## Why this design, in one paragraph

Phase 8 puts rule-based work on records in flight: drop, mask, rename,
derive, parse, sample, suppress duplicates, aggregate. The hard part is
not the transformations — it is that the same rule has to mean the same
thing in a Rust agent and a Go ingest tier, and that rules are pushed to
hosts over a channel deliberately built to be incapable of carrying
anything dangerous. So this design starts from the distribution channel
and the safety invariant it protects, and derives the language from
them, rather than designing a language and asking later how to ship it.

## The constraint everything else follows from

[`agent-management-design.md`](agent-management-design.md) states an
invariant plainly: every remotely editable field "degrades the agent's
behavior without ever cutting off its ability to receive the next
correction." That is why `ingest.endpoint` and TLS material are
permanently non-editable — a bad value there kills the only channel that
could fix it.

Processing rules are the first remotely editable thing that can execute.
A rule that panics, loops forever, or allocates without bound is not a
degraded setting; it is a broken agent.

### A correction to what the roadmap says

The roadmap change in #21 stated that a bad rule "strands the agent
exactly the way a corrupted `ingest.endpoint` would." Reading
`apply_override`'s actual semantics, that is too strong, and the
difference matters enough to write down rather than quietly soften:

**An override lives only in the running process's memory.** It is never
written to `agent.toml`. A restarted agent boots from its local config
alone and re-syncs on its next successful check-in.

So a rule set that crashes the agent produces a **crash-loop**, not a
strand:

```
boot (clean, no rules) → CheckIn → receive rules → apply → crash → boot …
```

The agent checks in on every iteration of that loop. The platform can
always push a corrected or cleared override, and the agent will take it.
That is a materially better failure mode than being stranded, and it
exists by accident — the "don't persist overrides" choice was made for
simplicity (no filesystem writes on read-only base images, no
reconcile-at-startup state machine), not for safety.

**This design promotes that accident to a constraint.** Persisting
overrides to disk would convert every crash-loop into a strand, because
the agent would apply the fatal rules before its first check-in and never
reach one. Anyone proposing offline-boot override persistence later must
solve this first. It is now load-bearing.

The residual harm is still real and still worth engineering away: a
crash-looping host ships almost nothing, and the loop runs at whatever
the check-in cadence is until a human notices.

## Decision 1: total evaluation, with apply-then-verify as a backstop

Two candidate guarantees were named in #21. This design takes **both**,
in priority order, because they solve different halves.

**Total evaluation** — the rule language is constructed so a rule set
*cannot* panic, loop unboundedly, or allocate without limit. This is the
primary guarantee, and it is an absence of the failure rather than a
recovery from it. It is purchasable only by keeping the language
declarative and typed, which the next section does.

**Apply-then-verify** — the agent treats a newly received rule set as
provisional: it records the version it is about to apply, applies it, and
marks it good once it has survived one full check-in interval. If it
boots and finds a provisional version recorded that never went good, it
reports the failure and runs *without* rules rather than reapplying them.

Note the tension with the constraint above: apply-then-verify needs a
small amount of state to survive a restart, which is exactly the
persistence the previous section forbids. The resolution is that what
persists is **a version stamp and a failure flag, never the rule set
itself** — a few bytes, and a host that cannot write even that simply
loses the backstop and keeps total evaluation. The agent must degrade to
"no rules" on a write failure, never to "apply anyway".

**Open:** whether the backstop is worth its complexity in the first
release, given total evaluation should make it unreachable. My view is
yes — "should be unreachable" is what every crash-loop was before it
happened — but it is a defensible cut for a v1.

## Decision 2: the rule shape

A rule is a **matcher** and an ordered list of **typed actions**. No
expressions, no arbitrary code, no user-supplied control flow.

```
rule
  match:   field, operator, value        (all must hold)
  actions: [ action, action, … ]         (applied in order)
```

Actions, and whether each is trivially total:

| Action | Effect | Total? |
|---|---|---|
| `drop` | discard the record | yes |
| `drop_fields` / `keep_fields` | remove or whitelist attributes | yes |
| `mask` | replace matched substring with a fixed token | yes, with a linear-time engine |
| `rename` | move a field | yes |
| `derive` | set a field from a literal or another field | yes |
| `parse_json` | parse `message` into fields | yes, with a depth and size cap |
| `parse_regex` | named captures into fields | yes, with a linear-time engine |
| `sample` | keep 1 in N | yes |
| `suppress_duplicates` | collapse identical records within a window | yes, with a bounded cache |
| `aggregate_count` | replace repeats with a count record | yes, with a bounded cache |

Deliberately absent: arbitrary expressions, loops, user-defined
functions, and anything resembling `eval`. Cribl's rule language is
JavaScript; this is less expressive on purpose. It is also the only
shape that can be pushed to ten thousand hosts and audited by reading it.

### Why regex does not break totality

Both implementation languages ship linear-time, non-backtracking regex
engines — Rust's `regex` crate and Go's `regexp` are both
finite-automata based, with no catastrophic backtracking to guard
against. This is a real piece of luck: the usual reason regex is unsafe
in a pushed rule set does not apply here, in either language,
without doing anything clever.

**Open, and it has a cost:** the agent currently has **no regex
dependency at all**. Adding the full `regex` crate is on the order of a
megabyte-plus of binary, against a project whose agent pitch is a small
static musl binary. `regex-lite` is far smaller and still linear-time,
at the cost of some syntax and speed. Alternatively `parse_regex` and
`mask` could be ingest-side only in v1, which sacrifices the "redact PII
before it leaves the host" claim in `positioning.md` — the one thing
that most needs to run on the agent. My recommendation is `regex-lite`
on the agent and full `regex` at ingest, with the conformance suite
(below) restricted to the syntax both accept.

## Decision 3: one spec, two implementations, one conformance suite

Rules run in Rust on the agent and in Go at ingest. "The same rule does
the same thing in both places" is the whole promise, and two
hand-written implementations will diverge — not maybe, eventually.

The deliverable that prevents it is a **language-neutral conformance
suite**: a directory of cases, each an input record, a rule set, and the
expected output record, in JSON. Both implementations run it in their own
CI. A case is added for every bug found in either.

This is the same discipline `/hack`'s fixtures already apply to ingest
shapes, applied to semantics instead. It should be built **first**, not
last — the suite is the specification, and the prose above is a summary
of it.

## Decision 4: distribution reuses the channel that exists

Fleet management already delivers desired state
([`agent-management-design.md`](agent-management-design.md)). Rules
become one more field on `DesiredOverride`:

```protobuf
repeated ProcessingRule rules = 8;
```

`extra_file_paths = 7` is the precedent to follow exactly: a repeated
field with no meaningful "unset", where the platform always submits the
complete desired list and an empty list unambiguously means "no rules
right now". The existing `version` stamp and `applied_override_version`
echo give rollout observability for free — you can already see which
hosts have taken a rule set and which have not.

**Open:** there is no staged rollout today. An edit goes to every agent
matching it on their next check-in. For batch sizes that is fine; for
executable rules it is the difference between breaking one host and
breaking all of them. A canary mechanism — apply to N hosts, require
them to report good, then widen — is not in the fleet design and would
be new work. My view is that this is the single most important thing to
add alongside rules, and it may deserve to gate the feature.

## Where each rule runs

Agent-side is the default and the cheaper place: data reduced before the
wire costs nothing to transport, store or index, and it is the only place
PII can be removed before it crosses the network.

Ingest-side exists for rules needing context the agent lacks, and for
changing behaviour without waiting for a fleet rollout.

**Open:** whether a rule declares where it runs, or whether the platform
decides. Explicit placement is simpler to reason about and to debug;
automatic placement is friendlier and much easier to get subtly wrong. I
lean explicit, with a validation error when a rule asks for something its
location cannot do.

## The motivating workload is real

The maintainer's own workstation, measured 2026-09-05 while the dev agent
ran against a live stack:

| rate | source | message |
|---|---|---|
| ~39/min | Discord | `Discord 1.0.155`, byte-identical every time |
| ~22/min | foreground_boost | `Checking active tasks` |

Those two are **308 of 325 journal entries in five minutes** — roughly
60% of one host's volume, from two processes saying nothing. Everything
else on that host, kernel firewall drops included, is single digits.

Today there is no way to do anything about it: the journald source's only
filter is a single-unit allowlist, so the choice is the whole journal or
one unit, with nothing in between.

**This is the v1 acceptance test.** A `suppress_duplicates` rule and a
`drop` rule should remove ~60% of that host's volume, measured
before-and-after against real data rather than a fixture. If the first
release cannot do that, it is not finished.

## Open questions, collected

1. Is apply-then-verify in v1, or is total evaluation alone enough?
2. `regex-lite` on the agent, full `regex` at ingest, and a conformance
   suite limited to their common syntax — or regex ingest-side only,
   giving up on-host redaction for v1?
3. Does a canary rollout gate the feature, or ship after it?
4. Explicit per-rule placement, or platform-decided?
5. Does `aggregate_count` emit a synthetic record, and if so what does it
   look like to a query that is not expecting one? This design does not
   answer that and should before anyone builds it.