Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web

End-to-end log pipeline for Linux hosts, per /docs/architecture.md:

- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
  mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
  end forwards to Redpanda unchanged, consumer normalizes and
  batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
  yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together

Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.
This commit is contained in:
2026-08-13 08:25:19 -07:00
commit b6b092c912
92 changed files with 7796 additions and 0 deletions
+100
View File
@@ -0,0 +1,100 @@
# sentry-agent
Distro-agnostic Linux log collector. Statically linked against musl, no
glibc runtime dependency. Tails journald (default) or a file, batches
lines, and ships them over mTLS gRPC to the ingest service.
## Workspace layout
- `sentry-parser` — pure-`std` RFC 5424 syslog parser with raw-passthrough
fallback. No I/O, easy to unit test in isolation.
- `sentry-agent` — the binary: config loading, sourcing (journald/file),
batching, mTLS gRPC client.
## Why journalctl, not libsystemd
The journald source shells out to `journalctl -f -o json` rather than
linking `libsystemd` via FFI. Statically linking libsystemd into a musl
binary is fragile — it pulls in dbus/libcap transitively and isn't designed
for static linking — and would undermine the no-glibc-runtime-deps goal
even where technically possible. `journalctl` ships on every systemd distro
this agent targets, so shelling out sidesteps the problem entirely. See
`/docs/architecture.md`.
## Building
Native build (whatever target your machine is):
```sh
cargo build --release
```
musl targets (what actually ships):
```sh
rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl
# x86_64: works with musl-gcc installed locally (musl-tools on Debian,
# musl on Arch, etc.) — the musl target is fully static by default.
cargo build --release --target x86_64-unknown-linux-musl
# aarch64 cross-compilation needs a cross toolchain; the boring, reliable
# option is `cross` (https://github.com/cross-rs/cross), which builds
# inside a Docker container with the right linker preinstalled:
cross build --release --target aarch64-unknown-linux-musl
```
Building requires `protoc` on PATH (used by `tonic-build`/`prost-build` at
compile time to generate the gRPC client from `/proto/sentry/logs/v1/logs.proto`).
Container build (see caveat below):
```sh
# from the repo root, not agent/
docker build -f agent/Dockerfile -t sentry-agent .
```
**Caveat:** the container image is provided for CI/completeness, but
journald sourcing needs `journalctl` and access to the host journal —
neither of which exist in the `scratch` image or are available to a
container without deliberately bind-mounting `/var/log/journal` (or
`/run/log/journal`) and the `journalctl` binary in. The intended Phase 0
deployment for journald sourcing is as a native binary managed by systemd
on the host, not containerized.
## Running
No CLI flags are required for the common case:
```sh
./sentry-agent
```
This uses `/etc/sentry-agent/agent.toml` if present, otherwise built-in
defaults: journald source (whole journal, no unit filter), service name
`default`, and mTLS material expected at
`/etc/sentry-agent/{ca,client,client-key}.pem`. mTLS is mandatory per the
project's transport requirements, so a from-scratch run with no certs in
place will fail fast with a clear error rather than connecting insecurely.
See `config/agent.example.toml` for all fields.
```sh
./sentry-agent --config /path/to/agent.toml
```
## Testing
```sh
cargo test --workspace
```
## Feature flags
- `journald` (default) — journalctl-based journald source.
- `file-tail` — polling-based file tailer (no inotify dependency; doesn't
follow rename-based log rotation yet).
Both can be enabled together; `[source].kind` in config picks which one
runs. Building without a feature and configuring that source at runtime
fails at startup with a clear error rather than silently doing nothing.