Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web

End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
2026-08-13 08:25:19 -07:00
commit b6b092c912
92 changed files with 7796 additions and 0 deletions
@@ -0,0 +1,10 @@
+# One-shot migration runner: bash + curl baked in, migrations/*.sql copied
+# in at build time. No runtime package install and no host volume mount
+# needed — works offline once built.
+#   docker build -f storage/Dockerfile -t sentry-clickhouse-migrate storage/
+FROM alpine:3.20
+RUN apk add --no-cache bash curl
+WORKDIR /storage
+COPY migrate.sh ./
+COPY migrations ./migrations
+ENTRYPOINT ["bash", "migrate.sh"]
@@ -0,0 +1,84 @@
+# storage
+
+ClickHouse schema and migration tooling for Sentry's analytical store.
+
+## Schema
+
+One table for Phase 0, `logs`:
+
+```sql
+CREATE TABLE logs
+(
+    `timestamp`  DateTime64(9, 'UTC'),
+    `host`       String,
+    `service`    String,
+    `severity`   LowCardinality(String),
+    `message`    String,
+    `attributes` Map(String, String)
+)
+ENGINE = MergeTree
+PARTITION BY toDate(timestamp)
+ORDER BY (service, timestamp)
+```
+
+Notes on choices that weren't fully specified by the task description:
+
+- **`DateTime64(9, 'UTC')`** (nanosecond precision) rather than second or
+  millisecond precision, to match the agent's `timestamp_unix_nano` field
+  end to end without truncation.
+- **`severity` as `LowCardinality(String)`**, not a numeric OTel
+  `SeverityNumber`. `/ingest`'s `normalize` package writes short text
+  values (`TRACE`/`DEBUG`/`INFO`/`WARN`/`ERROR`/`FATAL`/`UNSPECIFIED`).
+  `LowCardinality` gets you most of the storage/query efficiency of an enum
+  without committing to one at the schema level. Splitting into a proper
+  `SeverityNumber` + `SeverityText` pair (full OTel shape) is one of the
+  open questions already flagged in `/docs/architecture.md`.
+- **`PARTITION BY toDate(timestamp)`** (daily partitions) and
+  **`ORDER BY (service, timestamp)`** are exactly what the task asked for
+  — service-scoped queries over a time range are the dominant access
+  pattern this is optimized for.
+- No TTL/retention clause yet — also an open question in architecture.md,
+  deferred until storage sizing is a real concern.
+
+## Migration tooling: a plain SQL-file runner, not golang-migrate
+
+`migrate.sh` applies `migrations/*.sql` in filename order over
+ClickHouse's HTTP interface, tracking what's applied in a
+`schema_migrations` table. Chosen over `golang-migrate` for Phase 0
+because there's exactly one migration to run — pulling in a migration
+framework (another dependency, another thing to configure/vendor) for a
+single `CREATE TABLE` is exactly the kind of premature machinery this
+project's conventions say to avoid. Revisit `golang-migrate` once there's
+real schema churn across environments (rollback support, checksums,
+concurrent-apply safety become worth their cost at that point, not before).
+
+**Convention:** one DDL statement per migration file. The ClickHouse HTTP
+interface isn't reliably multi-statement, so `migrate.sh` doesn't try to
+split multi-statement files — keep each migration to a single statement.
+
+## Running
+
+```sh
+docker compose up -d          # starts a standalone ClickHouse for local work
+./migrate.sh                  # applies migrations/*.sql
+```
+
+Environment variables `migrate.sh` reads (all optional, matching
+`/ingest`'s ClickHouse defaults so the two stay in sync out of the box):
+
+| Var | Default |
+|---|---|
+| `CLICKHOUSE_HTTP` | `http://localhost:8123` |
+| `CLICKHOUSE_USER` | `default` |
+| `CLICKHOUSE_PASSWORD` | (empty) |
+| `CLICKHOUSE_DATABASE` | `sentry` |
+
+There's also a `Dockerfile` (bash + curl baked in, `migrations/` copied in
+at build time) used by the root-level `docker-compose.yml` as a one-shot
+init service — no runtime package install, no host volume mount needed.
+
+## Adding a migration
+
+Add `migrations/000N_description.sql` with the next sequential number and
+a single DDL statement. `migrate.sh` picks it up automatically — no
+registration step.
@@ -0,0 +1,20 @@
+# Standalone ClickHouse for local development against /storage in
+# isolation (e.g. iterating on migrations). The root-level docker-compose.yml
+# runs the full Phase 0 stack and defines its own clickhouse service
+# separately — this file is not included by it.
+services:
+  clickhouse:
+    image: clickhouse/clickhouse-server:24.8
+    container_name: sentry-clickhouse
+    ports:
+      - "8123:8123" # HTTP interface, used by migrate.sh
+      - "9000:9000" # native protocol, used by ingest
+    volumes:
+      - clickhouse-data:/var/lib/clickhouse
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+
+volumes:
+  clickhouse-data:
@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# Applies migrations/*.sql to ClickHouse in filename order, tracking what's
+# already been applied in a schema_migrations table. Talks to ClickHouse's
+# HTTP interface via curl rather than requiring the clickhouse-client
+# binary — nothing to install beyond curl, works identically on a dev
+# laptop or in CI.
+#
+# Convention: exactly one DDL statement per migration file. The ClickHouse
+# HTTP interface isn't reliably multi-statement, so keeping migrations to
+# one statement each avoids relying on that.
+set -euo pipefail
+
+CLICKHOUSE_HTTP="${CLICKHOUSE_HTTP:-http://localhost:8123}"
+CLICKHOUSE_USER="${CLICKHOUSE_USER:-default}"
+CLICKHOUSE_PASSWORD="${CLICKHOUSE_PASSWORD:-}"
+DATABASE="${CLICKHOUSE_DATABASE:-sentry}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+MIGRATIONS_DIR="${SCRIPT_DIR}/migrations"
+
+ch_exec() {
+    # $1 = SQL statement, $2 = optional database to scope the query to.
+    local sql="$1"
+    local db="${2:-}"
+    local url="${CLICKHOUSE_HTTP}/"
+    if [[ -n "$db" ]]; then
+        url="${CLICKHOUSE_HTTP}/?database=${db}"
+    fi
+    curl -sS -f -u "${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}" "$url" --data-binary "$sql"
+}
+
+echo "Ensuring database '${DATABASE}' exists..."
+ch_exec "CREATE DATABASE IF NOT EXISTS ${DATABASE}"
+
+echo "Ensuring schema_migrations table exists..."
+ch_exec "CREATE TABLE IF NOT EXISTS schema_migrations (version String, applied_at DateTime DEFAULT now()) ENGINE = MergeTree ORDER BY version" "$DATABASE"
+
+applied="$(ch_exec "SELECT version FROM schema_migrations FORMAT TabSeparated" "$DATABASE")"
+
+shopt -s nullglob
+for file in "${MIGRATIONS_DIR}"/*.sql; do
+    version="$(basename "$file")"
+    if grep -qx "$version" <<< "$applied"; then
+        echo "skip  ${version} (already applied)"
+        continue
+    fi
+    echo "apply ${version}"
+    ch_exec "$(cat "$file")" "$DATABASE" > /dev/null
+    ch_exec "INSERT INTO schema_migrations (version) VALUES ('${version}')" "$DATABASE" > /dev/null
+done
+
+echo "Migrations complete."
@@ -0,0 +1,12 @@
+CREATE TABLE IF NOT EXISTS logs
+(
+    `timestamp`  DateTime64(9, 'UTC'),
+    `host`       String,
+    `service`    String,
+    `severity`   LowCardinality(String),
+    `message`    String,
+    `attributes` Map(String, String)
+)
+ENGINE = MergeTree
+PARTITION BY toDate(timestamp)
+ORDER BY (service, timestamp)