Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web

End-to-end log pipeline for Linux hosts, per /docs/architecture.md:

- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
  mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
  end forwards to Redpanda unchanged, consumer normalizes and
  batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
  yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together

Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.

transport/Dockerfile

@@ -0,0 +1,7 @@
+# Built FROM the Redpanda image so `rpk` is already present -- no need for
+# a separate client install, and no Docker socket access needed since
+# provisioning happens over the network, not via `docker exec`.
+#   docker build -f transport/Dockerfile -t sentry-transport-provision transport/
+FROM docker.redpanda.com/redpandadata/redpanda:v24.2.7
+COPY provision-topics.sh /provision-topics.sh
+ENTRYPOINT ["/provision-topics.sh"]

transport/README.md

@@ -0,0 +1,26 @@
+# transport
+
+Redpanda for local development, plus the script that provisions the topic
+`/ingest` depends on.
+
+## Topic naming contract
+
+`ingest` defaults to `REDPANDA_TOPIC=sentry.logs.raw` (see
+`/ingest/internal/config`). `provision-topics.sh` defaults to the same
+name. These aren't wired together automatically — if you change one,
+change the other, or override `REDPANDA_TOPIC` consistently wherever
+both are invoked.
+
+## Running standalone
+
+```sh
+docker compose up -d
+REDPANDA_BROKERS=localhost:9092 ./provision-topics.sh
+```
+
+## In the full stack
+
+The root-level `docker-compose.yml` builds this directory's `Dockerfile`
+(FROM the Redpanda image itself, so `rpk` is already present) as a
+one-shot init service that runs after Redpanda reports healthy. See
+`/docs/phase-0-runbook.md`.

transport/docker-compose.yml

@@ -0,0 +1,34 @@
+# Standalone Redpanda for local development against /transport in
+# isolation. The root-level docker-compose.yml runs the full Phase 0 stack
+# and defines its own redpanda service separately — this file is not
+# included by it.
+#
+# Note advertise-kafka-addr is "localhost" here (host tools connect via the
+# mapped port), vs "redpanda" in the root compose (other containers connect
+# via the compose network's service DNS name). Getting this wrong is the
+# classic Redpanda/Kafka docker-compose footgun — clients can connect
+# initially but then fail on the broker's advertised address once they try
+# to actually produce/consume.
+services:
+  redpanda:
+    image: docker.redpanda.com/redpandadata/redpanda:v24.2.7
+    container_name: sentry-redpanda
+    command:
+      - redpanda
+      - start
+      - --smp=1
+      - --memory=1G
+      - --reserve-memory=0M
+      - --overprovisioned
+      - --node-id=0
+      - --check=false
+      - --kafka-addr=PLAINTEXT://0.0.0.0:9092
+      - --advertise-kafka-addr=PLAINTEXT://localhost:9092
+    ports:
+      - "9092:9092"
+      - "9644:9644" # admin API, used by rpk/healthchecks
+    volumes:
+      - redpanda-data:/var/lib/redpanda/data
+
+volumes:
+  redpanda-data:

transport/provision-topics.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Idempotently creates the topic ingest produces/consumes. Talks to
+# Redpanda over the network via `rpk`, not `docker exec` into the broker
+# container -- this way the same script works whether it's run from the
+# host (against the standalone compose in this directory), from inside a
+# sibling container on the root compose's network, or in CI.
+set -euo pipefail
+
+BROKERS="${REDPANDA_BROKERS:-localhost:9092}"
+TOPIC="${REDPANDA_TOPIC:-sentry.logs.raw}"
+PARTITIONS="${REDPANDA_TOPIC_PARTITIONS:-6}"
+
+echo "Waiting for Redpanda at ${BROKERS}..."
+until rpk cluster health --brokers "${BROKERS}" --exit-when-healthy > /dev/null 2>&1; do
+    sleep 1
+done
+
+if rpk topic list --brokers "${BROKERS}" | awk 'NR>1{print $1}' | grep -qx "${TOPIC}"; then
+    echo "Topic '${TOPIC}' already exists, skipping."
+else
+    echo "Creating topic '${TOPIC}' (${PARTITIONS} partitions)..."
+    rpk topic create "${TOPIC}" --brokers "${BROKERS}" --partitions "${PARTITIONS}" --replicas 1
+fi