cairnobs/storage/README.md

storage

ClickHouse schema and migration tooling for Sentry's analytical store.

Schema

One table, logs (Phase 0 columns plus record_id, added in migrations/0002_add_record_id.sql):

CREATE TABLE logs
(
    `timestamp`  DateTime64(9, 'UTC'),
    `host`       String,
    `service`    String,
    `severity`   LowCardinality(String),
    `message`    String,
    `attributes` Map(String, String),
    `record_id`  UUID DEFAULT generateUUIDv4()
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp)
ORDER BY (service, timestamp)
-- plus: INDEX record_id_idx record_id TYPE bloom_filter GRANULARITY 4

record_id (Phase 1) is the stable per-record identifier Tantivy's full-text search joins back to this table with — /ingest's gRPC front end assigns it once, server-side, before a record is produced to Redpanda (see /ingest/README.md for why it has to happen exactly once, upstream of both the ClickHouse-writer and Tantivy-indexer consumers). Added via ALTER TABLE ... ADD COLUMN + ADD INDEX rather than changing ORDER BY: ORDER BY (service, timestamp) is the proven time-range-scan access pattern from Phase 0 and shouldn't be disturbed for a fundamentally different access pattern (point lookups by ID). A data-skipping bloom filter index on record_id serves the WHERE record_id IN (...) lookup Tantivy-backed search results need, without touching the primary sort order. The DEFAULT generateUUIDv4() is a safety net, not the normal path — every row /ingest writes explicitly supplies its own record_id from the proto message; the default only matters for rows written some other way.

Notes on choices that weren't fully specified by the task description:

• DateTime64(9, 'UTC') (nanosecond precision) rather than second or millisecond precision, to match the agent's timestamp_unix_nano field end to end without truncation.
• severity as LowCardinality(String), not a numeric OTel SeverityNumber. /ingest's normalize package writes short text values (TRACE/DEBUG/INFO/WARN/ERROR/FATAL/UNSPECIFIED). LowCardinality gets you most of the storage/query efficiency of an enum without committing to one at the schema level. Splitting into a proper SeverityNumber + SeverityText pair (full OTel shape) is one of the open questions already flagged in /docs/architecture.md.
• PARTITION BY toDate(timestamp) (daily partitions) and ORDER BY (service, timestamp) are exactly what the task asked for — service-scoped queries over a time range are the dominant access pattern this is optimized for.
• No TTL/retention clause yet — also an open question in architecture.md, deferred until storage sizing is a real concern.

Migration tooling: a plain SQL-file runner, not golang-migrate

migrate.sh applies migrations/*.sql in filename order over ClickHouse's HTTP interface, tracking what's applied in a schema_migrations table. Chosen over golang-migrate for Phase 0 because there's exactly one migration to run — pulling in a migration framework (another dependency, another thing to configure/vendor) for a single CREATE TABLE is exactly the kind of premature machinery this project's conventions say to avoid. Revisit golang-migrate once there's real schema churn across environments (rollback support, checksums, concurrent-apply safety become worth their cost at that point, not before).

Convention: one DDL statement per migration file. The ClickHouse HTTP interface isn't reliably multi-statement, so migrate.sh doesn't try to split multi-statement files — keep each migration to a single statement.

Running

docker compose up -d                          # starts a standalone ClickHouse for local work
CLICKHOUSE_PASSWORD=sentry-dev-only ./migrate.sh   # applies migrations/*.sql

CLICKHOUSE_PASSWORD here has to match whatever docker-compose.yml set on the clickhouse service — found this the hard way running the Phase 0 runbook for real: the official ClickHouse image silently disables all network access (including the published port, not just container-to- container traffic) for the default user unless CLICKHOUSE_USER or CLICKHOUSE_PASSWORD is a genuinely non-empty value. An empty CLICKHOUSE_PASSWORD="" still triggers the lockdown — it has to actually have a value. Not a real secret, just what this image demands.

Environment variables migrate.sh reads (all optional except CLICKHOUSE_PASSWORD as of the above, matching /ingest's ClickHouse defaults so the two stay in sync out of the box):

Var	Default
CLICKHOUSE_HTTP	http://localhost:8123
CLICKHOUSE_USER	default
CLICKHOUSE_PASSWORD	(empty — override, see above)
CLICKHOUSE_DATABASE	sentry

There's also a Dockerfile (bash + curl baked in, migrations/ copied in at build time) used by the root-level docker-compose.yml as a one-shot init service — no runtime package install, no host volume mount needed.

Adding a migration

Add migrations/000N_description.sql with the next sequential number and a single DDL statement. migrate.sh picks it up automatically — no registration step.