search/src/consumer.rs's write-routing (built last pass) had no active- tenant check at all: IndexRegistry.resolve() would open-or-create an index directory for any syntactically-valid tenant_id, active or not -- unlike ClickHouse's chwriter.Registry (an active-tenants-only snapshot built at enterprise-ingest startup) or the read side (gated by searchclient.TenantChecker, a direct rbacstore query). search is AGPL core with no Postgres access and no enterprise/ import allowed, so it needed a network boundary instead -- the same shape ingest's TenantResolver already uses against enterprise-auth, just Rust calling Go instead of Go calling Go. New GET /internal/active-tenants endpoint on enterprise-auth (rbacstore.ListActiveTenantIDs + authhandler.handleActiveTenants), gated on a RoleService Bearer credential -- server-to-server auth, the same shape alerting presents to api, minted via the already-generic enterprise-auth -mint-service-token search. search/src/tenants.rs's ActiveTenantTracker polls it every 60s, blocking startup on the first fetch succeeding (fail-closed cold start -- a control-plane outage at boot must not silently accept every tenant_id) and keeping the last- known-good set on any later refresh failure (a transient blip shouldn't stop every tenant's indexing, only prevent the allowlist from growing/ shrinking until connectivity resumes). consumer.rs refuses any tagged record whose tenant isn't in the polled set, before ever calling resolve() -- IndexRegistry itself stays policy-free, matching the same mechanism/policy split clickhousewriter.Writer vs. chwriter.Registry already draws on the ClickHouse side. Off unless ENTERPRISE_AUTH_URL/ENTERPRISE_AUTH_SERVICE_TOKEN are both set (search/src/config.rs rejects exactly one being set) -- every existing deployment is unaffected. Verified with real HTTP round trips in this environment: tenants.rs's tests exercise real reqwest requests (actual Authorization: Bearer header, actual JSON parsing) against a hand-rolled dependency-free TCP test server, including both fail-closed paths (rejected first fetch, unreachable server). authhandler's new tests cover the credential-kind distinction this endpoint exists to enforce -- a real human session, even for a genuine Owner, must not satisfy a check meant for a service identity. One asymmetry remains, disclosed rather than fixed: chwriter.Registry's snapshot still never refreshes (stale until enterprise-ingest restarts), while ActiveTenantTracker's 60s poll gives Tantivy a materially tighter staleness window. Neither is a live per-write check -- that would mean a database/HTTP round trip per record, a throughput cost neither implementation accepts -- so both have some staleness window by design; the gap between the two windows is what's disclosed, not a claim either is fully live.
94 lines
4.2 KiB
Rust
94 lines
4.2 KiB
Rust
use anyhow::{Context, Result};
|
|
use std::path::PathBuf;
|
|
use std::time::Duration;
|
|
|
|
/// All via environment variables, same convention as /ingest and /api —
|
|
/// no config file format for this service either.
|
|
pub struct Config {
|
|
pub grpc_listen_addr: String,
|
|
pub redpanda_brokers: Vec<String>,
|
|
pub redpanda_topic: String,
|
|
pub index_path: PathBuf,
|
|
pub offsets_path: PathBuf,
|
|
pub commit_interval: Duration,
|
|
/// Phase 4: per-tenant index directories live under here, one
|
|
/// subdirectory per tenant_id, opened on demand by
|
|
/// registry::IndexRegistry for both the read side (SearchRequest.
|
|
/// tenant_id) and the write side (consumer.rs, routing on each
|
|
/// record's tenant_id Kafka header) -- `index_path` above stays the
|
|
/// single shared index an untagged record, or any deployment that
|
|
/// never turned on ingest's TenantResolver, still lands in. Default
|
|
/// matches the path convention deploy/operator's Tenant controller
|
|
/// and enterprise/internal/rbacstore's seeded default data source
|
|
/// already assume (`/var/lib/sentry-search/tenants/<id>`).
|
|
pub tenants_index_path: PathBuf,
|
|
/// Base URL of enterprise-auth's HTTP API, e.g.
|
|
/// `http://enterprise-auth:8082` -- same env var name and "empty
|
|
/// means off" shape as ingest/internal/config's own
|
|
/// ENTERPRISE_AUTH_URL. When set (together with
|
|
/// ENTERPRISE_AUTH_SERVICE_TOKEN below), tenants::ActiveTenantTracker
|
|
/// gates consumer.rs's write-routing on a polled active-tenant
|
|
/// allowlist -- see that module's doc comment for why this needed a
|
|
/// network call instead of direct Postgres access. When unset,
|
|
/// write-routing behaves exactly as it did before that tracker
|
|
/// existed: any syntactically-valid tenant_id is trusted.
|
|
pub enterprise_auth_url: Option<String>,
|
|
/// RoleService Bearer credential this process presents to
|
|
/// enterprise-auth's GET /internal/active-tenants -- minted via the
|
|
/// existing `enterprise-auth -mint-service-token search` (the flag
|
|
/// is already generic over caller name, no backend change needed to
|
|
/// mint one for a new caller). Required together with
|
|
/// enterprise_auth_url above; Config::load fails if exactly one of
|
|
/// the two is set, rather than silently running with the tracker
|
|
/// half-configured.
|
|
pub enterprise_auth_service_token: Option<String>,
|
|
}
|
|
|
|
impl Config {
|
|
pub fn load() -> Result<Self> {
|
|
let commit_interval_ms: u64 = getenv("COMMIT_INTERVAL_MS", "2000")
|
|
.parse()
|
|
.context("COMMIT_INTERVAL_MS must be a number")?;
|
|
|
|
let enterprise_auth_url = getenv_opt("ENTERPRISE_AUTH_URL");
|
|
let enterprise_auth_service_token = getenv_opt("ENTERPRISE_AUTH_SERVICE_TOKEN");
|
|
if enterprise_auth_url.is_some() != enterprise_auth_service_token.is_some() {
|
|
anyhow::bail!(
|
|
"ENTERPRISE_AUTH_URL and ENTERPRISE_AUTH_SERVICE_TOKEN must be set together, or neither -- got exactly one"
|
|
);
|
|
}
|
|
|
|
Ok(Self {
|
|
// Rust's SocketAddr parser needs a full address, unlike Go's
|
|
// net package (ingest/api's ":PORT" convention won't parse
|
|
// here).
|
|
grpc_listen_addr: getenv("GRPC_LISTEN_ADDR", "0.0.0.0:50052"),
|
|
redpanda_brokers: getenv("REDPANDA_BROKERS", "localhost:9092")
|
|
.split(',')
|
|
.map(str::to_string)
|
|
.collect(),
|
|
redpanda_topic: getenv("REDPANDA_TOPIC", "sentry.logs.raw"),
|
|
index_path: PathBuf::from(getenv("INDEX_PATH", "/var/lib/sentry-search/index")),
|
|
offsets_path: PathBuf::from(getenv(
|
|
"OFFSETS_PATH",
|
|
"/var/lib/sentry-search/offsets.json",
|
|
)),
|
|
commit_interval: Duration::from_millis(commit_interval_ms),
|
|
tenants_index_path: PathBuf::from(getenv(
|
|
"TENANTS_INDEX_PATH",
|
|
"/var/lib/sentry-search/tenants",
|
|
)),
|
|
enterprise_auth_url,
|
|
enterprise_auth_service_token,
|
|
})
|
|
}
|
|
}
|
|
|
|
fn getenv(key: &str, fallback: &str) -> String {
|
|
std::env::var(key).unwrap_or_else(|_| fallback.to_string())
|
|
}
|
|
|
|
fn getenv_opt(key: &str) -> Option<String> {
|
|
std::env::var(key).ok().filter(|v| !v.is_empty())
|
|
}
|