Observability

How we see what production is doing — logs, metrics, traces, and the playbook for chasing a 500 back to its root cause.

Stack Inventory

What is actually wired in, per service, with file evidence.

Service	Logs	Metrics	Traces	Errors	Product analytics
sirloin (Go)	log/slog (stdlib) shipped via OTLP logs (go.opentelemetry.io/otel/log, otlploghttp in apps/sirloin/go.mod)	OpenTelemetry metrics → OTLP HTTP (go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp)	OpenTelemetry traces → OTLP HTTP, with gRPC + HTTP middleware (otelgrpc, otelhttp); BUN auto-instrumented (bunotel, otelsql)	Sentry Go SDK (github.com/getsentry/sentry-go v0.44.1)	PostHog (github.com/posthog/posthog-go v1.11.2)
brain (NestJS)	pino + nestjs-pino + pino-http + pino-pretty (in apps/brain/package.json); auto-correlated with traces via @opentelemetry/instrumentation-pino	OpenTelemetry metrics → OTLP proto exporter (@opentelemetry/exporter-metrics-otlp-proto); endpoint set via BRAIN_OTEL_URL (.env.example:240); custom process-memory gauges in apps/brain/src/common/otel/metrics.ts	OpenTelemetry SDK Node (@opentelemetry/sdk-node, auto-instrumentations-node); OTLP HTTP exporters	Sentry NestJS (@sentry/nestjs, @sentry/profiling-node, @sentry/opentelemetry)	None — posthog-node is absent from apps/brain/package.json
brisket (Next.js)	Server: OTLP logs shipped via OpenTelemetry headers in BRISKET_OTEL_TRACES_LOGS_HEADERS (.env.example:63); browser: console + Sentry capture	OpenTelemetry metrics, exported to Axiom (BRISKET_OTEL_URL=https://api.axiom.co set in code)	@opentelemetry/sdk-node + auto-instrumentations; explicit trace/SpanStatusCode usage in apps/brisket/src	@sentry/nextjs (server + client)	posthog-js (useFeatureFlagEnabled, usePostHog across apps/brisket/src); posthog-node for server
fennec (React)	Browser console only (no log shipper in apps/fennec/package.json); SPA backed by nginx (apps/fennec/nginx.conf)	None observed	None observed	None — no @sentry/* in apps/fennec/package.json	None — no posthog-* in apps/fennec/package.json
flank (TS)	console / stdout (UI host only — the engine and its custom logger were removed)	None observed	None observed	None — no @sentry/* in apps/flank/package.json	None — no posthog-* in apps/flank/package.json
round (Go ML)	zerolog (apps/round/cmd/app/main.go: setupLogger returns zerolog.Logger)	None observed in go.mod	None observed	None observed	None observed
strip (Go SSR)	zerolog (apps/strip/cmd/app/main.go uses log.Info().Msg(...))	None observed	None observed	None observed	None observed
chuck (Strapi)	Strapi 5 default logger only — no logger override in apps/chuck/meat/config/	None — no metrics deps in apps/chuck/meat/package.json	None — no OTel deps in apps/chuck/meat/package.json	None — no @sentry/* in apps/chuck/meat/package.json	None — no posthog-* in apps/chuck/meat/package.json
shank (email)	n/a — build-time export only	n/a	n/a	n/a	n/a

Backends: OTLP traces/metrics/logs are exported to Axiom. The endpoint URL is configured per service via env: SIRLOIN_AXIOM_TOKEN / _DATASET / _METRICS_DATASET (.env.example:58-60), BRAIN_OTEL_URL / BRAIN_OTEL_METRICS_HEADERS / BRAIN_OTEL_TRACES_LOGS_HEADERS (.env.example:240-242), BRISKET_OTEL_URL / BRISKET_OTEL_METRICS_HEADERS / BRISKET_OTEL_TRACES_LOGS_HEADERS (.env.example:61-63). Sentry is the error sink for sirloin, brain, and brisket. PostHog is the product-analytics

• feature-flag sink for sirloin and brisket.

Log Levels and Conventions

Go services (sirloin, round, strip)

• sirloin uses Go stdlib log/slog with structured key/value pairs, OTLP-exported.
• round and strip use zerolog (apps/round/cmd/app/main.go, apps/strip/cmd/app/main.go). Levels: Debug, Info, Warn, Error, Fatal.
• Log level is configurable via env (round: setupLogger(level string)).
• Convention: log.Error().Err(err).Msg("failed to X") — error in err field, human message in Msg.

TS services (brain, brisket, flank, fennec)

• brain uses pino via nestjs-pino/pino-http. Prod = JSON, dev = pino-pretty. Auto-instrumented by OpenTelemetry so each log line carries trace_id/span_id.
• brisket uses Next.js + OpenTelemetry. Server logs ship via OTLP to Axiom (headers in BRISKET_OTEL_TRACES_LOGS_HEADERS, .env.example:63); browser uses console + Sentry capture.
• Levels: trace, debug, info, warn, error, fatal (pino defaults). TODO(@pawel): production minimum log level per service (brain pino config).

Metric Naming

TODO(@law): no documented convention found in code. Recommend OTel semantic conventions for HTTP/gRPC (http.server.duration, rpc.server.duration) and domain prefixes for custom metrics (sirloin.billing.invoice.failed, brain.generation.queued). File a follow-up if you want this enforced.

Brain exports these process-memory gauges through the same metrics datastream when BRAIN_OTEL_METRICS_HEADERS is configured:

• process.memory.rss — resident set size for the Node process.
• process.memory.external — native/external memory tracked by Node.
• process.memory.array_buffers — ArrayBuffer and Buffer backing memory.
• process.memory.heap_used — V8 heap used bytes.
• process.memory.heap_total — total V8 heap allocated bytes.
• process.memory.unaccounted — RSS not explained by V8 heap total plus Node-tracked external memory.

Brain also exports portable V8 and Sharp memory diagnostics:

• v8.heap.used_heap_size, v8.heap.total_heap_size, v8.heap.total_physical_size, v8.heap.total_available_size, v8.heap.heap_size_limit, v8.heap.malloced_memory, and v8.heap.peak_malloced_memory.
• v8.heap_space.used_size, v8.heap_space.space_size, v8.heap_space.physical_size, and v8.heap_space.available_size, grouped by v8.space.name.
• v8.code.code_and_metadata_size and v8.code.bytecode_and_metadata_size.
• v8.cpp_heap.used_size and v8.cpp_heap.committed_size when Node exposes C++ heap statistics.
• brain.sharp.queue, brain.sharp.process, brain.sharp.cache.memory, brain.sharp.cache.files, and brain.sharp.cache.items.

Brain also exports Node-specific runtime diagnostics:

• brain.runtime.active_resources — active resources from process.getActiveResourcesInfo(), grouped by node.resource.type.

Brain also exports file I/O limiter gauges from apps/brain/src/common/otel/metrics.ts, backed by apps/brain/src/modules/application/storage/services/file-io-limiter.service.ts:

• brain.file_io_limiter.active — active work items by limiter.name.
• brain.file_io_limiter.queued — queued work items by limiter.name.
• brain.file_io_limiter.limit — configured concurrency limit by limiter.name.
• brain.file_io_limiter.oldest_queued_ms — oldest queued wait by limiter.name.

Current limiter names are remote_downloads, video_transforms, provider_output_downloads, provider_preuploads, and archive_creation.

Brain also exports mediaflows BullMQ job-state gauges from the generation module. Each gauge uses queue.name=mediaflows:

• brain.queue.jobs.active — jobs claimed by Brain from BullMQ; downstream provider work may still be pending.
• brain.queue.jobs.waiting — jobs waiting in BullMQ.
• brain.queue.jobs.delayed — jobs delayed in BullMQ.
• brain.queue.jobs.failed — jobs failed in BullMQ.

For FOXY-202 native-memory leak checks, start with three Axiom chart groups in the brain metrics dataset:

1. RSS shape: process.memory.rss, process.memory.unaccounted, process.memory.external, process.memory.array_buffers, and process.memory.heap_total.
2. V8/native runtime: v8.heap.total_physical_size, v8.heap.malloced_memory, v8.heap.peak_malloced_memory, v8.cpp_heap.used_size, v8.code.code_and_metadata_size, and brain.runtime.active_resources.
3. Workload pressure: generation.media.total, round.infer.total, brain.file_io_limiter.active, brain.file_io_limiter.queued, brain.sharp.queue, and brain.sharp.process.

RSS/unaccounted growth while V8 heap, V8 malloced memory, and Sharp counters are flat points away from ordinary JS heap growth and toward native allocator, provider SDK, ffmpeg/gRPC, or platform-level RSS retention.

Trace Propagation

Trace IDs cross service boundaries via standard W3C traceparent headers.

Evidence (grep traceparent across apps/sirloin, apps/brain/src):

• TS code reads/writes traceparent and tracestate (e.g., normalizeTraceContextHeaderValue(headers.traceparent)).
• gRPC: otelgrpc interceptors on Sirloin and Brain’s Round client propagate across gRPC boundaries. Standard HTTP instrumentation covers Sirloin → Brain.
• Correlation ID fallback: x-correlation-id is read across NestJS / Fastify layers (req.headers['x-correlation-id'] ?? uuidv4()) and surfaced to logs as correlationId / req.id.

So a request hitting Brisket gets a traceparent, propagates over gRPC into Sirloin and then over HTTP to Brain. The trace terminates at the Sirloin/Brain → Round boundary: apps/round/go.mod declares no go.opentelemetry.io/* dependencies, and no OTel imports exist under apps/round/cmd or apps/round/internal.

Dashboards

TODO(@law): catalogue of Axiom dashboards, Sentry projects, PostHog dashboards (URLs and ownership). Suggested sections to fill:

• Axiom: per-service traces dashboard, error-rate dashboard, slow-query view.
• Sentry: one project per service (sirloin, brain, brisket).
• PostHog: funnels for signup → first generation → paid conversion.

Alerts and SLOs

TODO(@law): no alerts or SLO definitions found in this repo. If alerts live in Axiom / Sentry / PostHog UI rather than as code, document at minimum:

• Owning team / on-call.
• Pager destination.
• Threshold and rationale.

Expected failures that are deliberately not errors

Some spans look like failures but are the normal answer to a poll or a probe. Left alone they bury the real signal: these three sources accounted for roughly 135k of ~137k recorded error events, none of them an incident. Measured in Axiom against the beef-prod dataset over the 24h ending 2026-08-11 20:00 UTC, by counting spans and logs with severity_text == "error" or status.code == "ERROR" grouped by service and message. Each source is suppressed at a single narrow point, so anything outside the listed scope still alarms.

Signal	Why it is expected	Where it is suppressed
Brain GET /api/media/:id, HTTP 404	Sirloin polls this every 5s (job registered in apps/sirloin/internal/app/worker/worker.go) from dispatch until the workflow produces the media, so “not found” means “not ready yet”; the caller handles it in checkmediageneration.go.	Expected404SpanProcessor downgrades the span status to UNSET (apps/brain/src/common/otel/expected-404-span-processor.ts). Scoped to GET on that route with a 404.
S3 HeadObject, HTTP 404	Storage existence probes (fileExists / getObjectEtag) treat 404 as the answer, but the aws-sdk instrumentation still marks the span ERROR. Note that getObjectEtag swallows every error, not only a 404.	Same processor, same file. Scoped to a HeadObject call, or a HEAD whose host ends in a known S3 domain or carries any label starting with s3. That host match is deliberately broad, so an unrelated host such as s3-not-storage.example.com is also covered.
Brisket to Sentry ingest, HTTP 429	Sentry rate-limits once the quota is spent; HTTP auto-instrumentation traced every rejected envelope POST as an ERROR span.	ignoreOutgoingRequestHook skips the Sentry ingest host and our own OTLP endpoint (apps/brisket/otel.server.config.ts, predicate in apps/brisket/src/otel-runtime-config.ts).

A sustained 429 from Sentry means error reports are being dropped, not just untraced. Suppressing the span hides the noise, not the quota problem: check the Sentry plan usage when brisket error volume falls off a cliff.

Before adding a rule here, prefer fixing the caller. Suppress only where the failure is genuinely the expected answer, and keep the predicate narrow enough that a real regression on the same route still turns the span red.

Debugging Playbook — “I see a 500 in production”

Order assumes you have nothing but a Sentry alert or a user report.

1. Sentry — find the exception. Service → environment → time window. Grab the trace_id from the event tags (auto-attached for sirloin via getsentry/sentry-go + OTel, for brain via @sentry/opentelemetry).
2. Axiom — pivot on trace_id. Search across services for that trace ID. You will see the full request span tree: Brisket → Sirloin (gRPC) → Brain (HTTP) → Round/external. Identify the failing span.
3. Logs in Axiom. Filter logs by trace_id (sirloin slog → OTLP logs; brain pino → OTel logs). Correlate timestamps with the failing span. Fallback: filter by correlationId / x-correlation-id.
4. PostHog — user impact. Look up the affected user / session by distinct_id to confirm severity and reproduction path.
5. Reproduce locally. Capture the request payload from the trace, replay against a local stack (make dev-up-d). For billing or payment paths see the relevant ADRs in Decisions — many bugs in those flows are state-machine issues that need the right preconditions.

If the trace dead-ends at a service with no OTel (round, strip — currently), fall back to that service’s zerolog output by container/pod log search.

Sensitive Data in Logs

• Never log raw Chargebee card data, Primer tokens, or full Clerk JWTs.
• Email addresses currently appear in Sirloin.SubsAll.email and brain User.email. No global PII redaction layer exists today (see security-model.md → Logging hygiene for the partial sanitisers in apps/sirloin/internal/app/foxy360/server.go:526 and apps/sirloin/internal/app/services/media/listmediaexamples.go:53).
• See the Security Model page for the authoritative redaction rules.

Open follow-ups

1. Production minimum log level per service (TODO(@pawel) for brain).
2. Metric naming convention enforcement (TODO(@law)).
3. Dashboard inventory — Axiom / Sentry / PostHog URLs (TODO(@law)).
4. Alerts / SLOs / on-call ownership (TODO(@law)).
5. PII masking policy in logs (TODO(@law)).