Observability

How we see what production is doing — logs, metrics, traces, and the playbook for chasing a 500 back to its root cause.

Stack Inventory

What is actually wired in, per service, with file evidence.

Service	Logs	Metrics	Traces	Errors	Product analytics
sirloin (Go)	zerolog shipped via the OTLP log bridge in apps/sirloin/internal/pkg/tracing/zerologotelwriter.go	OpenTelemetry metrics → OTLP HTTP (go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp)	OpenTelemetry traces → OTLP HTTP, with gRPC + HTTP middleware (otelgrpc, otelhttp); BUN auto-instrumented (bunotel, otelsql)	Structured error logs and worker panic stacks → Axiom	PostHog (github.com/posthog/posthog-go in apps/sirloin/go.mod)
brain (NestJS)	pino + nestjs-pino + pino-http + pino-pretty; correlated through @opentelemetry/instrumentation-pino	OpenTelemetry metrics → OTLP proto exporter; endpoint set via BRAIN_OTEL_URL; process-memory gauges in apps/brain/src/common/otel/metrics.ts	OpenTelemetry Node SDK with native HTTP, Express, Nest, and Prisma instrumentation	Structured service and HTTP exception logs → Axiom	None observed
brisket (Next.js)	Server Pino → OTLP; browser reports → same-origin /api/telemetry → server Pino	OpenTelemetry metrics → Axiom	Node SDK + auto-instrumentation and explicit tRPC spans; browser performance timings are logs	Browser, render-boundary, and Next.js request errors → Axiom	posthog-js and posthog-node
fennec (React)	Browser console only (no log shipper in apps/fennec/package.json); SPA backed by nginx (apps/fennec/nginx.conf)	None observed	None observed	No exporter observed	None — no posthog-* in apps/fennec/package.json
flank (TS)	console / stdout (UI host only — the engine and its custom logger were removed)	None observed	None observed	No exporter observed	None — no posthog-* in apps/flank/package.json
round (Go ML)	zerolog (apps/round/cmd/app/main.go: setupLogger returns zerolog.Logger)	None observed in go.mod	None observed	None observed	None observed
strip (Go SSR)	zerolog (apps/strip/cmd/app/main.go uses log.Info().Msg(...))	None observed	None observed	None observed	None observed
chuck (Strapi)	Strapi 5 default logger only — no logger override in apps/chuck/meat/config/	None — no metrics deps in apps/chuck/meat/package.json	None — no OTel deps in apps/chuck/meat/package.json	No exporter observed	None — no posthog-* in apps/chuck/meat/package.json
shank (email)	n/a — build-time export only	n/a	n/a	n/a	n/a

Backends: OTLP traces/metrics/logs are exported to Axiom. The endpoint URL is configured per service via env: SIRLOIN_AXIOM_TOKEN / _DATASET / _METRICS_DATASET, BRAIN_OTEL_URL / BRAIN_OTEL_METRICS_HEADERS / BRAIN_OTEL_TRACES_LOGS_HEADERS and BRISKET_OTEL_URL / BRISKET_OTEL_METRICS_HEADERS / BRISKET_OTEL_TRACES_LOGS_HEADERS. Axiom is the error destination for sirloin, brain, and brisket. PostHog handles product analytics for sirloin and brisket. Brisket also uses PostHog feature flags.

Log Levels and Conventions

Go services (sirloin, round, strip)

• sirloin uses zerolog with structured fields and an OTLP log bridge.
• round and strip use zerolog (apps/round/cmd/app/main.go, apps/strip/cmd/app/main.go). Levels: Debug, Info, Warn, Error, Fatal.
• Log level is configurable via env (round: setupLogger(level string)).
• Convention: log.Error().Err(err).Msg("failed to X") — error in err field, human message in Msg.

TS services (brain, brisket, flank, fennec)

• brain uses pino via nestjs-pino/pino-http. Prod = JSON, dev = pino-pretty. Auto-instrumented by OpenTelemetry so each log line carries trace_id/span_id.
• brisket uses Next.js + OpenTelemetry. Server logs ship via OTLP to Axiom with BRISKET_OTEL_TRACES_LOGS_HEADERS; browser reports use /api/telemetry and the same server export path.
• Levels: trace, debug, info, warn, error, fatal (pino defaults). TODO(@pawel): production minimum log level per service (brain pino config).

Brain’s caught exceptions use the existing service logger with sanitized error details and diagnostic identifiers. HTTP logs omit request bodies, query values, and parameter objects. HTTP 4xx failures remain warnings; 5xx and unexpected failures use error severity. Fatal process errors trigger a bounded two-second telemetry flush before exit status 1. Native HTTP, Express, Nest, Prisma, and Pino instrumentation retain request trace correlation.

Browser error diagnostics

For the operator and agent lookup procedure, dataset selection, and copy-paste APL queries, see the Brisket client-log runbook.

Brisket reports manual captures, uncaught browser errors, unhandled promise rejections, and checkout/render boundary failures through /api/telemetry. The Node route validates bounded same-origin batches and writes structured Pino logs. Axiom credentials remain on the server. The route derives user_id from Clerk and attaches source=browser, runtime=browser, environment, and release. The separate browser_release label identifies the client build, which can differ from the ingestion server after a deployment. Client buffers and context clear when the signed-in account changes.

Start an investigation with service, release, path, exception name, and stack. For a Next.js render failure, correlate its digest with the server error log. Raw browser stacks are available; automatic browser source-map symbolication is not provided. Brisket retains private server source maps, but structured Error.stack values can still contain compiled frames because Next.js controls stack formatting. Use the matching release artifacts for manual symbolication. Sampled navigation and request timings are logs, not distributed browser traces. These signals do not provide session replay or profiling.

Delivery is best-effort. Client budgets, server request limits, invalid payloads, offline clients, and exporter failures can drop reports. Timing reports have a separate budget so they cannot consume the error allowance. Telemetry requests are excluded from capture to prevent reporting loops. Known analytics and media cancellations remain filtered. Validate delivery and alert routing in staging or a controlled cutover; the repository does not provision Axiom monitors or notification destinations. Keep rollback Sentry credentials available until cutover verification and the agreed observation window finish. Before removing them, record the deployed revision, representative Axiom events, cross-service trace continuity, monitor health, and notification delivery in the rollout ticket. Brisket also observes background process errors while preserving Next.js exit behavior. An immediate process exit can prevent its buffered OTLP log from arriving; the Pino record remains in service stdout.

Metric Naming

TODO(@law): no documented convention found in code. Recommend OTel semantic conventions for HTTP/gRPC (http.server.duration, rpc.server.duration) and domain prefixes for custom metrics (sirloin.billing.invoice.failed, brain.generation.queued). File a follow-up if you want this enforced.

Brain exports these process-memory gauges through the same metrics datastream when BRAIN_OTEL_METRICS_HEADERS is configured:

• process.memory.rss — resident set size for the Node process.
• process.memory.external — native/external memory tracked by Node.
• process.memory.array_buffers — ArrayBuffer and Buffer backing memory.
• process.memory.heap_used — V8 heap used bytes.
• process.memory.heap_total — total V8 heap allocated bytes.
• process.memory.unaccounted — RSS not explained by V8 heap total plus Node-tracked external memory.

Brain also exports portable V8 and Sharp memory diagnostics:

• v8.heap.used_heap_size, v8.heap.total_heap_size, v8.heap.total_physical_size, v8.heap.total_available_size, v8.heap.heap_size_limit, v8.heap.malloced_memory, and v8.heap.peak_malloced_memory.
• v8.heap_space.used_size, v8.heap_space.space_size, v8.heap_space.physical_size, and v8.heap_space.available_size, grouped by v8.space.name.
• v8.code.code_and_metadata_size and v8.code.bytecode_and_metadata_size.
• v8.cpp_heap.used_size and v8.cpp_heap.committed_size when Node exposes C++ heap statistics.
• brain.sharp.queue, brain.sharp.process, brain.sharp.cache.memory, brain.sharp.cache.files, and brain.sharp.cache.items.

Brain also exports Node-specific runtime diagnostics:

• brain.runtime.active_resources — active resources from process.getActiveResourcesInfo(), grouped by node.resource.type.

Brain also exports file I/O limiter gauges from apps/brain/src/common/otel/metrics.ts, backed by apps/brain/src/modules/application/storage/services/file-io-limiter.service.ts:

• brain.file_io_limiter.active — active work items by limiter.name.
• brain.file_io_limiter.queued — queued work items by limiter.name.
• brain.file_io_limiter.limit — configured concurrency limit by limiter.name.
• brain.file_io_limiter.oldest_queued_ms — oldest queued wait by limiter.name.

Current limiter names are storage_uploads, remote_downloads, video_transforms, provider_output_downloads, and provider_preuploads.

Brain also exports mediaflows BullMQ job-state gauges from the generation module. Each gauge uses queue.name=mediaflows:

• brain.queue.jobs.active — jobs claimed by Brain from BullMQ; downstream provider work may still be pending.
• brain.queue.jobs.waiting — jobs waiting in BullMQ.
• brain.queue.jobs.delayed — jobs delayed in BullMQ.
• brain.queue.jobs.failed — jobs failed in BullMQ.

For FOXY-202 native-memory leak checks, start with three Axiom chart groups in the brain metrics dataset:

1. RSS shape: process.memory.rss, process.memory.unaccounted, process.memory.external, process.memory.array_buffers, and process.memory.heap_total.
2. V8/native runtime: v8.heap.total_physical_size, v8.heap.malloced_memory, v8.heap.peak_malloced_memory, v8.cpp_heap.used_size, v8.code.code_and_metadata_size, and brain.runtime.active_resources.
3. Workload pressure: generation.media.total, round.infer.total, brain.file_io_limiter.active, brain.file_io_limiter.queued, brain.sharp.queue, and brain.sharp.process.

RSS/unaccounted growth while V8 heap, V8 malloced memory, and Sharp counters are flat points away from ordinary JS heap growth and toward native allocator, provider SDK, ffmpeg/gRPC, or platform-level RSS retention.

Trace Propagation

Trace IDs cross service boundaries via standard W3C traceparent headers.

Evidence (grep traceparent across apps/sirloin, apps/brain/src):

• TS code reads/writes traceparent and tracestate (e.g., normalizeTraceContextHeaderValue(headers.traceparent)).
• gRPC: otelgrpc interceptors on Sirloin and Brain’s Round client propagate across gRPC boundaries. Standard HTTP instrumentation covers Sirloin → Brain.
• Correlation ID fallback: x-correlation-id is read across NestJS / Fastify layers (req.headers['x-correlation-id'] ?? uuidv4()) and surfaced to logs as correlationId / req.id.

So a request hitting Brisket gets a traceparent, propagates over gRPC into Sirloin and then over HTTP to Brain. The trace terminates at the Sirloin/Brain → Round boundary: apps/round/go.mod declares no go.opentelemetry.io/* dependencies, and no OTel imports exist under apps/round/cmd or apps/round/internal.

Dashboards

TODO(@law): catalogue of Axiom dashboards and PostHog dashboards (URLs and ownership). Suggested sections to fill:

• Axiom: per-service traces dashboard, error-rate dashboard, slow-query view.
• PostHog: funnels for signup → first generation → paid conversion.

Alerts and SLOs

Alerts live in the Axiom UI as monitors on the beef-prod dataset, not as code. The ones below are the ones this doc owns; the rest are listed in Axiom under Monitors.

Monitor	Owner	Destination	Threshold	Why
Sirloin examples statements slow - PROD	@law	#log_beef-alerts-prod	20 or more Sirloin db spans on media.examples over 500 ms in 15 min, checked every 5 min	A statement that returns tens of thousands of example rows per call is slow before it is expensive. The bounded search and the hourly catalog load stay under this; the pre-fix search alone tripped it. See FOXY-1204.

Attributing Neon data transfer

Neon bills egress, and pg_stat_statements on the project hides query and queryid from the read-only roles. To find what is moving bytes:

1. Order pg_stat_statements by rows, keep calls, wal_bytes (writes have it, reads do not), and mean_exec_time.
2. Size a row with avg(pg_column_size(t.*)) on the suspect table, or a tablesample for large tables. Bytes per day is rows per call times width times calls per day.
3. Match the statement to its text in Axiom: Sirloin db spans carry attributes.db.statement, so group by the statement prefix and compare call rate and mean duration with the pg_stat_statements row. Brain emits no db spans; use attributes.http.route counts instead.

Full-table reads of media.examples were the 2026-09 case; the fix and the numbers are in Linear FOXY-1204.

Expected failures that are deliberately not errors

Some spans look like failures but are the normal answer to a poll or a probe. Suppress these at narrow instrumentation boundaries so unrelated failures remain visible.

Signal	Why it is expected	Where it is suppressed
Brain GET /api/media/:id, HTTP 404	Sirloin polls this every 5s (job registered in apps/sirloin/internal/app/worker/worker.go) from dispatch until the workflow produces the media, so “not found” means “not ready yet”; the caller handles it in checkmediageneration.go.	Expected404SpanProcessor downgrades the span status to UNSET (apps/brain/src/common/otel/expected-404-span-processor.ts). Scoped to GET on that route with a 404.
S3 HeadObject, HTTP 404	Storage existence probes (fileExists / getObjectEtag) treat 404 as the answer, but the aws-sdk instrumentation still marks the span ERROR. Note that getObjectEtag swallows every error, not only a 404.	Same processor, same file. Scoped to a HeadObject call, or a HEAD whose host ends in a known S3 domain or carries any label starting with s3. That host match is deliberately broad, so an unrelated host such as s3-not-storage.example.com is also covered.
Brisket OTLP export request	Tracing telemetry export would recursively generate telemetry about itself.	ignoreOutgoingRequestHook skips the configured OTLP host (apps/brisket/otel.server.config.ts, predicate in apps/brisket/src/otel-runtime-config.ts).

A sustained exporter failure can drop reports. Check exporter diagnostics, credentials, ingestion limits, and the last received event when a service’s telemetry volume falls unexpectedly. Export suppression is not delivery proof.

Before adding a rule here, prefer fixing the caller. Suppress only where the failure is genuinely the expected answer, and keep the predicate narrow enough that a real regression on the same route still turns the span red.

Debugging Playbook — “I see a 500 in production”

Start with an Axiom alert or a user report.

1. Axiom: find the error. Filter by service, environment, and a narrow time window. Read the error stack, release, and available correlation identifiers.
2. Axiom — pivot on trace_id. Search across services for that trace ID. Inspect the request span tree: Brisket → Sirloin (gRPC) → Brain (HTTP). Identify the failing span; Round itself has no exporter.
3. Logs in Axiom. Filter logs by trace_id where present (Brain/Brisket Pino → OTel logs). Sirloin’s zerolog bridge does not automatically attach active span context. Correlate timestamps with the failing span. Fallback: filter by correlationId / x-correlation-id.
4. PostHog — user impact. Look up the affected user / session by distinct_id to confirm severity and reproduction path.
5. Reproduce locally. Capture the request payload from the trace, replay against a local stack (make dev-up-d). For billing or payment paths see the relevant ADRs in Decisions — many bugs in those flows are state-machine issues that need the right preconditions.

If the trace dead-ends at a service with no OTel (round, strip — currently), fall back to that service’s zerolog output by container/pod log search.

Sensitive Data in Logs

• Never log raw Chargebee card data, Primer tokens, or full Clerk JWTs.
• Email addresses currently appear in Sirloin.SubsAll.email and brain User.email. No global PII redaction layer exists today (see security-model.md → Logging hygiene for the partial sanitisers in apps/sirloin/internal/app/foxy360/server.go:526 and apps/sirloin/internal/app/services/media/listmediaexamples.go:53).
• See the Security Model page for the authoritative redaction rules.

Open follow-ups

1. Production minimum log level per service (TODO(@pawel) for brain).
2. Metric naming convention enforcement (TODO(@law)).
3. Dashboard inventory: Axiom / PostHog URLs (TODO(@law)).
4. Alerts / SLOs / on-call ownership (TODO(@law)).
5. PII masking policy in logs (TODO(@law)).