Skip to content
Beef Docs

Brisket On-Call

Brisket is a customer-facing Next.js app. Most user-impacting incidents present as elevated tRPC error rates, a spike in 5xx from route handlers, or front-end performance regressions. This page lists what an on-call should watch and how to respond.

What’s Instrumented

Section titled “What’s Instrumented”
  • OpenTelemetry: apps/brisket/src/server/api/trpc.ts tracingMiddleware emits trpc.${type}.${path} spans with status codes; recordTrpcRequest (apps/brisket/src/lib/metrics.ts) emits a metric per request. Downstream protection emits trpc.downstream.rpc.fanout, downstream.rpc.calls.total, downstream.rpc.duration, downstream.rpc.queue.wait, downstream.rpc.in_flight, downstream.rpc.queue.depth, and downstream.rpc.rejected.total. Exporters are bootstrapped in apps/brisket/otel.server.config.ts (URLs from BRISKET_OTEL_URL; BRISKET_OTEL_TRACES_LOGS_HEADERS / BRISKET_OTEL_METRICS_HEADERS carry Axiom bearer authentication plus x-axiom-dataset for traces and logs or x-axiom-metrics-dataset for metrics) and gated by apps/brisket/src/otel-runtime-config.ts. Railway telemetry includes service.instance.id from RAILWAY_REPLICA_ID, so capacity is evaluated per replica rather than across the whole service.
  • Request protection: the tRPC client and server cap each batch at 10 procedures. apps/brisket/src/server/api/rpc-policy.ts permits 48 active and 128 queued downstream RPCs per Brisket process, with queued calls rejected after 2 seconds. Ordinary Sirloin calls time out after 10 seconds, selected external or billing operations after 60 seconds, and long-running operations after 120 seconds. Client disconnects cancel queued work and propagate to active Connect calls. The global pending-media read fetches at most 500 jobs in one RPC and rejects overflow instead of returning an incomplete list.
  • Error telemetry: server Pino logs and bounded browser batches through /api/telemetry reach Axiom when export is enabled. Render errors can carry a digest for cross-reference. Browser stacks remain raw; sampled navigation and resource timing logs provide limited performance diagnostics.
  • PostHog: product analytics + lifecycle events (account_created, logged_in) emitted from the Clerk webhook.
  • Health probe: GET /api/health — used by Railway healthcheck.

Top Alerts

Section titled “Top Alerts”

Targets below are starting points — calibrate against historical baselines.

# Alert Signal Threshold Why it matters
1 tRPC error rate (overall) Axiom: % of trpc.* spans with status=ERROR > 2 % over 5 min Indicates a broad outage (sirloin down, deploy regression, Clerk failure).
2 Checkout error rate Axiom: subscription.createPrimerCheckout errors > 5 % over 10 min Direct revenue impact.
3 Auth failure rate Axiom: middleware redirects / Clerk error logs > baseline × 3 over 5 min Sign-in broken.
4 /api/health failing Railway healthcheck red any Pod cannot serve traffic. Auto-restart, but page if repeats.
5 Webhook signature failures Axiom log filter Clerk webhook signature verification failed > 5 in 5 min Misconfigured CLERK_WEBHOOK_SECRET or tampering.
6 Sirloin Connect Unavailable (code 14) Axiom span tag sustained > 1 % for 5 min Upstream outage; brisket has no fallback.
7 Release error volume Axiom error logs grouped by release > N errors in first 30 min after deploy Bad deploy, candidate for rollback.
8 Build failures on release GitHub Actions / Railway any Production deploy stuck.
9 Core Web Vitals regression PostHog $web_vitals (sampled at 0.2 in features/analytics/posthog/PosthogProvider.tsx) LCP p75 > 2.5 s, INP p75 > 200 ms UX degradation.
10 OTel exporter failures Axiom ingest 401 / log line “Failed to export traces” any sustained Observability gap; on-call is flying blind.
11 Downstream RPC capacity warning Axiom downstream.rpc.in_flight, summed across rpc.method and grouped by service.instance.id at least 38 active calls for 5 consecutive min on one replica A replica has used about 80% of its 48-call limit and may begin queueing if demand or latency rises. Notify Slack.
12 Downstream RPC saturation critical Axiom increase of downstream.rpc.rejected.total, grouped by service.instance.id, where rpc.rejection.reason is queue_full or queue_timeout any increase in a 1 min window A replica has exhausted useful capacity and is shedding work. Page the on-call and notify Slack.

Configure alerts 11 and 12 against the production metrics dataset after this version is deployed, then test-fire both Slack and PagerDuty routes. The source repository supplies the signals and stable per-replica labels but does not manage Axiom monitors or notification integrations.

Dashboards

Section titled “Dashboards”
  • Axiom — Brisket overview: tRPC throughput, error rate, p50/p95/p99 by procedure. TODO(@marty): dashboard link.
  • Axiom error logs: compare browser and server errors by release. Configure monitors separately; the repository does not provision them.
  • PostHog: funnel for sign-up → first generation → paid checkout.
  • Railway — brisket service: deploy log, healthcheck status, build logs.

Triage Playbook

Section titled “Triage Playbook”
  1. Confirm scope — public outage (/api/health red, all users) vs. partial (one tRPC procedure spiking). Check Axiom by trpc.procedure tag, then compare trpc.downstream.rpc.fanout, downstream.rpc.queue.depth, and downstream.rpc.in_flight over the same incident window.
  2. Check upstreams — sirloin status, Chargebee status, Clerk status, Cloudflare Turnstile status. Brisket fails in user-visible ways when any of these degrade.
  3. Check recent deploys: compare the Railway deploy log with Axiom error volume. If the spike began at deploy time, rollback (see brisket-runbook).
  4. Open a war-room: paste Axiom links and the current error sample. Avoid speculation; lead with data.
  5. User comms — for >5-minute customer-visible issues, post status update via the standard channel. TODO(@marty): confirm status-page integration.

Escalation

Section titled “Escalation”
Tier Owner When
L1 Brisket on-call First responder; can rollback, restart, redeploy.
L2 Frontend lead Persistent client-side bug, deploy-time build issues.
L3 Sirloin on-call Confirmed upstream gRPC failure (sirloin returning systemic errors).
L3 Auth/identity owner Clerk webhook or session integrity issues.
L3 Billing owner Chargebee/Primer outage during checkout.

Owner directory: TODO(@marty): link to internal on-call rotation.

Useful Queries

Section titled “Useful Queries”

Axiom (tRPC error rate by procedure, last 15 min):

['beef-prod']
| where _time > ago(15m) and span_name startswith "trpc."
| summarize errors = countif(status == "ERROR"), total = count() by trpc_procedure
| extend error_rate = todouble(errors) / todouble(total)
| order by error_rate desc

TODO(@marty): exact Axiom field/dataset names; this is illustrative.

Anchors

Section titled “Anchors”
  • operations/observability — full observability standards.
  • operations/testing-strategy — pre-deploy gating.
  • standards/auth-model, standards/security-model — context for auth/billing alerts.
  • Companion docs: brisket-errors, brisket-runbook.