Skip to content
Beef Docs

Brain On-Call Quickstart

You are paged for brain. This page tells you what brain does, the top three things to check, and where to escalate. For deep-dive playbooks, jump to Brain Runbook.

What brain is, in one paragraph

Section titled “What brain is, in one paragraph”

Brain is the NestJS service that owns the fennec Postgres schema (characters, generation media, datasets, tags), drives generation pipelines via BullMQ on Redis, and brokers calls to round (gRPC) plus external AI providers (FAL, RunPod, Kling, Wavespeed, Vertex, OpenRouter, Hive). End-users authenticate via Clerk; sirloin and other internal services authenticate via static API keys.

Code: apps/brain/. Stage resolution: apps/brain/src/common/runtime/stage.ts.

First five minutes

Section titled “First five minutes”
  • Sign in to Axiom. Select the organization containing Brain’s configured datasets. For traces and logs, open the dataset from BRAIN_OTEL_TRACES_LOGS_HEADERS and filter service.name to brain. For metrics, use the dataset from BRAIN_OTEL_METRICS_HEADERS.
  • Open production Bull Board. For staging or a preview, append /queues to that environment’s current Brain domain.
  1. Identify the symptom class.
    • 5xx spike → Axiom error logs, grouped by exception and deployment.
    • 401/403 spike → likely Clerk or API-key issue (see Auth).
    • Slow generations / queue depth → Bull Board mediaflows queue.
    • Memory growth / OOM restarts → Axiom metrics process.memory.rss, process.memory.external, process.memory.array_buffers, process.memory.heap_used.
    • Webhook failures → Axiom filter on path:/api/webhook/*.
  2. Check recent deploys. Railway → brain → Deployments. A deploy in the last hour is the most common root cause.

Top alerts

Section titled “Top alerts”

These are the alert classes operators should expect. Specific thresholds and channels live in observability config (TODO(@pawel): link to alert defs).

Alert Usual cause First action
Brain 5xx rate > baseline Bad deploy, provider outage, Prisma error storm Check Axiom error logs; roll back if deploy-correlated.
mediaflows queue depth growing Worker crash, Redis unavailable, round down Bull Board → check Failed/Stalled; check round and Redis health.
RoundServiceUnavailableException spike round down or networking See Brain Runbook → Round outage.
Clerk 401 spike Clerk outage or wrong secret Clerk status; verify CLERK_SECRET_KEY.
Postgres connection errors (P1001/P1017) Neon saturation or pooler hiccup Neon dashboard; reduce queue concurrency if needed.
Processor exception in Axiom logs Provider regression, schema drift Inspect stack; coordinate with provider owners.

Common causes by symptom

Section titled “Common causes by symptom”
  • Generations stuck “in progress”: mediaflows worker crashed mid-job — onStalled log entries appear in Axiom. Restart brain.
  • API keys not accepted: AUTHORIZED_KEYS was rotated without redeploy. Both old and new keys can be present (comma-separated) during rotation.
  • Webhook 4xx from VI Generator: API key mismatch or DTO drift. Check webhook.http.controller.ts against the upstream payload.
  • All requests 502 on Railway: brain didn’t bind IPv6. Confirm logs show Listening on [::]:3000 — if not, the service was started with default binding and Railway can’t reach it (see main.ts comments).

Escalation

Section titled “Escalation”
  1. Brain owner: @pawel (per docs/src/content/docs/services/brain.md). TODO(@pawel): confirm rotation.
  2. Database: Neon owner — see Deployment Environment.
  3. Redis: TODO(@pawel) Upstash vs. Railway Redis owner.
  4. Round (ML): separate on-call — see Round.
  5. Sirloin (calls into brain): see Sirloin on-call (TODO(@pawel) link to sirloin-oncall once published).
  6. Clerk: external — open ticket in Clerk dashboard if API is implicated.

Communications

Section titled “Communications”
  • Status: TODO(@pawel) link to internal status channel.
  • Customer-facing: coordinate with sirloin/brisket on-call before posting; brain alone does not own customer messaging.

Don’t do during an incident

Section titled “Don’t do during an incident”
  • Don’t prisma migrate resolve --rolled-back in production without coordination — see Brain Runbook → Prisma migration rollback.
  • Don’t bulk-retry failed mediaflows jobs without checking root cause; you’ll multiply provider spend.
  • Don’t rotate AUTHORIZED_KEYS to a single new value without first appending — sirloin and admin tooling may be holding the old key.

TODO

Section titled “TODO”
  • TODO(@pawel): Link Axiom dashboards (logs, traces, BullMQ depth).
  • TODO(@pawel): Document SLOs (5xx rate, p95 latency, queue depth) once published.
  • TODO(@pawel): Confirm primary and secondary on-call rotation source of truth.