Skip to content
Beef Docs

Brain Errors

Brain converts thrown HttpException instances into the standard JSON error envelope via HttpExceptionFilter (apps/brain/src/common/filters/http-exception.filter.ts):

{
  "statusCode": 401,
  "timestamp": "2026-05-05T12:00:00.000Z",
  "path": "/api/character/abc",
  "message": "Invalid token"
}

5xx and unexpected errors are logged at error through Pino and exported to Axiom when telemetry is enabled. Expected HTTP failures use warn. Request context supports correlation across services. See Observability for log routing.

Exception-handling convention

Section titled “Exception-handling convention”

Layering rule for thrown errors:

  • Services throw domain exceptions, not Nest HttpException subclasses. The shared vocabulary lives in apps/brain/src/modules/domain/generation/errors/ — most commonly EntityNotFoundException and EntityUnableToProcessException (plain Error subclasses). Services must stay transport-agnostic because they can be called from HTTP controllers, the gRPC microservice, or BullMQ processors alike.
  • HTTP controllers catch by type and map: EntityNotFoundException → NotFoundException (404), EntityUnableToProcessException → BadRequestException (400). Anything unexpected propagates untouched and reaches HttpExceptionFilter as a 500 with an error log.
  • Never map by message text (e.g. regex on “not found”) — only by exception type. Message sniffing silently converts genuine server errors into 4xx and leaks internal error text to clients.

Canonical example: apps/brain/src/modules/domain/generation/controllers/examples.http.controller.ts.

HTTP errors

Section titled “HTTP errors”
Code Trigger Remediation Reference
401 Unauthorized — Invalid token Clerk token verification failed (expired, signature, OAuth oat_ exchange failed). Refresh client session; verify CLERK_SECRET_KEY matches the Clerk instance. apps/brain/src/modules/application/auth/strategies/clerk.strategy.ts
401 Unauthorized — No token Missing Authorization header on a non-public route. Caller must send Authorization: Bearer <jwt> (Clerk) or Authorization: Bearer <key> for @ApiKeyRoute(). clerk.strategy.ts
401 Unauthorized (api-key) API key not in AUTHORIZED_KEYS. Rotate or extend AUTHORIZED_KEYS; comma-separated string. apps/brain/src/modules/application/auth/strategies/api-key.strategy.ts
403 Forbidden — Role ‘X’ is not authorized Clerk user resolved but lacks roles required by @Roles(...) or default ROOT, ADMIN. Update user role in fennec via user service or sirloin admin tooling. apps/brain/src/modules/application/auth/guards/roles.guard.ts
400 Bad Request — Request body must include data PUT /application-settings/:id payload missing top-level data. Wrap payload as { "data": { ... } }. application-settings.http.controller.ts
400 Bad Request (Zod) Application settings body fails Zod schema; message contains formatApplicationSettingsZodError details. Fix payload to match the registered Zod schema for that settings key. application-settings.http.controller.ts
400 Bad Request (class-validator) DTO fails class-validator rules in any controller (CustomValidationPipe). Fix request body / query to satisfy DTO; message is an array of validator errors. apps/brain/src/common/pipes/pipes/custom-validation.pipe.ts
404 Not Found Resource lookups (character, media, generation, tag) returning empty. Verify resource id; confirm caller is querying the right environment. Multiple controllers, generally raised from services.
409 Conflict Unique-constraint violations surfaced from Prisma. Find the Request failed log by correlationId; it keeps only the error’s name, message and stack, not err.meta. A mapped 409 logs the service’s own message (for example An example already exists for generation <id>). An unmapped P2002 is a 500 whose message names the constraint (Unique constraint failed on the constraint: <name>). Resolve the duplicate before retry. Various services; underlying Prisma error code P2002.
501 Not Implemented — Endpoint deprecated POST /api/webhook/rp/training is intentionally retired. Stop calling it; RunPod training webhook flow is gone. webhook.http.controller.ts
500 Internal Server Error (HttpExceptionFilter fallback) Any non-HttpException thrown bubbles up. Triage in Axiom by correlationId; reproduce via repro endpoint. http-exception.filter.ts

Round client errors

Section titled “Round client errors”

Round-side failures are wrapped at apps/brain/src/modules/application/round/errors/:

Class Trigger Remediation
RoundServiceUnavailableException gRPC connection failure / health check fail. Confirm ROUND_HOST; check round Railway service status.
RoundServiceTimeoutException Call exceeded the configured deadline. Re-tune timeout; check round CPU/GPU pressure.
RoundServiceInvalidResponseException Round returned an unexpected proto shape. Regenerate stubs; verify proto compatibility (make generate-proto).
RoundServiceException (base) Generic gRPC error. Inspect inner cause.

A circuit breaker in apps/brain/src/modules/application/round/interceptors/circuit-breaker.interceptor.ts short-circuits round calls after sustained failure — expect RoundServiceUnavailableException while open.

Generation pipeline errors

Section titled “Generation pipeline errors”
Class Trigger Remediation
ContentModerationException Hive or internal moderation flagged the input. Logged at warn. Check moderation logs; user-facing message must not leak internal details.
Provider exceptions (FAL / RunPod / Wavespeed / Kling / Vertex) 4xx/5xx from upstream provider. Inspect provider logs; check API key validity and quota.
BullMQ stalled jobs Worker crash mid-job; emitted via OnWorkerEvent('stalled') in MediaFlowsProcessor. Inspect Bull Board (/queues); check Redis connection; resume via the dashboard.

describeFalError() in apps/brain/src/modules/application/fal/adapters/fal-generic.adapter.ts folds FAL’s error.body.detail (FastAPI-style 422 field errors) into the error message instead of the bare HTTP status text (e.g. Unprocessable Entity), capped at 500 characters. For a flat request-level error body ({ detail, error_type }) it appends the type: Gateway Timeout: Request timed out (request_timeout). The message becomes the workflow execution error and the Generation’s logs error entry. Only staff surfaces read either.

Provider rejection classification

Section titled “Provider rejection classification”

falFailure() in the same file classifies a failed Fal call from ApiError.status and body.detail[].type only, never from message text:

Fal response Generation.metadata.failure_reason Moderation row Generation health success rate
422 with a content_policy_violation detail EXTERNAL_MODERATION_FAILED Yes: provider: 'fal', flagged, external_moderation badge Excluded from the sample
Any other 422 EXTERNAL_VALIDATION_FAILED No Counted as a failure
5xx, network error, timeout, or a COMPLETED queue status carrying error/error_type Not set No Counted as a failure

The classification rides on RecoverableError.context (provider, provider_task_id, provider_endpoint, mediaFailureReason), so fallback nodes still run. call_model (apps/brain/src/modules/domain/workflow/nodes/gen/call-model.node.ts) writes the moderation row best-effort when Fal rejects the request, so a fallback that succeeds does not hide the block from moderators.

When the execution fails, GenerationMappingService.onExecutionFailed writes:

  • Generation.metadata: failure_reason, provider, provider_request_id, provider_endpoint. provider_endpoint is the model route call_model posted to (for Fal, the endpoint id such as bytedance/seedance-2.0/mini/reference-to-video). Every failed completion rewrites all four from that failure alone, clearing any an earlier failure left, and a successful completion removes them.
  • Generation.logs: [{ event: 'GENERATION_FLOW_EXECUTION', phase: 'ERROR', at, error }]. Rows written from 2026-06-12 until this change hold a phase-less [{ error }]. Fennec reads both.

Non-Fal providers are not classified. Their failures carry no failure_reason.

Fennec links a Fal request id to https://fal.ai/dashboard/recent-history?s_requestId=<provider_request_id>&s_endpointId=<url-encoded provider_endpoint> (apps/fennec/src/lib/providerRequestUrl.ts). Fal opens the request only when both parameters are set, so a failure stored without provider_endpoint shows the request id without a link. The Generation Details “Fail reason” box shows a shortened error: it drops the leading Node "<id>" failed:, Subworkflow "<name>" failed: and Provider <NAME> failed: prefixes (readableFailError in apps/fennec/src/pages/GenerationsDashboard/utils.ts). Then it parses by provider:

  • FAL: drops each <type> (<loc>): detail prefix, for example Unprocessable Entity: Input should be '480p' or '720p'.
  • WaveSpeed: turns WaveSpeed API error: <status> - <json> into the HTTP status text and the body’s message, for example Bad Request: Invalid size.
  • Any other provider: shows the whole message, without a leading Error: name and with whitespace collapsed.

Only Fal failures get a request link. The stored error and the Logs tab stay unchanged.

Storage (R2) errors

Section titled “Storage (R2) errors”

StorageService in apps/brain/src/modules/application/storage/services/storage.service.ts wraps R2 failures:

Class or log Trigger Remediation
StorageObjectNotFoundException The object key does not exist (NoSuchKey or 404). OnboardingHttpController maps it to 404. Check the key the caller sent; the upload may not have finished.
getImageBuffer retrying key: <key> (warn) One read attempt sent no headers or body bytes for 15 s, or lost its connection mid-body. Brain retries it, up to 3 attempts inside the 60 s budget, 100 then 200 ms apart. None for a single line. Many lines at once mean R2 is slow; compare Sirloin S3 HEAD span latency for the same minutes.
OperationTimedOutException: getImageBuffer timed out after 15000ms ... received no bytes The final attempt stalled too. Usually R2-side. Check R2 status and the retry lines before it.
OperationTimedOutException: getImageBuffer timed out after 60000ms The whole read exceeded its 60 s budget. Same as above. Calling routes such as POST /api/onboarding/kitsune/image-score still return 500.

Before ENGINE-106, a stalled body ended after 60 s as Error: aborted (ECONNRESET) with no retry. That error was logged as S3 Error Context for getImageBuffer with errorName=Error.

Prisma errors

Section titled “Prisma errors”

Prisma errors propagate as unknown/Error subtypes; the filter logs them at error for Axiom triage. Paging depends on separately configured Axiom monitors.

Prisma code Trigger Remediation
P2002 Unique-constraint violation. Map to 409 Conflict in service; surface offending field.
P2003 Foreign-key constraint failed. Validate referenced id exists.
P2025 Record not found on update/delete. Map to 404 Not Found.
P1001 Cannot reach database. Check DATABASE_URL, network, Neon status.
P1017 Server closed connection. Common with pooler; usually transient.

Clerk errors

Section titled “Clerk errors”

@clerk/backend wraps issues from Clerk Frontend API:

Symptom Trigger Remediation
oat_ token rejected OAuth opaque token; Clerk Frontend API /oauth/userinfo exchange failed. Verify CLERK_FRONTEND_API_URL reachable; check Clerk instance domain.
Session JWT rejected Wrong CLERK_SECRET_KEY for issuer. Match secret to the Clerk instance issuing the token.
Network timeout Clerk API outage. Check Clerk status page; Brain has no offline session cache.

BullMQ errors

Section titled “BullMQ errors”
Symptom Trigger Remediation
Jobs not draining Redis unreachable or auth wrong. Check REDIS_HOST/PORT/PASSWORD and Railway Redis service.
Jobs marked failed with MaxStalledCount Worker repeatedly crashed. Investigate via Bull Board; restart workers; inspect Axiom error logs for the crash stack.
Memory growth Backlog of completed/failed jobs above retention. removeOnComplete / removeOnFail is set in queue.config.ts; raise retention only if needed.

TODO

Section titled “TODO”
  • No controller maps Prisma P2002/P2025 to dedicated ConflictException/NotFoundException; unmapped Prisma errors propagate to the global filter as 500 (no PrismaClientKnownRequestError references in apps/brain/src/).
  • ContentModerationException is the sole moderation error subclass under apps/brain/src/modules/domain/generation/errors/; no other moderation subclasses exist (sibling exceptions: application-misconfigured, entity-not-found, entity-unable-to-process).
  • Round circuit-breaker defaults (apps/brain/src/modules/application/round/interceptors/circuit-breaker.interceptor.ts:37-43): errorThresholdPercentage: 50, volumeThreshold: 10, rollingCountTimeout: 10000ms (10 buckets × 1s), resetTimeout: 15000ms (half-open probe), per-call timeout: 35000ms.