gRPC Mesh Integration Playbook

gRPC is the primary inter-service protocol across the beef monorepo. REST is exposed only at edges (public API to chargebee/primer webhooks, public REST gateway on sirloin). Existing sirloin → brain application calls use HTTP; the character runtime uses gRPC in both directions. This page is the playbook for the internal mesh: who calls whom, how it’s secured, how protos are versioned and generated, what to watch for when it breaks.

1. Overview

• Transport: HTTP/2 over plaintext (h2c) — see Section 4.
• Schema source of truth: proto/ at the repo root, organised as proto/<service>/<version>/*.proto (proto-files-list).
• Codegen tool: buf v2 (buf.yaml, buf.gen.yaml) with local plugins (protoc-gen-go, protoc-gen-go-grpc, ts-proto, @bufbuild/protoc-gen-es). See Section 7.
• Generated stubs are checked in under each service’s internal/pkg/pb/ (Go), src/generated/ (brain), or pkg/ (brisket).
• Service auth on the wire: minimal. Network-trust + per-edge mechanisms (API key for sirloin → brain HTTP; private-mesh trust for the character runtime; Clerk JWT propagated as user_id in proto payloads for user-bound calls). See Section 5.
• RPC surface: see the generated index in /generated/proto-api.

The single biggest correctness invariant: every gRPC hop is plaintext, internal-only. No call is meant to traverse the public internet. If you find yourself wanting to expose a gRPC port externally, stop and route through sirloin’s REST gateway instead.

2. Topology

2.1 Service mesh

flowchart LR
  subgraph Edge[Public Edge]
    Browser[User browser]
    Admin[Admin browser]
    CB[Chargebee]
    PR[Primer]
  end

  subgraph Mesh[Internal mesh — plaintext h2c]
    BR[brisket Next.js]
    FE[fennec admin]
    ST[strip SSR]
    FL[flank]
    SI[(sirloin)]
    BN[(brain)]
    RD[(round ML)]
  end

  Browser -->|HTTPS REST tRPC| BR
  Admin -->|HTTPS| ST
  Browser -->|HTTPS Clerk| FE
  Browser -->|HTTPS Clerk| FL
  CB -->|HTTPS REST + Basic auth| SI
  PR -->|HTTPS REST + HMAC| SI

  BR -->|gRPC ConnectRPC| SI
  ST -->|gRPC| SI
  FL -->|HTTP + Clerk JWT| BN
  SI -->|HTTP + API key| BN
  SI -->|gRPC CharacterRuntimeService| BN
  SI -->|gRPC insecure| RD
  BN -->|gRPC insecure| RD

2.2 Direction notes

• brisket → sirloin is gRPC over Connect, not REST; see the root AGENTS.md. Two clients are constructed in apps/brisket/server/api/sirloin-api.ts (SirloinService, BillingService) using createGrpcTransport({ baseUrl: env.SIRLOIN_URL, peerMaxConcurrentStreams: 500 }). The transport is refreshed every 30 minutes to mitigate stale H2 connections.
• strip → sirloin is plain Go gRPC (apps/strip/cmd/app/main.go:165 — grpc.NewClient). user_id is passed inside proto request bodies, not via header metadata.
• sirloin → brain uses two transports. Existing application calls use HTTP with the shared SIRLOIN_BRAIN_API_KEY; brain validates it via apps/brain/src/modules/application/auth/strategies/api-key.strategy.ts. Recipe validation / health use CharacterRuntimeService on the private gRPC listener hosted by brain’s normal AppModule process. Character EXECUTE dispatch is HTTP Bull execute-by-id, not unary Run (UNIMPLEMENTED).
• Character control plane (Acquire / Commit) for Bull EXECUTE is Sirloin-local storage at wake-up plus the Brain→Sirloin HTTP character-completion webhook — deliberately the same completion channel as media-completion, not mid-flight CharacterRuntimeControlService RPCs (those belonged to the retired sync Run path). Brain dials the control service only for the read RPCs GetCharacterDataset, GetCharacterAttributes and GetCharacterApprovedCelebrity, never for Acquire / Commit. See Character Recipe Engine — Auth posture for the full rationale. CharacterRecipeService, CharacterSessionService, CharacterProjectionService (brisket / flank), and CharacterCapabilityService still rely on private-mesh reachability as the trust boundary: no per-call service credential on the character plane. Typed account and actor IDs scope the operation and are trusted from mesh-internal callers; lease fence tokens protect execution concurrency and are not authentication credentials.
• sirloin → round is gRPC, plaintext (apps/sirloin/cmd/app/main.go:237,263 — insecure.NewCredentials()).
• brain → round is gRPC, plaintext (apps/brain/src/modules/application/round/providers/round-grpc-client.provider.ts:19 — credentials.createInsecure()).
• flank → brain is HTTP with a Clerk session JWT (apps/flank/app/lib/brain-http-client.ts). Flank also retains narrow gRPC calls to Sirloin for CharacterRecipeService and StripService; the former bidirectional workflow link (FlankStorageService and FlankExecutionService) has been removed along with proto/flank/v1 and proto/sirloin/v5/flank.proto.
• fennec is a SPA — its gRPC needs are funneled through sirloin’s REST gateway. Confirmed: no @bufbuild/@connectrpc import or grpc dial in apps/fennec/src/.
• chuck (Strapi) is HTTP only; not part of the mesh.

3. Services + RPCs index

Per .proto file and owning service:

Proto file	Owner service (server)	Common callers (client)
proto/round/v1/round.proto	round	sirloin, brain
proto/sirloin/v5/sirloin.proto	sirloin	brisket, strip
proto/sirloin/v5/billing.proto	sirloin	brisket
proto/sirloin/v5/strip.proto	sirloin	strip
proto/brain/v1/character_runtime.proto	brain (CharacterRuntimeService), sirloin (CharacterRuntimeControlService)	sirloin, brain
proto/jobs/v1/jobs.proto	sirloin (JobService)	brain

The full RPC list, request/response shapes, and field-level docs are auto-generated from these protos; see /generated/proto-api, regenerated via bash scripts/gen-proto-docs.sh.

4. Transport security

Today: none on the wire. All inter-service gRPC uses h2c (HTTP/2 plaintext). Evidence:

Site	Code	Evidence
sirloin gRPC server	apps/sirloin/cmd/app/main.go:539	grpc.NewServer(...) — no grpc.Creds(...) option
sirloin → round client	apps/sirloin/cmd/app/main.go:237,263	grpc.WithTransportCredentials(insecure.NewCredentials())
brain → round client	apps/brain/.../round-grpc-client.provider.ts:19	credentials.createInsecure()
brain character-runtime server	apps/brain/.../character-runtime.grpc-server.ts	Server.bindAsync(..., ServerCredentials.createInsecure(), ...)
round gRPC server	apps/round/internal/app/server/server.go:63	grpc.NewServer(...) — no creds
strip → sirloin client	apps/strip/cmd/app/main.go:165	grpc.NewClient(...) (insecure)

Confidentiality / integrity therefore relies on the network boundary:

• Local dev: shared Docker network created by docker-compose.dev.yml.
• Production: Railway private network. TODO(@law): confirm the network guarantees (private VPC, no cross-tenant egress) against Railway docs and current platform config; cross-link to /standards/security-model/#in-transit.

flowchart LR
  subgraph Internet
    User[User]
  end
  subgraph Railway[Railway private network — plaintext within]
    direction LR
    SI[sirloin] -.h2c.-> RD[round]
    SI -.h2c.-> BN[brain]
    BN -.h2c.-> RD
    BR[brisket] -.h2c.-> SI
  end
  User ==HTTPS==> BR
  User ==HTTPS==> SI
  classDef plain fill:#fee,stroke:#c33;
  class SI,RD,BN,BR plain;

5. Service-to-service auth

Cross-link: see /standards/auth-model/ for the full table.

Caller → Callee	Mechanism	Evidence
sirloin → brain (HTTP)	API key header SIRLOIN_BRAIN_API_KEY	.env.example; apps/brain/.../api-key.strategy.ts
sirloin → brain character runtime (gRPC)	None (network-trust)	apps/sirloin/internal/app/services/characterruntimebrain.go
brain → sirloin character completion (HTTP)	X-Webhook-Secret	/webhooks/brain/character-completion; same pattern as media-completion
sirloin → round (gRPC)	None (network-trust)	apps/sirloin/cmd/app/main.go:237,263
brain → round (gRPC)	None (network-trust)	round-grpc-client.provider.ts:19
brain → sirloin jobs (gRPC jobs.v1.JobService)	Shared static token x-service-token (SIRLOIN_BRAIN_SERVICE_TOKEN), warn-and-allow while unset (rollout step 1); kinds validated against Sirloin’s enqueueable registry	apps/sirloin/internal/app/services/jobs_token.go; apps/brain/src/modules/application/sirloin/adapters/sirloin-jobs.adapter.ts
flank → brain (HTTP)	Clerk session JWT minted per request	apps/flank/app/lib/brain-http-client.ts, apps/flank/app/lib/auth.ts
flank → sirloin character recipes (gRPC)	None (network-trust); typed account/actor IDs scope writes	apps/flank/app/lib/grpc-client.ts
brisket → sirloin (gRPC)	Clerk JWT forwarded in headers via grpc-headers.ts; sirloin verifies user context	apps/brisket/server/api/sirloin-api.ts
strip → sirloin (gRPC)	user_id passed inside proto request body; verified by sirloin against Clerk session	apps/strip/cmd/app/main.go:165

flowchart TB
  subgraph Auth[Auth boundaries]
    direction TB
    User -->|Clerk JWT| BR[brisket]
    User -->|Clerk JWT| ST[strip]
    BR -->|Clerk JWT in metadata| SI[sirloin]
    ST -->|user_id in proto| SI
    SI -->|HTTP API key| BN[brain]
    SI -->|gRPC private mesh| BN
    BN -->|gRPC private mesh| SI
    SI -->|no auth| RD[round]
    FL[flank] -->|Clerk JWT| BN
    FL -->|gRPC private mesh| SI
    BN -->|no auth| RD
  end

Direction: there is no mTLS, no SPIFFE, no per-call signed identity. The backstop assumption is the private network. Anything that erodes that boundary (a public listener, a misconfigured ingress) becomes a critical finding immediately. TODO(@law): plan for mTLS or signed service tokens.

6. Versioning

Proto versions in use today:

Proto path	Version	Notes
proto/brain/v1/	v1	Character runtime execution and callbacks
proto/round/v1/	v1	Stable; embedding/face-detection RPCs
proto/sirloin/v5/	v5	Current; previous v1–v4 not retained in repo
proto/tbone/v1/	v1	Shared character recipe envelope

Compatibility rules (buf.yaml enforces):

breaking:
  use:
    - FILE

FILE-level breaking-change detection means each .proto is checked against its previous state for backwards-incompatible changes (renamed fields, deleted services, type changes). buf breaking is not wired into CI today (no buf breaking invocation in .github/workflows/). TODO(@law): add buf breaking --against '.git#branch=main' to the proto pipeline.

What is compatible (free to add):

• New RPCs on existing services.
• New fields with new tag numbers.
• New optional messages.

What requires a major bump (e.g. v5 → v6):

• Removing or renaming an RPC.
• Renumbering or repurposing a field tag.
• Changing field type, cardinality (singular ↔ repeated), or wrapper.
• Renaming an enum value visible on the wire.

flowchart LR
  P1[proto/sirloin/v5] -->|add field| P1
  P1 -->|add RPC| P1
  P1 -->|remove field| P2[proto/sirloin/v6]
  P1 -.compat.-> P1
  P2 -.parallel until v5 cleanup.-> P1

Coexistence: introducing a new major means generating both versions side-by-side until callers migrate. There is no shadowing/aliasing helper today; each service must serve both interfaces during the cutover. TODO(@law): write a migration runbook — none exists yet.

7. Codegen workflow

Source of truth: buf.yaml at repo root plus the scoped generation templates under proto/buf.gen.*.yaml, driven by make generate-proto. The root buf.gen.yaml is docs-only so plain buf generate does not write app stubs outside their intended consumers.

# Generate stubs for every service from /proto
make generate-proto

# Lint proto definitions
make lint-proto

buf.gen.yaml invokes local plugins (not BSR-hosted) to dodge rate limits. Output mapping:

Plugin	Output dir	Consumer
protoc-gen-go + protoc-gen-go-grpc	apps/round/internal/pkg/pb/	round
protoc-gen-go + protoc-gen-go-grpc	apps/sirloin/internal/pkg/pb/	sirloin
protoc-gen-go + protoc-gen-go-grpc	apps/strip/internal/pkg/pb/	strip
protoc-gen-ts_proto (outputServices=grpc-js)	apps/brain/src/generated/	brain
@bufbuild/protoc-gen-es	apps/brisket/pkg/	brisket
@bufbuild/protoc-gen-es	apps/flank/pkg/	flank

The Makefile scopes each buf generate invocation with --path, so cross-service outputs are not generated into unrelated app directories.

flowchart LR
  P[/proto/*.proto/]
  P --> BUF[buf generate]
  BUF --> GO_R[round/internal/pkg/pb]
  BUF --> GO_S[sirloin/internal/pkg/pb]
  BUF --> GO_ST[strip/internal/pkg/pb]
  BUF --> TS_B[brain/src/generated]
  BUF --> ES_BR[brisket/pkg]
  BUF --> ES_FL[flank/pkg]
  BUF --> CLEAN[Makefile cleanup]

How services consume the output:

• Go services import the package directly (import pb "github.com/.../internal/pkg/pb/sirloin/v5").
• brain (NestJS) imports from src/generated/...; gRPC clients are wired via providers (e.g. round-grpc-client.provider.ts).
• brisket constructs Connect transports against apps/brisket/pkg/sirloin/v5/... types.

CI drift guard: confirmed missing — no generate-proto invocation in .github/workflows/. TODO(@law): add a job that runs make generate-proto and fails on git diff.

8. Failure modes

Failure	Detection	Mitigation
Proto drift — generator not run after .proto edit	Compile error in dependent service; type mismatch on field rename	Run make generate-proto and commit; CI guard (TODO) to fail on diff
Schema-breaking change merged — field renumber, type swap	Old client deserialises garbage; production incidents	buf breaking in CI; bump major version (v5 → v6) and parallel-run
Unauthenticated cross-service call — caller skips API-key header	Brain returns 401; other gRPC servers on the mesh rely on network isolation	Network-level isolation; add interceptor (TODO) on every server enforcing peer identity
Missing deadline — client grpc.NewClient without WithTimeout per call	Slow/stuck upstream pins client goroutine; cascading queue depth	Mandate per-call context.WithTimeout; add deadline-required interceptor (TODO)
Retry storm — client retries on transient 5xx without backoff	Spike in grpc.code=UNAVAILABLE followed by sirloin saturation	Use bounded retry policy (none today — TODO(@law)); circuit breaker on hot RPCs
Stale H2 connection — long-lived stream goes half-closed	INTERNAL errors from brisket → sirloin after ~hours	brisket already mitigates with 30-min transport refresh (apps/brisket/server/api/sirloin-api.ts); replicate elsewhere
Insecure listener publicly exposed — gRPC port reachable from internet	Network scan; unexpected requests on :8080/:50051	Audit Railway service expose settings; never set PORT for gRPC services public-facing

9. Observability

Service	gRPC instrumentation	Evidence
sirloin (server)	OpenTelemetry stats handler + chained unary interceptors (rate-limit, auth, recovery)	apps/sirloin/cmd/app/main.go:539-542
sirloin (client → round)	otelgrpc.NewClientHandler()	apps/sirloin/cmd/app/main.go:238,264
round (server)	grpc.ChainUnaryInterceptor(recovery, logging, monitoring, validation, errorMapping) — no OTel handler	apps/round/internal/app/server/server.go:63-90
brain (client → round)	NestJS @opentelemetry/sdk-node auto-instruments gRPC	brain bootstrap
brisket (client → sirloin)	OTel via @opentelemetry/sdk-node; W3C traceparent injected by the propagateTraceContext Connect interceptor	apps/brisket/src/server/api/sirloin-api.ts

Trace propagation: traceparent and tracestate headers cross every gRPC boundary that has an interceptor. The chain currently breaks at round (no OTel imports observed). Cross-link: /operations/observability/#trace-propagation.

Logging: structured logs at every service. Correlation ID falls back to x-correlation-id if traceparent is absent.

10. Local dev

• Run a service against a local upstream: every gRPC client reads a *_GRPC_URL env var (SIRLOIN_GRPC_URL, ROUND_GRPC_URL, etc.). Point at localhost:<port> for a locally-running upstream, or use the docker-compose service name from inside the network (make dev-up-d).
• Mock a peer: round is the most commonly mocked. Use the generated Go interface (apps/sirloin/internal/pkg/pb/round/v1/round_grpc.pb.go) with mockery (config in apps/sirloin/.mockery.yml) to generate a mock, or stand up apps/round itself — model loading dominates start-up time but its gRPC surface is small.
• Authenticate Flank locally: configure CLERK_SECRET_KEY and CLERK_PUBLISHABLE_KEY; requireAuth() does not provide an environment-driven bypass.
• Inspect traffic: grpcurl -plaintext localhost:<port> list against any service that exposes reflection. Confirmed enabled on sirloin (apps/sirloin/cmd/app/main.go:572) and round (apps/round/cmd/app/main.go:80).

11. Runbook hooks

Per-service deep dives:

• services/sirloin — server config, RPC list, rate limits.
• services/brain — Sirloin/Brain HTTP and auth boundaries HTTP edge.
• services/round — TODO(@law): confirm page exists.
• services/flank — TODO(@law): confirm page exists.
• generated/proto-api — every RPC, every field.

When a gRPC incident fires, check in this order: (1) is the upstream healthy? (/services/<name>/runbook); (2) has a proto change shipped recently? (git log --since=24h -- proto/); (3) is the network path intact? (Railway private network status); (4) has the codegen pipeline drifted? (make generate-proto && git diff).