Primer Integration Playbook

Overview

Primer is the payment gateway that handles card tokenization, vaulting, 3DS/SCA, and capture for all checkout flows. Chargebee owns subscriptions and invoices; Primer owns the money movement. The split is formalised in ADR 003: Primer as Payment Gateway.

Owner: sirloin (Go). All Primer SDK calls live in apps/sirloin/internal/pkg/primer/ and are invoked from apps/sirloin/internal/app/services/billing/.
Production tier: critical. Outages block all paid signups, top-ups, renewals, and dunning retries. A circuit breaker (apps/sirloin/internal/pkg/primer/handler.go:67-94) trips after 5 consecutive 5xx/network failures over 30s. Requests that end because the caller canceled or its deadline passed do not count; the HTTP client timeout does.
Tooling: HTTP client built directly against Primer REST API v2.4 (apps/sirloin/internal/pkg/primer/handler.go:23). No vendor SDK; we own the wire protocol.

Architecture

The payment flow is a saga guarded by a per-invoice distributed lock. Confirmation arrives over three independent paths — webhook (fast), poller (durable), and client-side SubmitPaidInvoice (UI fallback) — all converging on the same idempotent processor.

sequenceDiagram
    autonumber
    participant FE as brisket (FE)
    participant SL as sirloin
    participant CB as Chargebee
    participant PR as Primer
    participant W as sirloin worker
    participant DB as Postgres

    FE->>SL: CreatePrimerCheckout
    SL->>CB: Create customer + pending subscription/invoice (auto-collection OFF)
    SL->>PR: CreateClientSession (orderId = invoiceId)
    SL-->>FE: clientToken
    FE->>PR: tokenize + authorize (3DS)
    PR-->>FE: payment.id

    par Fast path
        PR->>SL: POST /webhooks/primer/payments (HMAC signed)
        SL->>SL: VerifyWebhookSignature + skew check
    and Durable path
        W->>PR: ListSettledPaymentsPage (every N min)
    and UI fallback
        FE->>SL: SubmitPaidInvoice(paymentId)
    end

    SL->>SL: locker.TryLock(invoice:{id})
    SL->>PR: GetPayment / GetPaymentDetails
    SL->>CB: invoice.record_payment
    SL->>CB: subscription.change_term_end (activate)
    SL->>DB: insert purchase row (saga marker)
    SL->>SL: invalidate cache + emit analytics
    SL->>SL: locker.Release

The lock is acquired via PostgreSQL session-level advisory locks (apps/sirloin/internal/pkg/locks/advisory.go:32-72) keyed per invoice. Concurrent processors return already processed instead of double-booking.

Endpoints / SDKs used

All call sites go through *primer.Handler. The handler is a thin REST client defined in apps/sirloin/internal/pkg/primer/handler.go:122-152.

Method	Defined in	Called from
`CreateClientSession`	`pkg/primer/checkout.go:73`	`services/billing/createprimercheckout.go`
`UpdateClientSession`	`pkg/primer/checkout.go:134`	`services/billing/createprimercheckout.go`
`CreateVaultingClientSession`	`pkg/primer/checkout.go:169`	`services/billing/vaultingsession.go:132`
`CreateVaultingWithChargeSession`	`pkg/primer/checkout.go:303`	`services/billing/createprimercheckout.go`
`GetPayment`	`pkg/primer/payments.go`	`services/billing/submitpaidinvoice.go:533`, `worker/pollprimerpayments.go:294`, `foxy360/server.go:1182`
`GetPaymentDetails`	`pkg/primer/payments.go:454`	`services/billing/retrydunningpayment.go:118`, `worker/pollprimerfailedpayments.go:172`, `worker/pollprimerpayments.go:305`, `services/billing/successful_payment_velocity.go:92`
`ListSettledPaymentsPage`	`pkg/primer/payments.go`	`worker/pollprimerpayments.go:209`
`ListPayments` (`GET /payments`, cursor-paged)	`pkg/primer/listpayments.go:68`	`worker/syncprimerpayments.go` (`syncPrimerPaymentsWindow`)
`ListPaymentsPage` (failed)	`pkg/primer/payments.go`	`worker/pollprimerfailedpayments.go:497`
`FindPaymentByOrderID`	`pkg/primer/payments.go`	`worker/cleanupprimercheckouts.go:59`
`ListPaymentInstruments`	`pkg/primer/payments.go`	`services/billing/paymentmethods.go:48,75`
Set/Delete payment instrument	`pkg/primer/payments.go`	`services/billing/paymentmethods.go`

Background workers registered as River periodic jobs in RegisterRiver (apps/sirloin/internal/app/worker/river.go, params.Primer block):

TaskPollPrimerPayments — settled-payment reconciliation
TaskPollPrimerFailedPayments — failed-payment fraud signal collection
TaskCleanupPrimerCheckouts — voids expired pending checkouts
TaskRetryRenewalPayments — dunning-window renewal retries
TaskPollCreditNotes — refund credit-note polling
TaskRetryFailedOperations — transient billing-operation retries

TaskSyncPrimerPayments — the billing.primer_payments mirror behind the Strip payments page, roughly 1,500 ListPayments calls a day — is registered from its own params.PaymentsMirror block, not the params.Primer one. It needs SIRLOIN_PAYMENTS_MIRROR_ENABLED unset or not FALSE and a stage outside development and sandbox. That stage comes from SIRLOIN_STAGE first (apps/sirloin/internal/app/config/config.go:178 resolves it through env.StageValue, apps/sirloin/internal/pkg/env/stage.go:36-44); RAILWAY_ENVIRONMENT_NAME is consulted only when SIRLOIN_STAGE is blank, and only then does ParseStage read a name containing -pr- as sandbox (apps/sirloin/internal/pkg/env/stage.go:47-51). So a beef-pr-N name does not by itself keep a preview off this endpoint: a preview that inherits SIRLOIN_STAGE=staging from the environment it was cloned from resolves to staging and does issue these calls, with the flag as the only remaining stop (apps/sirloin/internal/pkg/env/stage_test.go:45-50 pins that combination). Read SIRLOIN_STAGE in the Railway environment, not its name, before assuming a preview is quiet. Staging and production both issue them.

Webhooks received

Both inbound endpoints are mounted on the dedicated webhook mux (webhookMux in apps/sirloin/cmd/app/main.go).

Path	Handler	Source file
`POST /webhooks/primer/payments`	`PrimerPaymentsWebhookHandler`	`services/billing/webhooks/primerpaymentwebhook.go:21`
`POST /webhooks/primer/disputes`	`PrimerDisputesWebhookHandler`	`services/billing/disputes/primerwebhook.go:24`

Signature verification — HMAC SHA-256, confirmed in apps/sirloin/internal/pkg/primer/webhook.go:52-90 (VerifyWebhookSignature, computeWebhookHMAC, webhookSignatureMatches). Both primary and secondary signature headers are accepted (see webhook_test.go:14-32). A 3-minute replay window is enforced via WebhookSignedAtWithinSkew (pkg/primer/webhook.go:107). Body capped at 1 MiB (maxPrimerWebhookBodyBytes).

Event types observed in code:

Payments webhook: PAYMENT.STATUS (webhooks/primerpaymentwebhook.go:23). Payload fields: eventType, signedAt, paymentId, customerId, status.
Disputes webhook: dynamic eventType string (e.g. dispute received). Payload fields: eventType, signedAt, paymentId, disputeId, reasonCode, amount, currency, status. TODO(@vlad): exhaustive list of Primer dispute event types — code branches only on the presence of fields, not on fixed enum values.

Idempotency:

Payments — handled inside the saga. The DB purchase row keyed by transactionID is the saga completion marker; replays short-circuit when the lock is released and the row exists (ADR 006).
Disputes — explicit dedup via repo.DisputeFraudEventExists(disputeId, eventType, status) in disputes/primerwebhook.go:134. Duplicate webhooks return 200 OK without re-emitting Slack alerts.

Failure modes

Failure	Detection	Mitigation
Webhook signature mismatch	`VerifyWebhookSignature` returns false → `401`. Logged with `webhook.provider=primer`.	Verify `SIRLOIN_PRIMER_WEBHOOK_SECRET` matches dashboard. Check clock skew (3-minute window). Sirloin runbook: Primer webhook backlog.
Empty webhook secret	Handler returns `500`, posts Slack `BillingAlert` (“Primer Webhook Misconfiguration”) — fail-loud + fail-closed. `disputes/primerwebhook.go:65-67`, `webhooks/primerpaymentwebhook.go:91-95`.	Set the secret in Railway env; redeploy.
Payment confirmation timeout	Webhook never arrives. `TaskPollPrimerPayments` reconciles within poll interval. Watch for `subscription.status='future'` past `checkout_expiration` (Payment saga stuck).	Manual `SubmitPaidInvoice` per affected invoice; or wait for the next poller tick.
Primer API down / 5xx	Circuit breaker opens after 5 consecutive failures (`pkg/primer/handler.go:30-34,67-94`). `Handler.State()` exposes state.	Breaker auto-recovers in 15s (`primerBreakerTimeout`). No code-level checkout-pause feature flag exists today; sustained outages require manual mitigation. TODO(@vlad): wire a Posthog/config flag to disable checkout.
Idempotency replay (double webhook)	Dispute path: `DisputeFraudEventExists` dedup. Payment path: per-invoice advisory lock + DB purchase row check inside `payments/processor.go:78,273`.	No action — designed for safe replay.
Partial saga rollback	Saga does not roll back by design (ADR 006: voiding Chargebee mutations loses money). If DB step fails after Chargebee activation, the saga marker is missing and retries succeed when DB recovers.	Persistent DB failure → manual intervention via admin tool. Per-user lock + idempotent replay through `payments/processor.go`.
API indexing delay	Primer payment search lags ~30s after authorization (comment in `submitpaidinvoice.go:276`).	`SubmitPaidInvoice` retries with extended attempts when in search mode.

Secrets & config

Variables consumed by sirloin (apps/sirloin/internal/app/config/config.go:69,284-286):

Variable	Purpose	Source
`SIRLOIN_PRIMER_API_KEY`	Primer REST API auth. Prefix selects environment: `prs_sbx_` → sandbox, `prs_` → production (`pkg/primer/handler.go:48-50`).	`.env.example`
`SIRLOIN_PRIMER_WEBHOOK_SECRET`	HMAC SHA-256 key for both payment and dispute webhooks. Empty value → fail-closed + Slack alert.	`.env.example`
`BRISKET_PRIMER_API_KEY`	Frontend client-side checkout (publishable).	`.env.example`
`SIRLOIN_FOXY360_PRIMER_API_KEY`	Foxy360 admin tooling Primer key — read-only access for the `foxy360_source_primer_query` MCP tool and `enrich-customer-data` script (`apps/sirloin/internal/app/foxy360/server.go:1157`, `apps/sirloin/cmd/scripts/enrich-customer-data/main.go:248`). TODO(@vlad): rotation runbook.	`.env.example`

Storage: TODO(@vlad) — Railway env is the deployment target per security-model.md, but the canonical secrets backend (Railway / Doppler / Vault) is not pinned in code.

Rotation procedure: TODO(@vlad). Documented procedure missing. Suggested steps until owner confirms: (1) generate new key in Primer dashboard, (2) add as SIRLOIN_PRIMER_API_KEY_NEW alongside old, (3) deploy with both loaded, (4) flip primary, (5) revoke old in dashboard. The webhook secret rotation must coordinate with Primer dashboard simultaneously to avoid the fail-closed window.

Sandbox vs prod

Switching is automatic from the API key prefix (apps/sirloin/internal/pkg/primer/handler.go:44-50):

Sandbox: keys prefixed prs_sbx_ → https://api.sandbox.primer.io
Production: keys prefixed prs_ → https://api.primer.io

No separate base-URL flag exists. Sandbox-only integration tests live behind the sandbox build tag (apps/sirloin/internal/app/services/billing/primersandbox_test.go:1) and run via go test -tags sandbox ./....

Test card numbers: see Primer test cards documentation. No pinned test card list lives in the repo today (grep returned no card numbers under apps/sirloin/internal/app/services/billing/); sandbox tests rely on the vendor docs.

Cost model

Primer’s per-transaction pricing (gateway fee + interchange) is not tracked in code or comments. TODO(@vlad): document tier, monthly minimum, per-tx fees, and 3DS surcharge. Owner: finance.

Dashboards

TODO(fill in): pin the canonical dashboards once authored.

Primer dashboard (vendor): payment volume, decline rate, dispute queue.
Axiom — sirloin: webhook 4xx/5xx on /webhooks/primer/*, signature failure rate, circuit breaker state.
Axiom — workers: TaskPollPrimerPayments lag, TaskPollPrimerFailedPayments fraud-event insertion rate, TaskCleanupPrimerCheckouts void count.
PostHog: checkout funnel — CreatePrimerCheckout → client-side authorize → PAYMENT.STATUS webhook → activation.

See observability for stack inventory and metric naming conventions.

Runbook hooks

The sirloin runbook contains the primary incident playbooks for Primer:

Primer webhook backlog — 401s on /webhooks/primer/payments, secret/clock skew triage, manual recovery via SubmitPaidInvoice.
Payment saga stuck — subscription stuck in future after payment recorded; per-user lock + idempotent replay through payments/processor.go.
Common Incidents index — full incident catalogue.

Related decisions: