Primer Integration Playbook

Overview

Primer is the payment gateway that handles card tokenization, vaulting, 3DS/SCA, and capture for all checkout flows. Chargebee owns subscriptions and invoices; Primer owns the money movement. The split is formalised in ADR 003: Primer as Payment Gateway.

• Owner: sirloin (Go). All Primer SDK calls live in apps/sirloin/internal/pkg/primer/ and are invoked from apps/sirloin/internal/app/services/billing/.
• Production tier: critical. Outages block all paid signups, top-ups, renewals, and dunning retries. A circuit breaker (apps/sirloin/internal/pkg/primer/handler.go:45-72) trips after 5 consecutive 5xx/network failures over 30s.
• Tooling: HTTP client built directly against Primer REST API v2.4 (apps/sirloin/internal/pkg/primer/handler.go:23). No vendor SDK; we own the wire protocol.

Architecture

The payment flow is a saga guarded by a per-invoice distributed lock. Confirmation arrives over three independent paths — webhook (fast), poller (durable), and client-side SubmitPaidInvoice (UI fallback) — all converging on the same idempotent processor.

sequenceDiagram
    autonumber
    participant FE as brisket (FE)
    participant SL as sirloin
    participant CB as Chargebee
    participant PR as Primer
    participant W as sirloin worker
    participant DB as Postgres

    FE->>SL: CreatePrimerCheckout
    SL->>CB: Create customer + pending subscription/invoice (auto-collection OFF)
    SL->>PR: CreateClientSession (orderId = invoiceId)
    SL-->>FE: clientToken
    FE->>PR: tokenize + authorize (3DS)
    PR-->>FE: payment.id

    par Fast path
        PR->>SL: POST /webhooks/primer/payments (HMAC signed)
        SL->>SL: VerifyWebhookSignature + skew check
    and Durable path
        W->>PR: ListSettledPaymentsPage (every N min)
    and UI fallback
        FE->>SL: SubmitPaidInvoice(paymentId)
    end

    SL->>SL: locker.TryLock(invoice:{id})
    SL->>PR: GetPayment / GetPaymentDetails
    SL->>CB: invoice.record_payment
    SL->>CB: subscription.change_term_end (activate)
    SL->>DB: insert purchase row (saga marker)
    SL->>SL: invalidate cache + emit analytics
    SL->>SL: locker.Release

The lock is acquired via PostgreSQL session-level advisory locks (apps/sirloin/internal/pkg/locks/advisory.go:32-72) keyed per invoice. Concurrent processors return already processed instead of double-booking.

Endpoints / SDKs used

All call sites go through *primer.Handler. The handler is a thin REST client defined in apps/sirloin/internal/pkg/primer/handler.go:111-141.

Method	Defined in	Called from
CreateClientSession	pkg/primer/checkout.go:73	services/billing/createprimercheckout.go
UpdateClientSession	pkg/primer/checkout.go:134	services/billing/createprimercheckout.go
CreateVaultingClientSession	pkg/primer/checkout.go:169	services/billing/vaultingsession.go:132
CreateVaultingWithChargeSession	pkg/primer/checkout.go:303	services/billing/createprimercheckout.go
GetPayment	pkg/primer/payments.go	services/billing/submitpaidinvoice.go:533, worker/pollprimerpayments.go:294, foxy360/server.go:1182
GetPaymentDetails	pkg/primer/payments.go:454	services/billing/retrydunningpayment.go:118, worker/pollprimerfailedpayments.go:172, worker/pollprimerpayments.go:305, services/billing/successful_payment_velocity.go:92
ListSettledPaymentsPage	pkg/primer/payments.go	worker/pollprimerpayments.go:209
ListPaymentsPage (failed)	pkg/primer/payments.go	worker/pollprimerfailedpayments.go:497
FindPaymentByOrderID	pkg/primer/payments.go	worker/cleanupprimercheckouts.go:59
ListPaymentInstruments	pkg/primer/payments.go	services/billing/paymentmethods.go:48,75
Set/Delete payment instrument	pkg/primer/payments.go	services/billing/paymentmethods.go

Background workers wired in apps/sirloin/internal/app/worker/worker.go:215-260:

• TaskPollPrimerPayments — settled-payment reconciliation
• TaskPollPrimerFailedPayments — failed-payment fraud signal collection
• TaskCleanupPrimerCheckouts — voids expired pending checkouts

Webhooks received

Both inbound endpoints are mounted on the dedicated webhook mux in apps/sirloin/cmd/app/main.go:467-469.

Path	Handler	Source file
POST /webhooks/primer/payments	PrimerPaymentsWebhookHandler	services/billing/webhooks/primerpaymentwebhook.go:21
POST /webhooks/primer/disputes	PrimerDisputesWebhookHandler	services/billing/disputes/primerwebhook.go:24

Signature verification — HMAC SHA-256, confirmed in apps/sirloin/internal/pkg/primer/webhook.go:52-90 (VerifyWebhookSignature, computeWebhookHMAC, webhookSignatureMatches). Both primary and secondary signature headers are accepted (see webhook_test.go:14-32). A 3-minute replay window is enforced via WebhookSignedAtWithinSkew (pkg/primer/webhook.go:107). Body capped at 1 MiB (maxPrimerWebhookBodyBytes).

Event types observed in code:

• Payments webhook: PAYMENT.STATUS (webhooks/primerpaymentwebhook.go:23). Payload fields: eventType, signedAt, paymentId, customerId, status.
• Disputes webhook: dynamic eventType string (e.g. dispute received). Payload fields: eventType, signedAt, paymentId, disputeId, reasonCode, amount, currency, status. TODO(@vlad): exhaustive list of Primer dispute event types — code branches only on the presence of fields, not on fixed enum values.

Idempotency:

• Payments — handled inside the saga. The DB purchase row keyed by transactionID is the saga completion marker; replays short-circuit when the lock is released and the row exists (ADR 006).
• Disputes — explicit dedup via repo.DisputeFraudEventExists(disputeId, eventType, status) in disputes/primerwebhook.go:134. Duplicate webhooks return 200 OK without re-emitting Slack alerts.

Failure modes

Failure	Detection	Mitigation
Webhook signature mismatch	VerifyWebhookSignature returns false → 401. Logged with webhook.provider=primer.	Verify SIRLOIN_PRIMER_WEBHOOK_SECRET matches dashboard. Check clock skew (3-minute window). Sirloin runbook: Primer webhook backlog.
Empty webhook secret	Handler returns 500, posts Slack BillingAlert (“Primer Webhook Misconfiguration”) — fail-loud + fail-closed. disputes/primerwebhook.go:65-67, webhooks/primerpaymentwebhook.go:91-95.	Set the secret in Railway env; redeploy.
Payment confirmation timeout	Webhook never arrives. TaskPollPrimerPayments reconciles within poll interval. Watch for subscription.status='future' past checkout_expiration (Payment saga stuck).	Manual SubmitPaidInvoice per affected invoice; or wait for the next poller tick.
Primer API down / 5xx	Circuit breaker opens after 5 consecutive failures (pkg/primer/handler.go:29-32,54-60). Handler.State() exposes state.	Breaker auto-recovers in 15s (primerBreakerTimeout). No code-level checkout-pause feature flag exists today; sustained outages require manual mitigation. TODO(@vlad): wire a Posthog/config flag to disable checkout.
Idempotency replay (double webhook)	Dispute path: DisputeFraudEventExists dedup. Payment path: per-invoice advisory lock + DB purchase row check inside payments/processor.go:78,273.	No action — designed for safe replay.
Partial saga rollback	Saga does not roll back by design (ADR 006: voiding Chargebee mutations loses money). If DB step fails after Chargebee activation, the saga marker is missing and retries succeed when DB recovers.	Persistent DB failure → manual intervention via admin tool. Per-user lock + idempotent replay through payments/processor.go.
API indexing delay	Primer payment search lags ~30s after authorization (comment in submitpaidinvoice.go:276).	SubmitPaidInvoice retries with extended attempts when in search mode.

Secrets & config

Variables consumed by sirloin (apps/sirloin/internal/app/config/config.go:69,284-286):

Variable	Purpose	Source
SIRLOIN_PRIMER_API_KEY	Primer REST API auth. Prefix selects environment: prs_sbx_ → sandbox, prs_ → production (pkg/primer/handler.go:48-50).	.env.example
SIRLOIN_PRIMER_WEBHOOK_SECRET	HMAC SHA-256 key for both payment and dispute webhooks. Empty value → fail-closed + Slack alert.	.env.example
BRISKET_PRIMER_API_KEY	Frontend client-side checkout (publishable).	.env.example
SIRLOIN_FOXY360_PRIMER_API_KEY	Foxy360 admin tooling Primer key — read-only access for the foxy360_source_primer_query MCP tool and enrich-customer-data script (apps/sirloin/internal/app/foxy360/server.go:1157, apps/sirloin/cmd/scripts/enrich-customer-data/main.go:248). TODO(@vlad): rotation runbook.	.env.example

Storage: TODO(@vlad) — Railway env is the deployment target per security-model.md, but the canonical secrets backend (Railway / Doppler / Vault) is not pinned in code.

Rotation procedure: TODO(@vlad). Documented procedure missing. Suggested steps until owner confirms: (1) generate new key in Primer dashboard, (2) add as SIRLOIN_PRIMER_API_KEY_NEW alongside old, (3) deploy with both loaded, (4) flip primary, (5) revoke old in dashboard. The webhook secret rotation must coordinate with Primer dashboard simultaneously to avoid the fail-closed window.

Sandbox vs prod

Switching is automatic from the API key prefix (apps/sirloin/internal/pkg/primer/handler.go:44-50):

• Sandbox: keys prefixed prs_sbx_ → https://api.sandbox.primer.io
• Production: keys prefixed prs_ → https://api.primer.io

No separate base-URL flag exists. Sandbox-only integration tests live behind the sandbox build tag (apps/sirloin/internal/app/services/billing/primersandbox_test.go:1) and run via go test -tags sandbox ./....

Test card numbers: see Primer test cards documentation. No pinned test card list lives in the repo today (grep returned no card numbers under apps/sirloin/internal/app/services/billing/); sandbox tests rely on the vendor docs.

Cost model

Primer’s per-transaction pricing (gateway fee + interchange) is not tracked in code or comments. TODO(@vlad): document tier, monthly minimum, per-tx fees, and 3DS surcharge. Owner: finance.

Dashboards

TODO(fill in): pin the canonical dashboards once authored.

• Primer dashboard (vendor): payment volume, decline rate, dispute queue.
• Axiom — sirloin: webhook 4xx/5xx on /webhooks/primer/*, signature failure rate, circuit breaker state.
• Axiom — workers: TaskPollPrimerPayments lag, TaskPollPrimerFailedPayments fraud-event insertion rate, TaskCleanupPrimerCheckouts void count.
• PostHog: checkout funnel — CreatePrimerCheckout → client-side authorize → PAYMENT.STATUS webhook → activation.

See observability for stack inventory and metric naming conventions.

Runbook hooks

The sirloin runbook contains the primary incident playbooks for Primer:

• Primer webhook backlog — 401s on /webhooks/primer/payments, secret/clock skew triage, manual recovery via SubmitPaidInvoice.
• Payment saga stuck — subscription stuck in future after payment recorded; per-user lock + idempotent replay through payments/processor.go.
• Common Incidents index — full incident catalogue.

Related decisions:

• ADR 003: Primer as Payment Gateway
• ADR 005: Distributed Locks for Payments
• ADR 006: Saga Pattern for Distributed Payments
• Billing flow