Billing

Billing

Purpose

Document subscription, payment, credit-application, dispute, dunning, and failed-operation recovery behavior.

Participants

• Sirloin owns all billing state and APIs.
• Chargebee owns subscription and invoice state.
• Primer processes payments and disputes.
• Postgres stores billing, fraud, failed-operation, and subscription-sync records.
• PostHog and Klaviyo receive analytics and lifecycle events.

Sequence

Checkout (CIT)

sequenceDiagram
  participant Brisket
  participant Sirloin
  participant Primer
  participant Chargebee
  participant DB as Postgres
  Brisket->>Sirloin: ReservePaymentAttempt (count quota check)
  Sirloin->>DB: Advisory lock + count + insert authorization_attempt
  Sirloin-->>Brisket: Allowed / blocked + remaining attempts
  Brisket->>Sirloin: Create checkout
  Sirloin->>Sirloin: Enforce velocity (amount + count windows)
  Sirloin->>Primer: Create payment session
  Sirloin-->>Brisket: Client token/order
  Brisket->>Primer: User pays
  Brisket->>Sirloin: Submit payment id
  Primer-->>Sirloin: Payment webhook fast path
  Sirloin->>Chargebee: Record payment and activate
  Chargebee-->>Sirloin: Invoice webhook fast path
  Sirloin->>Chargebee: Poll invoice/subscription state fallback
  Sirloin->>DB: Apply credits if idempotency checks pass
  Sirloin->>DB: Record payment_successful velocity event
  Primer-->>Sirloin: Dispute webhook when dispute changes

Every client session (subscription checkout, top-up and card vault) carries DefaultOrderedAllowedCardNetworks from apps/sirloin/internal/pkg/primer/payments.go: Visa, Mastercard, Amex and Diners. Discover is left out on purpose because Primer blocks it in every lane and the re-enable (FOXY-1262) was cancelled (TRUCOM-91); the Primer SDK in Brisket derives the hosted card field, the wallet sheets’ networks and the checkout saved-card filter from this list. Renewals and one-click top-ups charge saved methods without a client session and are not affected.

Renewal Dunning (MIT)

sequenceDiagram
  participant Worker
  participant Sirloin
  participant Chargebee
  participant Primer
  participant DB as Postgres
  Worker->>DB: Recover unfinished renewal operations (up to 60s)
  Worker->>Chargebee: List unpaid renewal invoices
  Worker->>DB: Acquire invoice lock
  Worker->>DB: Open renewal operation? Finish it, no new charge
  Worker->>DB: Check DunningAttemptExists
  Worker->>Sirloin: Resolve PSP routing (fail-closed)
  Sirloin->>Chargebee: Retrieve subscription
  Sirloin->>DB: Enforce velocity (amount + count windows)
  Sirloin->>DB: MIT count-quota gate (routed processor limits)
  Sirloin->>DB: Persist renewal operation (prepared, then submitting)
  Sirloin->>DB: Record authorization_attempt
  Sirloin->>Primer: Create renewal payment (X-Idempotency-Key = operation ID)
  Primer-->>Sirloin: Payment result, or no answer (operation stays unknown)
  Sirloin->>Chargebee: Record payment on invoice
  Sirloin->>DB: Record dunning_attempt once per operation, finalize operation

Every renewal charge and dunning cancellation is written to billing.renewal_payment_operations before its provider is called (FOXY-1045). Work that never left Sirloin closes as unsent and spends no slot, so a failed payment-method read or a deadline before the POST no longer burns a retry. A charge whose context ends before it has a connection counts as unsent too. A charge whose response was lost (timeout, 5xx, or a 409 for its own idempotency key) stays unknown: it blocks every new charge and the cancellation for that invoice, from every source, until recovery finds that exact payment by the renewal_operation_id in its Primer metadata, or billing on-call resolves it. A worker tick looks an unknown charge up only once its next_check_at is due. A scheduled charge still unknown with no Primer payment after three complete searches in a row is sent once more by a scheduled run, at least 15 minutes and at most a day after its original submission, as the same request under the same X-Idempotency-Key. Primer answers a key it already processed with a 409 IdempotencyKeyAlreadyExists and processes the request otherwise, so the resubmission either makes the first charge or learns that the first request landed. Only a 2xx answer settles it; any other answer leaves the charge unknown. It is sent at most once, never for a charge a user-driven retry started, and not at all if the invoice amount or the request no longer matches the original. The run that first learns a final outcome sends its events: the NSFW velocity event, the Klaviyo recovery or failure email, PostHog, and on a pickup decline the fraud flag. Recovery saves the outcome before sending them, so they go out at most once. Recovery reads the operation table itself, so an invoice that was paid or voided in the meantime cannot strand its work. A cancellation is persisted before the void, tracks void and cancel separately, reads Chargebee back before repeating either. A payment that turns up before the cancel closes it as superseded; one found after Chargebee already shows the subscription canceled closes it as canceled_after_payment, for billing on-call to reconcile.

The job runs inside a ten-minute budget measured from River worker entry to the last lock release. Ordinary work, recovery included, stops at 9m45s or when River cancels the job; finalizing in-flight work, the invoice unlock, and the River unlock each get at most five seconds after it. Work not reached is deferred to the next run and summarized in one renewal payment retry task completed log line. The renewal paths use their own Chargebee client: SDK retries off, each request bound to its context, 429/502/503/504 retried only while the deadline allows. A scheduled slot whose attempt cannot hide a payment, one an operation wrote or a failure with no possible payment behind it, is skipped before any provider read; any other recorded slot still gets the existing-payment check. The Primer invoice search fetches only payments whose summary names the invoice.

The unpaid list merges the local registry batch (up to 100 rows) with a newest-first walk over the Chargebee pages of the dunning window, which stops at the last page, on a page it already read, or after 20,000 invoices. The Chargebee walk hands the worker at most 50 invoices per run; the rest waits for the next run 15 minutes later. Each run first reads the invoices in their last slot, 14 days to 14 days and 1 hour old, oldest first, and keeps them ahead of the newest-first walk under the same cap, because that slot is open for one hour and a newest-first walk that fills the cap never reaches it. Until PAY-114 there was no such read, and on busy days in October 2026 about 370 invoices were canceled without their last attempt. Before reading a subscription, both listings drop invoices the scheduled guards would skip anyway: a scheduled hard decline, two scheduled APPLICATION_ERROR failures, or a current slot that already declined in Primer, was quota-blocked, or found no payment method. With SIRLOIN_DUNNING_SOFT_DECLINE_RETRY_ENABLED off, invoices of customers at the 3-decline cap are not dropped and take cap slots every run; the walk’s capped log field shows it. An invoice with a successful attempt stays in, and so does one whose renewal payment the payments mirror (billing.primer_payments, synced every 15 minutes) shows as pending, authorized or settled, so the worker’s existing-payment check can still record a charge that never reached Chargebee, including a manual or card-update charge that wrote no attempt. Where the mirror does not run, as on previews, that check waits for the invoice’s next slot. Once the window runs out, the cancel step runs the same Chargebee and Primer check under the invoice lock before it voids the invoice, and never charges: a settled payment is recorded with its credits instead of canceling, and an in-flight payment, a failed Primer read or a lock held elsewhere postpones the cancel to the next run. Until FOXY-1381 the walk read one page of 100, so from August 2026 invoices older than about a day dropped out before their next slot and were never retried.

NSFW Payment Method Enforcement

EMP (NSFW payment processor) only supports card payments — no Apple Pay or Google Pay. NMI supports cards and wallets but only accepts payment methods vaulted through its own gateway — a card vaulted via Stripe won’t work for NMI renewals. Three enforcement points prevent hard failures:

1. Frontend: Hide Vaulted Methods on SFW→NSFW Switch

When a user upgrades from SFW to NSFW tier, showVaultedMethods={false} forces fresh card entry. This does not apply to NSFW→NSFW plan changes (user already has a compatible method on file).

• brisket/.../single-plan-upgrade/index.tsx
• brisket/.../upgrade-selection/upgrade-button.tsx

2. Backend: Auto-Set Primary After NSFW Checkout

After a successful NSFW checkout payment, maybeSetNSFWCheckoutPrimary fetches the payment details from Primer, extracts the vaulted token, and sets it as the customer’s default payment instrument. This ensures the next renewal uses the EMP-compatible card.

• Fires only on subscription purchase (not renewals)
• Guards on access: "full" metadata (NSFW only)
• Non-fatal: all errors logged and swallowed

3. Backend: Card Preference for Scheduled NSFW Renewals

During scheduled dunning retries, resolveNSFWRenewalPaymentToken checks if the primary payment method is a wallet (APAY/GPAY). If so, it searches saved instruments for a card via SelectCardForNSFWRenewal and substitutes it. Falls back to the primary if no card is found (EMP rejects, enters dunning).

• Only activates for: scheduled retry (no client IP) + NSFW plan + wallet primary
• Manual retries skip substitution (user explicitly chose the method)
• instrumentLister interface enables testing without full Primer handler

CVV Re-entry on Vaulted Cards

A customer whose session routes to eMerchantPay is asked for the CVV again when paying with a stored card; nobody else is. The first release asked internal accounts only; the follow-up opened it to every eMerchantPay-routed customer once a production staff payment with forced 3DS settled on eMerchantPay at the first attempt. eMerchantPay’s 3D terminals carry Requires CVV, so a stored-card payment that goes through 3DS (the EEA issuer-country branch of the Primer workflow) is refused with code 320 when no CVV is sent and settles only on the Cybersource fallback. FOXY-1294 removed the prompt for everyone; FOXY-1319 restored it for eMerchantPay-routed customers after the data showed no conversion gain from removing it (evidence on FOXY-1319).

• requireVaultedCardCvv(sessionMetadata) in services/billing is the single rule. It reads the psp that setProcessorPreference wrote into the client-session metadata, so it always agrees with the route and cannot be handed a tier label. Subscription checkout and top-up call it on their session metadata after the processor preference is set; the vaulting session passes the decision to pkg/primer as a parameter, because the rule lives in services/billing
• Brisket reads cvvRecapture from the Primer SDK vault callback; with it true the CVV field renders on the selected saved card and vault.startPayment carries the CVV token, with it false the field is not rendered — no frontend gate exists. The field is created again whenever its container mounts (the card leaving a retry cooldown, a re-selection), a card whose network the vault does not report still gets one, and a field resolved for a card that is no longer selected is destroyed rather than used
• NMI and Cybersource never required a CVV on stored cards and stay without the prompt; the SFW lane never routes to eMerchantPay, so it never sets it
• An unassigned customer routes psp-absent and is not asked at checkout or top-up, unless a top-up repair locks them to eMerchantPay. The vaulting session resolves an unlocked full-access user to the eMerchantPay default and may set the option; it is inert there because Brisket renders that session with saved cards hidden and the $1 authorization is canceled
• Renewals are merchant-initiated, land on the CVV-off terminals and never carried a CVV
• Saved-card direct top-up on full-access plans (a token with no checkout UI) is refused independently of the CVV rule; nothing in Brisket calls it
• A per-card prompt resolved from Primer payment history was built as PR #3566 and dropped: it cost Primer calls on every saved-card list and still failed open for cards without history
• Accepted gaps, both visible in the locked-against-settled processor numbers in billing.primer_payments and sized on FOXY-1319: an unlocked full-access customer is not asked although the workflow may still send the card to eMerchantPay; a Cybersource lock on the Coastal merchant account, which an Apple Pay settle creates, is not asked although the locked workflow branches that account’s cards to eMerchantPay or NMI (FOXY-1179); an SSB or Esquire lock (FOXY-1364) routes its cards to Cybersource

Per-User Processor Assignment

Each user is locked to a specific payment processor (PSP) per tier, and on the NSFW lane to the merchant account (MID) that settled the first payment. Stored on users.credits as sfw_processor, nsfw_processor, and nsfw_processor_mid. Once locked, never auto-changed (first write wins).

How it works — both lanes are settle-then-lock (NSFW: FOXY-596, SFW: FOXY-877)

Neither lane pre-assigns a processor. The first payment is routed by a Primer workflow — the high-risk one for NSFW, Regular Access Traffic for SFW — and whichever processor actually settles the money is locked and replayed on all later payments. The merchant account is locked with it, on the NSFW lane only, for every processor on that lane (ADR). A first lock whose processor arrives without a MID (a renewal create response) fetches the payment to read it.

• Checkout (CIT): ResolveProcessor(ctx, userID, isNSFW) reads the lane’s stored processor and assigns nothing — processorRepo carries no writer, so the type system forbids it. No lock on either lane resolves to an empty psp: the checkout carries no processor and the lane’s Primer workflow routes it (card issuer, wallets, percentage split across processors and the NMI MIDs on the NSFW lane; Cybersource primary with a Stripe fallback on SFW). A temporary conservative price ceiling (min(NMI, EMP)) applies while an NSFW processor is unknown; an unassigned SFW checkout has none, because neither Cybersource nor Stripe caps a single transaction.
• On settle: LockProcessorFromPayment resolves the lane from the payment’s access label, reads the settled processor from Primer, and locks it first-wins — AssignNSFWProcessorWithMid with the MID on NSFW, AssignSFWProcessor on SFW. Each lane resolves processor names through its own table, so a name belonging to the other lane resolves to nothing and cannot be stored. Only subscription-linked settles lock — the payment must carry chargebee_subscription_id (a subscription CIT that vaults the token, or a renewal MIT that charges it), the only token-authoritative signal. Runs best-effort where those settles are observed (mainline/poller processFoundPayment, renewal). Failed payments, top-ups, and vault-only settles never lock.
• Renewal (MIT): both lanes replay the locked psp (and the MID on NSFW) and route processor-absent when there is none, so the workflow discovers and the settle records it. A credits-read error aborts the attempt on both lanes: routing processor-absent for a locked user would send them through discovery and can settle them off their lock, which first-write-wins then makes permanent. The next dunning tick re-reads.
• Vaulting / reserve: read-only — they never create a lock.
• Repair: RepairProcessorLockFromHistory recovers a lock from the user’s settled Primer history (ListPaymentsByCustomer) rather than guessing a default. Wired on the unlocked NSFW top-up and on both lanes’ renewals. Not on the SFW top-up: that lane’s tokens are not processor-homed, so the scan’s cost on a user-facing checkout buys nothing.
• Primer routing: sessions include metadata["psp"] (and, on the NSFW lane, metadata["psp_mid"] when the lock has one) so the workflow honors the locked route: Full Access NMI routes by MID, and once the FOXY-1426 Primer branches ship so does Full Access Cybersource; with the FOXY-1427 branch an EMP MID marks a lock written after FOXY-1364; an absent psp signals the discovery/recovery branches. Adding a processor to a lane’s table without the matching Primer branch routes it silently to Stripe, so the order is Discovery workflow, then locked branch, then table entry.
• Observability: billing_processor_lock_outcome_total carries outcome, lane and psp, one record per gated settle — read it per lane, since SFW is the larger one and would mask an NSFW failure in aggregate. History repair reports separately on billing_processor_repair_outcome_total. See Billing SLOs.
• Design record: Settle-Then-Lock SFW Processor Assignment explains why the lane tables are split, why neither lane assigns before a settle, and why a credits-read error aborts the attempt on both lanes.

Processors

Tier	Processor	Constant
SFW	Cybersource (primary), Stripe (fallback) — either can be the value that settles, so either can be the one locked	domain.PSPCybersource, domain.PSPStripe
NSFW	EMP, NMI (SSB / Esquire MIDs), Cybersource (Coastal, SSB and Esquire merchant accounts; pre-FOXY-1364 locks have no MID, later locks store the settled account)	domain.PSPEMP, domain.PSPNMI, domain.PSPCybersource

The lock is hard: vaulted-card renewals are processor-homed (an EMP-vaulted token cannot renew on NMI and vice versa), so a locked pair is never auto-changed. The psp_mid (the locked merchant account) is replayed on the NSFW lane so renewals stay on the merchant account that took the first payment; locks written before FOXY-1364 have none outside NMI and keep the workflows’ default branches. Operator overrides (StripUpdateUserProcessors) write only the lane whose value changed and treat an empty lane as “leave unchanged”. On the NSFW lane Strip sends a configured processor and MID pair: the MID can be set or cleared on its own, and a processor change without one clears the old MID.

Source Links

• apps/sirloin/internal/app/services/billing/createprimercheckout.go
• apps/sirloin/internal/app/services/billing/submitpaidinvoice.go
• apps/sirloin/internal/app/services/billing/events/poller.go
• apps/sirloin/internal/app/services/billing/events/credits.go
• apps/sirloin/internal/app/services/billing/disputes/primerwebhook.go
• apps/sirloin/internal/app/services/billing/webhooks/
• apps/sirloin/internal/app/services/billing/dunningretry.go
• apps/sirloin/internal/app/services/billing/processor_assignment.go
• apps/sirloin/internal/app/services/billing/reserve_payment_attempt.go
• apps/sirloin/internal/app/services/billing/successful_payment_velocity.go

State Transitions

Checkout creates a Primer session. Payment completion is resolved through Brisket’s submit call, Primer’s payment webhook fast path, and Primer polling fallback. Chargebee invoice/subscription state is accelerated by Chargebee invoice webhooks but still reconciled through polling. Credits are applied only after payment detection and idempotency checks. Failed operations are persisted for retry rather than silently dropped.

Invariants

• Sirloin is the billing owner.
• Primer and Chargebee webhooks are fast paths.
• Primer and Chargebee polling remain durable replay/fallback mechanisms.
• Primer disputes are webhook-driven and recorded as fraud audit events.
• Credit application must be idempotent.
• Failed operations must remain retryable and auditable.

Error Paths

Dunning retries can succeed, remain processing, no-op when already paid or not in dunning, fail retryably, fail on hard decline, or be blocked by fraud velocity rules. A Primer gateway reject (status-reason type GATEWAY_REJECTED, usually sent with an empty decline type and code) is non-retryable on every attempt source, like a hard decline (primer.IsNonRetryableDecline); with soft-decline retries on, a scheduled gateway reject ends the invoice’s remaining scheduled slots and sends payment_failed_update_required with failure kind hard_decline on that attempt, while manual and card-update retries stay allowed and send nothing. The 14-day access and cancellation timer is unchanged (TRUCOM-91). Fraud velocity checks (count quotas, amount windows, cooldowns) apply only to EMP-routed NSFW users; NMI-routed NSFW users skip these checks. CIT checkout can be blocked by ReservePaymentAttempt count-quota exhaustion (daily/weekly/monthly windows) — frontend shows differentiated toast messages. A subscription or upgrade checkout pauses with FailedPrecondition while the customer’s previous checkout payment is still pending or authorized in Primer; Brisket asks the customer to wait or pay again, and paying again resends the request with allow_payment_in_progress. MIT renewal can be blocked by velocity amount windows or count-quota gates; blocked attempts are recorded as quota_blocked dunning attempts to prevent retry spam. PSP routing fails closed if subscription data is missing.

Tests And Verification

• cd apps/sirloin && make run-tests
• cd apps/sirloin && make run-tests-all
• cd apps/sirloin && make lint

Related Decisions And Standards

• Billing Pricing
• Media Credit Costs
• Deferred Subscription Activation
• Chargebee Polling Over Webhooks
• Primer Payment Gateway
• Auto-Collection Off
• Distributed Locks For Payments
• Saga Pattern For Distributed Payments
• Authorization Attempt Quota