Billing Operational Runbook

Billing Operational Runbook

This document provides step-by-step procedures for common operational incidents involving the billing system.

Operational Architecture

Sirloin’s billing package owns subscription management, payment processing, checkout cleanup, renewal retries, product listing, credit allocation, cache invalidation, and billing analytics. The core package areas are:

• domain/: shared billing entities, sentinel errors, plan parsing, and analytics interfaces.
• chargebee/: Chargebee client wrapper and retry behavior.
• checkout/: Primer checkout creation and expired pending-subscription cleanup.
• payments/: unified payment recording, idempotency checks, and subscription activation.
• events/: Chargebee polling, credit extraction, refund detection, and analytics notifications.
• subscriptions/, renewals/, and products/: subscription lifecycle, renewal retry, and product display operations.

Operational incidents usually cross Chargebee, Primer, the Sirloin database, Redis/cache invalidation, and billing background workers. Treat Chargebee as the authoritative subscription and invoice source, and use local purchase records plus distributed locks to verify whether credits were applied exactly once.

Table of Contents

1. Orphaned Payments
2. Stuck Subscriptions
3. Chargebee Sync Drift
4. Circuit Breaker Triage
5. Rate Limit Spike
6. Fraud Alert Response
7. Unknown Renewal Payments

---

Orphaned Payments

Symptom

User reports: “I paid but my subscription isn’t active” or “Credits didn’t appear”

Root Causes

1. Primer webhook didn’t arrive (network issue)
2. Polling worker crashed (missing fallback detection)
3. Idempotency key mismatch (Primer txn ID not recorded)
4. Chargebee invoice not marked as paid
5. EventPoller hasn’t run yet (payment too recent; <15 seconds old)

Investigation

# 1. Check Chargebee for invoice status
chargebee-cli invoice get <invoice_id>
# Look for: status = "paid", payment_received, transaction_id

# 2. Check local DB for purchase record
SELECT * FROM purchases WHERE invoice_id = '<invoice_id>';
# If empty: payment not recorded locally

# 3. Check logs for payment recording errors
grep "invoice_id=<invoice_id>" logs/billing.log
# Look for: errors, rate limiting, lock failures

# 4. Check polling worker status
SELECT * FROM event_poller_state ORDER BY created_at DESC LIMIT 10;
# If > 5 min old: polling worker may be stuck

# 5. Check if payment is recent
SELECT TIMESTAMPDIFF(SECOND, paid_at, NOW()) FROM invoices WHERE id='<invoice_id>';
# If < 15 seconds: polling hasn't run yet; wait

Resolution

Case 1: Chargebee shows paid; DB shows no purchase

# Trigger manual event processing
curl -X POST http://localhost:8080/internal/billing/poll \
  -H "Content-Type: application/json" \
  -d '{"all_invoices": false, "since": "2026-04-08T12:00:00Z"}'

# Wait 30 seconds, check if credits appeared
SELECT credits FROM users WHERE id = '<user_id>';

Case 2: Chargebee shows unpaid

# Check if payment was actually captured in Primer
primer-cli transaction get <primer_transaction_id>
# If status = "authorized" (not "captured"):
#   → Contact Primer support; payment not actually settled

# If status = "captured":
#   → Payment was captured by Primer but not recorded in Chargebee
#   → See "Sync Drift" section below

Case 3: Polling worker stuck

# Check worker health
systemctl status billing-poller
# If not running: systemctl start billing-poller

# Check logs
journalctl -u billing-poller -n 100 --no-pager
# Look for: rate limit errors, API errors, crashes

# If rate-limited: Wait 60 seconds for backoff, then restart
systemctl restart billing-poller

Case 4: Idempotency check failed

# Verify transaction ID format
SELECT transaction_id FROM purchases WHERE invoice_id='<invoice_id>';
# Should match Primer transaction ID

# If mismatch: Data corruption issue
#   → Contact engineering; may need DB cleanup

Slack Alert: “ORPHANED PAYMENT RECOVERY EXHAUSTED - manual review required”

The Primer settled-payments poller (TaskPollPrimerPayments) auto-recovers payments that settled for a canceled subscription (checkout-retry orphans). Recovery retries on exponential backoff (1m → 30m cap), 10 attempts over ~2.5h; the polling cursor holds behind the payment so every attempt actually runs. Typical orphans clear on the first one or two attempts (within minutes); the ~2.5h figure is the worst-case ceiling before exhaustion, not the usual wait — the user-facing checkout-timeout toast intentionally quotes an optimistic ~30 min rather than that ceiling so it does not alarm users, so treat that copy as deliberate, not a stale number. This alert means all 10 attempts failed — the payment is now human-owned and the poller has moved on.

# 1. Pull the payment from Primer using the ID in the alert
primer-cli payment get <payment_id>
# Confirm status (SETTLED = money captured) and customerId

# 2. Check whether the user already got what they paid for
SELECT * FROM purchases WHERE user_id = '<customer_id>' ORDER BY created_at DESC LIMIT 5;
chargebee-cli subscription list --customer <customer_id>

# 3a. User has an active sub for the same plan → duplicate charge, refund via Primer
# 3b. User has nothing → manual recovery: create subscription + record payment
#     (mirror recoverOrphanedPayment steps), or refund if the user re-purchased

Note: the retry counter is in-memory — a deploy mid-cycle restarts the 10 attempts and can repeat the alert. The obligation row below is unique per payment, so the row, not the alert, is the record.

Obligation rows: refund_owed and recover_orphan_payment

Since TRUCOM-12 a settled checkout payment ends either delivered or with exactly one row in billing.failed_operations (status failed_permanent, so the retry worker never picks it up). Refunds stay manual. List the open ones:

SELECT id, created_at, operation_type, user_id, payment_id, subscription_id,
       operation_data->>'reason' AS reason, operation_data->>'amount' AS amount_cents,
       operation_data->>'active_subscription_id' AS active_subscription_id,
       operation_data->>'recovery_subscription_id' AS recovery_subscription_id,
       operation_data->>'first_payment_id' AS first_payment_id
FROM billing.failed_operations
WHERE operation_type IN ('refund_owed', 'recover_orphan_payment')
  AND resolved_at IS NULL
ORDER BY created_at DESC;

Row	Reason	What happened	What support does
refund_owed	duplicate_same_plan	The customer already had this plan from another payment (the DUPLICATE CHARGE log line)	Refund the payment in Primer
refund_owed	duplicate_other_plan	The customer bought another plan the payment was not upgrading from	Refund the payment in Primer
refund_owed	duplicate_on_activation	Pay again on a checkout whose payment had already settled: two active subscriptions on the same plan	Refund the payment and cancel the subscription in subscription_id
refund_owed	duplicate_same_order	Hourly reconciliation found another settled payment on one checkout order (a double submit); the order delivered once, for the payment in first_payment_id, which may be the later one	Check in Primer that it was not refunded already, then refund this payment
refund_owed	coupon_unavailable, amount_mismatch	Recovery could not reproduce the price paid	Refund the payment, or grant the plan by hand if the customer asks
recover_orphan_payment	record_failed, recovery_exhausted, no_customer_id	Recovery could not deliver for another reason	Check Primer and Chargebee, then deliver by hand or refund
recover_orphan_payment	duplicate_check_failed	The payment activated its plan, but the check for a second plan could not run	If the customer has two active subscriptions on the same plan, handle it like duplicate_on_activation; otherwise close the row
recover_orphan_payment	not_delivered	Hourly reconciliation found a settled payment with no purchase, older than the poller’s window, so recovery no longer sees it unless the poller cursor was stuck for over a day	Treat it like record_failed. Each hourly run closes the row on its own if the payment is delivered later, for example through the checkout page

When done, close the row: UPDATE billing.failed_operations SET resolved_at = now(), operation_status = 'completed' WHERE id = '<id>';

A checkout whose payment settled less than two hours ago (counted from the payment’s last update in Primer) is never canceled or voided by the cleanups (primer.SettledCheckoutHold); after that the stale sweep cancels it and recovery or reconciliation own the money. A payment has at most one of these rows (migration 183); a more specific row replaces a not_delivered one in place.

Prevention

• Monitor payment_recording_latency_seconds (alert if > 60s)
• Monitor events_polled_total (alert if 0 for > 5 min)
• Set up Primer webhook retry alerts

---

Stuck Subscriptions

Symptom

User’s subscription is in “future” or “non_renewing” state when it should be “active” or “cancelled”

Root Causes

1. Pending checkout never completed (user started but didn’t finish)
2. Activation failed (update_term_end API error)
3. Cancellation API error (returned error but was actually canceled)
4. Chargebee-side state mismatch (DB disagrees with Chargebee)

Investigation

# 1. Check local DB
SELECT id, status, start_date, current_term_end FROM subscriptions
WHERE customer_id='<customer_id>';

# 2. Check Chargebee
chargebee-cli subscription get <subscription_id>
# Compare status between DB and Chargebee

# 3. Check if subscription is pending checkout
SELECT * FROM subscriptions WHERE status='future' AND start_date > NOW() + INTERVAL '1 YEAR';
# If matches: this is a pending checkout

# 4. Check age
SELECT TIMESTAMPDIFF(DAY, created_at, NOW()) FROM subscriptions WHERE id='<subscription_id>';
# If > 1 day: eligible for cleanup

Resolution

Case 1: Subscription is future + start_date > 1 year (pending checkout)

# Is checkout still active?
curl -X GET http://localhost:8080/api/checkout/status?order_id=<invoice_id>
# If "expired": User didn't complete payment

# Option A: User wants to retry
# → Recreate checkout (new Primer session)
curl -X POST http://localhost:8080/api/billing/checkout \
  -d '{"user_id": "<user_id>", "item_price_id": "<item_price_id>"}'

# Option B: Cleanup (if > 1 day old)
# → Manually cancel
chargebee-cli subscription cancel <subscription_id>
# Then re-create if user wants to retry

Case 2: Subscription should be active but is still future

# Payment was recorded but activation failed?
SELECT * FROM purchases WHERE subscription_id='<subscription_id>';
# If empty: payment never recorded

# If purchase exists: activation failed
# → Try to manually activate
curl -X POST http://localhost:8080/internal/billing/activate \
  -d '{"subscription_id": "<subscription_id>"}'

# If successful: Check Chargebee status updated
chargebee-cli subscription get <subscription_id> | grep status

Case 3: Subscription should be cancelled but is still active

# Was cancellation API called?
grep "subscription_id=<subscription_id>" logs/billing.log | grep cancel
# If no matches: cancellation was never initiated

# If matches but status still "active":
# → Chargebee API returned success but didn't actually update
# → Try again
chargebee-cli subscription cancel <subscription_id> --force

# If still doesn't work: Chargebee issue
# → Contact Chargebee support with subscription ID

Case 4: DB/Chargebee mismatch

# Force sync from Chargebee to DB
curl -X POST http://localhost:8080/internal/billing/sync \
  -d '{"subscription_id": "<subscription_id>"}'

# Verify match
chargebee-cli subscription get <subscription_id> > /tmp/cb.json
curl http://localhost:8080/api/subscriptions/<subscription_id> > /tmp/db.json
diff /tmp/cb.json /tmp/db.json

Prevention

• Monitor subscriptions in “future” state (alert if > 1 day old + start_date not near now)
• Monitor activation errors in logs (alert if > 1% of payments)
• Test pending checkout cleanup regularly

---

Chargebee Sync Drift

Symptom

DB shows subscription in different state than Chargebee, or invoice amounts don’t match

Root Causes

1. Failed API call that partially succeeded (Chargebee changed, DB didn’t)
2. Stale cache (local cache hasn’t been invalidated)
3. Manual change in Chargebee UI (not synced back to DB)
4. Eventual consistency window (recent change, not synced yet)

Investigation

# 1. Get current state from both systems
chargebee-cli subscription get <subscription_id> | jq . > /tmp/chargebee.json
curl -X GET http://localhost:8080/api/subscriptions/<subscription_id> | jq . > /tmp/db.json

# 2. Compare critical fields
diff <(jq '.status, .current_term_end, .coupon_ids' /tmp/chargebee.json) \
     <(jq '.status, .current_term_end, .coupon_ids' /tmp/db.json)

# 3. Check invoice amounts
chargebee-cli invoice get <invoice_id> | jq '.total'
SELECT total_amount FROM invoices WHERE id='<invoice_id>';

Resolution

Case 1: Chargebee is newer (DB is stale)

# Option A: Invalidate cache + re-fetch
curl -X POST http://localhost:8080/internal/cache/invalidate \
  -d '{"customer_id": "<customer_id>"}'

# Then fetch again (will re-query Chargebee)
curl http://localhost:8080/api/subscriptions/<subscription_id>

# Option B: Full sync
curl -X POST http://localhost:8080/internal/billing/sync-all \
  -d '{"since": "2026-04-08T00:00:00Z"}'

Case 2: DB is newer (Chargebee is stale)

This should be rare (we sync from Chargebee, not push to it). But can happen if:

• Manual update was attempted but failed halfway
• Chargebee API returned success but didn’t apply

# Re-apply the update to Chargebee
chargebee-cli subscription update <subscription_id> \
  --new-field-name="<expected_value>"

# Then verify sync
curl -X POST http://localhost:8080/internal/billing/sync \
  -d '{"subscription_id": "<subscription_id>"}'

Case 3: Manual change in Chargebee UI

# User or support made manual changes in Chargebee
# Sync DB to match
curl -X POST http://localhost:8080/internal/billing/sync \
  -d '{"subscription_id": "<subscription_id>"}'

# Verify match
chargebee-cli subscription get <subscription_id> | jq '.status' > /tmp/cb_status
curl http://localhost:8080/api/subscriptions/<subscription_id> | jq '.status' > /tmp/db_status
diff /tmp/cb_status /tmp/db_status

Prevention

• Monitor sync errors in logs (alert if > 0 per hour)
• Monitor DB/Chargebee consistency gaps (audit query every 1 hour)
• Disable manual Chargebee changes for “system” subscriptions; route through API

---

Circuit Breaker Triage

Symptom

“Circuit breaker is open” error in logs; Chargebee API calls failing

Root Causes

1. Chargebee API is down (service issue)
2. Our API key is invalid or rate-limited (configuration issue)
3. Network connectivity problem (firewall, DNS, proxy)
4. Sustained error rate too high (threshold exceeded; e.g., > 50% failures)

Investigation

# 1. Check circuit breaker state
curl http://localhost:8080/internal/health/circuit-breaker

# Output might show: {"chargebee": {"state": "open", "error_rate": 0.75}}

# 2. Check error logs
grep "circuit.*open" logs/billing.log | tail -20

# 3. Test Chargebee API directly
curl -H "Authorization: Bearer $CHARGEBEE_API_KEY" \
  https://api.chargebee.com/api/v2/health

# 4. Check rate limiting
grep "rate.*limit\|429\|Please try after" logs/billing.log | tail -10

# 5. Check our API key configuration
echo $CHARGEBEE_API_KEY | head -c 10  # Show first 10 chars (don't log full key)
# If missing or changed recently: configuration issue

Resolution

Case 1: Chargebee API is actually down

# Wait for Chargebee to recover (~5-30 minutes typically)
# Monitor status page: https://status.chargebee.com

# In the meantime:
# - Primer payments may still work (Primer is independent)
# - Event polling will retry automatically (exponential backoff)
# - User-facing requests will fail; serve cached data if available

# Alert users (if outage > 30 minutes)
curl -X POST http://localhost:8080/internal/notifications/alert \
  -d '{"message": "Subscription operations temporarily unavailable"}'

Case 2: Rate limiting

# Chargebee rate limit is per API key, per minute
# Standard limit: ~60 requests/minute

# Check rate limit errors
grep "Please try after some time" logs/billing.log | wc -l

# Reduce request frequency if possible:
# - Increase polling interval (15s → 30s)
# - Batch operations (list instead of individual gets)
# - Implement caching more aggressively

# Contact Chargebee if sustained: request rate limit increase

Case 3: Invalid/rotated API key

# Check if key was recently rotated
git log --all --grep="API_KEY\|chargebee" --oneline | head -5

# If key was rotated but env var not updated:
export CHARGEBEE_API_KEY="<new_key>"
# Or update .env file and restart service
systemctl restart billing-service

# Verify key works
curl -H "Authorization: Bearer $CHARGEBEE_API_KEY" \
  https://api.chargebee.com/api/v2/items?limit=1

Case 4: Network connectivity

# Test DNS resolution
nslookup api.chargebee.com

# Test raw connectivity
nc -zv api.chargebee.com 443

# Check proxy settings (if behind proxy)
curl -v -x <proxy> https://api.chargebee.com/api/v2/health

# Check firewall rules
# If behind corporate firewall, ensure api.chargebee.com is whitelisted

Prevention

• Monitor circuit breaker state (alert if open > 5 minutes)
• Set up status page monitoring (Chargebee status page)
• Implement graceful degradation (cached data when CB unavailable)
• Test disaster scenario quarterly (simulate Chargebee down)

---

Rate Limit Spike

Symptom

Logs show "Please try after some time" errors; multiple retries with exponential backoff; user requests slow down

Root Causes

1. Sudden traffic spike (load test, marketing campaign, viral growth)
2. Polling worker making too many requests (config error, infinite loop)
3. Chargebee’s rate limit lowered (API downgrade, or our usage miscounted)
4. Retry storms (exponential backoff causing thundering herd)

Investigation

# 1. Check request volume in the last hour
grep "chargebee.*request" logs/billing.log | wc -l
# Compare to baseline

# 2. Count requests by operation
grep "chargebee.*request" logs/billing.log | \
  sed 's/.*operation=//' | cut -d' ' -f1 | sort | uniq -c

# 3. Check retry attempts
grep "retry.*attempt" logs/billing.log | tail -20

# 4. Check if polling worker is looping
grep "events_polled" logs/billing.log | tail -20 | awk '{print $1}' | uniq | wc -l
# Should be ~60 entries over 15 seconds; if > 100: looping

Resolution

Case 1: Normal traffic spike

# Chargebee will recover automatically (1-2 minutes)
# Our system retries with backoff; requests will eventually succeed

# Monitor recovery
grep "rate.*limit\|Please try after" logs/billing.log | tail -1 | awk '{print $1}'
# If timestamp is > 2 minutes ago: recovered

# Check if user requests are succeeding now
curl -X GET http://localhost:8080/api/subscriptions/<sub_id> -w "%{http_code}"
# Should be 200

Case 2: Polling worker misconfiguration

# Check polling frequency
ps aux | grep polling-worker
# Look for: --frequency=15s (should be 15+ seconds)

# If frequency too low: update config
systemctl stop billing-poller
# Edit /etc/billing/config.yaml: set frequency to 30s
systemctl start billing-poller

# Monitor again
grep "events_polled" logs/billing.log | tail -5

Case 3: Chargebee rate limit was lowered

# This is rare; would require Chargebee proactively lowering your limit
# Check account status: https://app.chargebee.com/settings/your-account

# If limit was lowered:
# Option A: Request increase (contact support)
# Option B: Reduce request frequency (increase poll interval, batch requests)

# Calculate required frequency reduction
current_rps = $(grep "chargebee.*request" logs/billing.log | wc -l) / 3600
new_rps = min(60 / 60, 1.0)  # 60 requests/minute = 1 RPS
reduction_factor = new_rps / current_rps

# Example: If we're doing 2 RPS and limit is 1 RPS, reduce by 50%

Case 4: Retry storms (thundering herd)

# If multiple requests all hit rate limit and retry simultaneously,
# backoff + jitter helps, but may still spike

# Check for synchronization:
grep "retry.*exponential" logs/billing.log | \
  sed 's/.*attempt=//' | sort | uniq -c | sort -rn | head
# If one attempt number dominates: synchronized retries

# Add random jitter to retry delay
# (Already implemented in retry/retry.go with exponential backoff)

# If problem persists: contact engineering for backoff tuning

Prevention

• Monitor chargebee_rate_limit_errors_total (alert if > 0)
• Implement request batching (list instead of individual lookups)
• Cache aggressively (extend TTLs, pre-warm cache)
• Load test with Chargebee to understand rate limit behavior

---

Fraud Alert Response

Symptom

Multiple failed payment attempts; potential fraudster; fraud alert triggered (alert from Primer or PostHog)

Root Causes

1. Testing (developer testing payment flows)
2. Card decline (legitimate user’s card being rejected)
3. Actual fraud (stolen card or account takeover)
4. Billing system error (charging multiple times unintentionally)

Investigation

# 1. Get user details
SELECT user_id, email, created_at FROM users WHERE id='<user_id>';

# 2. Check payment history
SELECT transaction_id, amount, status, created_at FROM transactions
WHERE customer_id='<chargebee_customer_id>'
ORDER BY created_at DESC LIMIT 10;

# 3. Check if account is newly created (more likely to be fraud)
SELECT TIMESTAMPDIFF(MINUTE, created_at, NOW()) FROM users WHERE id='<user_id>';
# If < 30 min: new account (higher fraud risk)

# 4. Check activity pattern (testing vs real usage)
SELECT * FROM subscriptions WHERE customer_id='<chargebee_customer_id>';
# Multiple subscriptions in short time = potential testing

# 5. Check IP/device changes
SELECT ip_address, COUNT(*) FROM login_attempts
WHERE user_id='<user_id>'
GROUP BY ip_address;
# Multiple IPs = possible account compromise

Resolution

Case 1: Testing (developer)

# If internal user: No action needed
# If external user: Contact support to explain

# To prevent: Use separate test account with test API key
# Make sure test environment doesn't use production Chargebee

Case 2: Legitimate card decline

# Card was declined (insufficient funds, expired, etc.)
# User will retry naturally; no action needed

# But: Check if we're charging multiple times on decline
SELECT COUNT(*) FROM transactions
WHERE customer_id='<id>' AND status='failed'
ORDER BY created_at DESC LIMIT 5;
# Should be 1 failed transaction per user attempt

# If multiple failures from single user attempt: Bug in payment retry logic
# Contact engineering

Case 3: Actual fraud

# Steps:
# 1. Freeze account (disable further payments)
curl -X POST http://localhost:8080/internal/users/<user_id>/freeze

# 2. Void any pending invoices
chargebee-cli invoice void <invoice_id>

# 3. Notify user (send email)
curl -X POST http://localhost:8080/internal/notifications/alert \
  -d '{"user_id": "<user_id>", "message": "We detected suspicious activity"}'

# 4. Contact payment processor (Primer)
# Report transaction ID to Primer support with details

# 5. Review transaction logs
grep "user_id=<user_id>" logs/billing.log logs/auth.log | tail -50
# Look for: unusual patterns, high-frequency attempts, geographic anomalies

# 6. Consider additional verification
# If high-value account: require 2FA, manual review before unlocking

Case 4: Billing system error (double-charging)

# Check for duplicate transactions
SELECT transaction_id, COUNT(*) FROM transactions
WHERE customer_id='<id>'
GROUP BY transaction_id
HAVING COUNT(*) > 1;

# If duplicates found: Data corruption issue
# 1. Contact engineering
# 2. Issue refund for duplicate charges
chargebee-cli credit-note create \
  --customer-id=<customer_id> \
  --amount=<duplicate_amount>

# 3. Fix root cause (idempotency check, locks, etc.)

Prevention

• Set up fraud alerts in Primer dashboard (monitor > N failed attempts)
• Monitor unusual payment patterns (>$1000/min, >10 charges/min)
• Require verification for high-value accounts
• Authorization attempt quota enforced per user (NSFW tier, production only), per processor in domain/processorlimits.go: eMerchantPay and unattributed or Cybersource full-access traffic 12 daily / 25 weekly / 50 monthly, NMI none. CIT reserves 1 slot for MIT renewal in each window the next renewal can land in, and always in the monthly window (effective CIT limits: 11/24/49 where it does)
• Regular fraud report reviews (quarterly)

---

Quota Exhaustion (Authorization Attempt Limits)

Symptom

User sees “checkout temporarily blocked due to too many payment attempts” or renewal retry logs show MIT blocked: authorization count quota exhausted.

Investigation

-- Check user's authorization attempts in last 24h / 7d / 30d
SELECT event_type, created_at, metadata->>'merchant_context' as ctx,
       metadata->>'amount' as amount
FROM fraud_events
WHERE user_id = '<user_id>'
  AND event_type = 'authorization_attempt'
  AND created_at > NOW() - INTERVAL '30 days'
ORDER BY created_at DESC;

-- Check if velocity block was triggered
SELECT * FROM fraud_events
WHERE user_id = '<user_id>'
  AND event_type = 'velocity_triggered'
ORDER BY created_at DESC LIMIT 5;

-- Check dunning attempts with quota_blocked status
SELECT * FROM dunning_attempts
WHERE invoice_id IN (
  SELECT invoice_id FROM chargebee_invoices WHERE customer_id = '<customer_id>'
)
ORDER BY created_at DESC LIMIT 10;

-- Count per window
SELECT
  COUNT(*) FILTER (WHERE created_at > NOW() - INTERVAL '1 day') as daily,
  COUNT(*) FILTER (WHERE created_at > NOW() - INTERVAL '7 days') as weekly,
  COUNT(*) FILTER (WHERE created_at > NOW() - INTERVAL '30 days') as monthly
FROM fraud_events
WHERE user_id = '<user_id>'
  AND event_type = 'authorization_attempt'
  AND metadata->>'merchant_context' = 'full';
-- EMP limits: 12 daily / 25 weekly / 50 monthly (CIT: 11/24/49 where the renewal slot is held)

Resolution

Legitimate user hit limit: Wait for window to expire. Daily resets after 24h from oldest attempt in window.

Fraud/abuse: No action needed — quota is working as intended. Monitor for escalation.

Bug (false blocks): Check if authorization_attempt rows are being written without corresponding payment attempts. Look for duplicate inserts or stale events.

---

NSFW Renewal Card Substitution Issues

Symptom

NSFW renewal fails with EMP hard decline despite user having a saved card. Or: renewal unexpectedly uses a different card than the user’s primary.

Investigation

# 1. Check if card substitution was attempted
grep "nsfw_renewal" logs/billing.log | grep "<customer_id>"
# Look for: nsfw_renewal_card_substitution (swapped) or nsfw_renewal_no_card_found

# 2. Check user's saved payment instruments
# Query Primer for customer's vaulted methods
primer-cli customer payment-instruments list <customer_id>
# Look for: PAYMENT_CARD type instruments that are not deleted/expired

# 3. Check if primary was auto-set after checkout
grep "nsfw_checkout_set_primary" logs/billing.log | grep "<customer_id>"
# If nsfw_checkout_set_primary_failed: Primer SetDefault API failed

# 4. Check what the user's current default instrument is
primer-cli customer payment-instruments list <customer_id> | grep '"default": true'

Resolution

Case 1: Card substitution worked but EMP still declined

Card may be expired or have insufficient funds. Check the decline code in the dunning attempt record. This is normal dunning behavior — not a substitution bug.

Case 2: No card found, wallet sent to EMP

User only has wallet methods saved. Expected behavior — EMP rejects, subscription enters dunning. User needs to add a card through checkout (frontend forces fresh card entry on SFW→NSFW switch).

Case 3: Auto-set primary failed after checkout

maybeSetNSFWCheckoutPrimary is non-fatal. If it failed, the user’s old primary remains. Next scheduled renewal will attempt card substitution via resolveNSFWRenewalPaymentToken. No manual action needed unless the user has no saved cards at all.

Prevention

• Monitor nsfw_renewal_no_card_found log frequency (rising trend = users without compatible cards)
• Monitor nsfw_checkout_set_primary_failed (Primer API issues)

---

Processor Assignment Issues

Symptom

• User routed to wrong PSP (e.g., NSFW payment hitting Cybersource instead of EMP/NMI)
• metadata["psp"] in Primer dashboard doesn’t match expected processor
• Checkout fails with processor-related error
• Reported as “user has no processor”: an empty column is the normal unlocked state on both lanes, not a missing assignment — see Resolution

Investigation

-- Check user's assigned processors
SELECT user_id, sfw_processor, nsfw_processor FROM users.credits WHERE user_id = '<user_id>';

-- Count unlocked users per lane. Both lanes are settle-then-lock, so a large
-- count is the expected steady state, not a backfill queue. Mirrors the guarded
-- writes: NULL and '' both count as unlocked, and soft-deleted rows are excluded
-- the way the writers exclude them.
SELECT
  COUNT(*) FILTER (WHERE sfw_processor IS NULL OR sfw_processor = '')   AS sfw_unlocked,
  COUNT(*) FILTER (WHERE nsfw_processor IS NULL OR nsfw_processor = '') AS nsfw_unlocked
FROM users.credits
WHERE deleted_at IS NULL;

Check Primer dashboard for the transaction’s metadata.psp field to confirm what was sent.

Resolution

Since FOXY-596 (NSFW) and FOXY-877 (SFW), neither lane pre-assigns a processor — the first payment is routed by that lane’s Primer workflow (high-risk for NSFW, Regular Access Traffic for SFW) and sirloin locks whatever processor actually settled, plus the MID on NSFW. So:

• An empty sfw_processor or nsfw_processor is normal, not a bug. An unlocked user is waiting for their first settle to lock them. Do NOT backfill either column — that reintroduces the blind-guess mislock this design removed, and first-write-wins makes the guess permanent.
• Wrong lock stored (the lane’s column points at a processor the user’s vaulted token can’t be charged on): the lock is first-wins and permanent, so manual SQL is the only override. Clear it only if the user has not successfully transacted on the stored processor, then let the next settle re-lock:

  -- NSFW: clear the processor and its merchant account together
  UPDATE users.credits SET nsfw_processor = '', nsfw_processor_mid = '' WHERE user_id = '<user_id>';
  
  -- SFW: this lane has no merchant account
  UPDATE users.credits SET sfw_processor = '' WHERE user_id = '<user_id>';

• ResolveProcessor returning error: check DB connectivity. resolveProcessor is lane-symmetric — a GetCredits error aborts checkout identically on both lanes, and renewals do the same: a lock-read failure aborts the attempt on either lane rather than routing processor-absent, because processor-absent on a locked user sends them through discovery routing and can settle them off their lock. The next dunning tick re-reads, roughly fifteen minutes later.

Prevention

• Watch billing_processor_lock_outcome_total (locked / already_locked / lost_race / unknown_processor / assign_error / …) split by lane, never in aggregate: SFW is the larger lane, so its healthy volume masks a total NSFW failure in a combined series. One record per gated settle, so the per-lane total is the number of settles the hook saw on that lane, and a lane going quiet means it stopped being called. The psp attribute says which processor each outcome concerns — none where there was no processor, other for an unmapped name (the vendor’s actual string is in the log line, not the metric).
• Watch billing_processor_repair_outcome_total (locked / no_history / unavailable) by lane for history-based repair. One record per repair, not per candidate payment. A rising unavailable rate is Primer or the database failing, not users without history — no_history is the benign one.
• These log lines carry what the counters cannot, and reading them by lane matters as much as reading the counters that way.
  • settled processor differs from locked psp (warn) is a locked user whose settle landed on the other processor of their own lane. On NSFW that is worth chasing — tokens are processor-homed there. On SFW it is the ordinary Cybersource↔Stripe fallback and is expected to appear at some steady rate; judge it by trend, not by presence. The trend is billing_processor_lock_drift_total split by lane, locked_psp and settled_psp — read it against billing_processor_lock_outcome_total{outcome="already_locked"} for the same lane, which is the denominator. The log line carries the user and payment when a specific case needs chasing; the counter is what you graph.
  • cross-lane settle on a locked user (debug) means the settled processor belongs to the other lane’s table, so the payment’s access label and the workflow that actually ran disagree. Rare by construction; a run of these points at a Primer trigger, not at a user.
  • settled processor is unknown to every lane (error) means Primer returned a name no lane maps: an unlocked settle cannot establish a lock until the table gains an entry. Already-locked users are unaffected — the hook reads the lane’s column and returns before it maps the settled name.
  • settled merchant differs from locked mid (warn) is an NSFW Cybersource user whose settle landed on another merchant account than the locked one, for example an SSB customer settling on Coastal. Wallet fallbacks between SSB and Esquire are expected; a settle on the SFW Coastal account is not. billing_processor_lock_mid_drift_total counts both, plus locks written on Coastal before SSB went live, so treat it as an upper bound and do not alert on it: triage by settled_mid in the log line. It fires only when the locked and settled MIDs are both non-empty and differ, so locks with an empty MID, legacy ones and those written after mid fetch failed, are silent. The check covers subscription-linked settles whose processor is in hand, meaning customer-initiated subscription payments and renewals that re-read an existing payment. Top-ups and vault-only settles are never checked, and a fresh renewal’s create response carries no MID.
  • mid fetch failed, locking without mid (warn) is a first NSFW lock written without a MID, by a settle or by payment-history repair, and only by the call that wrote it: reason=fetch_error (the Primer read failed or named no processor) or reason=no_mid (the payment carries none). The user routes as an existing user until the MID is set with the per-user SQL below. fetched processor differs from known processor, not locking (reason=processor_mismatch, with fetched_processor_name and fetched_mid) leaves the user unlocked; the next settle locks them.
• MID-only backfill for legacy NMI users is safe; never backfill nsfw_processor itself:

  UPDATE users.credits SET nsfw_processor_mid = '<MID>' WHERE nsfw_processor = 'nmi' AND (nsfw_processor_mid IS NULL OR nsfw_processor_mid = '');

• Never bulk-backfill a Cybersource or EMP MID: locks without one are the existing users, and an empty MID is what keeps them on the workflows’ default branches. Set one only for a single user whose lock was written after FOXY-1364 without it (mid fetch failed), with the merchant ID of the payment that locked them in Primer. In Strip (user → Payment processors → NSFW routing), only when the user’s current NSFW processor is that payment’s psp and the MID shows “none”: pick the pair with that payment’s MID. If the processor differs, an operator has re-homed the user since, and the MID must not be set at all. Strip does not check this for you; the SQL below does, and is the fallback. Guard on that payment’s own psp (cybersource or emp, from processor.name), never on both: if an operator re-homes the user before the fix, the lock points at the other processor, and the MID must not land there:

  UPDATE users.credits SET nsfw_processor_mid = '<MID from that Primer payment>' WHERE user_id = '<user_id>' AND nsfw_processor = '<psp of that Primer payment>' AND (nsfw_processor_mid IS NULL OR nsfw_processor_mid = '');

---

Unknown Renewal Payments

Symptom

• Axiom gauge billing_renewal_operation_unknown_age_seconds above 900 (an unknown renewal charge older than 15 minutes).
• A customer in dunning whose invoice is never retried; the worker logs invoice has unfinished renewal work - no new charge for it.
• renewal payment outcome unknown - held for recovery warnings.

Root Causes

A renewal charge reached Primer, or may have, and Sirloin never learned the result: a timeout, a 5xx, a dropped connection, or a 409 for the operation’s own idempotency key. Recovery looks the payment up by renewal_operation_id every run. It stays unknown when Primer has no such payment, or the customer’s payment list is too long to search completely. Neither proves no charge happened, so the hold stays. After three complete empty searches in a row, a scheduled run resubmits a scheduled charge once under the same idempotency key, between 15 minutes and a day after its original submission. The operation’s evidence then shows resubmit: sent with resubmitted_at, or request_changed when the invoice amount or the request no longer matched and nothing was sent. Only a 2xx answer settles a resubmission. A charge a manual or card-update retry started is never resubmitted. A charge still unknown after that needs an operator.

Investigation

1. List open operations, oldest first (read-only):

   SIRLOIN_DATABASE_URL=... go run ./cmd/scripts/renewal-operation-resolve --list

2. For the operation, read last_error, check_count and submitted_at. Search Primer’s dashboard for payments with metadata renewal_operation_id equal to the operation ID, and for the order ID <invoice>-attempt-<slot>.
3. If no payment is visible, ask Primer support to confirm whether a payment was created for the idempotency key (the operation ID). A missing search result is not evidence.

Resolution

Resolve under the invoice lock with the audited command. Each run records the operator, decision and evidence on the operation.

# Primer shows a settled payment carrying this operation ID:
SIRLOIN_DATABASE_URL=... SIRLOIN_PRIMER_API_KEY=... go run ./cmd/scripts/renewal-operation-resolve \
  --operation <id> --operator <name> --decision paid --primer-payment-id <pay_id> --evidence <pay_id> --confirm

# Primer shows a failed payment carrying this operation ID:
SIRLOIN_DATABASE_URL=... SIRLOIN_PRIMER_API_KEY=... go run ./cmd/scripts/renewal-operation-resolve \
  --operation <id> --operator <name> --decision declined --primer-payment-id <pay_id> --evidence <pay_id> --confirm

# Primer support confirms no payment exists for the key:
SIRLOIN_DATABASE_URL=... go run ./cmd/scripts/renewal-operation-resolve \
  --operation <id> --operator <name> --decision not_charged --evidence <support-ticket> --confirm

paid and declined read the payment back and refuse one that does not carry the operation ID or whose status does not match. After paid, the next renewal run records the payment in Chargebee, applies credits, records a successful dunning attempt under the original source and slot, and finalizes. After declined, it records a failed dunning attempt under the original source and slot, with the decline reason read back from Primer, and finalizes; no payment is recorded and no credits are applied. An operator-resolved outcome sends no customer email, analytics or NSFW velocity event. not_charged closes the operation without spending the slot; the next scheduled run charges with a new key. On the NSFW lane the authorization attempt the original charge reserved stays counted in the quota windows until it ages out, because nothing records which reservation it was; a customer at the limit can be held by quota until then.

A dunning cancellation closed as canceled_after_payment went through in Chargebee although a payment settled for the invoice: the customer paid and lost the subscription. Its evidence names the Primer payment. Record the payment on the invoice and reactivate the subscription by hand; the worker logs CRITICAL: subscription canceled for dunning but a payment settled for the invoice when it closes one.

If renewals cause duplicate submissions, lost state, leaked locks or a growing backlog, roll back only to a revision that honors unresolved operations (this one or later). Earlier revisions ignore billing.renewal_payment_operations and can charge an invoice whose earlier charge is still unknown. Keep the table and its rows until every operation is finalized.

Prevention

• The renewal job’s budget and Primer’s 30-second client timeout leave a charge in flight at the deadline unknown, by design; one cut off before it had a connection closes as unsent. Watch billing_renewal_deferrals_total, billing_renewal_run_duration_seconds and billing_renewal_missed_retry_windows_total for a queue that outgrows a run.

Quick Reference: Common Commands

# Check subscription status
chargebee-cli subscription get <subscription_id> | jq '.status, .coupon_ids'

# List unpaid invoices
chargebee-cli invoice list --filter='status:payment_due' --limit=50

# Void invoice
chargebee-cli invoice void <invoice_id>

# Create credit note (refund)
chargebee-cli credit-note create --customer-id=<id> --amount=<cents>

# Check payment transaction
chargebee-cli transaction get <transaction_id>

# Trigger manual event polling
curl -X POST http://localhost:8080/internal/billing/poll \
  -H "Content-Type: application/json" \
  -d '{"all_invoices": false}'

# Invalidate user cache
curl -X POST http://localhost:8080/internal/cache/invalidate \
  -d '{"customer_id": "<customer_id>"}'

# Sync subscription from Chargebee
curl -X POST http://localhost:8080/internal/billing/sync \
  -d '{"subscription_id": "<subscription_id>"}'

# Check circuit breaker status
curl http://localhost:8080/internal/health/circuit-breaker | jq .

# View billing logs
journalctl -u billing-service -n 100 -f

# Restart billing service
systemctl restart billing-service