Payments Alerts

Use this runbook when something posts in #payments-alerts, or when you change a payment check. The exact rules and the SQL behind every check are in the Linear document TRUCOM-10 rules and queries. Thresholds live in PostHog and Axiom; this page mirrors them, so change both together.

No application code is involved. Sirloin’s own billing Slack alerts read SIRLOIN_SLACK_WEBHOOK_URL_CS_REPORTS (apps/sirloin/internal/pkg/env/variables.go:102), which production does not set, so they still post nowhere.

Where it lives

Piece	Location
Dashboard	Payments: status table, trend charts, CEO tiles, the five health checks and LTV, in that order (see Reading the dashboard)
PostHog alerts	Alerts, one per health check, Slack integration “Foxy” to `#payments-alerts`
Monday post	Dashboard subscription to `#payments-alerts`, Mondays 09:00 UTC
Axiom monitor	Payments money at risk - PROD, notifier `#payments-alerts`
Warehouse	Rump DB billing source in PostHog: `dunning_attempts`, `failed_operations`, `primer_polling_state` and `chargebee_event_cursor` sync every 5 minutes, `primer_payments` every hour

Posts carry no customer data: labels name a processor, merchant account, reason, method, feed or billing operation id, and the Axiom post projects payment, invoice and subscription ids only.

Checks

Check	Fires when	Recovers when	Runs
Retry coverage	3 judged renewal cohort days in a row where under 50% of failed first attempts got scheduled attempt 2 within 30 hours	2 judged days in a row at 80% or more	daily 08:10 UTC
Renewals without a first attempt (row in the retry coverage check)	on a judged day, at least 10 and 2% of renewal invoices due that day (minimum 50) got no scheduled first attempt within 24 hours and were not paid or voided in that time	the next judged day under the limit	daily 08:10 UTC
Auth rate by processor and merchant account	7-day decline share above its own baseline by more than 5 pp and 3 standard errors	3 evaluations in a row at or below baseline + 2 pp	daily 08:10 UTC
Decline reason share	a renewal decline reason jumps to at least 5% of a processor’s first attempts and 3 times its prior share, or a new reason reaches 1%	3 daily evaluations below the larger of twice and 2 pp above the share frozen at first fire	daily 08:10 UTC
Checkout health by payment method	method x processor decline share 8 pp and 3 standard errors above baseline, or a method’s share of checkout attempts halves	back near baseline (3 evaluations for decline share, 1 for mix)	daily 08:10 UTC
Data freshness	a poller, the mirror, Chargebee events, the renewal worker or a warehouse sync is stale (sync limit 30 minutes, 90 for the hourly `primer_payments`); a billing operation failed for good (except a cancel that failed because Chargebee already had the subscription cancelled); a failed operation is 24 hours overdue	no recovery post; the dashboard shows it	hourly at :10
Axiom: money at risk	Sirloin logs a line starting with one of the prefixes in the monitor query, such as `DUPLICATE CHARGE during orphan recovery` or `SECURITY: payment amount does not match invoice total`	each line posts once	every minute

Each PostHog check posts when a row starts firing and again when it recovers. A row below its volume guard reports insufficient_data and keeps its previous state.

When something posts

Responder: Evgeny for Trust and Commerce until a rota exists. Reply in the post’s thread the same working day with what you found.

Retry coverage. Check that the renewal worker ran and that attempts 2 and later are being created. Treat as urgent: renewals are not being collected.
Renewals without a first attempt. The worker skipped due invoices. Check that it ran and picked up every invoice due that day. Treat as urgent: those customers are not being charged at all.
Auth rate or decline reason. Open Primer Observability for that processor and merchant account, confirm the change, and open a Linear issue with the row from the post.
Checkout health. Run a staff checkout with that method on production. Urgent if it fails.
Data freshness. Check the named poller, worker or warehouse sync. For a new failed operation, resolve each id in Chargebee and record the outcome in the thread.
Axiom line. Do what the line says (refund, reconcile, or record the payment) and note it in the thread. A RECOVERY FAILED line and an amount-mismatch line usually arrive together for one incident.

A routing change, such as moving traffic to a new merchant account, shifts the per-processor rows. New merchant accounts show insufficient_data until they have a baseline; expect posts on the affected rows in the first week and say so in the thread.

Reading the dashboard

Status table (top). One row per area: checkout, renewals, first attempts, retries, processors, decline reasons and data feeds. 🔴 means alert level (for decline rates: at least 5 pp worse than normal and beyond 3 standard errors), 🟡 a smaller but real change, 🟢 normal, ⚪ too little data. Each row shows the value now, the normal value, the change and the last 28 days.
Trend charts. Retry coverage, renewal declines by processor, decline reasons, checkout declines by processor and merchant account, and traffic share per route. The retry and decline-reason charts draw their alert threshold as a dashed line. “What moved” lists every route, method, processor and reason, last 7 days against the 28 before, worst first.
Alert detail. The five health-check tables at the bottom are what the alerts read. A green “Not firing” badge only means nothing changed state at the last check; a row can stay fired for days. The status table shows the standing picture.
Renewal rows leave out bulk days (more than 1.8 times the median of the 14 days before, such as the FOXY-1459 backfill on 30 September and 2-3 October). The status table and trends do not post anywhere; only the alerts do.

Replaying a past day

Every check, the status table and “What moved” read the as_of variable, saved empty; the trend charts always show the latest months. On the dashboard, type a date (2026-09-14, the 08:00 UTC evaluation of that day) or a minute (2026-09-19 15:00) to replay it. Never save the dashboard or an insight with a value: the alerts and the Monday post read the saved empty value. Pollers, the mirror, the Chargebee cursor and the overdue-operations row keep only their current state, so they show not replayable.

Changing the warehouse settings

primer_payments syncs hourly: each run re-reads about 2,300 rows whatever the interval, so a 5-minute sync costs about 650,000 warehouse rows a day. It runs at :01 every hour (sync time 08:01 UTC), so the 08:10 checks read the warehouse as of 08:01. Sirloin writes the table every 15 minutes (apps/sirloin/internal/pkg/riverjobs/kinds.go:59), so payments from the last 15 minutes before 08:00 can be missing until the 09:01 sync.
primer_payments syncs incrementally on synced_at with a 900-second lookback. Sirloin stamps synced_at when it reads a payment and writes the row later, for example after the renewal attempt is recorded (apps/sirloin/internal/app/services/billing/dunningretry.go:701), and the upsert keeps the later read (apps/sirloin/internal/pkg/storage/primerpayments.go:68). Without the lookback such a row is never synced.
dunning_attempts syncs on created_at, which is safe only while RecordDunningAttempt is the only writer in use (apps/sirloin/internal/pkg/storage/dunningattempts.go:180). If ReserveDunningAttempt or FinalizeDunningAttempt gain a caller, switch the table to updated_at, or the warehouse keeps the reserved outcome.
Renewal order ids contain -attempt- (apps/sirloin/internal/app/services/billing/dunningretry.go:542); every checkout query excludes them that way.

Known gaps

The PostHog data warehouse allocation was 88% used on 16 October 2026. When it runs out, every warehouse sync stops: the freshness check posts “Warehouse sync” and the other checks run on stale data until an organization admin raises the limit.
Renewal rows in the auth-rate check never fire before 5 December 2026, because the FOXY-1459 backfill distorts their baseline until then. The other checks watch renewals meanwhile.
Local invoice drift found by the unpaid-invoice reconciler writes no log line and is not covered.
Refunds and chargebacks are not measured; Primer and each processor’s dispute portal remain the source of truth.
Backlog: 25 revoke_credit_for_refund and 16 apply_renewal_credits operations failed for good between February and May 2026 and were marked alerted without posting. They are known and out of scope.

Rollback

Disable the five PostHog alerts and the Axiom monitor, delete the Monday subscription, and return the warehouse syncs to 6 hours. Payments, retries and routing are untouched.