Fal Spend to PostHog Rollout

Fal Spend to PostHog Rollout

Use this runbook to deploy and activate automated Fal spend reporting. The rollout crosses Brain, the Fennec Postgres warehouse source, the Fal billing console, and the PostHog GPU economics dashboard.

Prerequisites

• The Brain migration containing inference_provider_spend_breakdown is ready for the normal release pipeline.
• BRAIN_FAL_ADMIN_KEY is configured server-side and belongs to the production Fal workspace. Never place this key in PostHog.
• The operator can run the Brain administrative spend sync, inspect Fal billing, and edit the PostHog warehouse source and dashboard.

Procedure

1. Deploy the Brain migration and application through the normal release pipeline. Confirm that inference_provider_spend_breakdown exists before configuring PostHog.
2. Run a bounded manual backfill through POST /spend/sync with inclusive UTC from and to dates. The initial backfill must include at least 2026-08-01 through 2026-08-31; process later history in bounded chunks as needed.
3. Compare the stored inference_provider_spend rows for provider FAL with the Fal billing console. Reconcile representative sample days to the cent, then sum amount_cents for 2026-08-01 through 2026-08-31 and confirm the Brain total is exactly $412,378.12.
4. In the existing Fennec Postgres warehouse source, confirm that inference_provider_spend uses an incremental sync on updated_at. A cursor on id or created_at misses corrected daily totals.
5. Explicitly enable inference_provider_spend_breakdown, because the source does not automatically sync newly discovered schemas. Configure the table for full refresh: Brain deletes and recreates a day’s endpoint rows, so an append or incremental-only sync can retain stale endpoints.
6. Join inference_provider_spend_breakdown.inference_provider_spend_id to inference_provider_spend.id. Use endpoint_id, media_type, and amount_cents; preserve OTHER rather than dropping unclassified models.
7. Replace the hard-coded Fal series in tiles 1, 2, and 5 of GPU & Credit Economics | CEO Cockpit. Confirm that insight 6976800 still reconciles after the query change.

Zero-spend safety

The Fal usage API returns usage for the key’s workspace but does not include a workspace identifier in the response. Brain persists a complete zero day when the same requested window also contains positive spend. It skips a fully empty or all-zero window because that response cannot independently prove that the key has the intended workspace scope. Reconcile a genuinely all-zero window in the Fal billing console before repairing or replacing existing spend.

Model catalog failure safety

Brain classifies endpoint spend from the Fal model catalog. If that catalog request fails, the Fal provider writes no snapshots for the attempt rather than guessing a media type from the endpoint name. Alert on Failed to sync spend for FAL before enabling the scheduled sync. When it appears, verify that BRAIN_FAL_ADMIN_KEY can access GET /v1/models, inspect Fal availability and rate limits, then rerun POST /spend/sync for the missed UTC range after recovery. The scheduled lookback, seven days by default, only repairs missed days that are still inside its configured window.

Completion checks

Do not consider the rollout complete until all of these checks pass:

• The breakdown warehouse table reports a successful full refresh.
• Representative FAL totals in Brain and the dashboard match Fal billing to the cent.
• The Brain FAL total for August 2026 is $412,378.12.
• The dashboard updates from warehouse data without a manual Fal export.
• Daily and week-over-week image, video, model, and OTHER trends are visible.
• Monitoring alerts on Failed to sync spend for FAL, and a recovery sync has been tested with replacement bounded to the requested UTC range followed by reconciliation to Fal billing.

Rollback

Unset BRAIN_FAL_ADMIN_KEY in the target environment and redeploy Brain; the Fal adapter then skips spend fetching. Revert the application code through the normal release pipeline, but keep the new tables and historical rows so rollback does not destroy collected financial data. Revert the dashboard query to the last reconciled version if the warehouse feed is inaccurate.