Testing Hive Outages

Testing Hive Outages

How to exercise Brain’s Hive availability blocking (FOXY-1135) end to end on a local stack, on any developer machine. The behavior under test is described in Image And Video Moderation — Proactive Hive Availability Blocking.

mock_moderation_response replaces a Hive call before it is made, so it never feeds the incident tracker. Only a real Hive request that fails trips the block. The local setup makes real Hive requests fail on purpose with apps/flank/qa/hive-fault-proxy/, a Toxiproxy container. Its compose override gives it the network alias api.thehive.ai, so Brain’s Hive traffic goes through its hive proxy (:443, upstream is Hive’s IP resolved through public DNS at start). TLS passes through unchanged.

Toxiproxy’s API listens on 127.0.0.1:8474. Every fault is a call to that API, so the block and its recovery happen inside one running Brain process, without a restart.

Hosted environments are not covered: the faults need either the DNS alias or a Brain environment change, and QA cannot change environment variables there.

What To Expect

Signal	Meaning
Run Completed after a Hive failure, Brain logs provider: gemini	One of the first failures: the request falls back to Gemini. The fifth consecutive outage or auth failure on one adapter trips its block
Run Cancelled with Hive is currently unavailable (<adapter>); retrying automatically.	Outage block (5 consecutive timeouts, 5xx or connection failures on one adapter). A run that would generate a moderated output is cancelled before its first node, so no provider is called
Run Cancelled with Hive authentication is failing repeatedly for <adapter>; its API key needs to be reset.	Auth block (5 consecutive 401/403 responses on one adapter)
Brain log hive_outage_suspected	Outage tripped; logged once per adapter per incident
Brain log hive_outage_recovered	Two successful probes (5s apart) cleared the outage
Brain log hive_auth_blocked	Auth block tripped; there is no automatic recovery
Brain log hive_outage_superseded_by_auth	Probes hit 401s during an outage, so auth took over the block

The block is global: any blocked adapter cancels every gated run. The tracker is in memory, so each Brain replica has its own state, and a restart clears it. Local Brain runs one replica.

The Four QA Workflows

apps/flank/qa/hive-outage.qa-bundle.json holds four workflows. Flank Test runs are sandbox executions: no media is saved and Sirloin is not notified.

Workflow	Hive healthy	While blocked
QA Hive Outage Image Output	Completed	Cancelled with the block reason
QA Hive Outage Carousel Output	Completed	Cancelled; no panel reaches Hive
QA Hive Outage Skip Moderation	Completed	Completed; skipped targets bypass the gate
QA Hive Outage Call Model Gate	Failed with Unsupported inference provider: qa-nonexistent-provider (moderation passed; the provider is deliberately unknown so nothing is generated or billed)	Cancelled before any provider call. If this is the run that trips the block, it ends Failed with the provider error, not Completed

1. Start The Local Stack

You need Docker (Docker Desktop or Colima) running, a clone of dreamworld-research/beef, and the team’s filled-in root .env (ask the Brain owner; it is never committed). All commands run from the repository root.

1. Get the code.

   git fetch origin
   git checkout maintenance/FOXY-1135-hive-brain-integration-healthcheck
   git pull

2. Put the .env in the repository root. Check it has values (not just names) for these keys:

   grep -E '^(BRAIN_HIVE_VISUAL_MODERATION_API_KEY|BRAIN_HIVE_DEMOGRAPHICS_API_KEY|BRAIN_HIVE_CELEBRITY_RECOGNITION_API_KEY|BRAIN_GOOGLE_CLOUD_SERVICE_ACCOUNT_KEY|BRAIN_GOOGLE_CLOUD_PROJECT_ID|BRAIN_S3_BUCKET_NAME|BRAIN_OPENROUTER_API_KEY|STRIP_CLERK_PUBLISHABLE_KEY)=.+' .env | cut -d= -f1

   All eight names must print. A missing Hive key makes Brain skip Hive entirely, and a missing Google Cloud key disables the Gemini fallback.

3. Make Sirloin start. docker-compose.yml passes OPENROUTER_API_KEY only to Brain, and Sirloin refuses to start without it. Add one line to .env with the same value as BRAIN_OPENROUTER_API_KEY:

   OPENROUTER_API_KEY=<same value as BRAIN_OPENROUTER_API_KEY>

4. Stop any other Beef stack you have running, for example from another checkout or worktree. In that checkout run docker compose down (without --volumes, which would delete its database).

5. Tell Docker Compose to use the fault override for every command in this terminal:

   export COMPOSE_FILE=docker-compose.yml:apps/flank/qa/hive-fault-proxy/compose.hive-fault.yml

   Run this again in every new terminal you use for this test. Every docker compose command below relies on it.

6. Build and start everything. The first build takes 10–20 minutes.

   docker compose up -d --build

7. Wait until Brain and Sirloin are healthy. Repeat until both show (healthy):

   docker compose ps brain sirloin

   If sirloin keeps restarting, run docker compose logs --tail 20 sirloin and fix the variable it names in .env, then docker compose up -d sirloin.

8. Check that Brain’s Hive traffic goes through the fault proxy.

   docker compose logs hive-fault-proxy | grep ready
   docker compose exec brain getent hosts api.thehive.ai
   curl -s localhost:8474/proxies/hive

   Expected: the first line reads hive-fault-proxy ready: :443 -> api.thehive.ai (<IP>); the second shows the fault proxy’s container IP (a 172.x.x.x address), not Hive’s; the third prints JSON with "enabled":true.

Optional: work against a copy of staging data instead of an empty database. Follow Neon — Local Neon opt-in, or dump the Neon staging branch into the local rump container with pg_dump from a postgres:18 image (Neon and local rump run Postgres 18; an older pg_dump refuses). The tests below work on an empty database too.

2. Sign In And Import The Workflows

1. Open http://localhost:9960 (local Flank; pnpm dev outside Docker uses 3100 instead). It redirects to Sign in to Strip. Sign in with Continue with Google using your Foxy account.

2. Make your account an admin in the local Brain database. Flank shows only published workflows to other roles, and the imported workflows are drafts:

   docker compose exec rump psql -U sirloin -d fennec \
     -c "update fennec.users set role = 'ADMIN' where email = '<your Foxy email>'"

   Expected: UPDATE 1. UPDATE 0 means Brain has not seen your account yet; reload Flank once and run it again. Reload Flank afterwards.

3. Click Settings (or open http://localhost:9960/settings). Under Import configuration, click Import, then Click to select a .json export file, and choose apps/flank/qa/hive-outage.qa-bundle.json from your checkout.

4. The preview lists 4 workflows to create (or update, on a second import). Click Apply Changes. Expected: Import Complete.

5. Pick two test images. They must be storage keys of SFW images that exist in the bucket named by BRAIN_S3_BUCKET_NAME. On a staging data copy:

   docker compose exec rump psql -U sirloin -d rump -tAc \
     "select image_path from media.media where type = 'MEDIA_TYPE_IMAGE' and status = 'MEDIA_STATUS_AVAILABLE' and is_nsfw = false and deleted_at is null order by created_at desc limit 12"

   On an empty database, copy two image keys from staging Fennec → Media instead.

   Brain caches Hive results per image. Scenarios 2, 5, 8, 10 and 12 need a real Hive call, so give each one a pair of keys not yet used in this session. A first run that finishes in about 2s with no hive_ log line hit the cache; switch to new keys.

6. Build the run input and keep it in a text file; every Test run below uses it, with fresh keys where the note above says so:

   {"image_path": "<first image key>", "image_path_2": "<second image key>"}

7. How to run a workflow (used by every scenario): click Workflows in the sidebar, open the workflow by name, click Test run, paste the JSON into the Test Run Workflow dialog, and click Execute. The result appears in the dialog; Executions in the sidebar lists every run with its status and error.

8. Open a second terminal (run the export COMPOSE_FILE=... line from step 1.5 there too) and leave this running to watch the tracker:

   docker compose logs -f --since 1m brain | grep --line-buffered -E 'hive_(outage|auth)_|"provider":"(hive|gemini)"'

9. Run all four workflows once. They must match the Hive healthy column above, and the log terminal shows "provider":"hive". Do not start the scenarios until this baseline passes.

3. Fault Controls

Run these in the first terminal. Each takes effect at once, without restarting anything except where it says “recreate Brain”.

TOXI=http://localhost:8474/proxies/hive

# outage: connections to Hive are refused
curl -s -X POST $TOXI -d '{"enabled":false}'

# restore Hive
curl -s -X POST $TOXI -d '{"enabled":true}'

# timeout: Hive accepts the connection and never answers
curl -s -X POST $TOXI/toxics -d '{"name":"hang","type":"timeout","stream":"downstream","attributes":{"timeout":0}}'
curl -s -X DELETE $TOXI/toxics/hang

# slow Hive below the 30s request timeout
curl -s -X POST $TOXI/toxics -d '{"name":"slow","type":"latency","stream":"downstream","attributes":{"latency":8000}}'
curl -s -X DELETE $TOXI/toxics/slow

# auth: recreate Brain with an invalid key
BRAIN_HIVE_VISUAL_MODERATION_API_KEY=qa-invalid docker compose up -d --no-deps --force-recreate brain

# recreate Brain with the real key from .env
docker compose up -d --no-deps --force-recreate brain

# after recreating Brain, wait for (healthy) before the next run
docker compose ps brain

4. Scenarios

Run them in order; each one starts from a healthy state. If a step leaves Hive blocked, restore it (enable hive, remove toxics, or recreate Brain with the real key) before the next scenario.

#	Scenario	Steps	Expected
1	Baseline	Run all four workflows	Healthy column of the bundle table
2	Outage	Disable hive, run Image Output until hive_outage_suspected appears (at most 5 runs, fresh image keys each), then all four	Runs before the trip Completed via Gemini, each after about 15s; hive_outage_suspected once per adapter; then the While blocked column, each in about 1s
3	No alert storm	Keep hive disabled for 30s	No further hive_outage_suspected lines; probes keep running
4	Recovery	Enable hive	hive_outage_recovered within about 10s; runs complete again
5	Timeout	Add the hang toxic, run Image Output until it is Cancelled (at most 6 runs, fresh image keys each)	Runs before the trip Completed via Gemini, each after about 70–80s (30s request timeout plus retries within a 60s budget); hive_outage_suspected logged; the next run Cancelled at once
6	Timeout recovery	Remove hang	hive_outage_recovered within about 10s
7	Slow Hive	Add the slow toxic, run Image Output	Completed in about 20s, no tracker event
8	Auth block	Recreate Brain with an invalid key, run Image Output six times	Runs 1–5 Completed via Gemini, hive_auth_blocked after the fifth, run 6 Cancelled with the auth reason; still Cancelled 30s later; no probes
9	Auth recovery	Recreate Brain with the real key	Next run Completed
10	Restart during outage	Disable hive, trip the outage, recreate Brain	State resets: runs after the restart reach Hive again (slow, Gemini fallback) until 5 consecutive failures re-trip; enable hive to recover
11	Restart mid-run	Add hang, start a run, remove hang, recreate Brain within 10s	The in-flight execution reaches a terminal state; a fresh run completes
12	Auth supersedes outage	Recreate Brain with an invalid key, disable hive, trip the outage, enable hive	Probes now get 401s: about 25s later hive_outage_superseded_by_auth then hive_auth_blocked; runs Cancelled with the auth reason

Subworkflow propagation needs a SUBWORKFLOW-kind child, which Flank import cannot create. Create a child workflow of kind Subworkflow that contains a call_model with an image in its payload, call it from a parent through a subworkflow node, and run the parent while blocked: the parent is Cancelled with the Hive reason, not marked superseded.

5. Clean Up

1. curl -s -X POST localhost:8474/proxies/hive -d '{"enabled":true}' and delete any toxics you added.
2. Stop the stack: docker compose down (keeps the database) or docker compose down --volumes (deletes it).
3. unset COMPOSE_FILE so later docker compose commands run without the fault proxy.

Not Covered Here

These paths were not exercised by this setup:

• The legacy InputModerationCommand gate and the Sirloin media notification for a Hive-cancelled run. Both need a real, billed generation from Brisket while the block is active; check that the media ends up failed in Fennec.
• Multi-replica Brain, where each replica keeps its own tracker state.
• Hosted environments (Railway previews, staging), where QA cannot change Brain’s environment.