Creating Replay Vision Scanners

Creating Replay Vision scanners

A scanner is a standing LLM probe over session recordings. Once created and enabled, it runs on a Temporal schedule that sweeps every 5 minutes, applying its prompt to each new matching recording and recording the result as an observation (a queryable $recording_observed event). Each observation spends credits (1 credit = $0.01) from the org's budget for the current billing period, and an observation's price depends on the scanner's model — so budget in credits, not in observation counts.

That schedule is exactly why creation needs a gut-check: a scanner with a permissive query and full sampling starts spending automatically and can drain the whole period's budget within its first few sweeps. Creation itself does not check quota — that protection only kicks in at observation time, by which point the budget may already be gone.

First: is a scanner even the right thing?

A scanner is a standing watch over future recordings. If the user has specific sessions in front of them and a question about those sessions, they don't want a scanner at all — they want vision-scanners-inline-scan, which takes session_ids plus a prompt, saves nothing, and schedules nothing.

Use an inline scan when the sessions are already known: "what went wrong in these five recordings", "did any of yesterday's checkout sessions hit the coupon bug", anything you'd otherwise answer by creating a scanner and deleting it afterwards. It costs the same credits per session and reuses answers when the same question is asked twice, so re-asking is cheap.

Create a scanner only when the user wants recordings that haven't happened yet to be scanned automatically. If you find yourself planning to create a scanner, read its results once, and delete it, stop and run an inline scan instead — a throwaway scanner leaves a scheduled sweep running against every future recording that matches its query.

Core principle: size before you ship

Never create an enabled scanner blind. Estimate its monthly credit spend, check the remaining credit budget, and — when the projected spend is a meaningful fraction of what's left — show the user the numbers and get confirmation before creating. This is the heart of the skill; the rest is supporting detail.

The flow

Step 1: What should the scanner do?

Pick a scanner_type and write its scanner_config. Every type needs a prompt; the rest is type-specific:

scanner_type is locked after creation — to change it you delete and recreate, so confirm the type is right up front, and get the scanner_config shape right (a wrong shape is a create error, not a silent default — unknown keys are rejected too).

If the user's intent makes the type and prompt obvious, just proceed — don't interrogate them. When they only have a goal ("find where users get stuck in onboarding"), vision-scanners-draft drafts the type, config and query for you to review; it saves nothing. For a classifier, vision-scanners-suggest-tags proposes labels grounded in the project's data.

Step 2: Which sessions?

The query is a RecordingsQuery shape that selects which recordings the scanner watches. date_from and date_to are ignored (the schedule controls time), so don't bother setting them. Narrow the query to the sessions that actually matter — by event, URL, person property, duration, etc. A narrow query is the single biggest lever on cost.

When the target is one experiment's exposed population, that's its own job — use the scanning-experiments-with-replay-vision skill, which derives this query from the experiment's exposure criteria instead of hand-building it.

Two levers narrow it further, applied in this order:

• sampling_mode (default comprehensive) is a quality pre-filter on the matched sessions: focused keeps only the top sessions by surfacing score, balanced drops the lowest-quality ones, comprehensive keeps everything. Use it to spend the budget on sessions worth watching rather than shrinking coverage at random.
• sampling_rate (0..1, default 1.0) is a random downsample applied after that. Lower it to trade coverage for budget. Exactly 0 pauses scanning; non-zero rates below 0.0001 are rejected.

Which model?

model sets the price of every observation the scanner makes, so it's a cost lever as much as a quality one: gemini-3.5-flash-lite (2 credits), gemini-3-flash-preview (5 credits, the default) and gemini-3.8-flash (15 credits). Start at the default and only reach for gemini-3.8-flash when the cheaper tiers demonstrably miss what the scanner is looking for.

Step 3: Size it — the gut-check (do not skip)

Before creating, run both checks and reason about them together:

1. Estimate spend — call vision-scanners-estimate with the proposed query, sampling_rate, sampling_mode and model. It returns matched_sessions_in_window, the window_days measured, estimated_observations_per_month, credits_per_observation, estimated_credits_per_month, and other_enabled_scanners_monthly_credits (what the org's other enabled scanners are already projected to spend). When editing an existing scanner, pass its scanner_id so its own estimate isn't counted twice.
2. Check budget — call vision-quota-get for remaining and exhausted against the org's credit_limit (credits, 1 credit = $0.01; null when uncapped), plus the period_start/period_end of the current period.

Compare credits with credits over the same horizon, the way the product UI does — remaining is denominated in credits, not observations, so comparing it against estimated_observations_per_month understates the cost by the model's per-observation price. remaining is what's left for the rest of the current period, so prorate the monthly projection to that window rather than comparing a full month against it:

fleet_monthly    = estimated_credits_per_month + other_enabled_scanners_monthly_credits
period_days      = period_end - period_start        (in days)
days_left        = period_end - now                 (in days, floored at 0)
rest_of_period   = fleet_monthly * days_left / period_days

Then decide on rest_of_period against remaining:

• If it comfortably fits within remaining, proceed.
• If it's a large fraction of (or exceeds) remaining, stop and tell the user the concrete numbers (e.g. "This scanner is projected to spend ~X credits/month, about $Y; over the N days left this period that's ~R credits against the Z you have left."), then confirm before creating. Tightening the query, switching sampling_mode to focused, lowering sampling_rate, or picking a cheaper model are all ways to bring it down.
• Quote estimated_credits_per_month too, since it's what the scanner costs in a full period once this one resets. Mid-period a scanner can fit in remaining and still blow the next period's budget.
• If the org is already exhausted, say so. A new enabled scanner won't produce anything until the budget resets: its scheduled observations are silently skipped, and on-demand scans are rejected outright.
• If the estimate is a large fraction of remaining but the user still wants the scanner, offer a per-scanner cap: set credit_limit on create so this scanner can only ever spend that many credits per billing period. It stops scanning once the credits left can't cover another observation, then resumes when the period resets. Sessions it skipped while capped are not scanned later.

Confirmation here is a conversation step, not an API capability — surface the trade-off and let the user choose. When the projected volume is clearly small relative to the budget, you don't need to ask.

Step 4: Create

Call vision-scanners-create. Minimal example:

{
  "name": "Rage click monitor",
  "scanner_type": "monitor",
  "scanner_config": { "prompt": "Flag sessions where the user repeatedly clicks the same element in frustration." },
  "query": { "kind": "RecordingsQuery", "events": [{ "id": "$rageclick", "type": "events" }] },
  "sampling_rate": 1.0,
  "sampling_mode": "comprehensive",
  "model": "gemini-3-flash-preview",
  "enabled": true
}

name must be unique within the team. Set enabled: false if the user wants to create it paused (no schedule, no quota consumption) and turn it on later.

emits_signals: true is the other switch worth knowing: it augments the prompt with the Signals side mission and pushes one signal per finding into the PostHog Signals inbox, where findings corroborate across sessions into reports. Turn it on when the user wants the scanner to feed their inbox rather than just accumulate observations they have to go read.

After creation

• Show the scanner's PostHog URL from the response so the user can review it in the UI.
• Results take a few minutes to appear (rasterizing the recording to video + the LLM call are slow). Inspect them with vision-scanners-observations-list for one scanner over time, or vision-observations-list (requires session_id) for every scanner's findings on a single session. To dig into a recording, hand off to the investigating-replay skill.
• History is opt-in. A new scanner only sees recordings from now on. To cover the past, run vision-scanners-backfills-estimate for the window, give the user total_sessions and total_credits, and call vision-scanners-backfills-create only once they agree, passing that total_credits as max_total_credits. If the create is rejected because the window now costs more, estimate again and ask again. Watch it with vision-scanners-backfills-get.
• Notifications and digests. vision-alerts-create (plus a destination) notifies on findings; vision-scanners-scouts-create adds a scheduled scout that writes a report about the scanner's findings.
• Say how the scanner gets better. A first prompt is a guess, and the first sweep is what corrects it. Tell the user that rating results thumbs up or down with vision-observations-label-create turns into a config recommendation they can review on the scanner's Calibration tab. Most scanners are never rated, so they run forever on the first guess. The exploring-replay-vision-observations skill covers that loop.

Updating an existing scanner

vision-scanners-update is a partial update — send only changed fields. Re-run the Step 3 gut-check whenever you widen scope or raise the price: a broader query, a higher sampling_rate, a looser sampling_mode, or a pricier model all raise the monthly spend just like a fresh broad scanner would. Toggling enabled, tweaking the prompt, or narrowing the query don't need a re-estimate. Editing config bumps scanner_version; past observations keep a snapshot of the old config.

Gotchas

• One observation per (scanner, session). Re-running a scanner on a session it already observed — even a failed or ineligible one — is a no-op and won't produce a fresh scan. Use vision-observations-retry to replace a failed or ineligible observation.
• Ineligible ≠ failed. Observations can land ineligible (e.g. too_short, no_recording) — a terminal non-error outcome. Check error_reason when triaging why a scanner produced nothing.
• Provider/model are Google/Gemini only in the current version.