Skip to content

docs(replay-vision): document scanner impact tracking and cohort creation #18625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
inkeep wants to merge 6 commits into
base: master
Choose a base branch
from
+109 −37
Open

docs(replay-vision): document scanner impact tracking and cohort creation #18625

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b550ffc
docs(replay-vision): add impact tracking and cohort creation to scann…
inkeep[bot] Jul 17, 2026
68a1f73
docs(replay-vision): add scanner impact and cohorts section to observ…
inkeep[bot] Jul 17, 2026
5b5a50e
docs(replay-vision): add impact and cohort MCP tools to MCP page
inkeep[bot] Jul 17, 2026
481a495
docs(replay-vision): add impact and cohort MCP tools to mcp.mdx
inkeep[bot] Jul 17, 2026
0249f47
docs(cohorts): add Method 4 for creating cohorts from Replay Vision s…
inkeep[bot] Jul 17, 2026
13497a8
fix(docs): remove duplicate 'Impact and cohorts' section in Replay Vi…
inkeep[bot] Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions contents/docs/data/cohorts.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Groups require additional code in your app to set up, while cohorts are created

## How to create a cohort

Once you've captured identified events, there are three ways to create a new cohort:
Once you've captured identified events, there are several ways to create a new cohort:

### Method 1: Create a new cohort directly from an insight

Expand Down Expand Up @@ -132,6 +132,18 @@ Example prompts:

See the [MCP documentation](/docs/model-context-protocol) for setup instructions and the full list of available tools.

### Method 4: Create a cohort from Replay Vision scanner results

If you use [Replay Vision](/docs/replay-vision), you can save the users affected by a scanner's findings as a static cohort. This is useful for targeting those users with surveys, checking their retention, or excluding them from experiments.

1. Go to [Replay Vision](https://app.posthog.com/replay-vision) and open any scanner.

2. In the impact panel, click **Save as cohort** (for monitors) or click the cohort button next to a specific tag (for classifiers).

The cohort is a dated snapshot: it captures the users matched in the trailing window (default 30 days) and won't live-update as new observations come in. Calling it again creates a new cohort. See [scanner types](/docs/replay-vision/scanner-types) for details on what counts as "affected" for each type.

> **Note:** Cohort creation from scanners is capped at 10,000 users per request. Scorers require a score threshold, and summarizers don't support impact cohorts.

## Static and dynamic cohorts

You have two options when creating a cohort:
Expand All @@ -142,7 +154,7 @@ You have two options when creating a cohort:

### Static cohorts

Static cohorts are created from insights, by uploading a CSV of users, by selecting users individually, by duplicating a dynamic cohort, or by defining criteria for a one-time snapshot.
Static cohorts are created from insights, by uploading a CSV of users, by selecting users individually, by duplicating a dynamic cohort, by defining criteria for a one-time snapshot, or from [Replay Vision](/docs/replay-vision) scanner results.

When creating cohorts from CSVs, the [person profile](/docs/data/persons) associated with each row must already exist in PostHog.

Expand Down
8 changes: 8 additions & 0 deletions contents/docs/replay-vision/mcp.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,22 @@ For one scanner (across sessions):
- `vision-scanners-observations-list` – paginated list of a scanner's observations.
- `vision-scanners-observations-get` – fetch one of that scanner's observations by ID.

### Impact and cohorts

- `vision-scanners-impact-retrieve` – get affected sessions and users for a scanner over a trailing window. Monitors count verdict-yes observations. Classifiers require a `tag` parameter. Scorers require `min_score` and/or `max_score`.
- `vision-scanners-affected-cohort-create` – save the affected users as a static cohort. Same qualifiers as the impact endpoint. The cohort is a dated snapshot and doesn't live-update.

## Example prompts

Try these with your MCP-enabled agent:

- `How many users were affected by my "Dead-end pages" monitor in the last 30 days?`
- `Create a cohort from the users my "User intent" classifier tagged as support_seeking.`
- `Create a Replay Vision scanner that flags sessions where users get stuck on the /checkout page, narrow to enterprise customers, and start at 25% sampling. Estimate the cost first.`
- `Scan session abc123 with the "Dead-end pages" scanner and tell me what it found, including the reasoning.`
- `List the last 20 observations from my "Frustration score" scanner and summarize what's driving high scores.`
- `Find every Replay Vision observation for session abc123 and give me a one-line summary of each.`
- `How much Replay Vision quota do we have left this month?`
- `How many users did my "Dead-end pages" monitor affect in the last 14 days? Save them as a cohort.`

See the [MCP server docs](/docs/model-context-protocol) for client setup, and the [Replay Vision overview](/docs/replay-vision) for product concepts.
71 changes: 47 additions & 24 deletions contents/docs/replay-vision/observations.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,29 +36,29 @@ Every successful observation is captured as a `$recording_observed` event in you

### Event properties

| Property | Description |
| --- | --- |
| `scanner_name` | Human-readable name of the scanner |
| `scanner_id` | Scanner UUID |
| `scanner_type` | `monitor`, `classifier`, `scorer`, or `summarizer` |
| `scanner_version` | Version of the scanner config that produced this observation |
| `session_id` | The recording that was observed |
| `triggered_by` | `schedule` (background sweep) or `on_demand` |
| `triggered_by_user_id` | The user who triggered an on-demand observation; null for background sweeps |
| `model_used` | The model that produced the result |
| `provider_used` | The LLM provider behind `model_used` |
| `scanner_output_confidence` | The model's confidence (0.0 to 1.0) |
| `scanner_output_reasoning` | **Monitor, classifier, and scorer** – the model's reasoning text, including any inline citations. Summarizers don't have a separate reasoning field |
| `scanner_output_verdict` | **Monitor only** – `"yes"`, `"no"`, or `"inconclusive"` |
| `scanner_output_tags` | **Classifier only** – the array of vocabulary tags assigned |
| `scanner_output_tags_freeform` | **Classifier only** – tags outside the vocabulary, when the scanner allows them |
| `scanner_output_score` | **Scorer only** – the numeric score |
| `scanner_output_title` | **Summarizer only** – the generated one-line title |
| `scanner_output_summary` | **Summarizer only** – the generated prose summary |
| `scanner_output_intent` | **Summarizer only** – one sentence on what the user set out to do |
| `scanner_output_outcome` | **Summarizer only** – one sentence on how the session ended |
| `scanner_output_friction_points` | **Summarizer only** – array of named blockers or frustrations (empty if none) |
| `scanner_output_keywords` | **Summarizer only** – array of lowercase keywords describing the session |
| Property | Description |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scanner_name` | Human-readable name of the scanner |
| `scanner_id` | Scanner UUID |
| `scanner_type` | `monitor`, `classifier`, `scorer`, or `summarizer` |
| `scanner_version` | Version of the scanner config that produced this observation |
| `session_id` | The recording that was observed |
| `triggered_by` | `schedule` (background sweep) or `on_demand` |
| `triggered_by_user_id` | The user who triggered an on-demand observation; null for background sweeps |
| `model_used` | The model that produced the result |
| `provider_used` | The LLM provider behind `model_used` |
| `scanner_output_confidence` | The model's confidence (0.0 to 1.0) |
| `scanner_output_reasoning` | **Monitor, classifier, and scorer** – the model's reasoning text, including any inline citations. Summarizers don't have a separate reasoning field |
| `scanner_output_verdict` | **Monitor only** – `"yes"`, `"no"`, or `"inconclusive"` |
| `scanner_output_tags` | **Classifier only** – the array of vocabulary tags assigned |
| `scanner_output_tags_freeform` | **Classifier only** – tags outside the vocabulary, when the scanner allows them |
| `scanner_output_score` | **Scorer only** – the numeric score |
| `scanner_output_title` | **Summarizer only** – the generated one-line title |
| `scanner_output_summary` | **Summarizer only** – the generated prose summary |
| `scanner_output_intent` | **Summarizer only** – one sentence on what the user set out to do |
| `scanner_output_outcome` | **Summarizer only** – one sentence on how the session ended |
| `scanner_output_friction_points` | **Summarizer only** – array of named blockers or frustrations (empty if none) |
| `scanner_output_keywords` | **Summarizer only** – array of lowercase keywords describing the session |

The `timestamp` on the event is the moment the observation completed, not the moment the underlying recording was captured.

Expand Down Expand Up @@ -188,6 +188,29 @@ ORDER BY timestamp DESC

Confidence is self-reported, so this is a heuristic, not a guarantee – but it's often a useful filter for downstream automation.

## Scanner impact and cohorts

Each scanner tracks its impact: how many sessions and users its findings affected over a trailing window (default 30 days). These counts appear in the scanner overview panel.

What counts as "affected" depends on the scanner type:

| Scanner type | What counts as affected |
| -------------- | ------------------------------------------------------------------------------ |
| **Monitor** | Sessions where the verdict was `yes` |
| **Classifier** | Sessions tagged with a specific tag (tracked per-tag) |
| **Scorer** | Sessions scoring within a `min_score` and/or `max_score` threshold you specify |
| **Summarizer** | Not supported (narratives don't have categorical findings) |

### Creating cohorts from scanner results

You can save a scanner's affected users as a static [cohort](/docs/data/cohorts) with one click. The cohort is a snapshot at creation time and won't live-update as new observations come in.

For monitors, click **Save as cohort** in the impact panel. For classifiers, each tag has its own cohort button.

Cohort creation is capped at 10,000 users per request. If your scanner has more affected users, consider narrowing the trailing window or creating multiple cohorts over different time ranges.

Use these cohorts anywhere cohorts work in PostHog: funnel breakdowns, retention analysis, survey targeting, or experiment exclusion.

## Wiring observations into the rest of PostHog

### Insights and dashboards
Expand All @@ -206,7 +229,7 @@ The recipes above all become [insights](/docs/product-analytics/insights) by sav

### Joining to other PostHog data

Because `$recording_observed` events live alongside everything else in your project, you can join them to anything – pageviews, custom events, person properties, cohorts. A useful pattern: find the events the user fired *before* the moment a monitor flagged them, grouped by URL.
Because `$recording_observed` events live alongside everything else in your project, you can join them to anything – pageviews, custom events, person properties, cohorts. A useful pattern: find the events the user fired _before_ the moment a monitor flagged them, grouped by URL.

```sql
SELECT
Expand Down
Loading
Loading