Skip to content

Commit

Permalink
Merge pull request RasaHQ#11670 from RasaHQ/metrics-table
Browse files Browse the repository at this point in the history
Added metrics table and categories
  • Loading branch information
m-vdb authored Oct 14, 2022
2 parents 980e974 + 4bd020a commit b2be3a7
Show file tree
Hide file tree
Showing 2 changed files with 40 additions and 3 deletions.
8 changes: 5 additions & 3 deletions docs/docs/monitoring/analytics/example-queries.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@ description:

import useBaseUrl from "@docusaurus/useBaseUrl";

This section helps you to started with ways how to analyze
the conversations your assistant has. The examples use SQL queries
together with an example visualization in Metabase.
This section helps you get started with analyzing your assistant's conversations.
The examples use SQL queries together with an example visualization in Metabase.

For more metrics and categories of conversations,
see [Types of metrics](./getting-started-with-analytics#types-of-metrics).


<!--
Expand Down
35 changes: 35 additions & 0 deletions docs/docs/monitoring/analytics/getting-started-with-analytics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,41 @@ The pipeline will compute analytics as soon as the docker container
is successfully deployed and connected to your data warehouse and
Kafka instances.

## Types of metrics

Metrics collected from your assistant can broadly be categorized as

- User Analytics: Who are the users of the assistant, and how do they feel
about it? Examples: demographics, channels, sentiment analysis
- Usage Analytics: How is the assistant’s overall health and what kind of
traffic is coming to it? Examples: total number of sessions, time per
session, errors and error rates
- Conversation Analytics: What happened during the conversation?
Examples: number of messages sent, abandonment depth, number of topics
introduced by user, top N intents
- Business Analytics: How is the assistant performing with regard to business goals?
Examples: ROI of assistant per LoB, time comparison of assistant vs agent, containment rate

In this version of the Analytics pipeline, measurement of the following metrics
is possible

| Metric | Category | Meaning |
|--------|----------|---------|
| Number of conversations| Usage Analytics | Total number of conversations |
| Number of users | Usage Analytics | Total number of users |
| Number of sessions | Usage Analytics | Gross traffic to assistant |
| Channels used | Usage Analytics | Sessions by channel |
| User session count | User Analytics | Total number of user sessions or average sessions per user |
| Top N intents | Conversation Analytics | Top intents across all users |
| Avg response time | Conversation Analytics | Average response time for assistant |
| Containment rate | Business Analytics | % of conversations handled purely by assistant (not handed to human agent |
| Abandonment rate | Business Analytics | % of abandoned conversations |
| Escalation rate | Business Analytics | % of conversations escalated to human agent |


For examples of how you can extract these metrics,
see [Example queries](./example-queries).

## Prerequisites

- A production deployment of Kafka is required to set up Rasa Pro.
Expand Down

0 comments on commit b2be3a7

Please sign in to comment.