Virtual Assistant will require additional telemetry logged from the Bot Framework. This document attempts to describe the changes required. The schema for the events is from the Enterprise Template, the NEW tags indicate new columns/fields that need to be added. Based on the Virtual Assistant Bot Analytics documentation.
The elements striked through below are logged, but not consumed for the Virtual Assistant reports.
A new Application Insights Telemetry Initializer will be added in v4.2 Bot Framework SDK. Assume the UserID and ConversationID and ActivityID reside in all customEvents. These will manifest as user_id/session_id within the schema.
user_id= ChannelID + From.Idsession_id= ConversationIDcustomDimensions.activityId= The bot activity IDcustomDimensions.activityType= The bot activity typecustomDimensions.channelId= The bot activity channel ID
Logged From: TelemetryLoggerMiddleware (Enterprise Sample)
Logged when bot receives new message.
- UserID (From Telemetry Initializer)
- ConversationID (From Telemetry Initializer)
- ActivityID (From Telemetry Initializer)
- Channel (From Telemetry Initializer)
- ActivityType (From Telemetry Initializer)
- Text (Optional for PII)
- FromId
- FromName
- RecipientId
- RecipientName
- ConversationId
- ConversationName
- Locale
- NEW Geolocation per activity spec
Logged From: TelemetryLoggerMiddleware (Enterprise Sample)
Logged when bot sends a message.
- UserID (From Telemetry Initializer)
- ConversationID (From Telemetry Initializer)
- ActivityID (From Telemetry Initializer)
- Channel (From Telemetry Initializer)
- ActivityType (From Telemetry Initializer)
- ReplyToID
- Channel (Source channel - e.g. Skype, Cortana, Teams)
- RecipientId
- ConversationName
- Locale
- Text (Optional for PII)
- RecipientName (Optional for PII)
### CustomEvent: BotMessageUpdate
Logged From: TelemetryLoggerMiddleware
Logged when a message is updated by the bot (rare case)
### CustomEvent: BotMessageDelete
Logged From: TelemetryLoggerMiddleware
Logged when a message is deleted by the bot (rare case)
Logged From: TelemetryLuisRecognizer (Enterprise Sample)
Logs results from LUIS service.
- UserID (From Telemetry Initializer)
- ConversationID (From Telemetry Initializer)
- ActivityID (From Telemetry Initializer)
- Channel (From Telemetry Initializer)
- ActivityType (From Telemetry Initializer)
- Intent
- IntentScore
- Question
- ConversationId
- SentimentLabel
- SentimentScore
- LUIS entities
- NEW DialogId
Logged From: TelemetryQnaMaker (Enterprise Sample)
Logs results from QnA Maker service.
- UserID (From Telemetry Initializer)
- ConversationID (From Telemetry Initializer)
- ActivityID (From Telemetry Initializer)
- Channel (From Telemetry Initializer)
- ActivityType (From Telemetry Initializer)
- Username
- ConversationId
- OriginalQuestion
- Question
- Answer
- Score (Optional: if we found knowledge)
NEW
Note: Steps numbers may be skipped if no prompt is performed within the step.
Logged From: SDK when TelemetryClient configured
Logs individual steps from a Waterfall Dialog.
- UserID (From Telemetry Initializer)
- ConversationID (From Telemetry Initializer)
- ActivityID (From Telemetry Initializer)
- Channel (From Telemetry Initializer)
- ActivityType (From Telemetry Initializer)
- DialogId (This is the dialogId (string) passed into your Waterfall. You can consider this the "waterfall type")
- StepName (either method name or
StepXofYif lambda) - InstanceID (unique per instance of the dialog)
NEW
Logged From: SDK when TelemetryClient configured
Logs when a Waterfall Dialog completes.
- UserID (From Telemetry Initializer)
- ConversationID (From Telemetry Initializer)
- ActivityID (From Telemetry Initializer)
- Channel (From Telemetry Initializer)
- ActivityType (From Telemetry Initializer)
- DialogId (This is the dialogId (string) passed into your Waterfall. You can consider this the "waterfall type")
- InstanceID (unique per instance of the dialog)
NEW
Logged From: SDK when TelemetryClient configured
Logs when a Waterfall Dialog is canceled.
- UserID (From Telemetry Initializer)
- ConversationID (From Telemetry Initializer)
- ActivityID (From Telemetry Initializer)
- Channel (From Telemetry Initializer)
- ActivityType (From Telemetry Initializer)
- DialogId (This is the dialogId (string) passed into your Waterfall. You can consider this the "waterfall type")
- StepName - The step name (either the function name of
StepXofYif lambda) that the Waterfall was currently progressed to. - InstanceID (unique per instance of the dialog)
### CustomEvent: "Activity"
Logged From: Channel Service
Logged by the Channel Service when a message received.
### Exception: Bot Errors
Logged From: Channel Service
Logged by the channel when a call to the Bot returns a non-2XX Http Response.
| Item | Type | Description |
|---|---|---|
| Number of users who have enabled or registered VA | User Activity | Assumption: The registration process contains a Waterfall dialog. This will be the count of events for the WaterfallDialogConvert event |
| Number of Active (Engaged) VA users | User Activity | Distinct count of BotMessageReceived events on UserID |
| Number of Anonymous Users | User Activity | Distinct count of BotMessageReceived events on UserID minus Registered Users. |
| Number of new registerations | User Activity | Count of WaterfallDialogConvert events for the registeration dialog. |
| Average number of sessions per user | Engagment | BotMessageReceived : UserID by SessionID |
| Average time spent per session | Engagement | Based on BotMessageReceived |
| Average # of skills per person | Engagement | Based on WaterfallDialogStep distinct on UserID |
| Average number of interactions per user | Engagement | Based on BotMessageReceived |
| Number of Skills (per Region/Language/etc) | Engagement | Based on BotMessageReceived |
| Number of abandoned/cancelled Skills (last N sessions) | Churn | Based on WaterfallDialogCancel |
| Common (high frequency) abandoned/cancelled skills | Churn | WaterfallDialogStep / WaterfallDialogCancel |
| Skills with high interaction acount | Churn | WaterfallDialogStep |
| Item | Type | Description |
|---|---|---|
| Most popular/least popular skills | Skills Insight | WaterfallDialogStep |
| Skills take a long time to complete | Skills Insight | WaterfallDialogStep / WaterfallDialogComplete |
| Average number of Interations per skill | Skills Insight | WaterfallDialogStep / WaterfallDialogComplete |
| Duration of each stage | Skills Insight | WaterfallDialogStep |
| Skills that take higher interactions | Skills Insight | WaterfallDialogStep |
| Count user OK with recommendations | Skills Insight | ? Custom Event |
| Missed / retried utterance | Skills Insight | LuisIntent.INENTName |
| Skill successfully handled by IPA | Skills Insight | WaterfallDialogConvert |
| Skill transferred to Agent | Skills Insight | ? Custom Event |
| Number of times Skill Completed/Cancelled/Abandoned | Skills Insight | WaterfallDialogCancel / WaterfallDialogComplete / WaterfallDialogStep |
| Cancel / Abandon Correlation: Duration | Skills Insight | WaterfallDialogStep |
| Cancel / Abandon Correlation: Too many messages | Skills Insight | WaterfallDialogStep |
| Cancel / Abandon Correlation: Retry | Skills Insight | WaterfallDialogStep |
| Cancel / Abandon Correlation: Not happy | Skills Insight | Sentiment LuisIntent.INENTName |
*PRI2 Not in scope for 4.2
| Item | Type | Description |
|---|---|---|
| Identify user unsubscribe | Churn | ? Custom Event for Unsubscribe? Otherwise look-alike model |
| Identify user abandon | Churn | ? Assume abandon defined by duration. Look-alike model |
| Identify low consumption | Churn | Look-alike model |
| User preference | Recommendations | ? LUIS entities / intents? Look-alike? Need more information |
| Buy new product in future | Sales/Marketing | LUIS entities / Qna? |
- Be able to override IBotTelemetryClient (NullBotTelemetryCiient)
- Custom TelemetryClient (from override IBotTelemetryClient)
- Be able to override ILogger
- Initialize with Bot Configuration