Skip to content

Commit 2ae9dc3

Browse files
KaiqbKaiqb
authored
QnA-Telemtry (MicrosoftDocs#1615) (#1869)
1 parent a4a437a commit 2ae9dc3

File tree

5 files changed

+393
-2
lines changed

5 files changed

+393
-2
lines changed

‎articles/TOC.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@
5252
### [Add authentication to your bot](v4sdk/bot-builder-authentication.md)
5353
### [Implement custom storage for your bot](v4sdk/bot-builder-custom-storage.md)
5454
### [Add telemetry to your bot](v4sdk/bot-builder-telemetry.md)
55+
### [Add telemetry to your QnA bot](v4sdk/bot-builder-telemetry-QnAMaker.md)
5556
### [Use Direct Line Speech in your bot](directline-speech-bot.md)
5657
### [.NET](dotnet/TOC.md)
5758
### [Node.js](nodejs/TOC.md)

‎articles/v4sdk/bot-builder-telemetry-qnamaker.md‎

Lines changed: 388 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,388 @@
1+
---
2+
title: Add telemetry to your QnA bot | Microsoft Docs
3+
description: Learn how to integrate the new telemetry features into your QnA Maker enabled bot.
4+
keywords: telemetry, appinsights, Application Insights, monitor bot, QnA Maker
5+
author: WashingtonKayaker
6+
ms.author: kamrani
7+
manager: kamrani
8+
ms.topic: article
9+
ms.service: bot-service
10+
ms.date: 07/31/2019
11+
monikerRange: 'azure-bot-service-4.0'
12+
---
13+
14+
# Add telemetry to your QnAMaker bot
15+
16+
[!INCLUDE[applies-to](../includes/applies-to.md)]
17+
18+
19+
Telemetry logging was added to version 4.2 of the Bot Framework SDK. This enables bot applications to send event data to telemetry services such as [Application Insights](https://aka.ms/appinsights-overview). Telemetry offers insights into your bot by showing which features are used the most, detects unwanted behavior and offers visibility into availability, performance, and usage.
20+
21+
Two new components were added to the Bot Framework SDK that enable telemetry logging in QnA Maker enabled bots: `TelemetryLoggerMiddleware` and the `QnAMaker` class. `TelemetryLoggerMiddleware` is a middleware component that logs every time messages are received, sent, updated, or deleted, and the 'QnAMaker' class provides custom logging that extends telemetry capabilities.
22+
23+
In this article you will learn about:
24+
25+
* The code required to wire up telemetry in your bot
26+
27+
* The code required to enable the out-of-the-box QnA logging and reports that use the standard event properties.
28+
29+
* Modifying or extending the SDK's default event properties to enable a wide range of reporting needs.
30+
31+
32+
## Prerequisites
33+
34+
* The [QnA Maker sample code](https://aka.ms/cs-qna)
35+
36+
* A subscription to [Microsoft Azure](https://portal.azure.com/)
37+
38+
* An [Application Insights key](../bot-service-resources-app-insights-keys.md)
39+
40+
* Familiarity with [QnA Maker](https://qnamaker.ai/) is helpful.
41+
42+
* A [QnA Maker](https://aka.ms/create-qna-maker) account.
43+
44+
* A published QnA Maker knowledge base. If you do not have one, follow the steps in [Create and answer from KB](https://aka.ms/create-publish-query-in-portal) tutorial to create a QnA Maker knowledge base with questions and answers.
45+
46+
> [!NOTE]
47+
> This article will build on the [QnA Maker sample code](https://aka.ms/cs-qna) by stepping you through the steps required to incorporate telemetry.
48+
49+
## Wiring up telemetry in your QnA Maker bot
50+
51+
We will start with the [QnA Maker sample app](https://aka.ms/cs-qna) and add the code required to integrate telemetry into a bot using the QnA service. This will enable Application Insights to begin tracking requests.
52+
53+
1. Open the [QnA Maker sample app](https://aka.ms/cs-qna) in Visual Studio
54+
55+
2. Add the `Microsoft.Bot.Builder.Integration.ApplicationInsights.Core ` NuGet package. For more information on using NuGet, see [Install and manage packages in Visual Studio](https://aka.ms/install-manage-packages-vs):
56+
57+
3. Include the following statements in `Startup.cs`:
58+
```csharp
59+
using Microsoft.ApplicationInsights.Extensibility;
60+
using Microsoft.Bot.Builder.ApplicationInsights;
61+
using Microsoft.Bot.Builder.Integration.ApplicationInsights.Core;
62+
```
63+
64+
> [!NOTE]
65+
> If you're following along by updating the QnA Maker sample code you will notice that the using statement for `Microsoft.Bot.Builder.Integration.AspNet.Core` already exists in the QnA Maker sample.
66+
67+
4. Add the following code to the `ConfigureServices()` method in `Startup.cs`. This makes telemetry services available to your bot via [dependency injection (DI)](https://aka.ms/asp.net-core-dependency-interjection):
68+
```csharp
69+
// This method gets called by the runtime. Use this method to add services to the container.
70+
public void ConfigureServices(IServiceCollection services)
71+
{
72+
...
73+
// Create the Bot Framework Adapter with error handling enabled.
74+
services.AddSingleton<IBotFrameworkHttpAdapter, AdapterWithErrorHandler>();
75+
76+
// Add Application Insights services into service collection
77+
services.AddApplicationInsightsTelemetry();
78+
79+
// Add the standard telemetry client
80+
services.AddSingleton<IBotTelemetryClient, BotTelemetryClient>();
81+
82+
// Create the telemetry middleware to track conversation events
83+
services.AddSingleton<TelemetryLoggerMiddleware>();
84+
85+
// Add the telemetry initializer middleware
86+
services.AddSingleton<IMiddleware, TelemetryInitializerMiddleware>();
87+
88+
// Add telemetry initializer that will set the correlation context for all telemetry items
89+
services.AddSingleton<ITelemetryInitializer, OperationCorrelationTelemetryInitializer>();
90+
91+
// Add telemetry initializer that sets the user ID and session ID (in addition to other bot-specific properties, such as activity ID)
92+
services.AddSingleton<ITelemetryInitializer, TelemetryBotIdInitializer>();
93+
...
94+
}
95+
```
96+
97+
> [!NOTE]
98+
> If you are following along by updating the QnA Maker sample code you will notice that `services.AddSingleton<IBotFrameworkHttpAdapter, AdapterWithErrorHandler>();` already exists.
99+
100+
5. Instruct the adapter to use the middleware code that was added to the `ConfigureServices()` method. Open `AdapterWithErrorHandler.cs` and add `IMiddleware middleware` to the constructors parameter list. Add the `Use(middleware);` statement as the last line in the contructor:
101+
```csharp
102+
public AdapterWithErrorHandler(ICredentialProvider credentialProvider, ILogger<BotFrameworkHttpAdapter> logger, IMiddleware middleware, ConversationState conversationState = null)
103+
: base(credentialProvider)
104+
{
105+
...
106+
107+
Use(middleware);
108+
}
109+
```
110+
111+
6. Add the Application Insights instrumentation key in your `appsettings.json` file. The `appsettings.json` file contains metadata about external services the Bot uses while running. For example, CosmosDB, Application Insights and the QnA Maker service connection and metadata is stored there. The addition to your `appsettings.json` file must be in this format:
112+
113+
```json
114+
{
115+
"MicrosoftAppId": "",
116+
"MicrosoftAppPassword": "",
117+
"QnAKnowledgebaseId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
118+
"QnAEndpointKey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
119+
"QnAEndpointHostName": "https://xxxxxxxx.azurewebsites.net/qnamaker",
120+
"ApplicationInsights": {
121+
"InstrumentationKey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
122+
}
123+
}
124+
```
125+
> [!Note]
126+
>
127+
> * Details on getting the _Application Insights instrumentation key_ can be found in the article [Application Insights keys](../bot-service-resources-app-insights-keys.md).
128+
>
129+
> * You should already have a [QnA maker account](https://aka.ms/create-qna-maker), if needed you can find information on getting the QnA Knowledgebase Id, Endpoint Key and HostName values [here](https://aka.ms/bot-framework-emulator-qna-keys).
130+
>
131+
132+
133+
At this point the preliminary work to enable telemetry using Application Insights is done. You can run your bot locally using the bot emulator and then go into Application Insights to see what is being logged such as response time, overall app health, and general running information.
134+
135+
> [!TIP]
136+
> For information on Enabling / disabling activity event and personal information logging see [Add telemetry to your bot](bot-builder-telemetry.md#enabling-or-disabling-activity-logging)
137+
138+
Next we will see what needs to be included to add telemetry functionality to the QnA Maker service.
139+
140+
141+
## Enabling telemetry to capture usage data from the QnA Maker service
142+
143+
The QnA Maker service has built-in telemetry logging available so there is very little you need to do to start getting telemetry data from QnA Maker. First we will see how to incorporate telemetry into the QnA Maker code to enable the built-in telemetry logging, then we will learn how to replace or add additional properties to the existing event data to satisfy a wide range of reporting needs.
144+
145+
### Enabling default QnA logging
146+
147+
1. Create a private readonly field of type `IBotTelemetryClient` in your `QnABot` class in `QnABot.cs`:
148+
149+
```cs
150+
public class QnABot : ActivityHandler
151+
{
152+
private readonly IBotTelemetryClient _telemetryClient;
153+
...
154+
}
155+
```
156+
157+
2. Add an `IBotTelemetryClient` parameter to your `QnABot` class constructor in `QnABot.cs` and assign its value to the private field created in the previous step:
158+
159+
```cs
160+
public QnABot(IConfiguration configuration, ILogger<QnABot> logger, IHttpClientFactory httpClientFactory, IBotTelemetryClient telemetryClient)
161+
{
162+
...
163+
_telemetryClient = telemetryClient;
164+
}
165+
```
166+
167+
3. The _`telemetryClient`_ parameter is required when instantiating the new QnAMaker object in `QnABot.cs`:
168+
169+
```cs
170+
var qnaMaker = new QnAMaker(new QnAMakerEndpoint
171+
{
172+
KnowledgeBaseId = _configuration["QnAKnowledgebaseId"],
173+
EndpointKey = _configuration["QnAEndpointKey"],
174+
Host = _configuration["QnAEndpointHostName"]
175+
},
176+
null,
177+
httpClient,
178+
_telemetryClient);
179+
```
180+
181+
> [!TIP]
182+
> Make sure that the property names that you use in the `_configuration` entries match the property names you used in the AppSettings.json file and the values for those properties are obtained by selecting the _View Code_ button in https://www.qnamaker.ai/Home/MyServices:
183+
184+
185+
![AppSettings](media/AppSettings.json-QnAMaker.png)
186+
187+
#### Viewing Telemetry data logged from the QnA Maker default entries
188+
You can view the results of your QnA Maker bot usage in Application Insights after running your bot in the bot emulator by taking the following steps :
189+
190+
1. Go to the [Azure portal](https://portal.azure.com/)
191+
192+
2. Navigate to your Application Insights by clicking on __Monitor > Applications__.
193+
194+
3. Once in your Application Insights, click on _Logs (Analytics)_ on the navigation bar as shown below:
195+
196+
![Log Analytics](media/AppInsights-LogView-QnaBot.png)
197+
198+
4. Enter the following Kusto query and then select _Run_
199+
200+
```SQL
201+
customEvents
202+
| where name == 'QnaMessage'
203+
| extend answer = tostring(customDimensions.answer)
204+
| summarize count() by answer
205+
```
206+
5. Leave this page open in your browser, we will come back to it after adding a new custom property.
207+
208+
> [!TIP]
209+
> If you are new to the Kusto query language that is used to write log queries in Azure Monitor, but are familiar with SQL query language, you may find the [SQL to Azure Monitor log query cheat sheet](https://aka.ms/azureMonitor-SQL-cheatsheet) useful.
210+
211+
### Modifying or extending the default event properties
212+
If you need properties that are not defined in the `QnAMaker` class there are two ways of handling this, both require creating your own class derived from the `QnAMaker` class. The first is explained in the section below titled [Adding properties](#adding-properties) in which you add properties to the existing `QnAMessage` event. The second method allows you to create new events to which you can add properties as described in [Adding new events with custom properties](#adding-new-events-with-custom-properties).
213+
214+
> [!Note]
215+
> The `QnAMessage` event is part of the Bot Framework SDK and provides all of the out-of-the-box event properties that are logged to Application Insights.
216+
217+
218+
219+
#### Adding properties
220+
221+
The following demonstrates how you can derive from the `QnAMaker` class. The example shows adding the property "MyImportantProperty" to the `QnAMessage` event. The `QnAMessage` event is logged every time a QnA [GetAnswers](https://aka.ms/namespace-QnAMaker-GetAnswersAsync) call is performed.
222+
223+
After learning how to add custom properties we will learn how to create a new custom event and associate properties with it, then we will run the bot locally using the Bot Framework Emulator and see what is being logged in Application Insights using the Kusto query language.
224+
225+
1. Create a new class named `MyQnAMaker` in the `Microsoft.BotBuilderSamples` namespace that inherits from the `QnAMaker` class and save it as `MyQnAMaker.cs`. In order to inherit from the `QnAMaker` class you will need to add the `Microsoft.Bot.Builder.AI.QnA` using statement. Your code should appear as follows:
226+
227+
228+
```cs
229+
using Microsoft.Bot.Builder.AI.QnA;
230+
231+
namespace Microsoft.BotBuilderSamples
232+
{
233+
public class MyQnAMaker : QnAMaker
234+
{
235+
236+
}
237+
}
238+
```
239+
2. Add a class constructor to `MyQnAMaker`. Note that you will need two additional using statements for the constructors parameters `System.Net.Http` and `Microsoft.Bot.Builder`:
240+
241+
```cs
242+
...
243+
using Microsoft.Bot.Builder.AI.QnA;
244+
using System.Net.Http;
245+
using Microsoft.Bot.Builder;
246+
247+
namespace Microsoft.BotBuilderSamples
248+
{
249+
public class MyQnAMaker : QnAMaker
250+
{
251+
public MyQnAMaker(
252+
QnAMakerEndpoint endpoint,
253+
QnAMakerOptions options = null,
254+
HttpClient httpClient = null,
255+
IBotTelemetryClient telemetryClient = null,
256+
bool logPersonalInformation = false)
257+
: base(endpoint, options, httpClient, telemetryClient, logPersonalInformation)
258+
{
259+
260+
}
261+
}
262+
}
263+
```
264+
3. Add the new property to the QnAMessage event after the constructor and include the statements `System.Collections.Generic`, `System.Threading`, and `System.Threading.Tasks`:
265+
266+
```cs
267+
using Microsoft.Bot.Builder.AI.QnA;
268+
using System.Net.Http;
269+
using Microsoft.Bot.Builder;
270+
using System.Collections.Generic;
271+
using System.Threading;
272+
using System.Threading.Tasks;
273+
274+
namespace Microsoft.BotBuilderSamples
275+
{
276+
public class MyQnAMaker : QnAMaker
277+
{
278+
...
279+
280+
protected override async Task OnQnaResultsAsync(
281+
QueryResult[] queryResults,
282+
Microsoft.Bot.Builder.ITurnContext turnContext,
283+
Dictionary<string, string> telemetryProperties = null,
284+
Dictionary<string, double> telemetryMetrics = null,
285+
CancellationToken cancellationToken = default(CancellationToken))
286+
{
287+
var eventData = await FillQnAEventAsync(
288+
queryResults,
289+
turnContext,
290+
telemetryProperties,
291+
telemetryMetrics,
292+
cancellationToken)
293+
.ConfigureAwait(false);
294+
295+
// Add new property
296+
eventData.Properties.Add("MyImportantProperty", "myImportantValue");
297+
298+
// Log QnAMessage event
299+
TelemetryClient.TrackEvent(
300+
QnATelemetryConstants.QnaMsgEvent,
301+
eventData.Properties,
302+
eventData.Metrics
303+
);
304+
}
305+
306+
}
307+
}
308+
```
309+
310+
4. Modify your bot to use the new class, instead of creating a `QnAMaker` object you will create a `MyQnAMaker` object in `QnABot.cs`:
311+
312+
```cs
313+
var qnaMaker = new MyQnAMaker(new QnAMakerEndpoint
314+
{
315+
KnowledgeBaseId = _configuration["QnAKnowledgebaseId"],
316+
EndpointKey = _configuration["QnAEndpointKey"],
317+
Host = _configuration["QnAEndpointHostName"]
318+
},
319+
null,
320+
httpClient,
321+
_telemetryClient);
322+
```
323+
324+
##### Viewing telemetry data logged from the new property _MyImportantProperty_
325+
After running your bot in the emulator you can view the results in Application Insights by doing the following:
326+
327+
1. Switch back to your browser that has the _Logs (Analytics)_ view active.
328+
329+
2. Enter the following Kusto query and then select _Run_. This will give a count of the number of times the new property was executed:
330+
331+
```SQL
332+
customEvents
333+
| where name == 'QnaMessage'
334+
| extend MyImportantProperty = tostring(customDimensions.MyImportantProperty)
335+
| summarize count() by MyImportantProperty
336+
```
337+
338+
3. To show details instead of the count remove the last line and re-run the query:
339+
340+
```SQL
341+
customEvents
342+
| where name == 'QnaMessage'
343+
| extend MyImportantProperty = tostring(customDimensions.MyImportantProperty)
344+
```
345+
### Adding new events with custom properties
346+
If you need to log data to a different event than `QnaMessage`, you can create your own custom event with its own properties. To do this, we will add code to the end of the `MyQnAMaker` class as follows:
347+
348+
```CS
349+
public class MyQnAMaker : QnAMaker
350+
{
351+
...
352+
353+
// Create second event.
354+
var secondEventProperties = new Dictionary<string, string>();
355+
356+
// Create new property for the second event.
357+
secondEventProperties.Add(
358+
"MyImportantProperty2",
359+
"myImportantValue2");
360+
361+
// Log secondEventProperties event
362+
TelemetryClient.TrackEvent(
363+
"MySecondEvent",
364+
secondEventProperties);
365+
366+
}
367+
```
368+
## The Application Insights dashboard
369+
370+
Anytime you create an Application Insights resource in Azure, a new dashboard will automatically be created and associated with it. You can see that dashboard by selecting the button at the top of your Application Insights blade, labeled **Application Dashboard**.
371+
372+
![Application Dashboard Link](media/Application-Dashboard-Link.png)
373+
374+
375+
Alternatively, to view the data, go to the Azure portal. Click **Dashboard** on the left, then select the dashboard you want from the drop-down.
376+
377+
There you'll see some default information about your bot performance and any additional queries that you've pinned to your dashboard.
378+
379+
380+
## Additional Information
381+
382+
* [Add telemetry to your bot](bot-builder-telemetry.md)
383+
384+
* [What is Application Insights?](https://aka.ms/appinsights-overview)
385+
386+
* [Using Search in Application Insights](https://aka.ms/search-in-application-insights)
387+
388+
* [Create custom KPI dashboards using Azure Application Insights](https://aka.ms/custom-kpi-dashboards-application-insights)

0 commit comments

Comments
 (0)