Skip to content

Commit

Permalink
fullname and typo corrections
Browse files Browse the repository at this point in the history
  • Loading branch information
indam23 committed Aug 3, 2020
1 parent 02f7bf9 commit 7bceffd
Show file tree
Hide file tree
Showing 3 changed files with 17 additions and 15 deletions.
2 changes: 2 additions & 0 deletions .typo-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ excluded_words:
- asyncio
- bot
- bot's
- cdd
- CDD
- cmdline
- conveRT
- ConveRTFeaturizer
Expand Down
28 changes: 14 additions & 14 deletions docs/docs/conversation-driven-development.mdx
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
id: cdd
sidebar_label: CDD
title: Conversation Driven Development
id: conversation-driven-development
sidebar_label: Conversation-Driven Development
title: Conversation-Driven Development
---

## What is CDD?

Conversation-Driven Development (CDD) is the process of listening to your users and using those insights to improve your AI assistant. It is the overarching best practice approach for chatbot development.

Developing great AI assistants is challenging because users will always say something you didnt anticipate. The principle behind CDD is that in every conversation users are telling you—in their own words—exactly what they want. By practicing CDD at every stage of bot development, you orient your assistant towards real user language and behaviour.
Developing great AI assistants is challenging because users will always say something you didn't anticipate. The principle behind CDD is that in every conversation users are telling you—in their own words—exactly what they want. By practicing CDD at every stage of bot development, you orient your assistant towards real user language and behavior.

CDD includes the following actions:

Expand All @@ -19,22 +19,22 @@ CDD includes the following actions:
* **Track** when your assistant fails and measure its performance over time
* **Fix** how your assistant handles unsuccessful conversations

CDD is not a linear process; youll circle back to the same actions over and over as you develop and improve your bot.
CDD is not a linear process; you'll circle back to the same actions over and over as you develop and improve your bot.

Read more about these actions and the concept of CDD [here](https://blog.rasa.com/conversation-driven-development-a-better-approach-to-building-ai-assistants/).

You can also check out [Rasa X](x), a purpose-built tool for CDD.

## CDD in early stages of development

If youre at the earliest stage of bot development, it might seem like CDD has no role to play - after all, you have no conversations yet! However, there are CDD actions you can take at the very beginning of bot development:
If you're at the earliest stage of bot development, it might seem like CDD has no role to play - after all, you have no conversations yet! However, there are CDD actions you can take at the very beginning of bot development:

1. See the best practices for [NLU](http://s) and [Stories](http://s) for details on applying CDD in these areas.
2. Give your bot to test users early on.

CDD is all about listening to your users, so the earlier you find some, the better.

Test users can be anyone who doesnt already know how your bot works from the inside. People on the bot development team should not be test users, since they know exactly what the bot can and cant do. Dont overinstruct your test users; they should have only as much knowledge of the bots domain as your end users will have.
Test users can be anyone who doesn't already know how your bot works from the inside. People on the bot development team should not be test users, since they know exactly what the bot can and can't do. Don't overinstruct your test users; they should have only as much knowledge of the bot's domain as your end users will have.

3. Set up a CI/CD pipeline.

Expand All @@ -46,23 +46,23 @@ At this stage, you can [install Rasa X in local mode](http://x) to make it easie

## CDD with a bot in production

Once your bot is in production, youll have more conversations to gain insights from. Then you can fully apply CDD actions.
Once your bot is in production, you'll have more conversations to gain insights from. Then you can fully apply CDD actions.

* **Review**: Look in conversations for what users are really asking for.

Your test users had at least some instruction about what the bot was intended to do; real users often either have no idea, or ignore instructions given to them. You cant cater to every unexpected user behaviour, but you can try to address the main friction points you notice. Here are some things you could consider looking for:
Your test users had at least some instruction about what the bot was intended to do; real users often either have no idea, or ignore instructions given to them. You can't cater to every unexpected user behavior, but you can try to address the main friction points you notice. Here are some things you could consider looking for:

* Look at conversations where an “out_of_scope” intent or fallback behaviour occurred. These could indicate a potential new skill, or just a misclassified user utterance.
* Look at conversations where an “out_of_scope” intent or fallback behavior occurred. These could indicate a potential new skill, or just a misclassified user utterance.
* Look for user frustration, such as requests for transfer to a human.

* **Annotate**: Continue to follow [best practices for NLU](http://s) as you add new user utterances from real conversations to your training data. Be careful not to overfit your NLU model to utterances like those already in your training data. This can happen when you continuously add user utterances that were already predicted correctly and with high confidence to your training data. To avoid overfitting and help your model generalize to more diverse user utterances, add only user utterances that the model previously predicted incorrectly or with low confidence.

* **Test:** Add successful user conversations to your test conversations. Doing this consistently will help ensure you dont introduce regressions as you make other fixes to your bot.
* **Test:** Add successful user conversations to your test conversations. Doing this consistently will help ensure you don't introduce regressions as you make other fixes to your bot.

* **Track:** Look for clues to success and failure to help you track your bots performance.
* **Track:** Look for clues to success and failure to help you track your bot's performance.

Some metrics are external to your bot. For example, if you are building a bot to relieve demand on a customer service call center, one metric for success could be the reduction in traffic to the call center. Others you can get directly from conversations, such as whether a user reaches a certain action that represents achieving the user goal.

Automatically tracked metrics are by nature proxy metrics; the only way to get a true measure of success would be to individually review and rate every single conversation with your bot. While this clearly isnt realistic, just keep in mind that no metric is a perfect representation of your bots performance, so don't really only on metrics to see where your bot needs improvement.
Automatically tracked metrics are by nature proxy metrics; the only way to get a true measure of success would be to individually review and rate every single conversation with your bot. While this clearly isn't realistic, just keep in mind that no metric is a perfect representation of your bot's performance, so don't really only on metrics to see where your bot needs improvement.

* **Fix:** Continue to follow [best practices for Stories](http://s) as you expand and improve your bots skills. Let user demand guide which skills you add and which fixes you make. Make smaller changes frequently rather than making big changes only once in a while. This will help you gauge the effectiveness of changes youre making, since youll get user feedback more frequently. Your [CI/CD pipeline ](http://c) should allow you to do so with confidence.
* **Fix:** Continue to follow [best practices for Stories](http://s) as you expand and improve your bot's skills. Let user demand guide which skills you add and which fixes you make. Make smaller changes frequently rather than making big changes only once in a while. This will help you gauge the effectiveness of changes you're making, since you'll get user feedback more frequently. Your [CI/CD pipeline ](http://c) should allow you to do so with confidence.
2 changes: 1 addition & 1 deletion docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ module.exports = {
label: 'Best Practices',
collapsed: false,
items: [
'cdd',
'conversation-driven-development',
'generating-nlu-data',
'writing-stories',
],
Expand Down

0 comments on commit 7bceffd

Please sign in to comment.