cross train CLI

[feich-ms]

Fixes #687. The usage is listed here https://github.com/microsoft/botframework-cli/blob/cross-train/packages/cross-train/README.md

[feich-ms]

@vishwacsena, I pushed some commits today. It can convert luis/qnamaker recognizer to crosstrained recogizer. But I have a question here. What is the file name for the crosstrained recognizer? Currently, for luis build or qna build dialogs, we have below naming pricipals:

1. luis recognizer: fileName.en-us.lu.dialog
2. luis multi recognizer: fileName.lu.dialog
3. qna recognizer: botName.en-us.qna.dialog
4. qna multi recognizer: botName.qna.dialog

In my commits, I'm using the same name for luis recognizer and luis cross trained recognizer, which means I'm converting luis recognizer to crosstrained recognizer in the same file named fileName.en-us.lu.dialog. In such case, I cannot merge qna crosstrained recognizer into luis crosstrained recognizer file since they have different file names. We need the same name for both luis and qna crosstrained recognizer so that we can merge them. So the question is: which file name for crosstrained recognizer?

[vishwacsena]

@feich-ms starting with the simple use case, if i have dialogA and a dialogA.lu and dialogA.qna, then I would expect to find the crossTrained recognizer in dialogA.lu.qna.dialog. Having both .lu.qna would signify crossTrained. We could then extrapolate the convention and say dialogA.lu.dialog will be the multiLingual recognizer and dialogA.<locale>.lu.dialog be the individual luisRecognizer configuration.

This would also mean that if we detect dialogA.qna for dialogA then we should write out dialogA.qna.dialog with the multiLanguageRecognizer and dialogA.<locale>.qna.dialog as the qnamaker recognizer and have dialogA.lu.qna.dialog be the cross trained recognizer configuration.

Let's also simplify and prune the --dialog options to be --dialog = multiLanguage [default] | crossTrained. We can always add --dialog Luis | qnamaker for R10 or later.

[feich-ms]

@vishwacsena, I pushed more commits. I don't know whether current implementations meet your requirement. But I can show how to use it and you can have a try and then give feedbacks.

1. bf luis:cross-train or bf qnamaker:cross-train --in xxx --out xxx --config xxx
2. bf luis:build --in xxx --authoringKey xxx --botName xxx --out xxx --dialog crosstrained
3. bf qnamaker:build --in xxx --subscriptionKey xxx --botName YYY --out xxx --dialog crosstrained

In step 2, you need to specify --dialog to crosstrained and in step 3, qna build will search the crosstrained file with the same name. If step 2 doesn't specify crosstrained, step 3 will not be able to merge the crosstrained configuration in the same file (it will create a new crosstrained one with only qnamaker). In one word, if you want the completed crosstrained configuration with both Luis and QnA, you should specify crosstrained in both luis build and qna build cmd(Execution order does not matter).

Please notice that botName in qnamaker:build should be specified the same name of it's corresponding lu file (not botName in luis:build cmd) so that xxx.lu.qna.dialog file of both luis and qna can have the same file name and merge together because the qnamaker crosstrained file name is .lu.qna.dialog instead of .lu.qna.dialog.

I'm not confident that I 100% understand you requirements. Anyway, please let me know any feebacks at any time. I am glad to optimize.

I have another question about the dialog assets of luis and qna. In a bot, we have multiple lu files based on dialogs, and we publish them to multiple luis applications. But in QnA, all the qna files will be merged into one and published into one KB, this will make the luis and QnA mapping rule not 1 to 1. How about we still generate the same dialogs like luis, that means we generate qna .dialog per each qna file but actually we only published the merged one. We just need to add duplicated model id in qna settings file like below(All three xxx are the same id). In this way, we can uniform the luis and qna relationships and help to generate the crosstrained recognizer more easily which means we can generate the crosstraind configurations based on each LuisRecognizer and QnAMakerRecognizer pair it they are. Does this make sense?

{
    "qna": {
        "travelbot_en_us_qna": "xxx",
        "bookflight_en_us_qna": "xxx",
        "searchweather_en_us_qna": "xxx",
        "hostname": "https://qnamakerfeich.azurewebsites.net/qnamaker"
    }
}

[vishwacsena]

Verified all functionality.

[cleemullins]

IS this ready to merge?

[munozemilio]

@cleemullins I'll do a second pass since a lot of changes were added since my last review.
I will merge it after that

[munozemilio]

Minor nits added.

[feich-ms]

@munozemilio Thanks for all your good comments. They all very make sense. I fixed and resolved them. So I will merge this PR. Please let me know if you have any concerns.