forked from microsoft/BotBuilder-Samples
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
10 changed files
with
433 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
|
||
# Bot Framework Variables | ||
MICROSOFT_APP_ID= | ||
MICROSOFT_APP_PASSWORD= | ||
|
||
# Vision Api Variables | ||
# You need to add the MICROSOFT_VISION_API_KEY value in order for the bot to work. | ||
# You can obtain one from https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4 | ||
MICROSOFT_VISION_API_KEY= |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
# Image Caption Bot Sample | ||
|
||
A sample bot that illustrates how to use the Microsoft Cognitive Services Computer Vision API to analyze an image from a stream or a URL and return to the user the image caption. | ||
|
||
[![Deploy to Azure](http://azuredeploy.net/deploybutton.png)](https://azuredeploy.net?ptmpl=Node/ImageCaption/azuredeploy.json) | ||
|
||
### Prerequisites | ||
|
||
The minimum prerequisites to run this sample are: | ||
* Latest Node.js with NPM. Download it from [here](https://nodejs.org/en/download/). | ||
* The Bot Framework Emulator. To install the Bot Framework Emulator, download it from [here](https://aka.ms/bf-bc-emulator). Please refer to [this documentation article](https://docs.botframework.com/en-us/csharp/builder/sdkreference/gettingstarted.html#emulator) to know more about the Bot Framework Emulator. | ||
* Computer Vision App ID. You can obtain one from [Microsoft Cognitive Services Subscriptions Page](https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4). | ||
* **[Recommended]** Visual Studio Code for IntelliSense and debugging, download it from [here](https://code.visualstudio.com/) for free. | ||
|
||
|
||
### Code Highlights | ||
|
||
Microsoft Computer Vision API provides a number of methods that allows you to analyze an image. Check out [Computer Vision API - v1.0](https://dev.projectoxford.ai/docs/services/56f91f2d778daf23d8ec6739/operations/56f91f2e778daf14a499e1fa) for a complete reference of the methods available. In this sample we are using the 'analyze' endpoint with the 'visualFeatures' parameter set to 'Description' `https://api.projectoxford.ai/vision/v1.0/analyze/?visualFeatures=Description` | ||
|
||
The main components are: | ||
|
||
* [caption-service.js](caption-service.js): is the core component illustrating how to call the Computer Vision RESTful API. | ||
* [app.js](app.js): is the bot service listener receiving messages from the connector service and passing them down to caption-service.js. | ||
|
||
In this sample we are using the API to get the image description and send it back to the user. Check out the use of the `captionService.getCaptionFromStream(stream)` method in [app.js](app.js). | ||
|
||
````JavaScript | ||
if (hasImageAttachment(session)) { | ||
var stream = needle.get(session.message.attachments[0].contentUrl); | ||
captionService | ||
.getCaptionFromStream(stream) | ||
.then(caption => handleSuccessResponse(session, caption)) | ||
.catch(error => handleErrorResponse(session, error)); | ||
} | ||
```` | ||
and here is the implementation of `captionService.getCaptionFromStream(stream)` in [caption-service.js](caption-service.js) | ||
````JavaScript | ||
/** | ||
* Gets the caption of the image from an image stream | ||
* @param {stream} stream The stream to an image. | ||
* @return (Promise) Promise with caption string if succeeded, error otherwise | ||
*/ | ||
exports.getCaptionFromStream = stream => { | ||
return new Promise( | ||
(resolve, reject) => { | ||
const requestData = { | ||
url: VISION_URL, | ||
encoding: 'binary', | ||
headers: { 'content-type': 'application/octet-stream' } | ||
}; | ||
|
||
stream.pipe(request.post(requestData, (error, response, body) => { | ||
if (error) { | ||
reject(error); | ||
} | ||
else { | ||
resolve(extractCaption(JSON.parse(body))); | ||
} | ||
})); | ||
} | ||
); | ||
} | ||
```` | ||
|
||
### Outcome | ||
|
||
You will see the following when connecting the Bot to the Emulator and send it an image URL: | ||
|
||
Input: | ||
|
||
![Sample Outcome](images/bread-on-board.jpg) | ||
|
||
Output: | ||
|
||
![Sample Outcome](images/outcome-emulator-url.png) | ||
|
||
You can also choose to upload an image directly to the bot: | ||
|
||
![Sample Outcome](images/outcome-emulator-stream.png) | ||
|
||
### More Information | ||
|
||
To get more information about how to get started in Bot Builder for Node and and Microsoft Cognitive Services Computer Vision API please review the following resources: | ||
* [Bot Builder for Node.js Reference](https://docs.botframework.com/en-us/node/builder/overview/#navtitle) | ||
* [Microsoft Cognitive Services Computer Vision API](https://www.microsoft.com/cognitive-services/en-us/computer-vision-api) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
/*----------------------------------------------------------------------------- | ||
An image caption bot for the Microsoft Bot Framework. | ||
-----------------------------------------------------------------------------*/ | ||
|
||
// This loads the environment variables from the .env file | ||
require('dotenv-extended').load(); | ||
|
||
if (!process.env.MICROSOFT_VISION_API_KEY) { | ||
console.error("Missing MICROSOFT_VISION_API_KEY. Please set it in the '.env' file. You can obtain one from https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4"); | ||
process.exit() | ||
} | ||
|
||
const builder = require('botbuilder'), | ||
captionService = require('./caption-service'), | ||
needle = require("needle"), | ||
restify = require('restify'), | ||
validUrl = require('valid-url'); | ||
|
||
//========================================================= | ||
// Bot Setup | ||
//========================================================= | ||
|
||
// Setup Restify Server | ||
const server = restify.createServer(); | ||
server.listen(process.env.port || process.env.PORT || 3978, () => { | ||
console.log('%s listening to %s', server.name, server.url); | ||
}); | ||
|
||
// Create chat bot | ||
const connector = new builder.ChatConnector({ | ||
appId: process.env.MICROSOFT_APP_ID, | ||
appPassword: process.env.MICROSOFT_APP_PASSWORD | ||
}); | ||
|
||
const bot = new builder.UniversalBot(connector); | ||
server.post('/api/messages', connector.listen()); | ||
|
||
|
||
//========================================================= | ||
// Bots Events | ||
//========================================================= | ||
|
||
//Sends greeting message when the bot is first added to a conversation | ||
bot.on('conversationUpdate', message => { | ||
if (message.membersAdded) { | ||
message.membersAdded.forEach(identity => { | ||
if (identity.id === message.address.bot.id) { | ||
const reply = new builder.Message() | ||
.address(message.address) | ||
.text("Hi! I am ImageCaption Bot. I can understand the content of any image and try to describe it as well as any human. Try sending me an image or an image URL."); | ||
bot.send(reply); | ||
} | ||
}); | ||
} | ||
}); | ||
|
||
|
||
//========================================================= | ||
// Bots Dialogs | ||
//========================================================= | ||
|
||
// Gets the caption by checking the type of the image (stream vs URL) and calling the appropriate caption service method. | ||
bot.dialog('/', session => { | ||
if (hasImageAttachment(session)) { | ||
var stream = needle.get(session.message.attachments[0].contentUrl); | ||
captionService | ||
.getCaptionFromStream(stream) | ||
.then(caption => handleSuccessResponse(session, caption)) | ||
.catch(error => handleErrorResponse(session, error)); | ||
} | ||
else if (validUrl.isUri(session.message.text)) { | ||
captionService | ||
.getCaptionFromUrl(session.message.text) | ||
.then(caption => handleSuccessResponse(session, caption)) | ||
.catch(error => handleErrorResponse(session, error)); | ||
|
||
} | ||
else { | ||
session.send("Did you upload an image? I'm more of a visual person. Try sending me an image or an image URL"); | ||
} | ||
}); | ||
|
||
//========================================================= | ||
// Utilities | ||
//========================================================= | ||
const hasImageAttachment = session => { | ||
return ((session.message.attachments.length > 0) && (session.message.attachments[0].contentType.indexOf("image") !== -1)); | ||
} | ||
|
||
//========================================================= | ||
// Response Handling | ||
//========================================================= | ||
const handleSuccessResponse = (session, caption) => { | ||
if (caption) { | ||
session.send("I think it's " + caption); | ||
} | ||
else { | ||
session.send("Couldn't find a caption for this one"); | ||
} | ||
|
||
} | ||
|
||
const handleErrorResponse = (session, error) => { | ||
session.send("Oops! Something went wrong. Try again later."); | ||
console.error(error); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,136 @@ | ||
{ | ||
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#", | ||
"contentVersion": "1.0.0.0", | ||
"parameters": { | ||
"siteName": { | ||
"defaultValue": "BotBuilder-Samples", | ||
"type": "string" | ||
}, | ||
"hostingPlanName": { | ||
"type": "string" | ||
}, | ||
"siteLocation": { | ||
"type": "string" | ||
}, | ||
"sku": { | ||
"type": "string", | ||
"allowedValues": [ | ||
"Free", | ||
"Shared", | ||
"Basic", | ||
"Standard" | ||
], | ||
"defaultValue": "Free" | ||
}, | ||
"workerSize": { | ||
"type": "string", | ||
"allowedValues": [ | ||
"0", | ||
"1", | ||
"2" | ||
], | ||
"defaultValue": "0" | ||
}, | ||
"repoUrl": { | ||
"type": "string" | ||
}, | ||
"branch": { | ||
"type": "string" | ||
}, | ||
"Project": { | ||
"type": "string", | ||
"defaultValue": "Node/ImageCaption" | ||
}, | ||
"WEBSITE_NODE_DEFAULT_VERSION": { | ||
"type": "string", | ||
"defaultValue": "5.9.1" | ||
}, | ||
"MicrosoftAppId": { | ||
"type": "string" | ||
}, | ||
"MicrosoftAppPassword": { | ||
"type": "string" | ||
}, | ||
"MicrosoftVisionApiKey": { | ||
"type": "string" | ||
} | ||
}, | ||
"resources": [ | ||
{ | ||
"apiVersion": "2014-06-01", | ||
"name": "[parameters('hostingPlanName')]", | ||
"type": "Microsoft.Web/serverFarms", | ||
"location": "[parameters('siteLocation')]", | ||
"properties": { | ||
"name": "[parameters('hostingPlanName')]", | ||
"sku": "[parameters('sku')]", | ||
"workerSize": "[parameters('workerSize')]", | ||
"numberOfWorkers": 1 | ||
} | ||
}, | ||
{ | ||
"apiVersion": "2014-06-01", | ||
"name": "[parameters('siteName')]", | ||
"type": "Microsoft.Web/Sites", | ||
"location": "[parameters('siteLocation')]", | ||
"dependsOn": [ | ||
"[concat('Microsoft.Web/serverFarms/', parameters('hostingPlanName'))]" | ||
], | ||
"tags": { | ||
"[concat('hidden-related:', resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]": "empty" | ||
}, | ||
"properties": { | ||
"name": "[parameters('siteName')]", | ||
"serverFarm": "[parameters('hostingPlanName')]" | ||
}, | ||
"resources": [ | ||
{ | ||
"apiVersion": "2014-04-01", | ||
"type": "config", | ||
"name": "web", | ||
"dependsOn": [ | ||
"[concat('Microsoft.Web/Sites/', parameters('siteName'))]" | ||
], | ||
"properties": { | ||
"appSettings": [ | ||
{ | ||
"name": "Project", | ||
"value": "[parameters('Project')]" | ||
}, | ||
{ | ||
"name": "WEBSITE_NODE_DEFAULT_VERSION", | ||
"value": "[parameters('WEBSITE_NODE_DEFAULT_VERSION')]" | ||
}, | ||
{ | ||
"name": "MICROSOFT_APP_ID", | ||
"value": "[parameters('MicrosoftAppId')]" | ||
}, | ||
{ | ||
"name": "MICROSOFT_APP_PASSWORD", | ||
"value": "[parameters('MicrosoftAppPassword')]" | ||
}, | ||
{ | ||
"name": "MICROSOFT_VISION_API_KEY", | ||
"value": "[parameters('MicrosoftVisionApiKey')]" | ||
} | ||
] | ||
} | ||
}, | ||
{ | ||
"apiVersion": "2014-04-01", | ||
"name": "web", | ||
"type": "sourcecontrols", | ||
"dependsOn": [ | ||
"[resourceId('Microsoft.Web/Sites', parameters('siteName'))]", | ||
"[concat('Microsoft.Web/Sites/', parameters('siteName'), '/config/web')]" | ||
], | ||
"properties": { | ||
"RepoUrl": "[parameters('repoUrl')]", | ||
"branch": "[parameters('branch')]", | ||
"IsManualIntegration": true | ||
} | ||
} | ||
] | ||
} | ||
] | ||
} |
Oops, something went wrong.