diff --git a/Node/ImageCaption/.env b/Node/ImageCaption/.env new file mode 100644 index 0000000000..33b1ce002a --- /dev/null +++ b/Node/ImageCaption/.env @@ -0,0 +1,9 @@ + +# Bot Framework Variables +MICROSOFT_APP_ID= +MICROSOFT_APP_PASSWORD= + +# Vision Api Variables +# You need to add the MICROSOFT_VISION_API_KEY value in order for the bot to work. +# You can obtain one from https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4 +MICROSOFT_VISION_API_KEY= \ No newline at end of file diff --git a/Node/ImageCaption/README.md b/Node/ImageCaption/README.md new file mode 100644 index 0000000000..dd74267753 --- /dev/null +++ b/Node/ImageCaption/README.md @@ -0,0 +1,85 @@ +# Image Caption Bot Sample + +A sample bot that illustrates how to use the Microsoft Cognitive Services Computer Vision API to analyze an image from a stream or a URL and return to the user the image caption. + +[![Deploy to Azure](http://azuredeploy.net/deploybutton.png)](https://azuredeploy.net?ptmpl=Node/ImageCaption/azuredeploy.json) + +### Prerequisites + +The minimum prerequisites to run this sample are: +* Latest Node.js with NPM. Download it from [here](https://nodejs.org/en/download/). +* The Bot Framework Emulator. To install the Bot Framework Emulator, download it from [here](https://aka.ms/bf-bc-emulator). Please refer to [this documentation article](https://docs.botframework.com/en-us/csharp/builder/sdkreference/gettingstarted.html#emulator) to know more about the Bot Framework Emulator. +* Computer Vision App ID. You can obtain one from [Microsoft Cognitive Services Subscriptions Page](https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4). +* **[Recommended]** Visual Studio Code for IntelliSense and debugging, download it from [here](https://code.visualstudio.com/) for free. + + +### Code Highlights + +Microsoft Computer Vision API provides a number of methods that allows you to analyze an image. Check out [Computer Vision API - v1.0](https://dev.projectoxford.ai/docs/services/56f91f2d778daf23d8ec6739/operations/56f91f2e778daf14a499e1fa) for a complete reference of the methods available. In this sample we are using the 'analyze' endpoint with the 'visualFeatures' parameter set to 'Description' `https://api.projectoxford.ai/vision/v1.0/analyze/?visualFeatures=Description` + +The main components are: + +* [caption-service.js](caption-service.js): is the core component illustrating how to call the Computer Vision RESTful API. +* [app.js](app.js): is the bot service listener receiving messages from the connector service and passing them down to caption-service.js. + +In this sample we are using the API to get the image description and send it back to the user. Check out the use of the `captionService.getCaptionFromStream(stream)` method in [app.js](app.js). + +````JavaScript +if (hasImageAttachment(session)) { + var stream = needle.get(session.message.attachments[0].contentUrl); + captionService + .getCaptionFromStream(stream) + .then(caption => handleSuccessResponse(session, caption)) + .catch(error => handleErrorResponse(session, error)); + } +```` +and here is the implementation of `captionService.getCaptionFromStream(stream)` in [caption-service.js](caption-service.js) +````JavaScript +/** + * Gets the caption of the image from an image stream + * @param {stream} stream The stream to an image. + * @return (Promise) Promise with caption string if succeeded, error otherwise + */ +exports.getCaptionFromStream = stream => { + return new Promise( + (resolve, reject) => { + const requestData = { + url: VISION_URL, + encoding: 'binary', + headers: { 'content-type': 'application/octet-stream' } + }; + + stream.pipe(request.post(requestData, (error, response, body) => { + if (error) { + reject(error); + } + else { + resolve(extractCaption(JSON.parse(body))); + } + })); + } + ); +} +```` + +### Outcome + +You will see the following when connecting the Bot to the Emulator and send it an image URL: + +Input: + +![Sample Outcome](images/bread-on-board.jpg) + +Output: + +![Sample Outcome](images/outcome-emulator-url.png) + +You can also choose to upload an image directly to the bot: + +![Sample Outcome](images/outcome-emulator-stream.png) + +### More Information + +To get more information about how to get started in Bot Builder for Node and and Microsoft Cognitive Services Computer Vision API please review the following resources: +* [Bot Builder for Node.js Reference](https://docs.botframework.com/en-us/node/builder/overview/#navtitle) +* [Microsoft Cognitive Services Computer Vision API](https://www.microsoft.com/cognitive-services/en-us/computer-vision-api) diff --git a/Node/ImageCaption/app.js b/Node/ImageCaption/app.js new file mode 100644 index 0000000000..b58308c5f3 --- /dev/null +++ b/Node/ImageCaption/app.js @@ -0,0 +1,106 @@ +/*----------------------------------------------------------------------------- +An image caption bot for the Microsoft Bot Framework. +-----------------------------------------------------------------------------*/ + +// This loads the environment variables from the .env file +require('dotenv-extended').load(); + +if (!process.env.MICROSOFT_VISION_API_KEY) { + console.error("Missing MICROSOFT_VISION_API_KEY. Please set it in the '.env' file. You can obtain one from https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4"); + process.exit() +} + +const builder = require('botbuilder'), + captionService = require('./caption-service'), + needle = require("needle"), + restify = require('restify'), + validUrl = require('valid-url'); + +//========================================================= +// Bot Setup +//========================================================= + +// Setup Restify Server +const server = restify.createServer(); +server.listen(process.env.port || process.env.PORT || 3978, () => { + console.log('%s listening to %s', server.name, server.url); +}); + +// Create chat bot +const connector = new builder.ChatConnector({ + appId: process.env.MICROSOFT_APP_ID, + appPassword: process.env.MICROSOFT_APP_PASSWORD +}); + +const bot = new builder.UniversalBot(connector); +server.post('/api/messages', connector.listen()); + + +//========================================================= +// Bots Events +//========================================================= + +//Sends greeting message when the bot is first added to a conversation +bot.on('conversationUpdate', message => { + if (message.membersAdded) { + message.membersAdded.forEach(identity => { + if (identity.id === message.address.bot.id) { + const reply = new builder.Message() + .address(message.address) + .text("Hi! I am ImageCaption Bot. I can understand the content of any image and try to describe it as well as any human. Try sending me an image or an image URL."); + bot.send(reply); + } + }); + } +}); + + +//========================================================= +// Bots Dialogs +//========================================================= + +// Gets the caption by checking the type of the image (stream vs URL) and calling the appropriate caption service method. +bot.dialog('/', session => { + if (hasImageAttachment(session)) { + var stream = needle.get(session.message.attachments[0].contentUrl); + captionService + .getCaptionFromStream(stream) + .then(caption => handleSuccessResponse(session, caption)) + .catch(error => handleErrorResponse(session, error)); + } + else if (validUrl.isUri(session.message.text)) { + captionService + .getCaptionFromUrl(session.message.text) + .then(caption => handleSuccessResponse(session, caption)) + .catch(error => handleErrorResponse(session, error)); + + } + else { + session.send("Did you upload an image? I'm more of a visual person. Try sending me an image or an image URL"); + } +}); + +//========================================================= +// Utilities +//========================================================= +const hasImageAttachment = session => { + return ((session.message.attachments.length > 0) && (session.message.attachments[0].contentType.indexOf("image") !== -1)); +} + +//========================================================= +// Response Handling +//========================================================= +const handleSuccessResponse = (session, caption) => { + if (caption) { + session.send("I think it's " + caption); + } + else { + session.send("Couldn't find a caption for this one"); + } + +} + +const handleErrorResponse = (session, error) => { + session.send("Oops! Something went wrong. Try again later."); + console.error(error); +} diff --git a/Node/ImageCaption/azuredeploy.json b/Node/ImageCaption/azuredeploy.json new file mode 100644 index 0000000000..ddc9031f22 --- /dev/null +++ b/Node/ImageCaption/azuredeploy.json @@ -0,0 +1,136 @@ +{ + "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "parameters": { + "siteName": { + "defaultValue": "BotBuilder-Samples", + "type": "string" + }, + "hostingPlanName": { + "type": "string" + }, + "siteLocation": { + "type": "string" + }, + "sku": { + "type": "string", + "allowedValues": [ + "Free", + "Shared", + "Basic", + "Standard" + ], + "defaultValue": "Free" + }, + "workerSize": { + "type": "string", + "allowedValues": [ + "0", + "1", + "2" + ], + "defaultValue": "0" + }, + "repoUrl": { + "type": "string" + }, + "branch": { + "type": "string" + }, + "Project": { + "type": "string", + "defaultValue": "Node/ImageCaption" + }, + "WEBSITE_NODE_DEFAULT_VERSION": { + "type": "string", + "defaultValue": "5.9.1" + }, + "MicrosoftAppId": { + "type": "string" + }, + "MicrosoftAppPassword": { + "type": "string" + }, + "MicrosoftVisionApiKey": { + "type": "string" + } + }, + "resources": [ + { + "apiVersion": "2014-06-01", + "name": "[parameters('hostingPlanName')]", + "type": "Microsoft.Web/serverFarms", + "location": "[parameters('siteLocation')]", + "properties": { + "name": "[parameters('hostingPlanName')]", + "sku": "[parameters('sku')]", + "workerSize": "[parameters('workerSize')]", + "numberOfWorkers": 1 + } + }, + { + "apiVersion": "2014-06-01", + "name": "[parameters('siteName')]", + "type": "Microsoft.Web/Sites", + "location": "[parameters('siteLocation')]", + "dependsOn": [ + "[concat('Microsoft.Web/serverFarms/', parameters('hostingPlanName'))]" + ], + "tags": { + "[concat('hidden-related:', resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]": "empty" + }, + "properties": { + "name": "[parameters('siteName')]", + "serverFarm": "[parameters('hostingPlanName')]" + }, + "resources": [ + { + "apiVersion": "2014-04-01", + "type": "config", + "name": "web", + "dependsOn": [ + "[concat('Microsoft.Web/Sites/', parameters('siteName'))]" + ], + "properties": { + "appSettings": [ + { + "name": "Project", + "value": "[parameters('Project')]" + }, + { + "name": "WEBSITE_NODE_DEFAULT_VERSION", + "value": "[parameters('WEBSITE_NODE_DEFAULT_VERSION')]" + }, + { + "name": "MICROSOFT_APP_ID", + "value": "[parameters('MicrosoftAppId')]" + }, + { + "name": "MICROSOFT_APP_PASSWORD", + "value": "[parameters('MicrosoftAppPassword')]" + }, + { + "name": "MICROSOFT_VISION_API_KEY", + "value": "[parameters('MicrosoftVisionApiKey')]" + } + ] + } + }, + { + "apiVersion": "2014-04-01", + "name": "web", + "type": "sourcecontrols", + "dependsOn": [ + "[resourceId('Microsoft.Web/Sites', parameters('siteName'))]", + "[concat('Microsoft.Web/Sites/', parameters('siteName'), '/config/web')]" + ], + "properties": { + "RepoUrl": "[parameters('repoUrl')]", + "branch": "[parameters('branch')]", + "IsManualIntegration": true + } + } + ] + } + ] +} diff --git a/Node/ImageCaption/caption-service.js b/Node/ImageCaption/caption-service.js new file mode 100644 index 0000000000..081953e1d4 --- /dev/null +++ b/Node/ImageCaption/caption-service.js @@ -0,0 +1,78 @@ +// The exported functions in this module makes a call to Microsoft Cognitive Service Computer Vision API and return caption +// description if found. Note: you can do more advanced functionalities like checking +// the confidence score of the caption. For more info checkout the API documentation: +// https://www.microsoft.com/cognitive-services/en-us/Computer-Vision-API/documentation/AnalyzeImage + +const request = require('request').defaults({ encoding: null }); + +const VISION_URL = "https://api.projectoxford.ai/vision/v1.0/analyze/?visualFeatures=Description&subscription-key=" + process.env.MICROSOFT_VISION_API_KEY; + +/** + * Gets the caption of the image from an image stream + * @param {stream} stream The stream to an image. + * @return (Promise) Promise with caption string if succeeded, error otherwise + */ +exports.getCaptionFromStream = stream => { + return new Promise( + (resolve, reject) => { + const requestData = { + url: VISION_URL, + encoding: 'binary', + headers: { 'content-type': 'application/octet-stream' } + }; + + stream.pipe(request.post(requestData, (error, response, body) => { + if (error) { + reject(error); + } + else if (response.statusCode != 200) { + reject(body); + } + else { + resolve(extractCaption(JSON.parse(body))); + } + })); + } + ); +} + +/** + * Gets the caption of the image from an image URL + * @param {string} url The URL to an image. + * @return (Promise) Promise with caption string if succeeded, error otherwise + */ +exports.getCaptionFromUrl = url => { + return new Promise( + (resolve, reject) => { + const requestData = { + url: VISION_URL, + json: { "url": url } + }; + + request.post(requestData, (error, response, body) => { + if (error) { + reject(error); + } + else if (response.statusCode != 200) { + reject(body); + } + else { + resolve(extractCaption(body)); + } + }); + } + ); +} + +/** + * Extracts the caption description from the response of the Vision API + * @param {Object} body Response of the Vision API + * @return {string} Description if caption found, null otherwise. + */ +const extractCaption = body => { + if (body && body.description && body.description.captions && body.description.captions.length) { + return body.description.captions[0].text + } + + return null; +} \ No newline at end of file diff --git a/Node/ImageCaption/images/bread-on-board.jpg b/Node/ImageCaption/images/bread-on-board.jpg new file mode 100644 index 0000000000..eba53c7064 Binary files /dev/null and b/Node/ImageCaption/images/bread-on-board.jpg differ diff --git a/Node/ImageCaption/images/outcome-emulator-stream.png b/Node/ImageCaption/images/outcome-emulator-stream.png new file mode 100644 index 0000000000..3ab3e9f809 Binary files /dev/null and b/Node/ImageCaption/images/outcome-emulator-stream.png differ diff --git a/Node/ImageCaption/images/outcome-emulator-url.png b/Node/ImageCaption/images/outcome-emulator-url.png new file mode 100644 index 0000000000..32337d06d6 Binary files /dev/null and b/Node/ImageCaption/images/outcome-emulator-url.png differ diff --git a/Node/ImageCaption/package.json b/Node/ImageCaption/package.json new file mode 100644 index 0000000000..f5ee59af77 --- /dev/null +++ b/Node/ImageCaption/package.json @@ -0,0 +1,17 @@ +{ + "name": "botbuilder-sample-imagecaption", + "version": "1.0.0", + "description": "Bot Builder Sample - Image Caption", + "scripts": { + "start": "node app.js" + }, + "auhor": "", + "license": "MIT", + "dependencies": { + "botbuilder": "^3.2.3", + "dotenv-extended": "^1.0.3", + "needle": "^1.1.2", + "restify": "^4.1.1", + "valid-url": "^1.0.9" + } +} \ No newline at end of file diff --git a/README.md b/README.md index 6ac464c4ee..7d7b537ebe 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,6 @@ These examples emphasize the rich card support in Bot Framework. Sample | Description | C# | Node ------------ | ------------- | :-----------: | :-----------: -<<<<<<< HEAD Rich Cards | A sample bot to renders several types of cards as attachments. | [View Sample](/CSharp/cards-RichCards)[![Deploy to Azure][Deploy Button]][Deploy CSharp/RichCards] | [View Sample](/Node/cards-RichCards)[![Deploy to Azure][Deploy Button]][Deploy Node/RichCards] Carousel of Cards | A sample bot that sends multiple rich card attachments in a single message using the Carousel layout. | [View Sample](/CSharp/cards-CarouselCards)[![Deploy to Azure][Deploy Button]][Deploy CSharp/CarouselCards] | [View Sample](/Node/cards-CarouselCards)[![Deploy to Azure][Deploy Button]][Deploy Node/CarouselCards] @@ -34,7 +33,7 @@ Build bots with powerful algorithms leveraging intelligence APIs. Sample | Description | C# | Node ------------ | ------------- | :-----------: | :-----------: LUIS | A sample bot using LuisDialog to integrate with a LUIS.ai application. | [View Sample](/CSharp/intelligence-LUIS)[![Deploy to Azure][Deploy Button]][Deploy CSharp/LUIS] | [View Sample](/Node/intelligence-LUIS)[![Deploy to Azure][Deploy Button]][Deploy Node/LUIS] -Image Caption | A sample bot that gets an image caption using Microsoft Cognitive Services Vision API. | [View Sample](/CSharp/ImageCaption)[![Deploy to Azure][Deploy Button]][Deploy CSharp/ImageCaption] | +Image Caption | A sample bot that gets an image caption using Microsoft Cognitive Services Vision API. | [View Sample](/CSharp/ImageCaption)[![Deploy to Azure][Deploy Button]][Deploy CSharp/ImageCaption] | [View Sample](/Node/ImageCaption)[![Deploy to Azure][Deploy Button]][Deploy Node/ImageCaption] ## Demo These are bots designed to showcase end-to-end sample scenarios. They're great sources of code fragments if you're looking to have your bot lightup more complex features. @@ -71,3 +70,4 @@ Azure Search | Two sample bots that help the user navigate large amounts of cont [Deploy CSharp/ContosoFlowers]: https://azuredeploy.net?repository=https://github.com/microsoft/BotBuilder-Samples/tree/master/CSharp/demo-ContosoFlowers [Deploy Node/ContosoFlowers]: https://azuredeploy.net/?repository=https://github.com/microsoft/BotBuilder-Samples/tree/master/Node/demo-ContosoFlowers [Deploy CSharp/ImageCaption]: https://azuredeploy.net?ptmpl=CSharp/ImageCaption/azuredeploy.json +[Deploy Node/ImageCaption]: https://azuredeploy.net?ptmpl=Node/ImageCaption/azuredeploy.json