Skip to content

Commit

Permalink
Adding ImageCaption Nodejs sample
Browse files Browse the repository at this point in the history
  • Loading branch information
iassal authored and kgrashad committed Sep 28, 2016
1 parent 7d8bd0d commit 00a44fb
Show file tree
Hide file tree
Showing 10 changed files with 433 additions and 2 deletions.
9 changes: 9 additions & 0 deletions Node/ImageCaption/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@

# Bot Framework Variables
MICROSOFT_APP_ID=
MICROSOFT_APP_PASSWORD=

# Vision Api Variables
# You need to add the MICROSOFT_VISION_API_KEY value in order for the bot to work.
# You can obtain one from https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4
MICROSOFT_VISION_API_KEY=
85 changes: 85 additions & 0 deletions Node/ImageCaption/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Image Caption Bot Sample

A sample bot that illustrates how to use the Microsoft Cognitive Services Computer Vision API to analyze an image from a stream or a URL and return to the user the image caption.

[![Deploy to Azure](http://azuredeploy.net/deploybutton.png)](https://azuredeploy.net?ptmpl=Node/ImageCaption/azuredeploy.json)

### Prerequisites

The minimum prerequisites to run this sample are:
* Latest Node.js with NPM. Download it from [here](https://nodejs.org/en/download/).
* The Bot Framework Emulator. To install the Bot Framework Emulator, download it from [here](https://aka.ms/bf-bc-emulator). Please refer to [this documentation article](https://docs.botframework.com/en-us/csharp/builder/sdkreference/gettingstarted.html#emulator) to know more about the Bot Framework Emulator.
* Computer Vision App ID. You can obtain one from [Microsoft Cognitive Services Subscriptions Page](https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4).
* **[Recommended]** Visual Studio Code for IntelliSense and debugging, download it from [here](https://code.visualstudio.com/) for free.


### Code Highlights

Microsoft Computer Vision API provides a number of methods that allows you to analyze an image. Check out [Computer Vision API - v1.0](https://dev.projectoxford.ai/docs/services/56f91f2d778daf23d8ec6739/operations/56f91f2e778daf14a499e1fa) for a complete reference of the methods available. In this sample we are using the 'analyze' endpoint with the 'visualFeatures' parameter set to 'Description' `https://api.projectoxford.ai/vision/v1.0/analyze/?visualFeatures=Description`

The main components are:

* [caption-service.js](caption-service.js): is the core component illustrating how to call the Computer Vision RESTful API.
* [app.js](app.js): is the bot service listener receiving messages from the connector service and passing them down to caption-service.js.

In this sample we are using the API to get the image description and send it back to the user. Check out the use of the `captionService.getCaptionFromStream(stream)` method in [app.js](app.js).

````JavaScript
if (hasImageAttachment(session)) {
var stream = needle.get(session.message.attachments[0].contentUrl);
captionService
.getCaptionFromStream(stream)
.then(caption => handleSuccessResponse(session, caption))
.catch(error => handleErrorResponse(session, error));
}
````
and here is the implementation of `captionService.getCaptionFromStream(stream)` in [caption-service.js](caption-service.js)
````JavaScript
/**
* Gets the caption of the image from an image stream
* @param {stream} stream The stream to an image.
* @return (Promise) Promise with caption string if succeeded, error otherwise
*/
exports.getCaptionFromStream = stream => {
return new Promise(
(resolve, reject) => {
const requestData = {
url: VISION_URL,
encoding: 'binary',
headers: { 'content-type': 'application/octet-stream' }
};

stream.pipe(request.post(requestData, (error, response, body) => {
if (error) {
reject(error);
}
else {
resolve(extractCaption(JSON.parse(body)));
}
}));
}
);
}
````

### Outcome

You will see the following when connecting the Bot to the Emulator and send it an image URL:

Input:

![Sample Outcome](images/bread-on-board.jpg)

Output:

![Sample Outcome](images/outcome-emulator-url.png)

You can also choose to upload an image directly to the bot:

![Sample Outcome](images/outcome-emulator-stream.png)

### More Information

To get more information about how to get started in Bot Builder for Node and and Microsoft Cognitive Services Computer Vision API please review the following resources:
* [Bot Builder for Node.js Reference](https://docs.botframework.com/en-us/node/builder/overview/#navtitle)
* [Microsoft Cognitive Services Computer Vision API](https://www.microsoft.com/cognitive-services/en-us/computer-vision-api)
106 changes: 106 additions & 0 deletions Node/ImageCaption/app.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
/*-----------------------------------------------------------------------------
An image caption bot for the Microsoft Bot Framework.
-----------------------------------------------------------------------------*/

// This loads the environment variables from the .env file
require('dotenv-extended').load();

if (!process.env.MICROSOFT_VISION_API_KEY) {
console.error("Missing MICROSOFT_VISION_API_KEY. Please set it in the '.env' file. You can obtain one from https://www.microsoft.com/cognitive-services/en-us/subscriptions?productId=/products/54d873dd5eefd00dc474a0f4");
process.exit()
}

const builder = require('botbuilder'),
captionService = require('./caption-service'),
needle = require("needle"),
restify = require('restify'),
validUrl = require('valid-url');

//=========================================================
// Bot Setup
//=========================================================

// Setup Restify Server
const server = restify.createServer();
server.listen(process.env.port || process.env.PORT || 3978, () => {
console.log('%s listening to %s', server.name, server.url);
});

// Create chat bot
const connector = new builder.ChatConnector({
appId: process.env.MICROSOFT_APP_ID,
appPassword: process.env.MICROSOFT_APP_PASSWORD
});

const bot = new builder.UniversalBot(connector);
server.post('/api/messages', connector.listen());


//=========================================================
// Bots Events
//=========================================================

//Sends greeting message when the bot is first added to a conversation
bot.on('conversationUpdate', message => {
if (message.membersAdded) {
message.membersAdded.forEach(identity => {
if (identity.id === message.address.bot.id) {
const reply = new builder.Message()
.address(message.address)
.text("Hi! I am ImageCaption Bot. I can understand the content of any image and try to describe it as well as any human. Try sending me an image or an image URL.");
bot.send(reply);
}
});
}
});


//=========================================================
// Bots Dialogs
//=========================================================

// Gets the caption by checking the type of the image (stream vs URL) and calling the appropriate caption service method.
bot.dialog('/', session => {
if (hasImageAttachment(session)) {
var stream = needle.get(session.message.attachments[0].contentUrl);
captionService
.getCaptionFromStream(stream)
.then(caption => handleSuccessResponse(session, caption))
.catch(error => handleErrorResponse(session, error));
}
else if (validUrl.isUri(session.message.text)) {
captionService
.getCaptionFromUrl(session.message.text)
.then(caption => handleSuccessResponse(session, caption))
.catch(error => handleErrorResponse(session, error));

}
else {
session.send("Did you upload an image? I'm more of a visual person. Try sending me an image or an image URL");
}
});

//=========================================================
// Utilities
//=========================================================
const hasImageAttachment = session => {
return ((session.message.attachments.length > 0) && (session.message.attachments[0].contentType.indexOf("image") !== -1));
}

//=========================================================
// Response Handling
//=========================================================
const handleSuccessResponse = (session, caption) => {
if (caption) {
session.send("I think it's " + caption);
}
else {
session.send("Couldn't find a caption for this one");
}

}

const handleErrorResponse = (session, error) => {
session.send("Oops! Something went wrong. Try again later.");
console.error(error);
}
136 changes: 136 additions & 0 deletions Node/ImageCaption/azuredeploy.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
{
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"siteName": {
"defaultValue": "BotBuilder-Samples",
"type": "string"
},
"hostingPlanName": {
"type": "string"
},
"siteLocation": {
"type": "string"
},
"sku": {
"type": "string",
"allowedValues": [
"Free",
"Shared",
"Basic",
"Standard"
],
"defaultValue": "Free"
},
"workerSize": {
"type": "string",
"allowedValues": [
"0",
"1",
"2"
],
"defaultValue": "0"
},
"repoUrl": {
"type": "string"
},
"branch": {
"type": "string"
},
"Project": {
"type": "string",
"defaultValue": "Node/ImageCaption"
},
"WEBSITE_NODE_DEFAULT_VERSION": {
"type": "string",
"defaultValue": "5.9.1"
},
"MicrosoftAppId": {
"type": "string"
},
"MicrosoftAppPassword": {
"type": "string"
},
"MicrosoftVisionApiKey": {
"type": "string"
}
},
"resources": [
{
"apiVersion": "2014-06-01",
"name": "[parameters('hostingPlanName')]",
"type": "Microsoft.Web/serverFarms",
"location": "[parameters('siteLocation')]",
"properties": {
"name": "[parameters('hostingPlanName')]",
"sku": "[parameters('sku')]",
"workerSize": "[parameters('workerSize')]",
"numberOfWorkers": 1
}
},
{
"apiVersion": "2014-06-01",
"name": "[parameters('siteName')]",
"type": "Microsoft.Web/Sites",
"location": "[parameters('siteLocation')]",
"dependsOn": [
"[concat('Microsoft.Web/serverFarms/', parameters('hostingPlanName'))]"
],
"tags": {
"[concat('hidden-related:', resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]": "empty"
},
"properties": {
"name": "[parameters('siteName')]",
"serverFarm": "[parameters('hostingPlanName')]"
},
"resources": [
{
"apiVersion": "2014-04-01",
"type": "config",
"name": "web",
"dependsOn": [
"[concat('Microsoft.Web/Sites/', parameters('siteName'))]"
],
"properties": {
"appSettings": [
{
"name": "Project",
"value": "[parameters('Project')]"
},
{
"name": "WEBSITE_NODE_DEFAULT_VERSION",
"value": "[parameters('WEBSITE_NODE_DEFAULT_VERSION')]"
},
{
"name": "MICROSOFT_APP_ID",
"value": "[parameters('MicrosoftAppId')]"
},
{
"name": "MICROSOFT_APP_PASSWORD",
"value": "[parameters('MicrosoftAppPassword')]"
},
{
"name": "MICROSOFT_VISION_API_KEY",
"value": "[parameters('MicrosoftVisionApiKey')]"
}
]
}
},
{
"apiVersion": "2014-04-01",
"name": "web",
"type": "sourcecontrols",
"dependsOn": [
"[resourceId('Microsoft.Web/Sites', parameters('siteName'))]",
"[concat('Microsoft.Web/Sites/', parameters('siteName'), '/config/web')]"
],
"properties": {
"RepoUrl": "[parameters('repoUrl')]",
"branch": "[parameters('branch')]",
"IsManualIntegration": true
}
}
]
}
]
}
Loading

0 comments on commit 00a44fb

Please sign in to comment.