Show Facebook Computer Vision Tags Chrome Extension

Since April 2016, Facebook has been automatically adding alt tags to images you upload that are populated with keywords representing the content of your images:

<img csrc="https://facebook.com/some-url.jpg"
alt="Image may contain: golf, grass, outdoor and nature"
width="316" height="237">

They are labeling your images using a Deep ConvNet built by Facebook's FAIR team.

On one hand, this is really great. It improves accessibility for blind users who depend on screen readers which are only capable of processing text.

But I think a lot of internet users don't realize the amount of information that is now routinely extracted from photographs. Facebook (and Google, Apple, Amazon, etc) can easily tell from your photographs if you have a pet dog, if you collect cameras, if you play golf, if you have children, or if you are just really into sunglasses. There's nothing stopping them from using this information to show you relevant ads just based on photos of you - even if another user uploaded the photo and didn't even directly tag you!

This is a very simple Chrome Extension that I hacked together in a few minutes to make it easy to see the tags that Facebook is automatically applying to you and your friends' photos. Once you install it, all the photos you see on your Facebook timeline will automatically be overlaid with their tags, like this:

The goal is simply to make everyone aware of the kind of information that is routinely extracted from your own images today. This isn't some technology that is 'coming soon', but this is something in wide-spread production use now by every major internet company. You can even learn how to implement this kind of system yourself using widely available open-source software.

Here's some examples of the kinds of things that Facebook detects in your photos:

Objects in your photos

It's possible to tell what kinds of objects are in your photos and identify your interests based on that.

Activities you are doing

Whether you are eating, running, or playing golf, it's possible for an algorithm to tell what you are doing in your photographs.

When and where a photo was taken

You can tell if a photo was taken inside or outside, what the time of day was, what physical landmarks (mountains, trees, water, etc) were around, and so on.

Events you are attending

It's even possible to tell which kind of sporting event you are attending.

These are just a few examples. See what kinds of things Facebook has found in your own photos!

Installing this Chrome Extension

Option 1: From the Chrome Web Store

1. Update: Sorry, this is extension not currently available in the Chrome Web Store. Facebook filed a Trademark complaint due to the name of the extension having the word "Facebook" in it, so I had to remove it.

Option 2: From Source

1. Clone this repo so you have a copy in a folder locally.
2. Open chrome://extensions in the location or go to Tools > Extensions
3. Enable Developer mode by checking the checkbox in the upper-right corner.
4. Click on the button labelled Load unpacked extension....
5. Select the directory where you cloned this repo to.
6. Visit Facebook!

Installing in Firefox

Option 1: From the Add-ons website:

1. Click here and choose "+ Add to Firefox"

Option 2: From Source

1. Clone this repo so you have a copy in a folder locally.
2. Open about:debugging URL
3. Click on the button labelled Load Temporary Add-on.
4. Select the directory where you cloned this repo to, and select facebook.js
5. Visit Facebook!

Caveats

1. This only works for English-speaking Facebook users. I made no attempt to make this work for any other language. Sorry! But you could easily fork this and make it work for a different language. There's not a lot to this.

• Álvaro Sarasúa made a Spanish version of this extension here.
• Nicolas Raube made a German version of this extension here.

Press

• New York Magazine select/all - See What Facebook Thinks Is in Your Photos
• Nylon - You Can Now See What Information Facebook Gathers From Your Photos
• Mashable - Curious about what Facebook thinks is in your photos? This extension shows you.
• lifehacker - Show Facebook Computer Vision for Chrome Reveals Facebook's Hidden Photo Tags
• PetaPixel - This Chrome Add-On Shows You What Facebook ‘Sees’ In Your Photos
• DigitalTrends - Browser Extension Unveils the Hidden Tags Facebook Gives Your Photos
• The Daily Dot - This Chrome extension reveals Facebook's hidden image tags
• Metro UK - The amount of data Facebook detects in a single photo upload is terrifying
• DailyMail - Facebook's secret picture tags revealed: Plugin lets you see what the firm's AI thinks is in your snaps (and it's scarily accurate)
• t3n - Was Facebooks KI auf euren Fotos erkennt: Diese Erweiterung zeigt es euch
• BFM TV - Découvrez tout ce que Facebook sait de vous en analysant vos photos
• GQ France - Facebook affine son ciblage grâce à vos photos
• Meio Bit - Veja os objetos que a IA do Facebook reconhece em suas fotos
• CodigoFonte - Plugin para Firefox e Chrome exibe o que o Facebook vê em suas fotos
• 24h.com.vn - Facebook đang phân tích từng bức ảnh ra sao?
• ZDNet Korea - 브라우저에 페이스북의 눈을 달아보자
• unwire.hk - 表情動作場合都辨認到！加個插件即可看到 Facebook 圖像識別技術到底有幾犀利