Is it (F)ake?!

Image Classification using TensorFlow.js and Elasticsearch

This repo documents my exciting journey into learning machine learning practices for the web. It is featured in the talk of the same name at Devoxx UK 2024.

You can play the game and pit yourself against the models at https://is-it-fake.netlify.app/.

Structure

This repo contains two applications:

1. Is it (F)ake?! game, located in the cake-app folder where players can guess if the image is cake or not cake. This application is written using React, Elasticsearch Serverless Node.js client and Netlify functions.
2. The Model Classification application is written in vanilla JavaScript, HTML and CSS. It is present in the model-playground folder and contains several useful elements:

• A sample webpage showing predictions generated using the COCO-SD and MobileNet model on a sample image.
• The classify-images.js script includes a Node.js implementation generating image classifications of a set of source images for MobileNet and COCO-SSD. These results are persisted in Elasticsearch.
• A custom sequential image classification model is created and executed in custom-model.js. These results are also persisted in Elasticsearch, and the model is saved in the model folder.
• The script transfer-learning.js includes an example approach combining MobileNet with an additional classification head to determine the presence of cake in our image set. Just like our other scripts, the generated classifications are persisted in Elasticsearch.

How to run

See below for how to run the applications and tests.

cake-game

Start the application on http://localhost:5173/, ensuring you have Netlify CLI installed:

cd ./cake-game
npm install
npm install netlify-cli -g
netlify dev

Execute the component tests:

cd ./cake-game
npm install
npm run test-ct

model-classification-app

To start the model application on http://localhost:3000:

cd ./model-classification-app
npm install
npm run start

Each of the scripts for performing different types of image classification should be executed from the model-classification-app folder. To persist the classifications please ensure you have a simple Elasticsearch cluster available to persist the results, with environment variables ELASTIC_CLOUD_ID and ELASTIC_API_KEY exposed to your service.

Once the Elasticsearch prerequisite has been met you can run the scripts in turn similar to the below:

cd ./model-classification-app
npm install
node scripts/classify-images.js
node scripts/custom-model.js
node scripts/transfer-learning.js

Alternatively, to persist your results to an alternative store you need to override the respective functions in elasticsearch-util.js to send your classifications to the right place.

Learning Resources

1. TensorFlow.js API Reference
2. MobileNet
3. COCO-SSD
4. Kaggle Models

Machine Learning Explained

1. IBM: What are convolutional neural networks?
2. Kaggle: Basic Terminologies of Convolutional Neural Networks
3. National High Magnetic Field Laboratory: Convolution Kernels
4. Deeplizard: Max Pooling in Convolutional Neural Networks
5. Govinda Dumane: Introduction to Convolutional Neural Network (CNN) using Tensorflow
6. AI Wiki: Accuracy and Loss

Tutorials

1. Classification Model On Custom Dataset Using Tensorflow.js Made Simple: Here's What You Need to Know | Garima Nisha
2. Machine Learning for Web Developers (WebML) | Google for Developers
3. Training an image model with TensorFlow in Node.js | ????
4. TensorFlow.js: Make your own "Teachable Machine" using transfer learning with TensorFlow.js | Google Codelab
5. Google: Google AI for JavaScript developers with TensorFlow.js

Common Issues

1. CORS Issue with TensorFlow.js
2. TensorFlow incorrect tensor shape sample error
3. Tensor has disposed
4. TensorFlow.js TypeError: Cannot read property 'backend' of undefined
5. TensorFlow.js Cannot read property 'backend' of undefined in NodeJS
6. TensorFlow.js TypeError: Cannot read property 'backend' of undefined - tensor is already disposed when moveData is called

Sources

Thanks go to the fantastic bakers out there making hyper-realistic cakes that inspired this talk and have given me great family memories. Specific thanks go to these bakers whose images I used for training and classification:

1. https://www.thelondonbaker.com
2. https://hemussweetsensation.com
3. https://www.everythingjustbaked.com/
4. https://ninamariacharles.com
5. https://www.thesugardreams.net
6. https://juliemcallistercakes.com
7. https://www.thesweetstopofrva.com

Thanks also to Unsplash and their amazing contributors who allowed me to find non-cake images through the JavaScript wrapper unsplash-js of the Unsplash API.