Last updated: 29/01/2019 with TensorFlow v1.10
Changes: Added note that the train.py file is now located in the /object_detection/legacy folder and must be moved into the main folder before issuing the training command.
This GitHub repository is an step-by-step explanation about using Tensorflow framework for creating an object detector. This explanantion was created based on EdjeElectronics tutorial, so thanks to him for making a really good tutorial. Really apreciate it. My another inspirarion is Juan De Dios Santos with his tutorial on making an object detector to detect a pikachu. Really thanks to him too.
If you want to watch the oral tutotial you can visit this youtube link created by EdjeElectronics and Juan De Sios Santos below :
- How To Train an Object Detection Classifier Using TensorFlow 1.5 (GPU) on Windows 10
- Detecting Pikachu on Android Using Tensorflow Object Detection
The repository provides all the things needed to create an "Indonesian Plate Number" detection. The reason why i choose this object is because i live in Indonesia. If you want to create your own object detecion, you can alaso follow this explanation till the end.
This explaination was purposed to tell you about the way to make your own single/multiple object detection. This tutorial will guide you step-by-step on how to make a single object detection, but you can customize it to become a multiple object detection (just adding the other object dataset and do a simple configuration !!). Please follow this tutorial carefully so you shouldn't found an error in the future.
There are 2 types of tensorflow. First is tensorflow-GPU and the other is tensorflow-CPU. I use tensorflow-GPU because it can reduce the training time. The reason why tensorflow-GPU is faster than tensorflow-CPU can be found here.
You can install the tensorFlow-GPU by following the instructions in this YouTube Video by Mark Jay or in this tensorflow website.
If you want to use the tensorflow-GPU so you must install CUDA and cuDNN. You can choose and download the version you want here for CUDA and here for cuDNN. I recommand CUDA v9.0 and cuDNN v7.0, cause this version is compatible with the tensorflow-GPU 1.5.
Make sure to install Anaconda with Python 3.6 as instructed in the video, because the Anaconda virtual environment will be used for this tutorial.
The TensorFlow Object Detection API requires using the specific directory structure provided in its GitHub repository. It also requires several additional Python packages, specific additions to the PATH and PYTHONPATH variables, and a few extra setup commands to get everything set up to run or train an object detection model.
This portion of the tutorial goes over the full set up required. It is fairly meticulous, but follow the instructions closely, because improper setup can cause unwieldy errors down the road.
Create folder wherever you want, for example "C:\tf_detect". This folder will contain all the files you needed to create your own object detection. To do that, you can simply clone this repository to your folder or download and extract it to the folder. I have merged the tensorflow framework from this GitHub commit and from EdjeElectroincs GitHub, so if you prefered to use the another tensorflow GitHub commit, i can't make sure that it will work.
After you clone/download this repository, jump to "C:\tf_detect\models\research\object_detection" your folder will look like the picture below :
This repository contains the images, annotation data, .csv files, and TFRecords needed to train an "Indonesian Plate Number" detector. It also contains Python scripts that are used to generate the training data. It has scripts to test out the object detection classifier on images, videos, or a webcam.
This tutorial will use a fine tuning method or also well known as transfer leraning. So, to do that you must download the pre-trained network. TensorFlow provides several object detection models (pre-trained classifiers with specific neural network architectures) in its model zoo. You can download whatever model that you want. I use fasterRCNN Inception v2 for my experiment (you can see the experiment result screenshoot above).
After downloading the model, you should extract the model folder to the C:\tf_detect\models\research\object_detection folder. (Note: The model date and version will likely change in the future, but it should still work with this tutorial.)
If you want to train your own object detector, delete the following files (do not delete the folders):
- All files in \object_detection\images\train and \object_detection\images\test
- The “test_labels.csv” and “train_labels.csv” files in \object_detection\images
- All files in \object_detection\training
- All files in \object_detection\inference_graph
Now, you are ready to start from scratch in training your own object detector. I assume that you have deleted the file listed above, so follow this tutorial carefully.
From the Start menu in Windows, search "Anaconda Prompt", than Run as Administrator. Click "Yes" if windows ask to allow or deny the utility.
When the command terminal pops up, create a new virtual environment whatever that name, for example it called “tf_detect” by typing the following command:
C:\> conda create -n tf_detect pip python=3.5
Then, activate the environment you have created by typing:
C:\> activate tf_detect
Install tensorflow-gpu in this environment by typing:
(tf_detect) C:\> pip install --ignore-installed --upgrade tensorflow-gpu
Install the other necessary packages by typing the following commands:
(tf_detect) C:\> conda install -c anaconda protobuf
(tf_detect) C:\> pip install pillow
(tf_detect) C:\> pip install lxml
(tf_detect) C:\> pip install Cython
(tf_detect) C:\> pip install jupyter
(tf_detect) C:\> pip install matplotlib
(tf_detect) C:\> pip install pandas
(tf_detect) C:\> pip install opencv-python
(Note: The ‘pandas’ and ‘opencv-python’ packages are not needed by TensorFlow, but they are used in the Python scripts to generate TFRecords and to work with images, videos, and webcam feeds.)
A PYTHONPATH variable must be created that points to the \models, \models\research, and \models\research\slim directories. Do this by issuing the following commands (from any directory):
(tf_detect) C:\> set PYTHONPATH=C:\tf_detect\models;C:\tf_detect\models\research;C:\tf_detect\models\research\slim
(Note: Every time the "tf_detect" virtual environment is exited, the PYTHONPATH variable is reset and needs to be set up again.)
Next, compile the Protobuf files, which are used by TensorFlow to configure model and training parameters. Unfortunately, the short protoc compilation command posted on TensorFlow’s Object Detection API installation page does not work on Windows. Every .proto file in the \object_detection\protos directory must be called out individually by the command.
In the Anaconda Command Prompt, change directories to the \models\research directory and copy and paste the following command into the command line and press Enter:
protoc --python_out=. .\object_detection\protos\anchor_generator.proto .\object_detection\protos\argmax_matcher.proto .\object_detection\protos\bipartite_matcher.proto .\object_detection\protos\box_coder.proto .\object_detection\protos\box_predictor.proto .\object_detection\protos\eval.proto .\object_detection\protos\faster_rcnn.proto .\object_detection\protos\faster_rcnn_box_coder.proto .\object_detection\protos\grid_anchor_generator.proto .\object_detection\protos\hyperparams.proto .\object_detection\protos\image_resizer.proto .\object_detection\protos\input_reader.proto .\object_detection\protos\losses.proto .\object_detection\protos\matcher.proto .\object_detection\protos\mean_stddev_box_coder.proto .\object_detection\protos\model.proto .\object_detection\protos\optimizer.proto .\object_detection\protos\pipeline.proto .\object_detection\protos\post_processing.proto .\object_detection\protos\preprocessor.proto .\object_detection\protos\region_similarity_calculator.proto .\object_detection\protos\square_box_coder.proto .\object_detection\protos\ssd.proto .\object_detection\protos\ssd_anchor_generator.proto .\object_detection\protos\string_int_label_map.proto .\object_detection\protos\train.proto .\object_detection\protos\keypoint_box_coder.proto .\object_detection\protos\multiscale_anchor_generator.proto .\object_detection\protos\graph_rewriter.proto
This creates a name_pb2.py file from every name.proto file in the \object_detection\protos folder.
(Note: TensorFlow occassionally adds new .proto files to the \protos folder. If you get an error saying ImportError: cannot import name 'something_something_pb2' , you may need to update the protoc command to include the new .proto files.)
Finally, run the following commands from the C:\tensorflow1\models\research directory:
(tf_detect) C:\tf_detect\models\research> python setup.py build
(tf_detect) C:\tf_detect\models\research> python setup.py install
The TensorFlow Object Detection framework is now all set up to use pre-trained models for object detection, or to train a new one. You can test it out and verify your installation is working by launching the object_detection_tutorial.ipynb script with Jupyter. From the \object_detection directory, type this command:
(tf_detect) C:\tf_detect\models\research\object_detection> jupyter notebook object_detection_tutorial.ipynb
This opens the script in your default web browser and allows you to step through the code one section at a time. You can step through each section by clicking the “Run” button in the upper toolbar. The section is done running when the “In [ * ]” text next to the section populates with a number (e.g. “In [1]”).
(Note: part of the script downloads the ssd_mobilenet_v1 model from GitHub, which is about 74MB. This means it will take some time to complete the section, so be patient.)
Once you have stepped all the way through the script, you should see two labeled images at the bottom section the page. If you see this, then everything is working properly! If not, the bottom section will report any errors encountered. See the Appendix for a list of errors I encountered while setting this up.
Now that the TensorFlow Object Detection framework is ready, we need to provide the object dataset, it will used to train a new detection classifier.
TensorFlow needs minimum a hundreds of images of an object to train a good detection classifier. To train a robust classifier, the training images should have random objects in the image along with the desired objects, and should have a variety of backgrounds and lighting conditions. There should be some images where the desired object is partially obscured, overlapped with something else, or only halfway in the picture.
This is the sample dataset look like
You can use a smartphone to take pictures of the objects or download images of the objects from Google Image Search. I recommend having at least 200 pictures overall. I use a hundred of images and 2 minutes video for the dataset.
Make sure the images aren’t too large. They should be less than 200KB each, and their resolution shouldn’t be more than 720x1280. The larger the images are, the longer it will take to train the classifier. You can use the resizer.py script in this repository to reduce the size of the images.
After you have all the pictures you need, move 20% of them to the \object_detection\images\test directory, and 80% of them to the \object_detection\images\train directory. Make sure there are a variety of pictures in both the \test and \train directories.
Here comes the fun part! With all the pictures gathered, it’s time to label the desired objects in every picture. LabelImg is a great tool for labeling images, and its GitHub page has very clear instructions on how to install and use it.
Download and install LabelImg, point it to your \images\train directory, and then draw a box around each object in each image. Repeat the process for all the images in the \images\test directory. This will take a while!
LabelImg saves a .xml file containing the label data for each image. These .xml files will be used to generate TFRecords, which are one of the inputs to the TensorFlow trainer. Once you have labeled and saved each image, there will be one .xml file for each image in the \test and \train directories.
Also, you can check if the size of each bounding box is correct by running sizeChecker.py
(tf_detect) C:\tf_detect\models\research\object_detection> python sizeChecker.py --move
With the images labeled, it’s time to generate the TFRecords that serve as input data to the TensorFlow training model. This tutorial uses the xml_to_csv.py and generate_tfrecord.py scripts from Dat Tran’s Raccoon Detector dataset, with some slight modifications to work with our directory structure.
First, the image .xml data will be used to create .csv files containing all the data for the train and test images. From the \object_detection folder, issue the following command in the Anaconda command prompt:
(tf_detect) C:\tf_detect\models\research\object_detection> python xml_to_csv.py
This creates a train_labels.csv and test_labels.csv file in the \object_detection\images folder.
Next, open the generate_tfrecord.py file in a text editor. Replace the label map starting at line 31 with your own label map, where each object is assigned an ID number. This same number assignment will be used when configuring the labelmap.pbtxt file in Step 5b.
For example, say you are training a classifier to detect basketballs, shirts, and shoes. You will replace the following code in generate_tfrecord.py:
# TO-DO replace this with label map
def class_text_to_int(row_label):
if row_label == 'nine':
return 1
elif row_label == 'ten':
return 2
elif row_label == 'jack':
return 3
elif row_label == 'queen':
return 4
elif row_label == 'king':
return 5
elif row_label == 'ace':
return 6
else:
return None
With this:
# TO-DO replace this with label map
def class_text_to_int(row_label):
if row_label == 'chair':
return 1
elif row_label == 'table':
return 2
elif row_label == 'cup':
return 3
else:
return None
if you want a multiple object detection, but in this experiment i use this code :
# TO-DO replace this with label map
def class_text_to_int(row_label):
if row_label == 'PlateNumber':
return 1
else:
return None
Then, generate the TFRecord files by issuing these commands from the \object_detection folder:
python generate_tfrecord.py --csv_input=images\train_labels.csv --image_dir=images\train --output_path=train.record
python generate_tfrecord.py --csv_input=images\test_labels.csv --image_dir=images\test --output_path=test.record
These generate a train.record and a test.record file in \object_detection. These will be used to train the new object detection classifier.
The last thing to do before training is to create a label map and edit the training configuration file.
The label map tells the trainer what each object is by defining a mapping of class names to class ID numbers. Use a text editor to create a new file and save it as labelmap.pbtxt in the C:\tf_detect\models\research\object_detection\training folder. (Make sure the file type is .pbtxt, not .txt !) In the text editor, copy or type in the label map in the format below (the example below is the label map for my Plate Number Detector):
item {
id: 1
name: 'PlateNumber'
}
The label map ID numbers should be the same as what is defined in the generate_tfrecord.py file. For the chair, table, and cup detector example mentioned in Step 4, the labelmap.pbtxt file will look like:
item {
id: 1
name: 'chair'
}
item {
id: 2
name: 'table'
}
item {
id: 3
name: 'cup'
}
Finally, the object detection training pipeline must be configured. It defines which model and what parameters will be used for training. This is the last step before running training!
Navigate to C:\tf_detect\models\research\object_detection\samples\configs and copy the faster_rcnn_inception_v2_pets.config file into the \object_detection\training directory. Then, open the file with a text editor. There are several changes to make to the .config file, mainly changing the number of classes and examples, and adding the file paths to the training data.
Make the following changes to the faster_rcnn_inception_v2_pets.config file. Note: The paths must be entered with single forward slashes (NOT backslashes), or TensorFlow will give a file path error when trying to train the model! Also, the paths must be in double quotation marks ( " ), not single quotation marks ( ' ).
-
Line 9. Change num_classes to the number of different objects you want the classifier to detect. For the above basketball, shirt, and shoe detector, it would be num_classes : 3 .
-
Line 110. Change fine_tune_checkpoint to:
- fine_tune_checkpoint : "C:/tf_detect/models/research/object_detection/faster_rcnn_inception_v2_coco_2018_01_28/model.ckpt"
-
Lines 126 and 128. In the train_input_reader section, change input_path and label_map_path to:
- input_path : "C:/tf_detect/models/research/object_detection/train.record"
- label_map_path: "C:/tf_detect/models/research/object_detection/training/labelmap.pbtxt"
-
Line 132. Change num_examples to the number of images you have in the \images\test directory.
-
Lines 140 and 142. In the eval_input_reader section, change input_path and label_map_path to:
- input_path : "C:/tf_detect/models/research/object_detection/test.record"
- label_map_path: "C:/tf_detect/models/research/object_detection/training/labelmap.pbtxt"
Save the file after the changes have been made. That’s it! The training job is all configured and ready to go!
UPDATE 9/26/18: As of version 1.9, TensorFlow has deprecated the "train.py" file and replaced it with "model_main.py" file. I haven't been able to get model_main.py to work correctly yet (I run in to errors related to pycocotools). Fortunately, the train.py file is still available in the /object_detection/legacy folder. Simply move train.py from /object_detection/legacy into the /object_detection folder and then continue following the steps below.
Here we go! From the \object_detection directory, issue the following command to begin training:
python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
If everything has been set up correctly, TensorFlow will initialize the training. The initialization can take up to 30 seconds before the actual training begins. When training begins, it will look like this:
Each step of training reports the loss. It will start high and get lower and lower as training progresses. For my training on the Faster-RCNN-Inception-V2 model, it started at about 3.0 and quickly dropped below 0.8. I recommend allowing your model to train until the loss consistently drops below 0.05, which will take about 40,000 steps, or about 2 hours (depending on how powerful your CPU and GPU are). Note: The loss numbers will be different if a different model is used. MobileNet-SSD starts with a loss of about 20, and should be trained until the loss is consistently under 2.
You can view the progress of the training job by using TensorBoard. To do this, open a new instance of Anaconda Prompt, activate the tensorflow1 virtual environment, change to the C:\tf_detect\models\research\object_detection directory, and issue the following command:
(tf_detect) C:\tf_detect\models\research\object_detection>tensorboard --logdir=training
This will create a webpage on your local machine at YourPCName:6006, which can be viewed through a web browser. The TensorBoard page provides information and graphs that show how the training is progressing. One important graph is the Loss graph, which shows the overall loss of the classifier over time.
The training routine periodically saves checkpoints about every five minutes. You can terminate the training by pressing Ctrl+C while in the command prompt window. I typically wait until just after a checkpoint has been saved to terminate the training. You can terminate training and start it later, and it will restart from the last saved checkpoint. The checkpoint at the highest number of steps will be used to generate the frozen inference graph.
Now that training is complete, the last step is to generate the frozen inference graph (.pb file). From the \object_detection folder, issue the following command, where “XXXX” in “model.ckpt-XXXX” should be replaced with the highest-numbered .ckpt file in the training folder:
python export_inference_graph.py --input_type image_tensor --pipeline_config_path training/faster_rcnn_inception_v2_pets.config --trained_checkpoint_prefix training/model.ckpt-XXXX --output_directory inference_graph
This creates a frozen_inference_graph.pb file in the \object_detection\inference_graph folder. The .pb file contains the object detection classifier.
The object detection classifier is all ready to go! I’ve written Python scripts to test it out on an image, video, or webcam feed.
Before running the Python scripts, you need to modify the NUM_CLASSES variable in the script to equal the number of classes you want to detect. (For my Pinochle Card Detector, there are six cards I want to detect, so NUM_CLASSES = 6.)
To test your object detector, move a picture of the object or objects into the \object_detection folder, and change the IMAGE_NAME variable in the Object_detection_image.py to match the file name of the picture. Alternatively, you can use a video of the objects (using Object_detection_video.py), or just plug in a USB webcam and point it at the objects (using Object_detection_webcam.py).
To run any of the scripts, type “idle” in the Anaconda Command Prompt (with the “tensorflow1” virtual environment activated) and press ENTER. This will open IDLE, and from there, you can open any of the scripts and run them.
If everything is working properly, the object detector will initialize for about 10 seconds and then display a window showing any objects it’s detected in the image!
If you encounter errors, please check out the Appendix: it has a list of errors that I ran in to while setting up my object detection classifier. You can also trying Googling the error. There is usually useful information on Stack Exchange or in TensorFlow’s Issues on GitHub.
It appears that the TensorFlow Object Detection API was developed on a Linux-based operating system, and most of the directions given by the documentation are for a Linux OS. Trying to get a Linux-developed software library to work on Windows can be challenging. There are many little snags that I ran in to while trying to set up tensorflow-gpu to train an object detection classifier on Windows 10. This Appendix is a list of errors I ran in to, and their resolutions.
This error occurs when you try to run object_detection_tutorial.ipynb or train.py and you don’t have the PATH and PYTHONPATH environment variables set up correctly. Exit the virtual environment by closing and re-opening the Anaconda Prompt window. Then, issue “activate tensorflow1” to re-enter the environment, and then issue the commands given in Step 2e.
You can use “echo %PATH%” and “echo %PYTHONPATH%” to check the environment variables and make sure they are set up correctly.
Also, make sure you have run these commands from the \models\research directory:
setup.py build
setup.py install
This occurs when the protobuf files (in this case, preprocessor.proto) have not been compiled. Re-run the protoc command given in Step 2f. Check the \object_detection\protos folder to make sure there is a name_pb2.py file for every name.proto file.
This occurs when you try to run the
“protoc object_detection/protos/*.proto --python_out=.”
command given on the TensorFlow Object Detection API installation page. Sorry, it doesn’t work on Windows! Copy and paste the full command given in Step 2f instead. There’s probably a more graceful way to do it, but I don’t know what it is.
4. Unsuccessful TensorSliceReader constructor: Failed to get "file path" … The filename, directory name, or volume label syntax is incorrect.
This error occurs when the filepaths in the training configuration file (faster_rcnn_inception_v2_pets.config or similar) have not been entered with backslashes instead of forward slashes. Open the .config file and make sure all file paths are given in the following format:
“C:/path/to/model.file”
5. ValueError: Tried to convert 't' to a tensor and failed. Error: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].
The issue is with models/research/object_detection/utils/learning_schedules.py Currently it is
rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
range(num_boundaries),
[0] * num_boundaries))
Wrap list() around the range() like this:
rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
list(range(num_boundaries)),
[0] * num_boundaries))