[Slim] Imagenet training not utilizing multiple GPUs efficiently

[redserpent7]

Hi,

I am running some tests on Slim's imagenet training using Inception Resnet V2. The training is done on AWS ec2 instances (p2.xlarge and p2.8xlarge): Here are the specs for both:

1. p2.xlarge: GPUs (1), vCPUs (4), Ram (61GB)
2. p2.8xlarge: GPUs (8), vCPUs (32), Ram (488GB)

The GPUs are all Nvidia Tesla K80

Tensorflow seems to detect and loads the training on all GPUs according to both the training output and nvidia-smi. However there does not seem to be much difference in execution times.

On the p2.xlarge instance, TF/Slim reported an average of 3.05 sec/step.
On the p2.8xlarge instance it reported an average of 2.96 sec/step

I was expecting the time to drop significantly but given the above results I do not see a huge benefit running the training on multiple GPUs.

Both instances have a copy of the same exact training datasets and scripts. I am running the training using this command:

DATASET_DIR=/imagenet2
TRAIN_DIR=/imagenet2/train_logs
python train_image_classifier.py \
	--train_dir=${TRAIN_DIR} \
	--dataset_name=imagenet \
	--dataset_split_name=train \
	--dataset_dir=${DATASET_DIR} \
	--max_number_of_steps=20000 \
	--model_name=inception_resnet_v2

Both instances running Tensorflow 1.0.1 running from binary as VM
Both instances are running Ubuntu 14.04 x64

Regards

[tobiajo]

Have you tried to set num_clones to the number of GPUs?

[drpngx]

@tfboyd for performance.

[tfboyd]

That script does not detect the number of GPUs automatically. You have to set it. To double check run nvidia-smi or watch -n 3 nvidia-smi (I think that is correct) and make sure you see 8 python processes and all 8 GPUs are active. I have not run that model on slim but I have trained some of the other models and you should see a speedup.

[redserpent7]

Ok I have set num_clones to 8 which is the number of GPUs I have. I can confirm that using nvidia-smi it shows me that all GPUs are being fully utilized. Volatile GPU-Util shows each GPU at ~88% - 98% and I can see all GPUs running the training process with 10937MiB.

However, the situation became worse. On a single GPU training reports an average of 3.1 sec/step. On 8 GPUs its now reporting an average of 3.4 sec/step.

So I am really confused now as it seems not using num_clones gave better results

[tfboyd]

I will try to reproduce it today locally and go from there. I have run a few models via slim but not this one.

[tfboyd]

I have not forgotten you. I had some personal items and some cleanup to do on another task.

[tfboyd]

Ahh I forgot until I setup the example and ran it. What you are seeing is 100% correct and makes sense. Slim is reporting time for the step. You are running synchronized training so your images/sec would be

• (32/3.1) * 1(GPU) = 10.3 images/sec
• (32/3.4) * 8(GPUs) = 74 images/sec

[redserpent7]

@tfboyd interesting. So can you explain this equation? Namely what does the number 32 represent?

[tfboyd]

@redserpent7 Apologies and I mean that with sincerity. It makes me crazy when people do not explain things. And I apologize again if I am to detailed but that is the best way I can think of to explain it.

32 is the batch-size, which is the default and very popular so I assume that is what you used.

With one GPU you did 32 images in 3.1 seconds, which would be 10.2 images per second
With 8 GPUs you did 8 GPUS * 32 images per GPU for a total of 256 images in 3.4 seconds, which is 74 images per second.

What happens when you add more GPUs is that you are processing a larger batch size. So instead of 32 images in 3.1 seconds you are processing 256 images in 3.4 seconds. You could add this formula to the script to get images per second and account for the number of clones

I definitely understand how this is confusing. I setup a VM and setup the code before I realized the answer to your question. I know I marked this closed but please, as I said, feel free to ask more questions. I am happy to help. Best of luck.

[redserpent7]

@tfboyd thaks so mucb for the info much appreciated. Well it all make sense now. So I am now trying to calculate the number of steps that I will require. I first started on 1 GPU for 10000 steps. This gave me a top-1 accuracy of 0.0022. I then tried running 100000 steps which assuming a linear progress should yeild 0.022 accuracy and using 8 GPUs should give me 8 times that.

Am I correct in my assumption?