Inconsistency of the result in GPU and CPU? Will crop affect the result a lot?

[shuangao]

I am trying to see the accuracy of the validation set. However I found the result for standalone validation accuracy is not consistent with the accuracy while training.

Like here, I output the top1 top3 and top5 accuracy (with GPU)

I0121 00:26:24.870959 22427 solver.cpp:247] Iteration 30000, Testing net (#0)
I0121 00:29:49.700218 22427 solver.cpp:298]     Test net output #0: loss = 1.92555 (* 1 = 1.92555 loss)
I0121 00:29:49.847443 22427 solver.cpp:298]     Test net output #1: top 1 accuracy = 0.62146
I0121 00:29:49.847465 22427 solver.cpp:298]     Test net output #2: top 3 accuracy = 0.790921
I0121 00:29:49.847482 22427 solver.cpp:298]     Test net output #3: top 5 accuracy = 0.849442

However, when I want to see the accuracy of the model on iteration 30000 with the same validation set, I found the result is not the same: (CPU and python wrapper)
top1: 0.430361445783
top3: 0.632449799197
top5: 0.721124497992
My Code

net = caffe.Classifier('models/fine_tune.prototxt',
                       'models/fine_tune.caffemodel') // initialize the model with 30000 iteration one
net.set_phase_test()
net.set_mode_cpu()
mean = BinaryProbToNPY('models/food256_mean.binaryproto',256,256,3) // load binaryproto
net.set_mean('data', mean)  
net.set_raw_scale('data', 255)  
net.set_channel_swap('data', (2,1,0))  

path = np.genfromtxt('test/val.txt',usecols=(0),delimiter=' ',dtype=None) // load the validation set path
inputs = [caffe.io.load_image('test/val/'+im_f) for im_f in path]
# Will crop strategy cause such a big difference here? I use the whole image instead of crops.
# I cropped the images in training
scores = net.predict(inputs,oversample = False)
np.save('scores.npy',scores) #save result
# true labels in validation set
true_label = np.genfromtxt('test/val.txt',usecols=(1),delimiter=' ',dtype=None)
top_k = np.asarray([score.argsort()[-1::-1] for score in scores])
# top 1,3,5 accuracy
print top_n_acc(top_k,true_label,1)
print top_n_acc(top_k,true_label,3)
print top_n_acc(top_k,true_label,5)

Here is the output

0.430361445783
0.632449799197
0.721124497992

I am not sure whether this is caused by my crop strategy. If not, Hopefully there is no mistakes in my code.

[shelhamer]

See #1728 committed to dev in 6b0ad49. There was an issue with setting phase and device through pycaffe. You need to call caffe.set_phase_test() (or train) before making the net.

For a suggestion on running nets through Python to match training and test results, see https://groups.google.com/d/msg/caffe-users/r2w27xzSLu8/T5G6yJIbIoAJ.

Closing since the fix is in master and dev. Post to caffe-users for pycaffe usage questions. Thanks.