Fail training Visual Genome dataset

[andrew-yang0722]

I tried to train with Visual Genome dataset as you described.
I just follow the default parameter of preprocess.py to make data for training.
I faced error at the beginning.
What can I check?

Successfully loaded /disk3/daum/iu/github/densecap/data/models/vgg-16/VGG_ILSVRC_16_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
total number of parameters in net: 135734875
total number of parameters in CNN: 14454528
/home/andrew/torch/install/bin/luajit: /home/andrew/torch/install/share/lua/5.1/nn/Container.lua:67:
In 3 module of nn.Sequential:
./densecap/modules/BoxSampler.lua:104: attempt to call method 'scatter' (a nil value)
stack traceback:
./densecap/modules/BoxSampler.lua:104: in function 'forward'
./densecap/modules/BoxSamplerHelper.lua:78: in function 'forward'
./densecap/LocalizationLayer.lua:426: in function 'f'
./densecap/LocalizationLayer.lua:228: in function 'timeit'
./densecap/LocalizationLayer.lua:425: in function <./densecap/LocalizationLayer.lua:383>
[C]: in function 'xpcall'
/home/andrew/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
/home/andrew/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
./densecap/DenseCapModel.lua:253: in function 'forward'
./densecap/DenseCapModel.lua:406: in function 'forward_backward'
train.lua:90: in function 'lossFun'
train.lua:120: in main chunk
[C]: in function 'dofile'
...drew/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

WARNING: If you see a stack trace below, it doesn't point to the place where this error occured. Please use only the one above.
stack traceback:
[C]: in function 'error'
/home/andrew/torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors'
/home/andrew/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
./densecap/DenseCapModel.lua:253: in function 'forward'
./densecap/DenseCapModel.lua:406: in function 'forward_backward'
train.lua:90: in function 'lossFun'
train.lua:120: in main chunk
[C]: in function 'dofile'
...drew/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

[jcjohnson]

The relevant part of the error message is

./densecap/modules/BoxSampler.lua:104: attempt to call method 'scatter' (a nil value)

which makes me think that you are running a very old version of torch. Try updating all the core torch packages:

luarocks install torch
luarocks install cutorch

[agude]

I too get the same error. Our docker images were built (using luarocks install torch and
luarocks install cutorch) within the last few days. Additionally, updating them by running those commands in the container does not solve the issue.

Are there other packages that might be out of date?

[soumith]

no, scatter and gather are defined just in torch / cutorch.
I'll do a test run of densecap tonight with a fresh install of torch and see if I can reproduce it.

[lingxiangwu]

I came across the same problem today exactly after I updated the cutorch.

[soumith]

i reproduced the issue. looking into it.

[soumith]

The quickest way to fix this for you is to install cutorch / cunn versions 1.0-0

luarocks remove cutorch --force
luarocks remove cunn --force

luarocks install cutorch 1.0-0
luarocks install cunn 1.0-0

The current cutorch master introduced some minor breaking changes to the API because we had to move to proper types on CUDA.

https://github.com/torch/cutorch#api-changes-and-versioning

I am working on fixing / making sure that the densecap code runs as is on the cutorch / cunn master. I'll send a PR to densecap / torch-rnn today

This is a high priority issue. Thanks for bring this to attention.

[jcjohnson]

Awesome, thanks @soumith! These look like good API changes in the long run.

[soumith]

actually, deleted my previous comment. stick to cutorch 1.0-0. Will fix this later today in cutorch master and report back.

[agude]

Thanks @soumith! I'm giving pinning the versions a try now.

[agude]

Pinning the versions allows me to train, so I'll use that for now!

[lingxiangwu]

Now that works. Thank you @soumith

[soumith]

Now cutorch master should work fine with densecap, with PR: torch/cutorch#461 and torch/cutorch#459 , torch/nn#924 , torch/cunn#319

Sorry, it took longer than expected. The work was quite involved.