GPU OOM error

I used the code of branch tf-1.2 and my own training data. I am running Tensorflow 1.3 on single GPU. When I train the GNMT model, step-time is unbearable: about 50s for batch_size=128. How to make training faster?

I reinstall tensorflow 1.3 and now the following OOM error arises:

created eval model with fresh parameters, time 1.82s
2017-09-20 06:50:14.984896: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/cpu:0"]]
2017-09-20 06:50:14.984994: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/cpu:0"]]
2017-09-20 06:50:14.985225: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/cpu:0"]]
2017-09-20 06:50:14.985266: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/cpu:0"]]
2017-09-20 06:50:14.985344: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/cpu:0"]]
2017-09-20 06:50:14.985382: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/cpu:0"]]
eval dev: perplexity 309009.57, time 14s, Wed Sep 20 06:50:14 2017.
2017-09-20 06:50:26.386888: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.73GiB. Current allocation summary follows.
2017-09-20 06:50:26.386953: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.386963: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.386972: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 1, Chunks in use: 0 1.0KiB allocated for chunks. 8B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.386979: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.386986: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.386993: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387002: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 1, Chunks in use: 0 30.0KiB allocated for chunks. 512B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387011: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 1, Chunks in use: 0 52.2KiB allocated for chunks. 9.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387017: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387024: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387030: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387037: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387043: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387049: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387069: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 2, Chunks in use: 0 8.00MiB allocated for chunks. 1.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387078: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 1, Chunks in use: 0 9.50MiB allocated for chunks. 512.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387084: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387090: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387095: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387107: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387114: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 2, Chunks in use: 0 2.22GiB allocated for chunks. 54.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-09-20 06:50:26.387119: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 1.73GiB was 256.00MiB, Chunk State:
2017-09-20 06:50:26.387134: I tensorflow/core/common_runtime/bfc_allocator.cc:666] Size: 991.25MiB | Requested Size: 6.50MiB | in_use: 0, prev: Size: 11.50MiB | Requested Size: 6.50MiB | in_use: 1, next: Size: 760.99MiB | Requested Size: 760.99MiB | in_use: 1
2017-09-20 06:50:26.387143: I tensorflow/core/common_runtime/bfc_allocator.cc:666] Size: 1.26GiB | Requested Size: 48.00MiB | in_use: 0, prev: Size: 760.99MiB | Requested Size: 760.99MiB | in_use: 1
2017-09-20 06:50:26.387150: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0000 of size 1280
2017-09-20 06:50:26.387154: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0500 of size 1280
2017-09-20 06:50:26.387157: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0a00 of size 1280
2017-09-20 06:50:26.387162: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0f00 of size 256
2017-09-20 06:50:26.387165: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1000 of size 256
2017-09-20 06:50:26.387169: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1100 of size 16384
2017-09-20 06:50:26.387173: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5100 of size 256
2017-09-20 06:50:26.387177: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5200 of size 16384
2017-09-20 06:50:26.387180: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e9200 of size 256
2017-09-20 06:50:26.387184: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e9300 of size 16384
2017-09-20 06:50:26.387188: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050ed300 of size 256
2017-09-20 06:50:26.387192: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050ed400 of size 16384
2017-09-20 06:50:26.387195: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f1400 of size 256
2017-09-20 06:50:26.387199: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f1500 of size 16384
2017-09-20 06:50:26.387203: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f5500 of size 256
2017-09-20 06:50:26.387210: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f5600 of size 256
2017-09-20 06:50:26.387214: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f5700 of size 16384
2017-09-20 06:50:26.387217: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f9700 of size 256
2017-09-20 06:50:26.387221: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f9800 of size 256
2017-09-20 06:50:26.387225: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f9900 of size 256
2017-09-20 06:50:26.387229: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050f9a00 of size 4096
2017-09-20 06:50:26.387233: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050faa00 of size 256
2017-09-20 06:50:26.387236: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050fab00 of size 16384
2017-09-20 06:50:26.387240: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050feb00 of size 256
2017-09-20 06:50:26.387243: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050fec00 of size 16384
2017-09-20 06:50:26.387247: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205102c00 of size 256
2017-09-20 06:50:26.387251: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205102d00 of size 16384
2017-09-20 06:50:26.387255: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205106d00 of size 256
2017-09-20 06:50:26.387258: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205106e00 of size 256
2017-09-20 06:50:26.387262: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205106f00 of size 16384
2017-09-20 06:50:26.387265: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120510af00 of size 16384
2017-09-20 06:50:26.387269: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120510ef00 of size 16384
2017-09-20 06:50:26.387272: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205112f00 of size 256
2017-09-20 06:50:26.387276: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113000 of size 256
2017-09-20 06:50:26.387280: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113100 of size 256
2017-09-20 06:50:26.387283: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113200 of size 256
2017-09-20 06:50:26.387287: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113300 of size 256
2017-09-20 06:50:26.387290: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113400 of size 256
2017-09-20 06:50:26.387294: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113500 of size 256
2017-09-20 06:50:26.387297: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113600 of size 256
2017-09-20 06:50:26.387301: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113700 of size 256
2017-09-20 06:50:26.387304: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113800 of size 256
2017-09-20 06:50:26.387308: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113900 of size 256
2017-09-20 06:50:26.387311: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113a00 of size 256
2017-09-20 06:50:26.387315: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113b00 of size 256
2017-09-20 06:50:26.387318: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113c00 of size 256
2017-09-20 06:50:26.387322: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113d00 of size 256
2017-09-20 06:50:26.387325: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113e00 of size 256
2017-09-20 06:50:26.387329: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205113f00 of size 256
2017-09-20 06:50:26.387332: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205114000 of size 4096
2017-09-20 06:50:26.387339: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205115000 of size 16384
2017-09-20 06:50:26.387343: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205119000 of size 16384
2017-09-20 06:50:26.387347: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120511d000 of size 16384
2017-09-20 06:50:26.387350: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205121000 of size 16384
2017-09-20 06:50:26.387354: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205125000 of size 16384
2017-09-20 06:50:26.387358: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205129000 of size 16384
2017-09-20 06:50:26.387361: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120512d000 of size 4096
2017-09-20 06:50:26.387365: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120512e000 of size 16384
2017-09-20 06:50:26.387368: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205132000 of size 16384
2017-09-20 06:50:26.387372: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205136000 of size 16384
2017-09-20 06:50:26.387375: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120513a000 of size 16384
2017-09-20 06:50:26.387379: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120513e000 of size 16384
2017-09-20 06:50:26.387382: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205142000 of size 16384
2017-09-20 06:50:26.387386: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205146000 of size 4096
2017-09-20 06:50:26.387389: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205147000 of size 16384
2017-09-20 06:50:26.387393: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120514b000 of size 16384
2017-09-20 06:50:26.387396: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120514f000 of size 16384
2017-09-20 06:50:26.387400: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205153000 of size 256
2017-09-20 06:50:26.387403: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205153100 of size 16384
2017-09-20 06:50:26.387407: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205157100 of size 16384
2017-09-20 06:50:26.387410: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120515b100 of size 16384
2017-09-20 06:50:26.387414: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120515f100 of size 16384
2017-09-20 06:50:26.387417: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205163100 of size 16384
2017-09-20 06:50:26.387421: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205167100 of size 16384
2017-09-20 06:50:26.387425: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b100 of size 256
2017-09-20 06:50:26.387428: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b200 of size 256
2017-09-20 06:50:26.387432: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b300 of size 256
2017-09-20 06:50:26.387435: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b400 of size 256
2017-09-20 06:50:26.387439: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b500 of size 256
2017-09-20 06:50:26.387442: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b600 of size 256
2017-09-20 06:50:26.387446: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b700 of size 256
2017-09-20 06:50:26.387449: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b800 of size 256
2017-09-20 06:50:26.387453: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516b900 of size 256
2017-09-20 06:50:26.387456: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516ba00 of size 256
2017-09-20 06:50:26.387460: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516bb00 of size 256
2017-09-20 06:50:26.387466: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516bc00 of size 256
2017-09-20 06:50:26.387470: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516bd00 of size 256
2017-09-20 06:50:26.387474: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516be00 of size 256
2017-09-20 06:50:26.387477: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516bf00 of size 256
2017-09-20 06:50:26.387481: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516c000 of size 256
2017-09-20 06:50:26.387484: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516c100 of size 256
2017-09-20 06:50:26.387488: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516c200 of size 4096
2017-09-20 06:50:26.387491: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120516d200 of size 16384
2017-09-20 06:50:26.387495: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205171200 of size 16384
2017-09-20 06:50:26.387498: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205175200 of size 16384
2017-09-20 06:50:26.387502: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205179200 of size 4096
2017-09-20 06:50:26.387505: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517a200 of size 256
2017-09-20 06:50:26.387509: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517a300 of size 256
2017-09-20 06:50:26.387512: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517a400 of size 256
2017-09-20 06:50:26.387516: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517a500 of size 256
2017-09-20 06:50:26.387519: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517a600 of size 256
2017-09-20 06:50:26.387523: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517a700 of size 256
2017-09-20 06:50:26.387557: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517a800 of size 16384
2017-09-20 06:50:26.387564: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517e800 of size 256
2017-09-20 06:50:26.387567: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120517e900 of size 16384
2017-09-20 06:50:26.387571: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205182900 of size 256
2017-09-20 06:50:26.387575: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205182a00 of size 16384
2017-09-20 06:50:26.387578: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205186a00 of size 256
2017-09-20 06:50:26.387582: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205186b00 of size 16384
2017-09-20 06:50:26.387585: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120518ab00 of size 256
2017-09-20 06:50:26.387589: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120518ac00 of size 16384
2017-09-20 06:50:26.387593: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120518ec00 of size 256
2017-09-20 06:50:26.387596: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120518ed00 of size 256
2017-09-20 06:50:26.387600: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120518ee00 of size 16384
2017-09-20 06:50:26.387603: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205192e00 of size 256
2017-09-20 06:50:26.387607: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205192f00 of size 256
2017-09-20 06:50:26.387611: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205193000 of size 256
2017-09-20 06:50:26.387614: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205193100 of size 4096
2017-09-20 06:50:26.387618: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205194100 of size 256
2017-09-20 06:50:26.387628: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205194200 of size 16384
2017-09-20 06:50:26.387635: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205198200 of size 256
2017-09-20 06:50:26.387641: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205198300 of size 16384
2017-09-20 06:50:26.387647: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120519c300 of size 256
2017-09-20 06:50:26.387654: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120519c400 of size 16384
2017-09-20 06:50:26.387658: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051a0400 of size 256
2017-09-20 06:50:26.387662: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051a0500 of size 256
2017-09-20 06:50:26.387665: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051a0600 of size 16384
2017-09-20 06:50:26.387669: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051a4600 of size 16384
2017-09-20 06:50:26.387672: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051a8600 of size 16384
2017-09-20 06:50:26.387676: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051ac600 of size 16384
2017-09-20 06:50:26.387680: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b0600 of size 16384
2017-09-20 06:50:26.387683: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b4600 of size 16384
2017-09-20 06:50:26.387687: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8600 of size 256
2017-09-20 06:50:26.387690: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8700 of size 256
2017-09-20 06:50:26.387694: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8800 of size 256
2017-09-20 06:50:26.387698: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8900 of size 256
2017-09-20 06:50:26.387701: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8a00 of size 256
2017-09-20 06:50:26.387705: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8b00 of size 256
2017-09-20 06:50:26.387708: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8c00 of size 256
2017-09-20 06:50:26.387712: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8d00 of size 256
2017-09-20 06:50:26.387716: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8e00 of size 256
2017-09-20 06:50:26.387719: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b8f00 of size 256
2017-09-20 06:50:26.387723: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b9000 of size 256
2017-09-20 06:50:26.387726: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b9100 of size 256
2017-09-20 06:50:26.387730: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b9600 of size 256
2017-09-20 06:50:26.387733: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051b9700 of size 4096
2017-09-20 06:50:26.387737: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051ba700 of size 16384
2017-09-20 06:50:26.387740: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051be700 of size 16384
2017-09-20 06:50:26.387744: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051c2700 of size 16384
2017-09-20 06:50:26.387748: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051c6700 of size 4096
2017-09-20 06:50:26.387751: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051c7700 of size 256
2017-09-20 06:50:26.387755: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051c7800 of size 9472
2017-09-20 06:50:26.387759: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051d1500 of size 6656
2017-09-20 06:50:26.387765: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x12051b9200 of size 1024
2017-09-20 06:50:26.387772: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x12051c9d00 of size 30720
2017-09-20 06:50:26.387776: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x12051d2f00 of size 53504
2017-09-20 06:50:26.387781: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12051e0000 of size 33554432
2017-09-20 06:50:26.387786: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12072e0000 of size 4194304
2017-09-20 06:50:26.387792: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12076e0000 of size 4194304
2017-09-20 06:50:26.387799: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1207ae0000 of size 4194304
2017-09-20 06:50:26.387805: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1207ee0000 of size 4194304
2017-09-20 06:50:26.387811: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12082e0000 of size 4194304
2017-09-20 06:50:26.387815: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1208ae0000 of size 4194304
2017-09-20 06:50:26.387819: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x12086e0000 of size 4194304
2017-09-20 06:50:26.387823: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1208ee0000 of size 4194304
2017-09-20 06:50:26.387827: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12092e0000 of size 67108864
2017-09-20 06:50:26.387831: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x120d2e0000 of size 54525952
2017-09-20 06:50:26.387835: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12106e0000 of size 79691776
2017-09-20 06:50:26.387839: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12152e0000 of size 50331648
2017-09-20 06:50:26.387843: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12182e0000 of size 33554432
2017-09-20 06:50:26.387846: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x121a2e0000 of size 33554432
2017-09-20 06:50:26.387850: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x121c2e0000 of size 50331648
2017-09-20 06:50:26.387853: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x121f2e0000 of size 33554432
2017-09-20 06:50:26.387857: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12212e0000 of size 67108864
2017-09-20 06:50:26.387861: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12252e0000 of size 797954048
2017-09-20 06:50:26.387865: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1254bdd000 of size 1349529600
2017-09-20 06:50:26.387869: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12a52e0000 of size 50331648
2017-09-20 06:50:26.387873: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12a82e0000 of size 50331648
2017-09-20 06:50:26.387877: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12ab2e0000 of size 33554432
2017-09-20 06:50:26.387880: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12ad2e0000 of size 83886080
2017-09-20 06:50:26.387884: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12b22e0000 of size 50331648
2017-09-20 06:50:26.387887: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12b52e0000 of size 50331648
2017-09-20 06:50:26.387891: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12b82e0000 of size 50331648
2017-09-20 06:50:26.387895: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12bb2e0000 of size 33554432
2017-09-20 06:50:26.387898: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12bd2e0000 of size 50331648
2017-09-20 06:50:26.387902: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12c02e0000 of size 33554432
2017-09-20 06:50:26.387905: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12c22e0000 of size 33554432
2017-09-20 06:50:26.387909: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12c42e0000 of size 83886080
2017-09-20 06:50:26.387916: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12c92e0000 of size 50331648
2017-09-20 06:50:26.387920: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12cc2e0000 of size 50331648
2017-09-20 06:50:26.387924: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12cf2e0000 of size 93310976
2017-09-20 06:50:26.387928: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12d4bdd000 of size 1349529600
2017-09-20 06:50:26.387932: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13252e0000 of size 1145311232
2017-09-20 06:50:26.387936: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1369721000 of size 33554432
2017-09-20 06:50:26.387939: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x136b721000 of size 50331648
2017-09-20 06:50:26.387943: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x136f0a1000 of size 12058624
2017-09-20 06:50:26.387947: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13adb62000 of size 797954048
2017-09-20 06:50:26.387951: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13dd45f000 of size 1206390784
2017-09-20 06:50:26.387955: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x136e721000 of size 9961472
2017-09-20 06:50:26.387958: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x136fc21000 of size 1039405056
2017-09-20 06:50:26.387962: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x14254e0000 of size 797954048
2017-09-20 06:50:26.387966: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1454ddd000 of size 1406860288
2017-09-20 06:50:26.387970: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size:
2017-09-20 06:50:26.387977: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 85 Chunks of size 256 totalling 21.2KiB
2017-09-20 06:50:26.387982: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 1280 totalling 3.8KiB
2017-09-20 06:50:26.387987: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 9 Chunks of size 4096 totalling 36.0KiB
2017-09-20 06:50:26.387991: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 6656 totalling 6.5KiB
2017-09-20 06:50:26.387996: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 9472 totalling 9.2KiB
2017-09-20 06:50:26.388001: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 54 Chunks of size 16384 totalling 864.0KiB
2017-09-20 06:50:26.388005: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 4194304 totalling 24.00MiB
2017-09-20 06:50:26.388010: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 12058624 totalling 11.50MiB
2017-09-20 06:50:26.388015: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 9 Chunks of size 33554432 totalling 288.00MiB
2017-09-20 06:50:26.388020: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 11 Chunks of size 50331648 totalling 528.00MiB
2017-09-20 06:50:26.388025: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 54525952 totalling 52.00MiB
2017-09-20 06:50:26.388029: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 67108864 totalling 128.00MiB
2017-09-20 06:50:26.388034: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 79691776 totalling 76.00MiB
2017-09-20 06:50:26.388039: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 83886080 totalling 160.00MiB
2017-09-20 06:50:26.388044: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 93310976 totalling 88.99MiB
2017-09-20 06:50:26.388049: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 797954048 totalling 2.23GiB
2017-09-20 06:50:26.388053: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1145311232 totalling 1.07GiB
2017-09-20 06:50:26.388057: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1206390784 totalling 1.12GiB
2017-09-20 06:50:26.388066: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1349529600 totalling 1.26GiB
2017-09-20 06:50:26.388070: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1406860288 totalling 1.31GiB
2017-09-20 06:50:26.388075: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 8.31GiB
2017-09-20 06:50:26.388083: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit: 11332668621
InUse: 8925298432
MaxInUse: 10212418304
NumAllocs: 51911
MaxAllocSize: 2147483648

2017-09-20 06:50:26.388101: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ************___******x************************************xx
2017-09-20 06:50:26.388119: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[1664,279617]
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/ec2-user/nmt/nmt.py", line 495, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/ec2-user/nmt/nmt.py", line 488, in main
run_main(FLAGS, default_hparams, train_fn, inference_fn)
File "/home/ec2-user/nmt/nmt.py", line 481, in run_main
train_fn(hparams, target_session=target_session)
File "nmt/train.py", line 329, in train
sample_tgt_data)
File "nmt/train.py", line 250, in run_full_eval
eval_model, eval_sess, model_dir, hparams, summary_writer)
File "nmt/train.py", line 192, in run_internal_eval
summary_writer, "test")
File "nmt/train.py", line 500, in _internal_eval
ppl = model_helper.compute_perplexity(model, sess, label)
File "nmt/model_helper.py", line 276, in compute_perplexity
loss, predict_count, batch_size = model.eval(sess)
File "nmt/model.py", line 201, in eval
self.batch_size])
File "/usr/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1664,279617]
[[Node: dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](dynamic_seq2seq/decoder/output_projection/Tensordot/Reshape, dynamic_seq2seq/decoder/output_projection/Tensordot/Reshape_1)]]
[[Node: dynamic_seq2seq/truediv/_283 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_1596_dynamic_seq2seq/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

Caused by op u'dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul', defined at:
File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/ec2-user/nmt/nmt.py", line 495, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/ec2-user/nmt/nmt.py", line 488, in main
run_main(FLAGS, default_hparams, train_fn, inference_fn)
File "/home/ec2-user/nmt/nmt.py", line 481, in run_main
train_fn(hparams, target_session=target_session)
File "nmt/train.py", line 285, in train
single_cell_fn)
File "nmt/train.py", line 143, in create_eval_model
single_cell_fn=single_cell_fn)
File "nmt/gnmt_model.py", line 54, in init
single_cell_fn=single_cell_fn)
File "nmt/attention_model.py", line 55, in init
single_cell_fn=single_cell_fn)
File "nmt/model.py", line 96, in init
res = self.build_graph(hparams, scope=scope)
File "nmt/model.py", line 234, in build_graph
encoder_outputs, encoder_state, hparams)
File "nmt/model.py", line 351, in _build_decoder
logits = self.output_layer(outputs.rnn_output)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 450, in call
outputs = self.call(inputs, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/layers/core.py", line 137, in call
[0]])
File "/usr/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 2456, in tensordot
ab_matmul = matmul(a_reshape, b_reshape)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 1844, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 1289, in _mat_mul
transpose_b=transpose_b, name=name)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1664,279617]
[[Node: dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](dynamic_seq2seq/decoder/output_projection/Tensordot/Reshape, dynamic_seq2seq/decoder/output_projection/Tensordot/Reshape_1)]]
[[Node: dynamic_seq2seq/truediv/_283 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_1596_dynamic_seq2seq/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

I am using smaller batch size and using sampled_softmax_loss in case someone meets the same problem of a large vocabulary size.

[kmario23]

Hi @ZixuanLiang ,
I also faced the same issue with a vocabulary size of 65K, batch size of 64/128 with 512/1024 units.
So, I scaled down the batch and model size; it runs now but still not sure how to run it in a faster mode with a reasonably good model.

[sushant097]

hi @ZixuanLiang, @kmario23
I got same error ResourceExhaustedError: OOM when allocating tensor with shape [12360,17191].
Did you find the solution ? Please mention me.
Here some screen shot:

[sushant097]

hi @ZixuanLiang,
i resolve this issue by reducing batch_size=52; inside nmt/nmt.py
Only to reduce memory use is to reduce batch_size.
And if not resolve then, please edit this:
nmt/inference.py

config = tf.ConfigProto()
  config.gpu_options.allocator_type = 'BFC'
  with tf.Session(
      graph=infer_model.graph, config=config) as sess: # change config to utils.get_config_proto()

[SeanThomasWilliams]

Turning off the dynamic memory allocation, coupled with tuning the batch size shown above, allowed me to squeeze out a few more WPS on my model

diff --git a/nmt/utils/misc_utils.py b/nmt/utils/misc_utils.py                                                                                                                                                                                                                                                                                            
index a680a5c..9a44bc8 100644                                                                                                                                                                                                                                                                                                                             
--- a/nmt/utils/misc_utils.py                                                                                                                                                                                                                                                                                                                             
+++ b/nmt/utils/misc_utils.py                                                                                                                                                                                                                                                                                                                             
@@ -141,7 +141,8 @@ def get_config_proto(log_device_placement=False, allow_soft_placement=True,                                                                                                                                                                                                                                                           
   config_proto = tf.ConfigProto(                                                                                                                                                                                                                                                                                                                         
       log_device_placement=log_device_placement,                                                                                                                                                                                                                                                                                                         
       allow_soft_placement=allow_soft_placement)                                                                                                                                                                                                                                                                                                         
-  config_proto.gpu_options.allow_growth = True                                                                                                                                                                                                                                                                                                           
+                                                                                                                                                                                                                                                                                                                                                         
+  #config_proto.gpu_options.allow_growth = True                                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                                                          
   # CPU threads options                                                                                                                                                                                                                                                                                                                                  
   if num_intra_threads:                                         

[dadashkarimi]

I tried to use different tokenization approach and the problem has been resolved. The problem was because I was creating a large number of useless words and tensorflow failed to handle e.g., 190,000 vocabs. my english vocab is around 20,000 for english-bulgarian paris and the other side is around 40,000. I recommend to use nltk or regex for tokenization and excluding tokens with less than 10 occurrences in the corpus.