Transformer Training fails on custom dataset

[karthikb23]

Hi,
I'm training Transformer on a custom dataset on CPU. I've used spm encoding and have followed instructions to a T, but the training always fails with the below error trace. The same error occurs regardless of BPE, SPM or raw encodings. Kindly help!

step: 400, loss: 5.6811
step: 500, loss: 5.3085
Traceback (most recent call last):
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[55,128] = 128 is not in [0, 128)
	 [[{{node sinusoid_posisiton_embedder/embedding_lookup}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "transformer_main.py", line 308, in <module>
    main()
  File "transformer_main.py", line 293, in main
    step = _train_epoch(sess, epoch, step, smry_writer)
  File "transformer_main.py", line 273, in _train_epoch
    _eval_epoch(sess, epoch, mode='eval')
  File "transformer_main.py", line 193, in _eval_epoch
    fetches_ = sess.run(fetches, feed_dict=feed_dict)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[55,128] = 128 is not in [0, 128)
	 [[node sinusoid_posisiton_embedder/embedding_lookup (defined at /home/karthik/installs/texar/texar/modules/embedders/position_embedders.py:327) ]]

Caused by op 'sinusoid_posisiton_embedder/embedding_lookup', defined at:
  File "transformer_main.py", line 308, in <module>
    main()
  File "transformer_main.py", line 96, in main
    src_pos_embeds = pos_embedder(sequence_length=src_seq_len)
  File "/home/karthik/installs/texar/texar/module_base.py", line 116, in __call__
    return self._template(*args, **kwargs)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 360, in __call__
    return self._call_func(args, kwargs)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 311, in _call_func
    result = self._func(*args, **kwargs)
  File "/home/karthik/installs/texar/texar/modules/embedders/position_embedders.py", line 327, in _build
    outputs = tf.nn.embedding_lookup(embedding, inputs)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/ops/embedding_ops.py", line 316, in embedding_lookup
    transform_fn=None)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/ops/embedding_ops.py", line 133, in _embedding_lookup_and_transform
    result = _clip(array_ops.gather(params[0], ids, name=name),
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 3273, in gather
    return gen_array_ops.gather_v2(params, indices, axis, name=name)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3748, in gather_v2
    "GatherV2", params=params, indices=indices, axis=axis, name=name)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/home/karthik/installs/miniconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): indices[55,128] = 128 is not in [0, 128)
	 [[node sinusoid_posisiton_embedder/embedding_lookup (defined at /home/karthik/installs/texar/texar/modules/embedders/position_embedders.py:327) ]]

[ZhitingHu]

what's the max_decoding_length are you using?

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[55,128] = 128 is not in [0, 128)
	 [[node sinusoid_posisiton_embedder/embedding_lookup (defined at /home/karthik/installs/texar/texar/modules/embedders/position_embedders.py:327) ]]

It looks cuz the position embedding index exceeds the max

[karthikb23]

Yeah, was my first guess.
The max_decoding_length is set to 128 (was previously 256, but had the same issue).
So no clue why 128 is being indexed... Any help please?

[haoransh]

Could you please check whether there are input sentences longer than max_decoding_length (128 maybe) in your validation set and test set?
Our provided scripts don't perform sequence cutoff on the validation set and test set for fair comparison. So you may need to do this yourself. Or you can simply increase the max_decoding_length to make it work. It seems the error in your trace has nothing to do with decoding. I think the input sentences may be too long.

  File "transformer_main.py", line 308, in <module>
    main()
  File "transformer_main.py", line 96, in main
    src_pos_embeds = pos_embedder(sequence_length=src_seq_len)

[karthikb23]

Great! This fix worked - thanks!
However, now a new error shows up.

Traceback (most recent call last):
  File "transformer_main.py", line 308, in <module>
    main()
  File "transformer_main.py", line 293, in main
    step = _train_epoch(sess, epoch, step, smry_writer)
  File "transformer_main.py", line 273, in _train_epoch
    _eval_epoch(sess, epoch, mode='eval')
  File "transformer_main.py", line 210, in _eval_epoch
    eval_bleu = bleu_wrapper(ref_fn, hyp_fn, case_sensitive=True)
  File "/home/karthik/texar/examples/transformer/bleu_tool.py", line 200, in bleu_wrapper
    return compute_bleu(ref_tokens, hyp_tokens)
  File "/home/karthik/texar/examples/transformer/bleu_tool.py", line 137, in compute_bleu
    bp = math.exp(1 - 1. / ratio) if ratio < 1.0 else 1.0
ZeroDivisionError: float division by zero

[ZhitingHu]

Could you check if the generated output is empty (translation_length=0)

texar/examples/transformer/bleu_tool.py

Lines 134 to 137 in 821c7db

	ratio = translation_length / reference_length
	if ratio == 0:
	bp = 0
	bp = math.exp(1 - 1. / ratio) if ratio < 1.0 else 1.0

[haoransh]

Thanks for your catching, we will modify the bleu_tool.py to make it more robust.

[karthikb23]

Thanks for the really quick responses! All set now.