Problems with transformer model training

[kudou1994]

Description

Q1：Problems with transformer model training, when i use the transformer_base or transformer_big and 100,000 or 1,000,000 or 10,000,000 dataset. The loss function does not converge when it reaches 1.2(loss=1.2)
INFO:tensorflow:loss = 1.3547997, step = 1191000 (34.544 sec)
Q2: Why are the all data evaluation at 20 or 30 percent complete?
INFO:tensorflow:Restoring parameters from train/model.ckpt-120000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Finished evaluation at 2018-10-08-01:04:22
Q3: When i use universal_transformer, The loss function does not converge when it reaches 4.3(loss=4.3)
INFO:tensorflow:global_step/sec: 3.36792
INFO:tensorflow:loss = 4.30654, step = 120600 (29.692 sec)

Environment information

OS: <Ubuntu16.04>

$ pip freeze | grep tensor
tensor2tensor==1.9.0
tensorboard==1.10.0
tensorflow==1.10.1
tensorflow-gpu==1.10.1


$ python -V
Python 3.6.4 :: Anaconda, Inc.

For bugs: reproduction and error logs

# Steps to reproduce:
...

# Error logs:
INFO:tensorflow:loss = 1.3547997, step = 1191000 (34.544 sec)

[crystal0913]

i have the same question about Q1 and Q3.
Here is my answer of Q2:
Evaluation [30/100] not means 30 percent complete, "100" is the number of steps in evaluation, see source, it seems like the eval has finished after 30 steps. This depends on your batch_size, the smaller the batch_size, the more steps you need to eval.

[MostafaDehghani]

Hi, The convergence problem of the Universal Transformer is solved in #1194.
(Really sorry for the delay in fixing this issue!)

[colmantse]

@MostafaDehghani in addition, i notice when training UT and transformer with 8 gpu, transformer tends to converge faster, in fact, at 10k steps 8 gpu, transformer outperform UT for 1 bleu. May i know if this is the case for the original experiment as well, that only transformer converges would UT be able to catch up?