Modify serving utils

[Drunkar]

When I serve a transformer seq2se2 model exprted with decode_hparams return_beams=True, the decoding process of t2t-query-server raised an error:

Traceback (most recent call last):
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/bin/t2t-query-server", line 4, in <module>
    __import__('pkg_resources').run_script('tensor2tensor==1.10.0', 't2t-query-server')
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/pkg_resources/__init__.py", line 664, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1444, in run_script
    exec(code, namespace, namespace)
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensor2tensor-1.10.0-py3.6.egg/EGG-INFO/scripts/t2t-query-server", line 17, in <module>
    tf.app.run()
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensor2tensor-1.10.0-py3.6.egg/EGG-INFO/scripts/t2t-query-server", line 12, in main
    query.main(argv)
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensor2tensor_optuna-1.10.0-py3.6.egg/tensor2tensor/serving/query.py", line 89, in main
    outputs = serving_utils.predict([inputs], problem, request_fn)
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensor2tensor_optuna-1.10.0-py3.6.egg/tensor2tensor/serving/serving_utils.py", line 162, in predict
    for prediction in predictions
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensor2tensor_optuna-1.10.0-py3.6.egg/tensor2tensor/serving/serving_utils.py", line 162, in <listcomp>
    for prediction in predictions
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensor2tensor_optuna-1.10.0-py3.6.egg/tensor2tensor/serving/serving_utils.py", line 97, in _decode
    return output_decoder.decode(output_ids, strip_extraneous=True)
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensor2tensor_optuna-1.10.0-py3.6.egg/tensor2tensor/data_generators/text_encoder.py", line 536, in decode
    ids = strip_ids(ids, list(range(self._num_reserved_ids or 0)))
  File "/home/uuuzzz/anaconda3/envs/tensor2tensor/lib/python3.6/site-packages/tensor2tensor_optuna-1.10.0-py3.6.egg/tensor2tensor/data_generators/text_encoder.py", line 104, in strip_ids
    while ids and ids[-1] in ids_to_strip:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This caused by a difference of output_ids in _decode() function of tensor2tensor/serving/serving_utils.py.

So I added logic for return_beams=True into decoding and printing functions.

output_ids example

• with model return_beams=False

[2402   31 2402  132    1    0    0    0    0    0    0    0    0    0
    0    0    0    0    0]

• with model return_beams=True

[[2402   31 2402  132    1    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0]
 [2402   31 2402   31 2402  132    1    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0]]

printed result example

• with model return_beams=False -> Not changed.
• with model return_beams=True

Input:
q

Output (Scores [-5.256,-5.810,-6.794,-7.576,-7.585]):
['Yang', 'Ya', 'Ya?', 'Yang?', 'Yang man']

[lukaszkaiser]

Thanks, this is very helpful!

[lukaszkaiser]

Again, great thanks for helping everyone overcome this bug!