Skip to content
This repository has been archived by the owner on Sep 6, 2022. It is now read-only.

Multi-GPU training is not working #18

Open
prajwaljpj opened this issue Apr 20, 2020 · 15 comments
Open

Multi-GPU training is not working #18

prajwaljpj opened this issue Apr 20, 2020 · 15 comments

Comments

@prajwaljpj
Copy link

prajwaljpj commented Apr 20, 2020 •
edited
Loading

I have a machine with
2x Nvidia RTX 2080 Ti
8 Core Intel i7 processor
32Gb of RAM

The training code (non-Docker version) when CUDA_VISIBLE_DEVICES=0,1 causes a memory leak in eval_step.
python run_common_voice.py --mode train --data_dir
These are the warnings I get. I am not able to pinpoint which object is causing the retracing error.

Performing evaluation. [949/1811]
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:38.737240 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:38.740701 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:38.743986 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:38.747186 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
2020-04-21 00:39:43.431398: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-04-21 00:39:44.193788: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perf$
rmance.
W0421 00:39:49.856330 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_$ 2 inside a tf.function to get the best performance.
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.859219 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.859964 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perf$
rmance.
W0421 00:39:49.861165 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_$ 2 inside a tf.function to get the best performance.
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.863494 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.864265 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perf$
rmance.
W0421 00:39:49.865403 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_$ 2 inside a tf.function to get the best performance.
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.867894 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.868691 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perfo
rmance.
W0421 00:39:49.869868 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v 2 inside a tf.function to get the best performance.
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perfo
rmance.
W0421 00:40:01.864544 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v 2 inside a tf.function to get the best performance.
WARNING:tensorflow:5 out of the last 5 calls to <function run_evaluate..eval_step at 0x7fc4885dc598> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.functi
on has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for
more details.
W0421 00:40:37.296950 140487075895104 def_function.py:586] 5 out of the last 5 calls to <function run_evaluate..eval_step at 0x7fc4885dc598> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python obj
ects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensor
flow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:6 out of the last 6 calls to <function run_evaluate..eval_step at 0x7fc4885dc598> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.functi
on has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for
more details

Is this a tensorflow issue?
@NAM-hj
Copy link

NAM-hj commented Apr 21, 2020 •
edited
Loading

I have same issue.
My system is

RAM : 128GB
GPU : GTX 1080ti * 4
OS : ubuntu 18.04
NVIDIA Driver : 440.82
CUDA : 10.1
CUDNN : 7.6.5
python : 3.6.9
tensorflow & tensorflow-gpu : 2.1.0
(And I do not change any param in run_common_voice.py)

When I run the run_common_voice.py code. These are shown.

  1. At the 0th epoch
    Eval_step is running with retracing warning and then, I got the OOM error.

  2. Disable evaluation at the 0th epoch.
    2-1. When there is retracing warning (slow)
    Epoch: 0, Batch: 60, Global Step: 60, Step Time: 26.0310, Loss: 165.6244
    2-2. When there is no retracing warning (fast)
    Epoch: 0, Batch: 62, Global Step: 62, Step Time: 6.3741, Loss: 164.6387

    Then I get the OOM error after this line
    Epoch: 0, Batch: 226, Global Step: 226, Step Time: 5.9092, Loss: 142.7257
    ...

I think some of the tf.function? affect to speed of the training.

Does the retracing warning have a connection with OOM error?
--> If so, how can I solve the retracing warning?
--> If not, how can I solve the OOM error?

Thank you

@prajwaljpj
Copy link
Author

@Nambee Did single GPU training work for you?

@NAM-hj
Copy link

NAM-hj commented Apr 22, 2020 •
edited
Loading

@Nambee Did single GPU training work for you?

No it does not work.

To see the progress, I print some logs in 'run_evaluate' func which is inside of 'run_training' func. (I attach this code at the end of this comment. (I only added 'print' functions.))
After 432 batches, OOM error has occurred. (+ The total eval_dataset loop count is 486.)

CUDA_VISIBLE_DEVICE=1 python run_common_voice.py --mode train --data_dir english_data/feature

...
tensorflow.org/api_docs/python/tf/function for more details.
Performing evaluation.2-2
Performing evaluation.2-3
-------------------- [432] ------------------
Performing evaluation.2-1
eval_step : <tensorflow.python.eager.def_function.Function object at 0x7f6dac6b07b8>
Type : eval_step : <class 'tensorflow.python.eager.def_function.Function'>
input type : <class 'tuple'>
Performing evaluation.2-1-1
Performing evaluation.2-1-2
Performing evaluation.2-1-3: Tensor("Identity:0", shape=(), dtype=float32, device=/job:localhost/replica:0/task:0/device:CPU:0)
Performing evaluation.2-1-4: {'WER': <tf.Tensor 'Identity_1:0' shape= dtype=float32>, 'Accuracy': <tf.Tensor 'Identity_2:0' shape= dtype=float32>, 'CER': <tf.Tensor 'Identity_3:0' shape= dtype=float32>}
2020-04-22 15:55:35.613508: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
...
(0) Resource exhausted: OOM when allocating tensor with shape[8,4088,303,37] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node transducer/dense_1/BiasAdd-0-TransposeNHWCToNCHW-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

[[replica_3/StringsByteSplit_1/RaggedGetItem/strided_slice_4/stack_1/_1212]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[8,4088,303,37] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node transducer/dense_1/BiasAdd-0-TransposeNHWCToNCHW-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

def run_evaluate(model,
                 optimizer,
                 loss_fn, 
                 eval_dataset,
                 batch_size,
                 strategy,
                 metrics=[],
                 fp16_run=False,
                 gpus=[]):

    @tf.function(experimental_relax_shapes=True)
    def eval_step(dist_inputs):
        def step_fn(inputs):
            (mel_specs, pred_inp, 
             spec_lengths, label_lengths, labels) = inputs

            outputs = model([mel_specs, pred_inp], 
                training=False)

            loss = loss_fn(labels, outputs,
                spec_lengths=spec_lengths,
                label_lengths=label_lengths)
            loss *= (1. / batch_size)

            if fp16_run:
                loss = optimizer.get_scaled_loss(loss)

            if metrics is not None:
                metric_results = run_metrics(mel_specs, labels,
                    metrics=metrics)
                metric_results = {name: result * (1. / max(len(gpus), 1)) for name, result in metric_results.items()}

            return loss, metric_results
        print('Performing evaluation.2-1-1')
        losses, metrics_results = strategy.experimental_run_v2(step_fn, args=(dist_inputs,))
        print('Performing evaluation.2-1-2')
        mean_loss = strategy.reduce(
            tf.distribute.ReduceOp.SUM, losses, axis=0)
        print('Performing evaluation.2-1-3:',mean_loss)
        mean_metrics = {name: strategy.reduce(
            tf.distribute.ReduceOp.SUM, result, axis=0) for name, result in metrics_results.items()}
        print('Performing evaluation.2-1-4:',mean_metrics)        
        return mean_loss, mean_metrics

    print('Performing evaluation.')

    loss_object = tf.keras.metrics.Mean()
    metric_objects = {fn.__name__: tf.keras.metrics.Mean() for fn in metrics}
    print('Performing evaluation.2 ')
    cnt = 0
    for batch, inputs in enumerate(eval_dataset):
        cnt = cnt +1
        print('-------------------- ['+str(cnt)+'] ------------------')
        print('Performing evaluation.2-1')
        print('eval_step : ',eval_step)
        print('Type : eval_step : ',type(eval_step))
        print('input type : ',type(inputs))
        loss, metrics_results = eval_step(inputs)
        print('Performing evaluation.2-2')
        loss_object(loss)
        print('Performing evaluation.2-3')
        for metric_name, metric_result in metrics_results.items():
            metric_objects[metric_name](metric_result)
    print('Performing evaluation.3')
    metrics_final_results = {name: metric_object.result() for name, metric_object in metric_objects.items()}
    print('Performing evaluation. finish')
    return loss_object.result(), metrics_final_results

@prajwaljpj
Copy link
Author

@Nambee From this log, you can see that you are running out of gpu memory, reduce the batch size to 8 or lower should fix the problem.
2020-04-22 15:55:35.613508: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

But still it looks like due to eager execution, the memory requirement keeps growing and only at eval step. My system fails to allocate GPU memory after 19000 Batches at Epoc 0.
@noahchalifour is there a way to fix this?

@NAM-hj
Copy link

NAM-hj commented Apr 23, 2020 •
edited
Loading

Oh, it's a different issue. sorry.
I thought you end up with OOM error too.

Can you run the run_common_voice.py without the OOM error?
I got the OOM error for eval_step and train_step too.
(I disabled the eval step to see the train_step can work.)

@prajwaljpj
Copy link
Author

Oh, it's a different issue. sorry.
I thought you end up with OOM error too.

Can you run the run_common_voice.py without the OOM error?
I got the OOM error for eval_step and train_step too.
(I disabled the eval step to see the train_step can work.)

Yes it worked for me. Even though you use CUDA_VISIBLE_DEVICES=0 to specify one GPU you have to change the strategy = None in run_common_voice.py.

@NAM-hj
Copy link

NAM-hj commented Apr 27, 2020

@prajwaljpj Thank you for your advice. Retracing errors are gone when I disable strategy. I still got the OOM error, I should reduce some factors. Again, Thank you!

@prajwaljpj
Copy link
Author

@Nambee Strategy part is not implemented for eval. If you see the training function there is a condition which implements strategy and experimental_run. You have to make a similar change for eval. also Try reducing batch size to 2.

@NAM-hj
Copy link

NAM-hj commented Apr 28, 2020

@prajwaljpj Yes, I did that already. But I apply it only for small datasets. (Because I need feasibility now)
I will expand it in the future. Thank you for your kind consideration.

@noahchalifour
Copy link
Owner

Can someone please let me know if this is resolved in the latest commit? I do not have a multi GPU machine to test on. Thanks

@stefan-falk
Copy link

Could this be related to #29 ?

@stefan-falk
Copy link

It does seem so.

First off, there seems to be an error, gpus is not defined at this point and run_evaluate() does not expose an argument gpus.

rnnt-speech-recognition/run_rnnt.py

Line 570 in a0d972f

gpus=gpus)

If I run the training with CUDA_VISIBLE_DEVICES=0 it does seem to work. However, running with multiple GPUs gives me the exception as described in #29.

for completeness, click to expand full error log 
/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
  from numba.decorators import jit as optional_jit
/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
  from numba.decorators import jit as optional_jit
2020-05-26 09:14:30.736191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-26 09:14:30.748386: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.749173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.749232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.750058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties: 
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.750112: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.750888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.750927: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.751427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 3 with properties: 
pciBusID: 0000:05:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.751570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:30.752638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 09:14:30.753673: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 09:14:30.753866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 09:14:30.754997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 09:14:30.755618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 09:14:30.757804: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 09:14:30.757899: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.759345: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.760097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.760844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.761589: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.762328: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.763068: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.763805: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.764521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2, 3
2020-05-26 09:14:30.764770: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-26 09:14:30.770070: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 4200000000 Hz
2020-05-26 09:14:30.770492: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560e6150fef0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-26 09:14:30.770507: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-26 09:14:31.020514: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.037961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.041811: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.049635: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.050189: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560e60e72d20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-26 09:14:31.050199: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050203: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050206: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (2): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050209: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (3): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.051527: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.051949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.051989: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.052409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties: 
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.052448: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.052867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.052904: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.053326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 3 with properties: 
pciBusID: 0000:05:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.053353: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:31.053366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 09:14:31.053377: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 09:14:31.053387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 09:14:31.053397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 09:14:31.053407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 09:14:31.053418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 09:14:31.053452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.053895: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.054339: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.054782: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.055227: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.055669: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.056126: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.056579: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.057003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2, 3
2020-05-26 09:14:31.057025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:31.059325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-26 09:14:31.059335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 1 2 3 
2020-05-26 09:14:31.059340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N Y Y Y 
2020-05-26 09:14:31.059344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 1:   Y N Y Y 
2020-05-26 09:14:31.059347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 2:   Y Y N Y 
2020-05-26 09:14:31.059350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 3:   Y Y Y N 
2020-05-26 09:14:31.060102: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.060567: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061033: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061486: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061942: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.062368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9449 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-05-26 09:14:31.062688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.063131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10161 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2020-05-26 09:14:31.063473: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.064690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10161 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
2020-05-26 09:14:31.065011: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.065455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10161 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
4 Physical GPU, 4 Logical GPUs
WARNING:tensorflow:From /home/sfalk/tmp/rnnt-speech-recognition/model.py:59: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
W0526 09:14:32.108052 140106746382080 deprecation.py:317] From /home/sfalk/tmp/rnnt-speech-recognition/model.py:59: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9e97d820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.108385 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9e97d820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:From /home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/ops/rnn_cell_impl.py:962: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
W0526 09:14:32.109819 140106746382080 deprecation.py:317] From /home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/ops/rnn_cell_impl.py:962: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9009e730>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.227335 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9009e730>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c4811f9d0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.490125 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c4811f9d0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c48086070>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.669947 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c48086070>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383bc4f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.804272 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383bc4f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383a7d00>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.951039 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383a7d00>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c3830e190>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.074690 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c3830e190>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c382f8250>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.202479 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c382f8250>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c381bd820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.890956 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c381bd820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c086b71f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:34.015121 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c086b71f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
I0526 09:14:34.344151 140106746382080 run_rnnt.py:490] Using word-piece encoder with vocab size: 4341
Model: "encoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, None, 240)]       0         
_________________________________________________________________
batch_normalization (BatchNo (None, None, 240)         960       
_________________________________________________________________
rnn (RNN)                    (None, None, 640)         8527872   
_________________________________________________________________
dropout (Dropout)            (None, None, 640)         0         
_________________________________________________________________
layer_normalization (LayerNo (None, None, 640)         1280      
_________________________________________________________________
rnn_1 (RNN)                  (None, None, 640)         11804672  
_________________________________________________________________
dropout_1 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_1 (Layer (None, None, 640)         1280      
_________________________________________________________________
time_reduction (TimeReductio (None, None, 1280)        0         
_________________________________________________________________
rnn_2 (RNN)                  (None, None, 640)         17047552  
_________________________________________________________________
dropout_2 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_2 (Layer (None, None, 640)         1280      
_________________________________________________________________
rnn_3 (RNN)                  (None, None, 640)         11804672  
_________________________________________________________________
dropout_3 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_3 (Layer (None, None, 640)         1280      
_________________________________________________________________
rnn_4 (RNN)                  (None, None, 640)         11804672  
_________________________________________________________________
dropout_4 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_4 (Layer (None, None, 640)         1280      
_________________________________________________________________
rnn_5 (RNN)                  (None, None, 640)         11804672  
_________________________________________________________________
dropout_5 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_5 (Layer (None, None, 640)         1280      
_________________________________________________________________
rnn_6 (RNN)                  (None, None, 640)         11804672  
_________________________________________________________________
dropout_6 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_6 (Layer (None, None, 640)         1280      
_________________________________________________________________
rnn_7 (RNN)                  (None, None, 640)         11804672  
_________________________________________________________________
dropout_7 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_7 (Layer (None, None, 640)         1280      
=================================================================
Total params: 96,414,656
Trainable params: 96,414,176
Non-trainable params: 480
_________________________________________________________________
Model: "prediction_network"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, None)]            0         
_________________________________________________________________
embedding (Embedding)        (None, None, 500)         2170500   
_________________________________________________________________
rnn_8 (RNN)                  (None, None, 640)         10657792  
_________________________________________________________________
dropout_8 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_8 (Layer (None, None, 640)         1280      
_________________________________________________________________
rnn_9 (RNN)                  (None, None, 640)         11804672  
_________________________________________________________________
dropout_9 (Dropout)          (None, None, 640)         0         
_________________________________________________________________
layer_normalization_9 (Layer (None, None, 640)         1280      
=================================================================
Total params: 24,635,524
Trainable params: 24,635,524
Non-trainable params: 0
_________________________________________________________________
Model: "transducer"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
mel_specs (InputLayer)          [(None, None, 240)]  0                                            
__________________________________________________________________________________________________
pred_inp (InputLayer)           [(None, None)]       0                                            
__________________________________________________________________________________________________
encoder (Model)                 (None, None, 640)    96414656    mel_specs[0][0]                  
__________________________________________________________________________________________________
prediction_network (Model)      (None, None, 640)    24635524    pred_inp[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_ExpandDims (TensorF [(None, None, 1, 640 0           encoder[1][0]                    
__________________________________________________________________________________________________
tf_op_layer_ExpandDims_1 (Tenso [(None, 1, None, 640 0           prediction_network[1][0]         
__________________________________________________________________________________________________
tf_op_layer_AddV2 (TensorFlowOp [(None, None, None,  0           tf_op_layer_ExpandDims[0][0]     
                                                                 tf_op_layer_ExpandDims_1[0][0]   
__________________________________________________________________________________________________
dense (Dense)                   (None, None, None, 6 410240      tf_op_layer_AddV2[0][0]          
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, None, None, 4 2782581     dense[0][0]                      
==================================================================================================
Total params: 124,243,001
Trainable params: 124,242,521
Non-trainable params: 480
__________________________________________________________________________________________________
Starting training.
Performing evaluation.
Traceback (most recent call last):
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2292, in _convert_inputs_to_signature
    flatten_inputs[index] = ops.convert_to_tensor(
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1341, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 321, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 261, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Attempt to convert a value (PerReplica:{
  0: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-9.8887777e+00, -9.5391264e+00, -9.2146311e+00, ...,
          1.4807711e+00,  1.4137149e+00,  1.5833356e+00],
        [-2.5297828e+00, -1.0314496e+00, -4.4551528e-01, ...,
         -8.5550594e-01, -3.8671780e-01, -6.2595654e-01],
        [-9.2890608e-01, -9.3925929e-01, -1.0737282e+00, ...,
         -4.6040058e-01, -1.3226795e-01, -4.6705770e-01],
        ...,
        [-8.9524627e-02, -1.3095784e-01,  4.4763446e-02, ...,
         -7.6179504e-03,  3.1356859e-01,  1.3805485e-01],
        [-1.0855615e-01, -3.6668968e-01, -3.5269606e-01, ...,
         -1.6952515e-01, -4.3339968e-01, -2.3297167e-01],
        [-3.4607446e-01, -4.6576285e-01, -2.4114418e-01, ...,
         -3.9931583e-01, -6.5470409e-01, -5.9117317e-02]],

       [[-1.0188073e+01, -9.7674351e+00, -9.2495003e+00, ...,
          2.7845190e+00,  3.0497322e+00,  2.9723659e+00],
        [-5.8545446e-01, -9.0612173e-01, -1.4650891e+00, ...,
         -1.2318707e+00, -1.2984281e+00, -1.2217040e+00],
        [-4.9245834e-01, -6.1498523e-01, -9.3534470e-01, ...,
         -1.1765385e+00, -1.4464822e+00, -6.3945484e-01],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]],

       [[-9.7931051e+00, -9.2576466e+00, -8.7187033e+00, ...,
          2.6441813e+00,  2.5935564e+00,  2.7019000e+00],
        [-1.1656429e+00, -2.4396741e-01,  2.2748601e-01, ...,
         -1.1423826e+00, -1.9891844e+00, -1.8549285e+00],
        [-1.1924456e+00, -1.5766211e+00, -1.9995271e+00, ...,
         -6.9014835e-01, -1.1843119e+00, -1.7883348e+00],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]],

       ...,

       [[-9.2142944e+00, -8.7442112e+00, -8.0906887e+00, ...,
          2.5637565e+00,  2.6928663e+00,  2.9714472e+00],
        [-1.4353551e+00, -1.3912488e+00, -1.1410775e+00, ...,
         -6.8289948e-01, -7.4661875e-01, -9.4346857e-01],
        [-1.3514730e+00, -1.3447710e+00, -1.1697345e+00, ...,
         -8.4204197e-01, -6.4308357e-01, -6.6116714e-01],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]],

       [[-9.4380770e+00, -9.2610064e+00, -9.3578424e+00, ...,
          1.3151474e+00,  1.3194189e+00,  1.6185846e+00],
        [-1.4748455e+00, -9.6358955e-01, -7.5807011e-01, ...,
         -1.2112403e+00, -5.9023190e-01, -2.0361996e-01],
        [-1.2093054e+00, -1.8930241e+00, -1.3843757e+00, ...,
         -3.7091255e-01, -6.0484600e-01, -1.9562192e+00],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]],

       [[-9.2918978e+00, -9.2315750e+00, -8.9223146e+00, ...,
          2.7278879e+00,  2.8609688e+00,  2.7423246e+00],
        [-5.9187198e-01, -1.8667960e-01,  2.1728778e-01, ...,
         -9.7960901e-01, -2.1147966e-01, -8.8720083e-01],
        [-8.6280262e-01, -8.0901504e-01, -5.0346565e-01, ...,
         -7.1112490e-01, -1.2980080e-01,  3.8867188e-01],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]]], dtype=float32)>,
  1: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.0592985 ,  -9.730883  ,  -9.271258  , ...,   2.8219585 ,
           2.6701574 ,   3.145619  ],
        [ -0.9512273 ,  -0.57793355,  -0.37342393, ...,  -0.36957645,
          -0.58688927,  -0.6034956 ],
        [ -1.0401095 ,  -1.0510697 ,  -1.1356157 , ...,  -0.2553835 ,
          -0.40022898,  -1.2278776 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.3389225 ,  -9.18263   ,  -8.9984045 , ...,   2.095945  ,
           2.1119766 ,   2.3430693 ],
        [ -0.9921645 ,  -0.9324732 ,  -0.67674816, ...,  -0.8709469 ,
          -0.87235546,  -0.8106818 ],
        [ -2.0774326 ,  -2.127688  ,  -1.135358  , ...,  -0.97692585,
          -1.1275706 ,  -0.88771343],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[-10.804321  , -10.686319  , -10.408899  , ...,   2.3665676 ,
           2.5379932 ,   2.420307  ],
        [ -1.1987064 ,  -1.067873  ,  -0.9529356 , ...,  -0.30030394,
          -0.12877178,  -0.16108751],
        [ -0.45661426,  -0.7980306 ,  -1.1557912 , ...,  -1.5581841 ,
          -0.4684105 ,  -0.44000912],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       ...,

       [[ -9.445232  ,  -9.460578  ,  -9.285489  , ...,   1.8623145 ,
           1.9304647 ,   2.195097  ],
        [ -1.0719146 ,  -1.0908513 ,  -0.9582559 , ...,  -0.99088955,
          -0.6627326 ,  -0.6685376 ],
        [ -1.8296288 ,  -1.8624439 ,  -1.321793  , ...,  -0.70586205,
          -0.97963333,  -1.346128  ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.328621  ,  -9.586381  ,  -9.817234  , ...,   3.6245484 ,
           3.8879015 ,   3.8824553 ],
        [ -2.2337832 ,  -1.1850247 ,  -0.43120062, ...,  -1.3237286 ,
          -0.8165655 ,  -0.5502229 ],
        [ -4.9042144 ,  -1.1433594 ,  -0.3725183 , ...,   0.19899416,
           0.02039671,   0.09885693],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.299686  ,  -8.799611  ,  -8.434992  , ...,   3.097272  ,
           3.1601548 ,   3.1777468 ],
        [ -1.1876402 ,  -1.0250216 ,  -0.64193165, ...,  -0.32103252,
          -0.06696558,  -0.6231766 ],
        [ -0.5330059 ,  -0.6796452 ,  -0.77790976, ...,  -0.9301033 ,
          -0.8343253 ,  -0.95349455],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]]], dtype=float32)>,
  2: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.20004   ,  -9.797416  ,  -9.144037  , ...,   2.1522975 ,
           2.177644  ,   2.4891276 ],
        [ -0.6829766 ,  -1.0327927 ,  -1.9740211 , ...,  -1.109499  ,
          -1.2662077 ,  -1.4992175 ],
        [ -0.50412035,  -0.75838935,  -1.4640208 , ...,  -1.1935134 ,
          -1.90482   ,  -1.4951177 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.057947  ,  -8.835877  ,  -8.432611  , ...,   2.7456422 ,
           2.6064804 ,   2.9408405 ],
        [ -0.7960679 ,  -0.7536596 ,  -0.56650114, ...,  -0.9725952 ,
          -0.9330125 ,  -0.8314767 ],
        [ -1.273035  ,  -1.0706124 ,  -0.8415742 , ...,  -1.0601668 ,
          -1.3329258 ,  -2.0243492 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -8.598597  ,  -8.078391  ,  -7.316219  , ...,   3.475574  ,
           3.5632377 ,   3.4421172 ],
        [ -0.15228844,   0.27448   ,   0.84499   , ...,  -1.2552052 ,
          -0.435256  ,  -0.70891285],
        [ -0.08234251,  -0.10229468,  -0.20772624, ...,   0.49269104,
          -0.33324862,  -0.30197382],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       ...,

       [[-10.674739  , -10.312547  ,  -9.784449  , ...,   1.9142385 ,
           2.0289752 ,   2.207048  ],
        [ -2.3002534 ,  -1.4031292 ,  -0.7912599 , ...,   0.46009445,
           0.26866436,   0.23870277],
        [ -0.8622447 ,  -0.8987092 ,  -0.7318269 , ...,  -0.51101494,
           0.02473354,   0.04263115],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[-11.913138  , -11.890651  , -11.718769  , ...,  -0.6516266 ,
          -0.11420345,  -0.2889433 ],
        [ -2.3279111 ,  -0.83327794,  -0.38966238, ...,  -0.18499136,
          -0.31412745,  -0.78526115],
        [ -1.2105832 ,  -0.65170074,  -0.30061066, ...,  -0.5946727 ,
          -0.5343251 ,  -0.26280165],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.108984  ,  -8.4880495 ,  -7.7428994 , ...,   3.0713623 ,
           3.0152838 ,   3.4003892 ],
        [ -0.19017792,  -0.29030776,  -0.33207464, ...,  -0.0172677 ,
          -0.24603939,  -0.9277673 ],
        [ -0.6175966 ,  -0.8139999 ,  -1.1248744 , ...,  -0.649323  ,
          -0.6167655 ,  -0.5438595 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]]], dtype=float32)>,
  3: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[ -8.988267  ,  -8.965929  ,  -8.728106  , ...,   2.1489723 ,
           2.334958  ,   2.5260453 ],
        [ -0.9514227 ,  -0.8912209 ,  -0.42839706, ...,  -0.7747984 ,
          -0.88131857,  -1.1858144 ],
        [ -1.6392798 ,  -1.8752912 ,  -1.3590232 , ...,  -0.71193075,
          -0.46426392,  -0.79004574],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.117433  ,  -8.756327  ,  -8.3021965 , ...,   2.6770406 ,
           2.9165506 ,   2.619464  ],
        [ -0.8334371 ,  -0.83099353,  -0.7541424 , ...,  -1.08567   ,
          -0.3741293 ,  -0.14404964],
        [ -1.1120342 ,  -1.2024816 ,  -1.1394764 , ...,  -1.3766332 ,
          -1.0821438 ,  -0.808341  ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -8.535436  ,  -8.446434  ,  -8.433824  , ...,   1.7634034 ,
           1.9151568 ,   2.234317  ],
        [ -0.73761785,  -1.0498412 ,  -1.1566993 , ...,  -1.9087753 ,
          -1.4135947 ,  -0.43615532],
        [ -3.1867118 ,  -1.3442845 ,  -0.6116022 , ...,  -1.2935882 ,
          -1.5357218 ,  -1.4339061 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       ...,

       [[ -8.752668  ,  -8.795702  ,  -9.234339  , ...,   1.4543443 ,
           1.308625  ,   1.8872037 ],
        [ -0.8426125 ,  -0.5590738 ,  -0.64102423, ...,  -0.2699995 ,
          -0.34920692,  -0.23712015],
        [ -0.49331522,  -1.201171  ,  -1.1223062 , ...,  -1.4179659 ,
          -0.8049512 ,  -0.8301816 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.640016  ,  -9.385439  ,  -9.126853  , ...,   2.6549459 ,
           2.8241823 ,   3.1816008 ],
        [ -1.9176644 ,  -1.4502705 ,  -0.9663389 , ...,  -0.6905961 ,
          -0.6539016 ,  -0.5099206 ],
        [ -1.4909953 ,  -1.6704174 ,  -1.1156216 , ...,  -1.4263935 ,
          -0.81988144,  -1.1464152 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[-11.273719  , -10.959808  , -10.497703  , ...,   1.7455099 ,
           1.9854834 ,   2.464494  ],
        [ -2.0011683 ,  -1.4769925 ,  -0.9517286 , ...,  -0.25176716,
          -0.5848155 ,  -0.5404253 ],
        [ -1.4961203 ,  -1.4866172 ,  -1.359714  , ...,  -1.1029644 ,
          -1.4994144 ,  -0.89810133],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]]], dtype=float32)>
}) with an unsupported type (<class 'tensorflow.python.distribute.values.PerReplica'>) to a Tensor.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_rnnt.py", line 586, in <module>
    app.run(main)
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "run_rnnt.py", line 532, in main
    run_training(
  File "run_rnnt.py", line 347, in run_training
    checkpoint_model()
  File "run_rnnt.py", line 304, in checkpoint_model
    eval_loss, eval_metrics_results = run_evaluate(
  File "run_rnnt.py", line 433, in run_evaluate
    loss, metrics_results = eval_step(inputs)
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 647, in _call
    self._stateful_fn._function_spec.canonicalize_function_inputs(  # pylint: disable=protected-access
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2235, in canonicalize_function_inputs
    inputs = _convert_inputs_to_signature(
  File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2296, in _convert_inputs_to_signature
    raise ValueError("When input_signature is provided, all inputs to "
ValueError: When input_signature is provided, all inputs to the Python function must be convertible to tensors:
  inputs: (
    (PerReplica:{
  0: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-9.8887777e+00, -9.5391264e+00, -9.2146311e+00, ...,
          1.4807711e+00,  1.4137149e+00,  1.5833356e+00],
        [-2.5297828e+00, -1.0314496e+00, -4.4551528e-01, ...,
         -8.5550594e-01, -3.8671780e-01, -6.2595654e-01],
        [-9.2890608e-01, -9.3925929e-01, -1.0737282e+00, ...,
         -4.6040058e-01, -1.3226795e-01, -4.6705770e-01],
        ...,
        [-8.9524627e-02, -1.3095784e-01,  4.4763446e-02, ...,
         -7.6179504e-03,  3.1356859e-01,  1.3805485e-01],
        [-1.0855615e-01, -3.6668968e-01, -3.5269606e-01, ...,
         -1.6952515e-01, -4.3339968e-01, -2.3297167e-01],
        [-3.4607446e-01, -4.6576285e-01, -2.4114418e-01, ...,
         -3.9931583e-01, -6.5470409e-01, -5.9117317e-02]],

       [[-1.0188073e+01, -9.7674351e+00, -9.2495003e+00, ...,
          2.7845190e+00,  3.0497322e+00,  2.9723659e+00],
        [-5.8545446e-01, -9.0612173e-01, -1.4650891e+00, ...,
         -1.2318707e+00, -1.2984281e+00, -1.2217040e+00],
        [-4.9245834e-01, -6.1498523e-01, -9.3534470e-01, ...,
         -1.1765385e+00, -1.4464822e+00, -6.3945484e-01],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]],

       [[-9.7931051e+00, -9.2576466e+00, -8.7187033e+00, ...,
          2.6441813e+00,  2.5935564e+00,  2.7019000e+00],
        [-1.1656429e+00, -2.4396741e-01,  2.2748601e-01, ...,
         -1.1423826e+00, -1.9891844e+00, -1.8549285e+00],
        [-1.1924456e+00, -1.5766211e+00, -1.9995271e+00, ...,
         -6.9014835e-01, -1.1843119e+00, -1.7883348e+00],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]],

       ...,

       [[-9.2142944e+00, -8.7442112e+00, -8.0906887e+00, ...,
          2.5637565e+00,  2.6928663e+00,  2.9714472e+00],
        [-1.4353551e+00, -1.3912488e+00, -1.1410775e+00, ...,
         -6.8289948e-01, -7.4661875e-01, -9.4346857e-01],
        [-1.3514730e+00, -1.3447710e+00, -1.1697345e+00, ...,
         -8.4204197e-01, -6.4308357e-01, -6.6116714e-01],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]],

       [[-9.4380770e+00, -9.2610064e+00, -9.3578424e+00, ...,
          1.3151474e+00,  1.3194189e+00,  1.6185846e+00],
        [-1.4748455e+00, -9.6358955e-01, -7.5807011e-01, ...,
         -1.2112403e+00, -5.9023190e-01, -2.0361996e-01],
        [-1.2093054e+00, -1.8930241e+00, -1.3843757e+00, ...,
         -3.7091255e-01, -6.0484600e-01, -1.9562192e+00],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]],

       [[-9.2918978e+00, -9.2315750e+00, -8.9223146e+00, ...,
          2.7278879e+00,  2.8609688e+00,  2.7423246e+00],
        [-5.9187198e-01, -1.8667960e-01,  2.1728778e-01, ...,
         -9.7960901e-01, -2.1147966e-01, -8.8720083e-01],
        [-8.6280262e-01, -8.0901504e-01, -5.0346565e-01, ...,
         -7.1112490e-01, -1.2980080e-01,  3.8867188e-01],
        ...,
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
        [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
          0.0000000e+00,  0.0000000e+00,  0.0000000e+00]]], dtype=float32)>,
  1: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.0592985 ,  -9.730883  ,  -9.271258  , ...,   2.8219585 ,
           2.6701574 ,   3.145619  ],
        [ -0.9512273 ,  -0.57793355,  -0.37342393, ...,  -0.36957645,
          -0.58688927,  -0.6034956 ],
        [ -1.0401095 ,  -1.0510697 ,  -1.1356157 , ...,  -0.2553835 ,
          -0.40022898,  -1.2278776 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.3389225 ,  -9.18263   ,  -8.9984045 , ...,   2.095945  ,
           2.1119766 ,   2.3430693 ],
        [ -0.9921645 ,  -0.9324732 ,  -0.67674816, ...,  -0.8709469 ,
          -0.87235546,  -0.8106818 ],
        [ -2.0774326 ,  -2.127688  ,  -1.135358  , ...,  -0.97692585,
          -1.1275706 ,  -0.88771343],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[-10.804321  , -10.686319  , -10.408899  , ...,   2.3665676 ,
           2.5379932 ,   2.420307  ],
        [ -1.1987064 ,  -1.067873  ,  -0.9529356 , ...,  -0.30030394,
          -0.12877178,  -0.16108751],
        [ -0.45661426,  -0.7980306 ,  -1.1557912 , ...,  -1.5581841 ,
          -0.4684105 ,  -0.44000912],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       ...,

       [[ -9.445232  ,  -9.460578  ,  -9.285489  , ...,   1.8623145 ,
           1.9304647 ,   2.195097  ],
        [ -1.0719146 ,  -1.0908513 ,  -0.9582559 , ...,  -0.99088955,
          -0.6627326 ,  -0.6685376 ],
        [ -1.8296288 ,  -1.8624439 ,  -1.321793  , ...,  -0.70586205,
          -0.97963333,  -1.346128  ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.328621  ,  -9.586381  ,  -9.817234  , ...,   3.6245484 ,
           3.8879015 ,   3.8824553 ],
        [ -2.2337832 ,  -1.1850247 ,  -0.43120062, ...,  -1.3237286 ,
          -0.8165655 ,  -0.5502229 ],
        [ -4.9042144 ,  -1.1433594 ,  -0.3725183 , ...,   0.19899416,
           0.02039671,   0.09885693],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.299686  ,  -8.799611  ,  -8.434992  , ...,   3.097272  ,
           3.1601548 ,   3.1777468 ],
        [ -1.1876402 ,  -1.0250216 ,  -0.64193165, ...,  -0.32103252,
          -0.06696558,  -0.6231766 ],
        [ -0.5330059 ,  -0.6796452 ,  -0.77790976, ...,  -0.9301033 ,
          -0.8343253 ,  -0.95349455],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]]], dtype=float32)>,
  2: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.20004   ,  -9.797416  ,  -9.144037  , ...,   2.1522975 ,
           2.177644  ,   2.4891276 ],
        [ -0.6829766 ,  -1.0327927 ,  -1.9740211 , ...,  -1.109499  ,
          -1.2662077 ,  -1.4992175 ],
        [ -0.50412035,  -0.75838935,  -1.4640208 , ...,  -1.1935134 ,
          -1.90482   ,  -1.4951177 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.057947  ,  -8.835877  ,  -8.432611  , ...,   2.7456422 ,
           2.6064804 ,   2.9408405 ],
        [ -0.7960679 ,  -0.7536596 ,  -0.56650114, ...,  -0.9725952 ,
          -0.9330125 ,  -0.8314767 ],
        [ -1.273035  ,  -1.0706124 ,  -0.8415742 , ...,  -1.0601668 ,
          -1.3329258 ,  -2.0243492 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -8.598597  ,  -8.078391  ,  -7.316219  , ...,   3.475574  ,
           3.5632377 ,   3.4421172 ],
        [ -0.15228844,   0.27448   ,   0.84499   , ...,  -1.2552052 ,
          -0.435256  ,  -0.70891285],
        [ -0.08234251,  -0.10229468,  -0.20772624, ...,   0.49269104,
          -0.33324862,  -0.30197382],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       ...,

       [[-10.674739  , -10.312547  ,  -9.784449  , ...,   1.9142385 ,
           2.0289752 ,   2.207048  ],
        [ -2.3002534 ,  -1.4031292 ,  -0.7912599 , ...,   0.46009445,
           0.26866436,   0.23870277],
        [ -0.8622447 ,  -0.8987092 ,  -0.7318269 , ...,  -0.51101494,
           0.02473354,   0.04263115],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[-11.913138  , -11.890651  , -11.718769  , ...,  -0.6516266 ,
          -0.11420345,  -0.2889433 ],
        [ -2.3279111 ,  -0.83327794,  -0.38966238, ...,  -0.18499136,
          -0.31412745,  -0.78526115],
        [ -1.2105832 ,  -0.65170074,  -0.30061066, ...,  -0.5946727 ,
          -0.5343251 ,  -0.26280165],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.108984  ,  -8.4880495 ,  -7.7428994 , ...,   3.0713623 ,
           3.0152838 ,   3.4003892 ],
        [ -0.19017792,  -0.29030776,  -0.33207464, ...,  -0.0172677 ,
          -0.24603939,  -0.9277673 ],
        [ -0.6175966 ,  -0.8139999 ,  -1.1248744 , ...,  -0.649323  ,
          -0.6167655 ,  -0.5438595 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]]], dtype=float32)>,
  3: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[ -8.988267  ,  -8.965929  ,  -8.728106  , ...,   2.1489723 ,
           2.334958  ,   2.5260453 ],
        [ -0.9514227 ,  -0.8912209 ,  -0.42839706, ...,  -0.7747984 ,
          -0.88131857,  -1.1858144 ],
        [ -1.6392798 ,  -1.8752912 ,  -1.3590232 , ...,  -0.71193075,
          -0.46426392,  -0.79004574],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.117433  ,  -8.756327  ,  -8.3021965 , ...,   2.6770406 ,
           2.9165506 ,   2.619464  ],
        [ -0.8334371 ,  -0.83099353,  -0.7541424 , ...,  -1.08567   ,
          -0.3741293 ,  -0.14404964],
        [ -1.1120342 ,  -1.2024816 ,  -1.1394764 , ...,  -1.3766332 ,
          -1.0821438 ,  -0.808341  ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -8.535436  ,  -8.446434  ,  -8.433824  , ...,   1.7634034 ,
           1.9151568 ,   2.234317  ],
        [ -0.73761785,  -1.0498412 ,  -1.1566993 , ...,  -1.9087753 ,
          -1.4135947 ,  -0.43615532],
        [ -3.1867118 ,  -1.3442845 ,  -0.6116022 , ...,  -1.2935882 ,
          -1.5357218 ,  -1.4339061 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       ...,

       [[ -8.752668  ,  -8.795702  ,  -9.234339  , ...,   1.4543443 ,
           1.308625  ,   1.8872037 ],
        [ -0.8426125 ,  -0.5590738 ,  -0.64102423, ...,  -0.2699995 ,
          -0.34920692,  -0.23712015],
        [ -0.49331522,  -1.201171  ,  -1.1223062 , ...,  -1.4179659 ,
          -0.8049512 ,  -0.8301816 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[ -9.640016  ,  -9.385439  ,  -9.126853  , ...,   2.6549459 ,
           2.8241823 ,   3.1816008 ],
        [ -1.9176644 ,  -1.4502705 ,  -0.9663389 , ...,  -0.6905961 ,
          -0.6539016 ,  -0.5099206 ],
        [ -1.4909953 ,  -1.6704174 ,  -1.1156216 , ...,  -1.4263935 ,
          -0.81988144,  -1.1464152 ],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]],

       [[-11.273719  , -10.959808  , -10.497703  , ...,   1.7455099 ,
           1.9854834 ,   2.464494  ],
        [ -2.0011683 ,  -1.4769925 ,  -0.9517286 , ...,  -0.25176716,
          -0.5848155 ,  -0.5404253 ],
        [ -1.4961203 ,  -1.4866172 ,  -1.359714  , ...,  -1.1029644 ,
          -1.4994144 ,  -0.89810133],
        ...,
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
           0.        ,   0.        ]]], dtype=float32)>
}, PerReplica:{
  0: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[   0,  500,    7, 1030, 2607,   29,  108, 2559, 4196, 4131,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,   70,   75, 2098,   14, 1025,  228,   26, 1178,    8,    5,
        3233, 4131,    0,    0,    0,    0,    0,    0],
       [   0,   12, 1849, 4117, 3926,  156,   64,   10,  932,   29,  108,
         302, 4131,    0,    0,    0,    0,    0,    0],
       [   0,    1, 3220, 4196, 4130, 2585, 1846,    1,  418,   45,   10,
         164, 4131,    0,    0,    0,    0,    0,    0],
       [   0,    1,  426,   56,  215, 1333,  173,  304, 4131,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,  108, 2186,  290,  600,   22,  931,  266,  631,   56,   42,
        2335, 4199, 4131,    0,    0,    0,    0,    0],
       [   0,   23, 2684,  121,    5,   88, 4185, 2864,   99,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,    4,  486,  471,   12,    5,   69,   30, 1286,   26,  495,
         397,  332,    0,    0,    0,    0,    0,    0]], dtype=int32)>,
  1: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[   0,   20,  701,   16, 2598,  442,   20,  502, 2628,  494, 1363,
        1566,    0,    0,    0,    0,    0,    0,    0],
       [   0,  399,   26,  356,   52, 2941, 4131,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,    1,  575, 1465, 4117, 3062, 4130,   25,   22, 2190, 2940,
        4193, 4131,    0,    0,    0,    0,    0,    0],
       [   0,    9,   97, 3737,   17,   59, 3619,  474, 1846, 4196, 4131,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,   84, 1332,  177, 3837, 4117,  939, 1316, 4200, 4131,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,    3,  123, 2750, 4117,    3, 1462, 4117,   27,    5,  875,
        4131,    0,    0,    0,    0,    0,    0,    0],
       [   0, 3019, 1525, 4007,  592,  664,   16,  289,   11,   51, 4028,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0, 2321, 4117,   25,   59, 1002,   18,  365,  150,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0]], dtype=int32)>,
  2: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[   0,  143,   12,  108,  539,    7,   34,   67,   25, 2746, 4148,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,    9, 2373,  195,  496,  190,    4, 3921,   74, 2677, 1078,
          45,    1,  532,   19,   23,  140, 4131,    0],
       [   0,   23,   18, 4026,  456, 2204,   15,   17,   35,  831,  470,
         524,    0,    0,    0,    0,    0,    0,    0],
       [   0,    5,  512,    2,   36,   18,  724,   21,   20,  688, 4131,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,   78,   68,   36,   89,   20,  157, 1279, 1855, 1910,  611,
         528,  747,    8,  139, 2423, 4200, 4131,    0],
       [   0,    3,   28,   12,    3,  582, 4131,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,   92, 2640,  186,    4, 1235,   10, 1718,  459, 4131,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,  614, 1674, 2151,  492,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0]], dtype=int32)>,
  3: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[   0, 1044, 4130,   25,    1, 2312,  269, 1781, 4117,   21,  191,
        4079, 4131,    0,    0,    0,    0,    0,    0],
       [   0,   35, 2797, 4117, 1290, 4196, 4130, 1517,  737,   12, 1411,
        2873, 4182, 4131,    0,    0,    0,    0,    0],
       [   0,   49,  641, 1525, 3145,   15,   25,   23,  411, 1628,  403,
        4131,    0,    0,    0,    0,    0,    0,    0],
       [   0,    9, 2824,  335,  646,   25,   13, 3178,  633, 2679,   29,
           5,  916,   34, 1558,  181, 4182, 4131,    0],
       [   0,   92,  112, 4025, 4182, 4118,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,   37,  273,  280, 1296,   19,   23,  140, 4131,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0],
       [   0,   26, 3288,   15,  493, 1239, 1424,  117,   37,    5,  517,
         266,  138, 4131,    0,    0,    0,    0,    0],
       [   0,  932,   29,    1, 1383,  416, 4117,   12,    4, 1620, 1170,
        1737,  420, 1138, 4117,    2,    5, 1785, 4131]], dtype=int32)>
}, PerReplica:{
  0: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([267, 225, 193, 132, 140, 159, 105, 137], dtype=int32)>,
  1: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([166, 126, 146, 132, 135, 180, 184, 163], dtype=int32)>,
  2: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([137, 239, 157, 172, 201, 112, 127, 124], dtype=int32)>,
  3: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([200, 174, 151, 185, 165, 176, 204, 209], dtype=int32)>
}, PerReplica:{
  0: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([ 9, 12, 12, 12,  8, 13,  8, 12], dtype=int32)>,
  1: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([11,  6, 12, 10,  9, 11, 10,  8], dtype=int32)>,
  2: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([10, 17, 11, 10, 17,  6,  9,  4], dtype=int32)>,
  3: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([12, 13, 11, 17,  5,  8, 13, 18], dtype=int32)>
}, PerReplica:{
  0: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[ 500,    7, 1030, 2607,   29,  108, 2559, 4196, 4131,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [  70,   75, 2098,   14, 1025,  228,   26, 1178,    8,    5, 3233,
        4131,    0,    0,    0,    0,    0,    0],
       [  12, 1849, 4117, 3926,  156,   64,   10,  932,   29,  108,  302,
        4131,    0,    0,    0,    0,    0,    0],
       [   1, 3220, 4196, 4130, 2585, 1846,    1,  418,   45,   10,  164,
        4131,    0,    0,    0,    0,    0,    0],
       [   1,  426,   56,  215, 1333,  173,  304, 4131,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [ 108, 2186,  290,  600,   22,  931,  266,  631,   56,   42, 2335,
        4199, 4131,    0,    0,    0,    0,    0],
       [  23, 2684,  121,    5,   88, 4185, 2864,   99,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [   4,  486,  471,   12,    5,   69,   30, 1286,   26,  495,  397,
         332,    0,    0,    0,    0,    0,    0]], dtype=int32)>,
  1: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[  20,  701,   16, 2598,  442,   20,  502, 2628,  494, 1363, 1566,
           0,    0,    0,    0,    0,    0,    0],
       [ 399,   26,  356,   52, 2941, 4131,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [   1,  575, 1465, 4117, 3062, 4130,   25,   22, 2190, 2940, 4193,
        4131,    0,    0,    0,    0,    0,    0],
       [   9,   97, 3737,   17,   59, 3619,  474, 1846, 4196, 4131,    0,
           0,    0,    0,    0,    0,    0,    0],
       [  84, 1332,  177, 3837, 4117,  939, 1316, 4200, 4131,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [   3,  123, 2750, 4117,    3, 1462, 4117,   27,    5,  875, 4131,
           0,    0,    0,    0,    0,    0,    0],
       [3019, 1525, 4007,  592,  664,   16,  289,   11,   51, 4028,    0,
           0,    0,    0,    0,    0,    0,    0],
       [2321, 4117,   25,   59, 1002,   18,  365,  150,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0]], dtype=int32)>,
  2: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[ 143,   12,  108,  539,    7,   34,   67,   25, 2746, 4148,    0,
           0,    0,    0,    0,    0,    0,    0],
       [   9, 2373,  195,  496,  190,    4, 3921,   74, 2677, 1078,   45,
           1,  532,   19,   23,  140, 4131,    0],
       [  23,   18, 4026,  456, 2204,   15,   17,   35,  831,  470,  524,
           0,    0,    0,    0,    0,    0,    0],
       [   5,  512,    2,   36,   18,  724,   21,   20,  688, 4131,    0,
           0,    0,    0,    0,    0,    0,    0],
       [  78,   68,   36,   89,   20,  157, 1279, 1855, 1910,  611,  528,
         747,    8,  139, 2423, 4200, 4131,    0],
       [   3,   28,   12,    3,  582, 4131,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [  92, 2640,  186,    4, 1235,   10, 1718,  459, 4131,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [ 614, 1674, 2151,  492,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0]], dtype=int32)>,
  3: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[1044, 4130,   25,    1, 2312,  269, 1781, 4117,   21,  191, 4079,
        4131,    0,    0,    0,    0,    0,    0],
       [  35, 2797, 4117, 1290, 4196, 4130, 1517,  737,   12, 1411, 2873,
        4182, 4131,    0,    0,    0,    0,    0],
       [  49,  641, 1525, 3145,   15,   25,   23,  411, 1628,  403, 4131,
           0,    0,    0,    0,    0,    0,    0],
       [   9, 2824,  335,  646,   25,   13, 3178,  633, 2679,   29,    5,
         916,   34, 1558,  181, 4182, 4131,    0],
       [  92,  112, 4025, 4182, 4118,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [  37,  273,  280, 1296,   19,   23,  140, 4131,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0],
       [  26, 3288,   15,  493, 1239, 1424,  117,   37,    5,  517,  266,
         138, 4131,    0,    0,    0,    0,    0],
       [ 932,   29,    1, 1383,  416, 4117,   12,    4, 1620, 1170, 1737,
         420, 1138, 4117,    2,    5, 1785, 4131]], dtype=int32)>
}))
  input_signature: (
    [TensorSpec(shape=(None, None, 240), dtype=tf.float32, name=None), TensorSpec(shape=(None, None), dtype=tf.int32, name=None), TensorSpec(shape=(None,), dtype=tf.int32, name=None), TensorSpec(shape=(None,), dtype=tf.int32, name=None), TensorSpec(shape=(None, None), dtype=tf.int32, name=None)])

@stefan-falk stefan-falk mentioned this issue May 26, 2020
Open
@ChristopheZhao
Copy link

I can train the model use multi-gpus by add a decorator @tf.function refer this link tensorflow/tensorflow#29911 i also add line of “os.environ['CUDA_VISIBLE_DEVICES'] = "{your gpus}” in my code.

@stefan-falk
Copy link

Maybe take a look at https://github.com/usimarit/TiramisuASR

@ChristopheZhao
Copy link

It's the multi-gpu training code what i modified,but the loss value from negative to nan after trained some batches.
image

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ChristopheZhao @prajwaljpj @noahchalifour @NAM-hj @stefan-falk