Describe the bug
Backward LSTM results mismatch between TF and ONNXRuntime.
in TF runtime (tf.keras), say 1,2,3 are time frames.. output of reverse LSTM will be op3 op2 op1. where as in ONNXRuntime it will be op1 op2 op3. Either ONNX converters need to add a reverseV2 operator after backward LSTM or need to handle on the ONNXRuntime.
Urgency
I hack this with change in keras2onnx, but I would like to get confirmation on the expected behavior.
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS mojave
- ONNX Runtime installed from (source or binary): pip
- ONNX Runtime version: 1.6
- Python version: 3.6
- Visual Studio version (if applicable):
- GCC/Compiler version (if compiling from source): gcc 11
- CUDA/cuDNN version: N/A
- GPU model and memory: N/A
To Reproduce
simple_lstm.txt
-rename the above file to .h5
-run with some random input on tf.keras backend and on onnxruntime and compare results.
-outputs are just reverse to each other.
Alternatively, you can try this:
code for reverse LSTM:
max_features = 10
maxlen = 10
model = Sequential()
model.add(Embedding(max_features, 128, input_length=maxlen))
model.add(LSTM(64, go_backwards=True, return_sequences = True))
model.add(Dense(1, activation='sigmoid'))
model.compile('adam', 'binary_crossentropy', metrics=['accuracy'])
from keras.models import load_model
model.save('simple_rev_lstm.h5')
To compare results with other inference engines and ONNXRuntime
# pip install onnxruntime
#env TF_KERAS=1
import numpy as np
import onnxruntime
from tensorflow.keras.models import load_model as load_model_tf_keras
np.random.seed(0)
input_data = np.random.randint(10, size=(2, 10))
# with CustomObjectScope({'GlorotUniform': glorot_uniform()}):
sess = onnxruntime.InferenceSession("simple_rev_lstm.onnx")
result = sess.run(["dense_1"], {'embedding_1_input': input_data.astype(np.float32)})
print("ONNX Runtime")
print(np.asarray(result[0]))
model = load_model_tf_keras('simple_rev_lstm.h5')
result = model.predict(input_data)
print("TF Runtime")
print(result)
OUTPUT:
ONNX Runtime
[[[0.5028585 ]
[0.51010513]
[0.5096171 ]
[0.50567615]
[0.50046337]
[0.50202006]
[0.5010086 ]
[0.49436894]
[0.4989641 ]
[0.50032514]]
[[0.50235206]
[0.5045772 ]
[0.504677 ]
[0.5032916 ]
[0.5016299 ]
[0.50043035]
[0.4991624 ]
[0.50064933]
[0.5028503 ]
[0.5012636 ]]]
TF Runtime
[[[0.50032514]
[0.4989641 ]
[0.49436894]
[0.5010086 ]
[0.50202006]
[0.50046337]
[0.50567615]
[0.5096171 ]
[0.51010513]
[0.5028585 ]]
[[0.5012636 ]
[0.5028503 ]
[0.50064933]
[0.4991624 ]
[0.50043035]
[0.5016299 ]
[0.5032916 ]
[0.504677 ]
[0.5045773 ]
[0.50235206]]]
Observation:
The values are in reverse to each other on axis 1.
Not sure which is correct behavior.
Expected behavior
outputs should be same
Describe the bug
Backward LSTM results mismatch between TF and ONNXRuntime.
in TF runtime (tf.keras), say 1,2,3 are time frames.. output of reverse LSTM will be op3 op2 op1. where as in ONNXRuntime it will be op1 op2 op3. Either ONNX converters need to add a reverseV2 operator after backward LSTM or need to handle on the ONNXRuntime.
Urgency
I hack this with change in keras2onnx, but I would like to get confirmation on the expected behavior.
System information
To Reproduce
simple_lstm.txt
-rename the above file to .h5
-run with some random input on tf.keras backend and on onnxruntime and compare results.
-outputs are just reverse to each other.
Alternatively, you can try this:
code for reverse LSTM:
To compare results with other inference engines and ONNXRuntime
OUTPUT:
Observation:
The values are in reverse to each other on axis 1.
Not sure which is correct behavior.
Expected behavior
outputs should be same