Cannot start inference session with tf.keras model converted to onnx

[mxdillon]

Describe the bug
When training a simple tf.keras model and converting to onnx with tf2onnx,
sess = rt.InferenceSession("model.onnx") gives the error

onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for the node sequential/dense/Relu:Relu(6)

This is similar to #967, but I am specifying --opset 10 in my conversion

Urgency

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS

• ONNX Runtime installed from (source or binary): pip (whl)

• ONNX Runtime version: 1.1.0

• Python version: 3.7.2 64-bit

• Visual Studio version (if applicable): Version: 1.41.1
  Commit: 26076a4de974ead31f97692a0d32f90d735645c0
  Date: 2019-12-18T14:57:51.166Z
  Electron: 6.1.5
  Chrome: 76.0.3809.146
  Node.js: 12.4.0
  V8: 7.6.303.31-electron.0
  OS: Darwin x64 19.2.0

• GCC/Compiler version (if compiling from source): na

• CUDA/cuDNN version: na

• GPU model and memory: na

To Reproduce
Run the code below using the package versions specified

onnx==1.6.0
onnxruntime==1.1.0
tensorflow==1.15.0
tf2onnx==1.5.4

import os
import tf2onnx
import tensorflow as tf
import onnx
import onnxruntime

mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

x_train = x_train[..., tf.newaxis]
x_test = x_test[..., tf.newaxis]

train_ds = (
    tf.data.Dataset.from_tensor_slices((x_train, y_train)).shuffle(10000).batch(32)
)
test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)

model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(128, activation="relu"),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation="softmax"),
    ]
)

model.compile(
    optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)

history = model.fit(train_ds, epochs=1)

tf.keras.experimental.export_saved_model(model, "./saved_model")
command = f"python -m tf2onnx.convert --opset 10 --fold_const --verbose --saved-model ./saved_model --output model.onnx"
os.system(command)

# load model for inference
model = onnx.load("model.onnx")

onnx.checker.check_model(model)

sess = rt.InferenceSession("model.onnx")

Expected behavior
I expect the inference session to start.
I am using onnxruntime==1.1.0 and tf2onnx==1.5.4 which are listed as compatible in https://github.com/Microsoft/onnxruntime/blob/master/docs/Versioning.md#tool-compatibility
The process doesn't throw an error at 'onnx.checker.check_model(model)' so there isn't an issue with the onnx model.

[pranavsharma]

The generated model has a Relu node whose input (sequential/dense/BiasAdd:0) is of type double. We have implemented only float support for Relu. If you enable verbose logging you'll see this error message: "2020-01-10 18:33:16.3760603 [I:onnxruntime:Default, kernel_registry.cc:273 onnxruntime::KernelRegistry::TryFindKernel] Relu kernel is not supported in CPUExecutionProvider Encountered following errors: (Op: Relu Incompatible types. )".

Can you use float instead of double?

---

node {
input: "sequential/dense/BiasAdd:0"
output: "sequential/dense/Relu:0"
name: "sequential/dense/Relu"
op_type: "Relu"
}

initializer {
dims: 10
data_type: 11
name: "sequential/dense_1/BiasAdd/ReadVariableOp/2__cf__2:0"
raw_data: "\352\t\001\177\330o\263\277\0322u[\022\234?\217\371r\234\356GZ?\263g\031BP\225\237\277\236\2705\374\313an\277\217H\234\323\321\260\227?\325\216y|\242\224\220\277\024\237-#\316\216\263?\007\325\024?\205#\240\277\274[l\343\274)\206?"
}

[mxdillon]

Do you know how I could use floats? Putting tf.keras.backend.set_floatx('float32') before defining the model still gives the same error.

Also what is the syntax for enabling verbose logging? sess_options.session_log_verbosity_level gives an error.

[pranavsharma]

Also what is the syntax for enabling verbose logging? sess_options.session_log_verbosity_level gives an error.

import onnxruntime as ort
ort.set_default_logger_severity(0)

[mxdillon]

Thanks, any ideas about overwriting the default use of doubles with floats?

data_type: 11

Does this attribute need to be set to a different integer?

[pranavsharma]

Using https://www.tensorflow.org/api_docs/python/tf/keras/backend/set_floatx should've worked. Not sure why it's not working. May be try on keras github.
Regarding the data_type attribute, you don't have to set it manually; it's taken care of as part of the conversion process.

[faxu]

mxdillon can you check if the latest build from master works as expected?

[faxu]

Closing per comments above. Please reopen if there are still blockers.