Endpoint returning 500 for all input formats

[o4k741x]

Hey guys

Sagemaker version used: 1.2.3
Python version(s) used: 2.7, 3.6
Training image: Sagemaker's TF training image

I am having problems with querying an endpoint with different formats.
I have trained and deployed a Keras model on Sagemaker. I haven't found an example where you try to populate anything other than one tensor, so this might be where the problem lies.
Here's the following serving_input_fn:

def serving_input_fn(params):
 user = tf.placeholder(tf.int64, shape=[1])
 item = tf.placeholder(tf.int64, shape=[1])
 return build_raw_serving_input_receiver_fn({
 USER_TENSOR_NAME: user, ITEM_TENSOR_NAME: item
 })()

I'm creating the endpoint like:

predictor = RealTimePredictor(endpoint='itemembd-3-tensorflow-endpoint',
 sagemaker_session=sagemaker_session,
 deserializer=tf_json_deserializer,
 serializer=tf_json_serializer)

# also tried:

Dictionary as such:
predictor.predict({'user': 10, 'item': 10})
predictor.predict({'user': [10], 'item': [10]})

Gives me following output:

[2018-04-18 22:03:05,068] ERROR in serving: Unsupported request data format: {u'item': 10, u'user': 10}.

[2018-04-18 22:07:33,356] ERROR in serving: Unsupported request data format: {u'item': [10], u'user': [10]}.

Valid formats: tensor_pb2.TensorProto, dict<string, tensor_pb2.TensorProto> and predict_pb2.PredictRequest
Traceback (most recent call last):
Valid formats: tensor_pb2.TensorProto, dict<string, tensor_pb2.TensorProto> and predict_pb2.PredictRequest
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/container_support/serving.py", line 180, in _invoke
self.transformer.transform(content, input_content_type, requested_output_content_type)
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 267, in transform
return self.transform_fn(data, content_type, accepts), accepts
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 192, in f
prediction = self.predict_fn(input)
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 207, in predict_fn
return self.proxy_client.request(data)
File "/usr/local/lib/python2.7/dist-packages/tf_container/proxy_client.py", line 67, in request
return request_fn(data)
File "/usr/local/lib/python2.7/dist-packages/tf_container/proxy_client.py", line 97, in predict
request = self._create_predict_request(data)
File "/usr/local/lib/python2.7/dist-packages/tf_container/proxy_client.py", line 114, in _create_predict_request
input_map = self._create_input_map(data)
File "/usr/local/lib/python2.7/dist-packages/tf_container/proxy_client.py", line 211, in _create_input_map
raise ValueError(msg.format(data))
ValueError: Unsupported request data format: {u'item': 10, u'user': 10}.
Valid formats: tensor_pb2.TensorProto, dict<string, tensor_pb2.TensorProto> and predict_pb2.PredictRequest
[2018-04-18 22:03:05,068] ERROR in serving: Unsupported request data format: {u'item': 10, u'user': 10}.
Valid formats: tensor_pb2.TensorProto, dict<string, tensor_pb2.TensorProto> and predict_pb2.PredictRequest
10.32.0.2 - - [18/Apr/2018:22:03:05 +0000] "POST /invocations HTTP/1.1" 500 0 "-" "AHC/2.0"

I thought this was fixed in 1.1.0?

#62

Even when I try to follow the error message and use these types dict<string, tensor_pb2.TensorProto> like so:

user = tf.make_tensor_proto(values=np.asarray([10]), shape=[1], dtype=tf.float64)
item = tf.make_tensor_proto(values=np.asarray([10]), shape=[1], dtype=tf.float64)
d = {'user': user, 'item': item}
predictor.predict(d)

I get:

TypeErrorTraceback (most recent call last)
<ipython-input-35-9b9c7393b590> in <module>()
 5 item = tf.make_tensor_proto(values=np.asarray([10]), shape=[1], dtype=tf.int64)
 6 d = {'user': user, 'item': item}
----> 7 predictor.predict(d)

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/sagemaker/predictor.pyc in predict(self, data)
 72 """
 73 if self.serializer is not None:
---> 74 data = self.serializer(data)
 75 
 76 request_args = {

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/sagemaker/tensorflow/predictor.pyc in __call__(self, data)
 85 return json_format.MessageToJson(data)
 86 else:
---> 87 return json_serializer(data)
 88 
 89 

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/sagemaker/predictor.pyc in __call__(self, data)
 246 if not len(data.keys()) > 0:
 247 raise ValueError("empty dictionary can't be serialized")
--> 248 return _json_serialize_python_object(data)
 249 
 250 # files and buffers

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/sagemaker/predictor.pyc in _json_serialize_python_object(data)
 264 
 265 def _json_serialize_python_object(data):
--> 266 return _json_serialize_object(data)
 267 
 268 

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/sagemaker/predictor.pyc in _json_serialize_object(data)
 272 
 273 def _json_serialize_object(data):
--> 274 return json.dumps(data)
 275 
 276 

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/json/__init__.pyc in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, encoding, default, sort_keys, **kw)
 242 cls is None and indent is None and separators is None and
 243 encoding == 'utf-8' and default is None and not sort_keys and not kw):
--> 244 return _default_encoder.encode(obj)
 245 if cls is None:
 246 cls = JSONEncoder

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/json/encoder.pyc in encode(self, o)
 205 # exceptions aren't as detailed. The list call should be roughly
 206 # equivalent to the PySequence_Fast that ''.join() would do.
--> 207 chunks = self.iterencode(o, _one_shot=True)
 208 if not isinstance(chunks, (list, tuple)):
 209 chunks = list(chunks)

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/json/encoder.pyc in iterencode(self, o, _one_shot)
 268 self.key_separator, self.item_separator, self.sort_keys,
 269 self.skipkeys, _one_shot)
--> 270 return _iterencode(o, 0)
 271 
 272 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

/home/ec2-user/anaconda3/envs/tensorflow_p27/lib/python2.7/json/encoder.pyc in default(self, o)
 182 
 183 """
--> 184 raise TypeError(repr(o) + " is not JSON serializable")
 185 
 186 def encode(self, o):

TypeError: dtype: DT_INT64
tensor_shape {
 dim {
 size: 1
 }
}
int64_val: 10
 is not JSON serializable

The only input that works is passing in one single tensor_proto:

item = tf.make_tensor_proto(values=np.asarray([10]), shape=[1], dtype=tf.int64)
predictor.predict(item)

which gives the following output (due to multiple tensors needing to get populated):

 ERROR in serving: AbortionError(code=StatusCode.INVALID_ARGUMENT, details="input size does not match signature")
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/container_support/serving.py", line 180, in _invoke
self.transformer.transform(content, input_content_type, requested_output_content_type)
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 267, in transform
return self.transform_fn(data, content_type, accepts), accepts
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 192, in f
prediction = self.predict_fn(input)
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 207, in predict_fn
return self.proxy_client.request(data)
File "/usr/local/lib/python2.7/dist-packages/tf_container/proxy_client.py", line 67, in request
return request_fn(data)
File "/usr/local/lib/python2.7/dist-packages/tf_container/proxy_client.py", line 99, in predict
result = stub.Predict(request, self.request_timeout)
File "/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py", line 309, in __call__
self._request_serializer, self._response_deserializer)
File "/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py", line 195, in _blocking_unary_unary
raise _abortion_error(rpc_error_call)
AbortionError: AbortionError(code=StatusCode.INVALID_ARGUMENT, details="input size does not match signature")
[2018-04-18 22:12:40,641] ERROR in serving: AbortionError(code=StatusCode.INVALID_ARGUMENT, details="input size does not match signature")
10.32.0.2 - - [18/Apr/2018:22:12:40 +0000] "POST /invocations HTTP/1.1" 500 0 "-" "AHC/2.0"

I'm basically stuck with the endpoint saying that it requires some format, and the serializer not being able to serialize this format :(

[ChoiByungWook]

Hello,

Thanks for bringing this issue to our attention.

Can you please provide a minimal repo?

Thanks!

[o4k741x]

Sure

https://github.com/OktayGardener/itemrecs

[o4k741x]

Hello. I've switched to BYO model for now, starting with some raw python to learn how this works.

I found something here. With 'deploy' on the RealTimePredictor object, the user is able to create its own serializer and deserializer, maybe a quick solution is to give this freedom for now?

I found this however:

https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/estimator.py#L472

https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/estimator.py#L488

Which blocks me of using your TF Docker wrapper class and making me not able to pass in these arguments. Otherwise, I would have been able to serve this model right now. I might just build all of this myself for transparency reasons after learning how all of these things work. Thanks again.

[o4k741x]

Okay I've found one problem that could help.

I use
tf.make_tensor_proto()
like:

from tensorflow.core.framework import tensor_pb2
from tensorflow import make_tensor_proto
import tensorflow as tf 

Also this works:

import google.protobuf.json_format as json_format
data = make_tensor_proto(1, dtype=tf.int64, shape=[1])
json_format.MessageToJson(data)

'{\n "dtype": "DT_INT64", \n "tensorShape": {\n "dim": [\n {\n "size": "1"\n }\n ]\n }, \n "int64Val": [\n "1"\n ]\n}'

But:

import json
d = dict()
item = 'item'.encode('utf-8')
d[item] = make_tensor_proto(1, dtype=tf.int64, shape=[1])
# d = d.encode('utf-8')
predictor.predict(d)

Doesn't work because this is not satisfied:

https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/predictor.py#L84

Breaks and decays to this else statement:

 in __call__(self, data)
 85 return json_format.MessageToJson(data)
 86 else:
---> 87 return json_serializer(data)
 88 

But you're importing

from tensorflow.python.framework.tensor_util import make_tensor_proto

and checking for tensor_pb2.TensorProto while the return value of

from tensorflow import make_tensor_proto
make_tensor_proto(1, dtype=tf.int64, shape=[1]).__class__.__name__

is TensorProto.

[lukmis]

Hi @OktayGardener,

This looks like we are missing this exact scenario at this moment.
As you showed dict<string, tensor_pb2.TensorProto> can't be passed from the SDK.
I think you basically would need to provide your own input_fn() to be able to use it smoothly. you would still use:
predictor.predict({'user': 10, 'item': 10})
but in the input_fn() you would deserialize it back into the expected dictionary.
Please refer to the README for details how to provide your own input_fn() to the standard container.

[o4k741x]

Hey @lukmis. In the minimal repo I provided per request from @ChoiByungWook, I am actually doing this, check it out here: https://github.com/OktayGardener/itemrecs/blob/master/itemembd.py#L109. Also, is serving_input_fn getting fed by my _input_fn? Doesn't this have to do with serializers/deserializers that you guys have locked when using sagemaker API for Tensorflow Estimator?

I think it would greatly help if I could get an explanation of what order of execution these operations have, who gets fed what and when. Another thing I'm clueless of is how I can add more stuff to 'params'.

[winstonaws]

@OktayGardener We have a big gap in our documentation on our TensorFlow execution logic; sorry for the pain that has caused. However, the code is open source. The fastest way to get an understanding of what's happening may be to read the file. here's the most relevant part:

https://github.com/aws/sagemaker-tensorflow-containers/blob/master/src/tf_container/trainer.py#L67

The customer_script field is your module. We basically take the different pieces provided by functions in your module, assemble them into the relevant TensorFlow configuration objects, and call train_and_evaluate.

[carverdo]

Hi all,
Yesterday I found myself in the same tensor_pb2.TensorProto / serializer loop-trauma as Oktay.

I can't really make that much sense of the #67 reference above and wanted to ask if there was any further follow up, or progress on the issue?

[carverdo]

Answering my own question (in case anyone else falls into a similar trap)...

I needed to add an input_fn into the entrypoint.py to tease apart ser/deserialization from the requirement to use a tensor_proto format.

def input_fn(serialized_input, content_type):
deserial = json.loads(serialized_input)
return {k: tf.make_tensor_proto(v, dtype=tf.float32, shape=[1]) for k, v in deserial.items()}