Tensorflow error when using input of type sagemaker.session.s3_input

[rohitgmathews]

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Tensorflow
• Framework Version: 1.12.0
• Python Version: 2.7
• CPU or GPU: CPU (ml.m5.xlarge)
• Python SDK Version:
• Are you using a custom image: No

Describe the problem

Sagemaker tensorflow fails when trying to train on data that is provided in the form of s3_input

For example:
Given

s3_input_train = sagemaker.s3_input(
 s3_data='s3://my_bucket/path/to/prefix,
 content_type='csv',
 distribution='ShardedByS3Key')

the following fail while looking for training_data in /opt/ml/input/data/training/

estimator.fit({'train': s3_input_train})
(or)
# Assume s3_eval_train was also created similar to s3_input_train
estimator.fit({'train': s3_input_train, 'validation': s3_eval_train})

But this succeeds,
estimator.fit(s3_input_train)

Minimal repro / logs

2019-03-12 15:12:29,570 ERROR - container_support.training - uncaught exception during training: [Errno 2] No such file or directory: '/opt/ml/input/data/training/'
Traceback (most recent call last):
 File "/usr/local/lib/python2.7/dist-packages/container_support/training.py", line 36, in start
 fw.train()
 File "/usr/local/lib/python2.7/dist-packages/tf_container/train_entry_point.py", line 173, in train
 train_wrapper.train()
 File "/usr/local/lib/python2.7/dist-packages/tf_container/trainer.py", line 69, in train
 estimator = self._build_estimator(run_config=run_config)
 File "/usr/local/lib/python2.7/dist-packages/tf_container/trainer.py", line 92, in _build_estimator
 return self.customer_script.estimator_fn(run_config, hyperparameters)
 File "/opt/ml/code/tensorflow_entry_point.py", line 26, in estimator_fn
 feature_columns = [tf.feature_column.numeric_column(INPUT_TENSOR_NAME, shape=get_shape())]
 File "/opt/ml/code/tensorflow_entry_point.py", line 20, in get_shape
 filename0 = os.listdir(training_dir)[0]

• Exact command to reproduce:

s3_input_train = sagemaker.s3_input(
 s3_data='s3://my_bucket/path/to/prefix,
 content_type='csv',
 distribution='ShardedByS3Key')
estimator.fit({'train': s3_input_train})

[mvsusp]

Hi @rohitgmathews,

I noticed in your code that you are using the channel name train. That means that your data will be available under the folder /opt/ml/input/data/train. Would you mind trying that code change?

Thanks for using SageMaker.

[rohitgmathews]

Hi @mvsusp,

That worked. Thank you. Quick question if you don't mind.

Why is it that training_dir is set to None when train_input_fn(training_dir, params) is called?

[mvsusp]

Hi @rohitgmathews,

I'm glad that it worked. It seems that you are using the legacy TensorFlow container instead of the TensorFlow script mode container. I suggest you to use the latter instead for an optimal and simple to use experience.

For more information about the TF script mode container:

• https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_script_mode_quickstart/tensorflow_script_mode_quickstart.ipynb
• https://sagemaker.readthedocs.io/en/stable/using_tf.html#training-with-tensorflow-estimator

Thanks for using SageMaker!

[rohitgmathews]

Thanks @mvsusp
I will try that :)

[ismailgmutlu]

Hi @mvsusp

I was using script mode, but I was having trouble getting preprocessing done for inference with Tensorflow Serving. Therefore, I need to use legacy Tensorflow version for my estimator, because I need python based endpoints for inference. Is there any way of working around the issue @rohitgmathews mentioned in legacy mode? How can I prevent training_dir setting to None. Can you provide any examples of estimators trained in legacy mode.