Stalled PyTorch training job on SageMaker with custom image

[mattmcclean]

Describe the bug
I have found an issue training a PyTorch based model with SageMaker. The training job stalls and does not return any log at a certain point. I have tried to run the same training job locally on a SageMaker Notebook Instance in local mode and it works fine.

To reproduce
Here is the code I used to train the model:

import boto3
from sagemaker.pytorch import PyTorch
import sagemaker as sage

# upload training data to S3
sess = sage.Session()
data_location = sess.upload_data(path, key_prefix=prefix)

client = boto3.client('sts')
account = client.get_caller_identity()['Account']
my_session = boto3.session.Session()
region = my_session.region_name
algorithm_name = 'fastai2-oxford-pets-sagemaker-example'
ecr_image = '{}.dkr.ecr.{}.amazonaws.com/{}-{}:latest'.format(account, region, algorithm_name, 'training')

# run the training job
hyperparameters = {'epochs': 2}
instance_type = 'ml.p3.2xlarge'
estimator = PyTorch(source_dir='container/oxford-pets',
 entry_point='oxford-pets.py',
 role=role,
 train_instance_count=1,
 train_instance_type=instance_type,
 framework_version='1.4.0',
 py_version='py3',
 hyperparameters=hyperparameters,
 image_name=ecr_image)

estimator.fit(data_location)

Expected behavior
I expect the remote training job to run in the same way as the local training job.

Screenshots or logs

Here are the logs from CloudWatch when running the training job on SageMaker:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| timestamp | message |
|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1584809751611 | bash: cannot set terminal process group (-1): Inappropriate ioctl for device |
| 1584809751611 | bash: no job control in this shell |
| 1584809752611 | 2020-03-21 16:55:52,398 sagemaker-containers INFO Imported framework sagemaker_pytorch_container.training |
| 1584809752611 | 2020-03-21 16:55:52,421 sagemaker_pytorch_container.training INFO Block until all host DNS lookups succeed. |
| 1584809754612 | 2020-03-21 16:55:53,875 sagemaker_pytorch_container.training INFO Invoking user training script. |
| 1584809754612 | 2020-03-21 16:55:54,132 sagemaker-containers INFO Module default_user_module_name does not provide a setup.py. |
| 1584809754612 | Generating setup.py |
| 1584809754612 | 2020-03-21 16:55:54,132 sagemaker-containers INFO Generating setup.cfg |
| 1584809754612 | 2020-03-21 16:55:54,132 sagemaker-containers INFO Generating MANIFEST.in |
| 1584809754612 | 2020-03-21 16:55:54,132 sagemaker-containers INFO Installing module with the following command: |
| 1584809754612 | /opt/conda/bin/python -m pip install . |
| 1584809755613 | Processing /tmp/tmpnauetv9h/module_dir |
| 1584809755613 | Building wheels for collected packages: default-user-module-name Building wheel for default-user-module-name (setup.py): started Building wheel for default-user-module-name (setup.py): finished with status 'done' Created wheel for default-user-module-name: filename=default_user_module_name-1.0.0-py2.py3-none-any.whl size=4363 sha256=0a635403d69c368f1f1cd92132ecda70c3f413161b66290a1e8f51480ff25087 Stored in directory: /tmp/pip-ephem-wheel-cache-tnzgd27m/wheels/0c/79/09/07c6c63bdf9a903552bed453a9418666c007f39c160a72555f |
| 1584809755613 | Successfully built default-user-module-name |
| 1584809756613 | Installing collected packages: default-user-module-name |
| 1584809756613 | Successfully installed default-user-module-name-1.0.0 |
| 1584809756613 | 2020-03-21 16:55:56,318 sagemaker-containers INFO Invoking user script |
| 1584809756613 | Training Env: |
| 1584809756614 | { "additional_framework_parameters": {}, "channel_input_dirs": { "training": "/opt/ml/input/data/training" }, "current_host": "algo-1", "framework_module": "sagemaker_pytorch_container.training:main", "hosts": [ "algo-1" ], "hyperparameters": { "epochs": 2 }, "input_config_dir": "/opt/ml/input/config", "input_data_config": { "training": { "TrainingInputMode": "File", "S3DistributionType": "FullyReplicated", "RecordWrapperType": "None" } }, "input_dir": "/opt/ml/input", "is_master": true, "job_name": "fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-48-47-522", "log_level": 20, "master_hostname": "algo-1", "model_dir": "/opt/ml/model", "module_dir": "s3://sagemaker-eu-west-1-934676248949/fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-48-47-522/source/sourcedir.tar.gz", "module_name": "oxford-pets", "network_interface_name": "eth0", "num_cpus": 8, "num_gpus": 1, "output_data_dir": "/opt/ml/output/data", "output_dir": "/opt/ml/output", "output_intermediate_dir": "/opt/ml/output/intermediate", "resource_config": { "current_host": "algo-1", "hosts": [ "algo-1" ], "network_interface_name": "eth0" }, "user_entry_point": "oxford-pets.py" |
| 1584809756614 | } |
| 1584809756614 | Environment variables: |
| 1584809756614 | SM_HOSTS=["algo-1"] |
| 1584809756614 | SM_NETWORK_INTERFACE_NAME=eth0 |
| 1584809756614 | SM_HPS={"epochs":2} |
| 1584809756615 | SM_USER_ENTRY_POINT=oxford-pets.py |
| 1584809756615 | SM_FRAMEWORK_PARAMS={} |
| 1584809756615 | SM_RESOURCE_CONFIG={"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"} |
| 1584809756615 | SM_INPUT_DATA_CONFIG={"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}} |
| 1584809756615 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data |
| 1584809756615 | SM_CHANNELS=["training"] |
| 1584809756615 | SM_CURRENT_HOST=algo-1 |
| 1584809756615 | SM_MODULE_NAME=oxford-pets |
| 1584809756615 | SM_LOG_LEVEL=20 |
| 1584809756615 | SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main |
| 1584809756615 | SM_INPUT_DIR=/opt/ml/input |
| 1584809756615 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config |
| 1584809756615 | SM_OUTPUT_DIR=/opt/ml/output |
| 1584809756615 | SM_NUM_CPUS=8 |
| 1584809756615 | SM_NUM_GPUS=1 |
| 1584809756615 | SM_MODEL_DIR=/opt/ml/model |
| 1584809756615 | SM_MODULE_DIR=s3://sagemaker-eu-west-1-934676248949/fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-48-47-522/source/sourcedir.tar.gz |
| 1584809756615 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1"],"hyperparameters":{"epochs":2},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-48-47-522","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-eu-west-1-934676248949/fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-48-47-522/source/sourcedir.tar.gz","module_name":"oxford-pets","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"},"user_entry_point":"oxford-pets.py"} |
| 1584809756615 | SM_USER_ARGS=["--epochs","2"] |
| 1584809756615 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate |
| 1584809756615 | SM_CHANNEL_TRAINING=/opt/ml/input/data/training |
| 1584809756615 | SM_HP_EPOCHS=2 |
| 1584809756615 | PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages |
| 1584809756616 | Invoking script with the following command: |
| 1584809756616 | /opt/conda/bin/python oxford-pets.py --epochs 2 |
| 1584809769641 | [2020-03-21 16:56:08.752 algo-1:46 INFO json_config.py:90] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json. |
| 1584809769641 | Downloading: "https://download.pytorch.org/models/resnet50-19c8e357.pth" to /root/.cache/torch/checkpoints/resnet50-19c8e357.pth |
| 1584809769641 | [2020-03-21 16:56:08.752 algo-1:46 INFO hook.py:170] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries. |
| 1584809769641 | #015 0%| | 0.00/97.8M [00:00<?, ?B/s]#015 3%|▎ | 2.52M/97.8M [00:00<00:03, 26.4MB/s]#015 5%|▌ | 5.25M/97.8M [00:00<00:03, 26.9MB/s]#015 8%|▊ | 7.75M/97.8M [00:00<00:03, 26.2MB/s]#015 11%|█ | 10.6M/97.8M [00:00<00:03, 27.1MB/s]#015 14%|█▎ | 13.3M/97.8M [00:00<00:03, 27.5MB/s]#015 16%|█▋ | 15.9M/97.8M [00:00<00:03, 27.6MB/s]#015 19%|█▉ | 18.7M/97.8M [00:00<00:02, 27.8MB/s]#015 22%|██▏ | 21.2M/97.8M [00:00<00:02, 27.3MB/s]#015 24%|██▍ | 23.9M/97.8M [00:00<00:02, 27.5MB/s]#015 27%|██▋ | 26.6M/97.8M [00:01<00:02, 27.8MB/s]#015 30%|██▉ | 29.3M/97.8M [00:01<00:02, 27.9MB/s]#015 33%|███▎ | 31.9M/97.8M [00:01<00:02, 27.8MB/s]#015 35%|███▌ | 34.6M/97.8M [00:01<00:02, 27.8MB/s]#015 38%|███▊ | 37.4M/97.8M [00:01<00:02, 28.0MB/s]#015 41%|████ | 40.1M/97.8M [00:01<00:02, 28.2MB/s]#015 44%|████▍ | 42.8M/97.8M [00:01<00:02, 28.2MB/s]#015 47%|████▋ | 45.6M/97.8M [00:01<00:01, 28.4MB/s]#015 49%|████▉ | 48.3M/97.8M [00:01<00:01, 27.7MB/s]#015 52%|█████▏ | 50.9M/97.8M [00:01<00:01, 27.7MB/s]#015 55%|█████▍ | 53.6M/97.8M [00:02<00:01, 27.9MB/s]#015 58%|█████▊ | 56.3M/97.8M [00:02<00:01, 27.1MB/s]#015 60%|██████ | 59.0M/97.8M [00:02<00:01, 27.5MB/s]#015 63%|██████▎ | 61.8M/97.8M [00:02<00:01, 27.8MB/s]#015 66%|██████▌ | 64.5M/97.8M [00:02<00:01, 28.1MB/s]#015 69%|██████▉ | 67.2M/97.8M [00:02<00:01, 28.3MB/s]#015 72%|███████▏ | 70.0M/97.8M [00:02<00:01, 28.3MB/s]#015 74%|███████▍ | 72.8M/97.8M [00:02<00:00, 28.5MB/s]#015 77%|███████▋ | 75.5M/97.8M [00:02<00:00, 28.0MB/s]#015 80%|████████ | 78.2M/97.8M [00:02<00:00, 28.1MB/s]#015 83%|████████▎ | 81.0M/97.8M [00:03<00:00, 28.4MB/s]#015 86%|████████▌ | 83.7M/97.8M [00:03<00:00, 28.3MB/s]#015 88%|████████▊ | 86.4M/97.8M [00:03<00:00, 28.4MB/s]#015 91%|█████████ | 89.2M/97.8M [00:03<00:00, 28.4MB/s]#015 94%|█████████▍| 91.9M/97.8M [00:03<00:00, 27.8MB/s]#015 97%|█████████▋| 94.6M/97.8M [00:03<00:00, 28.0MB/s]#015100%|█████████▉| 97.3M/97.8M [00:03<00:00, 28.0MB/s]#015100%|██████████| 97.8M/97.8M [00:03<00:00, 27.9MB/s] |
| 1584809769641 | [2020-03-21 16:56:08.752 algo-1:46 INFO hook.py:215] Saving to /opt/ml/output/tensors |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Here are the logs when running in local mode. It ran successfully with the same Docker image:

Creating tmpsp4f4h0t_algo-1-636se_1 ... 
Attaching to tmpsp4f4h0t_algo-1-636se_12mdone
algo-1-636se_1 | 2020-03-21 16:44:10,168 sagemaker-containers INFO Imported framework sagemaker_pytorch_container.training
algo-1-636se_1 | 2020-03-21 16:44:10,201 sagemaker_pytorch_container.training INFO Block until all host DNS lookups succeed.
algo-1-636se_1 | 2020-03-21 16:44:10,202 sagemaker_pytorch_container.training INFO Invoking user training script.
algo-1-636se_1 | 2020-03-21 16:44:10,348 sagemaker-containers INFO Module default_user_module_name does not provide a setup.py. 
algo-1-636se_1 | Generating setup.py
algo-1-636se_1 | 2020-03-21 16:44:10,348 sagemaker-containers INFO Generating setup.cfg
algo-1-636se_1 | 2020-03-21 16:44:10,348 sagemaker-containers INFO Generating MANIFEST.in
algo-1-636se_1 | 2020-03-21 16:44:10,349 sagemaker-containers INFO Installing module with the following command:
algo-1-636se_1 | /opt/conda/bin/python -m pip install . 
algo-1-636se_1 | Processing /tmp/tmp5fvc5oxc/module_dir
algo-1-636se_1 | Building wheels for collected packages: default-user-module-name
algo-1-636se_1 | Building wheel for default-user-module-name (setup.py) ... done
algo-1-636se_1 | Created wheel for default-user-module-name: filename=default_user_module_name-1.0.0-py2.py3-none-any.whl size=4364 sha256=9b9736c6048e647a361000af0385eaf0507f14fbaf41ed0e8dddd7c383462a08
algo-1-636se_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-nahh2sra/wheels/29/99/43/3d5edfdcc8f0955107dd1c3bbb438930ef19cddf2d205fb4b8
algo-1-636se_1 | Successfully built default-user-module-name
algo-1-636se_1 | Installing collected packages: default-user-module-name
algo-1-636se_1 | Successfully installed default-user-module-name-1.0.0
algo-1-636se_1 | 2020-03-21 16:44:12,595 sagemaker-containers INFO Invoking user script
algo-1-636se_1 | 
algo-1-636se_1 | Training Env:
algo-1-636se_1 | 
algo-1-636se_1 | {
algo-1-636se_1 | "additional_framework_parameters": {},
algo-1-636se_1 | "channel_input_dirs": {
algo-1-636se_1 | "training": "/opt/ml/input/data/training"
algo-1-636se_1 | },
algo-1-636se_1 | "current_host": "algo-1-636se",
algo-1-636se_1 | "framework_module": "sagemaker_pytorch_container.training:main",
algo-1-636se_1 | "hosts": [
algo-1-636se_1 | "algo-1-636se"
algo-1-636se_1 | ],
algo-1-636se_1 | "hyperparameters": {
algo-1-636se_1 | "epochs": 2
algo-1-636se_1 | },
algo-1-636se_1 | "input_config_dir": "/opt/ml/input/config",
algo-1-636se_1 | "input_data_config": {
algo-1-636se_1 | "training": {
algo-1-636se_1 | "TrainingInputMode": "File"
algo-1-636se_1 | }
algo-1-636se_1 | },
algo-1-636se_1 | "input_dir": "/opt/ml/input",
algo-1-636se_1 | "is_master": true,
algo-1-636se_1 | "job_name": "fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-44-07-259",
algo-1-636se_1 | "log_level": 20,
algo-1-636se_1 | "master_hostname": "algo-1-636se",
algo-1-636se_1 | "model_dir": "/opt/ml/model",
algo-1-636se_1 | "module_dir": "s3://sagemaker-eu-west-1-934676248949/fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-44-07-259/source/sourcedir.tar.gz",
algo-1-636se_1 | "module_name": "oxford-pets",
algo-1-636se_1 | "network_interface_name": "eth0",
algo-1-636se_1 | "num_cpus": 8,
algo-1-636se_1 | "num_gpus": 1,
algo-1-636se_1 | "output_data_dir": "/opt/ml/output/data",
algo-1-636se_1 | "output_dir": "/opt/ml/output",
algo-1-636se_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
algo-1-636se_1 | "resource_config": {
algo-1-636se_1 | "current_host": "algo-1-636se",
algo-1-636se_1 | "hosts": [
algo-1-636se_1 | "algo-1-636se"
algo-1-636se_1 | ]
algo-1-636se_1 | },
algo-1-636se_1 | "user_entry_point": "oxford-pets.py"
algo-1-636se_1 | }
algo-1-636se_1 | 
algo-1-636se_1 | Environment variables:
algo-1-636se_1 | 
algo-1-636se_1 | SM_HOSTS=["algo-1-636se"]
algo-1-636se_1 | SM_NETWORK_INTERFACE_NAME=eth0
algo-1-636se_1 | SM_HPS={"epochs":2}
algo-1-636se_1 | SM_USER_ENTRY_POINT=oxford-pets.py
algo-1-636se_1 | SM_FRAMEWORK_PARAMS={}
algo-1-636se_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-636se","hosts":["algo-1-636se"]}
algo-1-636se_1 | SM_INPUT_DATA_CONFIG={"training":{"TrainingInputMode":"File"}}
algo-1-636se_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
algo-1-636se_1 | SM_CHANNELS=["training"]
algo-1-636se_1 | SM_CURRENT_HOST=algo-1-636se
algo-1-636se_1 | SM_MODULE_NAME=oxford-pets
algo-1-636se_1 | SM_LOG_LEVEL=20
algo-1-636se_1 | SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
algo-1-636se_1 | SM_INPUT_DIR=/opt/ml/input
algo-1-636se_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
algo-1-636se_1 | SM_OUTPUT_DIR=/opt/ml/output
algo-1-636se_1 | SM_NUM_CPUS=8
algo-1-636se_1 | SM_NUM_GPUS=1
algo-1-636se_1 | SM_MODEL_DIR=/opt/ml/model
algo-1-636se_1 | SM_MODULE_DIR=s3://sagemaker-eu-west-1-934676248949/fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-44-07-259/source/sourcedir.tar.gz
algo-1-636se_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1-636se","framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1-636se"],"hyperparameters":{"epochs":2},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-44-07-259","log_level":20,"master_hostname":"algo-1-636se","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-eu-west-1-934676248949/fastai2-oxford-pets-sagemaker-example-t-2020-03-21-16-44-07-259/source/sourcedir.tar.gz","module_name":"oxford-pets","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-636se","hosts":["algo-1-636se"]},"user_entry_point":"oxford-pets.py"}
algo-1-636se_1 | SM_USER_ARGS=["--epochs","2"]
algo-1-636se_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
algo-1-636se_1 | SM_CHANNEL_TRAINING=/opt/ml/input/data/training
algo-1-636se_1 | SM_HP_EPOCHS=2
algo-1-636se_1 | PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages
algo-1-636se_1 | 
algo-1-636se_1 | Invoking script with the following command:
algo-1-636se_1 | 
algo-1-636se_1 | /opt/conda/bin/python oxford-pets.py --epochs 2
algo-1-636se_1 | 
algo-1-636se_1 | 
epoch train_loss valid_loss error_rate time 
algo-1-636se_1 | Downloading: "https://download.pytorch.org/models/resnet50-19c8e357.pth" to /root/.cache/torch/checkpoints/resnet50-19c8e357.pth
0 0.902371 0.234592 0.077131 00:31 
100%|██████████| 97.8M/97.8M [00:03<00:00, 27.5MB/s]
1 0.323855 0.182979 0.064953 00:39 
algo-1-636se_1 | INFO:__main__:Finished training
algo-1-636se_1 | INFO:__main__:Saving the model.
algo-1-636se_1 | INFO:__main__:Model saved
algo-1-636se_1 | 2020-03-21 16:45:34,886 sagemaker-containers INFO Reporting training SUCCESS
tmpsp4f4h0t_algo-1-636se_1 exited with code 0
Aborting on container exit...
===== Job Complete =====

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 1.51.3
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch
• Framework version: 1.4.0
• Python version: 3.6
• CPU or GPU: GPU
• Custom Docker image (Y/N): Y

Additional context
Add any other context about the problem here.

[mmaybeno]

This seems to be a common problem that not only you have come across. The author of #1353 and I both have had similar issues. I've done several days of debugging and still haven't found what is causing it. I suspect it is something with their base pytorch image. I've run the same code in several other instances without issue, even in a sagemaker notebook instance. I would love understanding what is going on as it is impacting my options to scale up.

[ajaykarpur]

Hi @mattmcclean, would you be able to share your Dockerfile for your custom Docker image as well?

@mmaybeno, are you observing this issue with a custom Docker image, or with the base PyTorch image itself?

[mmaybeno]

Custom docker image as well. I had issues with the provided one because of this issue: aws/sagemaker-pytorch-training-toolkit#181. Main difference between mine and the provided dockerfile from the sagemaker-pytorch-container git repo is that I did not use the custom builds of pytorch and torchvision. Instead I installed them via pip. Also building locally fails unless you copy the build artifacts from the specific locations.

The diff is something to this effect

diff --git a/docker/1.4.0/py3/Dockerfile.gpu b/docker/1.4.0/py3/Dockerfile.gpu
index 8be56d4..72176f7 100644
--- a/docker/1.4.0/py3/Dockerfile.gpu
+++ b/docker/1.4.0/py3/Dockerfile.gpu
@@ -106,8 +106,8 @@ RUN conda install -c pytorch magma-cuda101==2.5.1 \
 WORKDIR /opt/pytorch
 
 # Copy workaround script for incorrect hostname
-COPY changehostname.c /
-COPY start_with_right_hostname.sh /usr/local/bin/start_with_right_hostname.sh
+COPY docker/build_artifacts/changehostname.c /
+COPY docker/build_artifacts/start_with_right_hostname.sh /usr/local/bin/start_with_right_hostname.sh
 
 WORKDIR /root
 
@@ -124,10 +124,10 @@ RUN pip install \
 --no-cache-dir fastai==1.0.59 \
 awscli \
 scipy==1.2.2 \
- && pip install --no-cache-dir -U https://pytorch-aws.s3.amazonaws.com/pytorch-1.4.0/py3/gpu/torch-1.4.0-cp36-cp36m-manylinux1_x86_64.whl \
- && pip uninstall -y torchvision \
- && pip install --no-cache-dir -U \
- https://torchvision-build.s3.amazonaws.com/1.4.0/py3/gpu/torchvision-0.5.0-cp36-cp36m-linux_x86_64.whl
+ torch==1.4.0 \
+ torchvision==0.5.0 \
+ sagemaker==1.50.17 \
+ "sagemaker-pytorch-training<2"

[mattmcclean]

Hi @mattmcclean, would you be able to share your Dockerfile for your custom Docker image as well?

My Dockerfile can be found here. I tried building a custom Docker image but also tried the standard PyTorch GPU training container and both have the same issue using the 1.4.0 training container.

[hjuhel-cdpq]

Hi,
I'm the author of #1353.
We have directly reached AWS support. They were aware of the issue and have fixed it in a new release of the pytorch container. Can you try rebuilding (with a fresh version of the base python docker). According to AWS, this is the version to use

[mmaybeno]

Resolved for me, at least for the epoch issue. Need to reinstall pytorch as their build still has this issue: aws/sagemaker-pytorch-training-toolkit#181.

Current dockerfile that builds off the now working pytorch image.

FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3

RUN pip uninstall torch -y && pip install --no-cache-dir -U torch

[shreyapandit]

Closing this issue since error has since been resolved