"No such file or directory: '/opt/ml/model/model.joblib'" occured at deploy in the local mode of windows #846

Description

@xnaiman

System Information

  • Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Scikit-Learn
  • Framework Version: 0.20.0 (official sagemaker-scikit-learn-container)
  • Python Version: 3.6
  • CPU or GPU: CPU
  • Python SDK Version: 1.26.0
  • Are you using a custom image: No

Describe the problem

No such file or directory: /opt/ml/model/model.joblib occured at deploy in the local mode of windows.
(Raised after avoiding issue #844 and #845)

Minimal repro / logs

Training

  • Logs
Windows Support for Local Mode is Experimental
Creating tmpri2jv76o_algo-1-4cuqr_1 ... done
Attaching to tmpri2jv76o_algo-1-4cuqr_1
algo-1-4cuqr_1 | 2019-06-12 23:57:11,801 sagemaker-containers INFO Imported framework sagemaker_sklearn_container.training
algo-1-4cuqr_1 | 2019-06-12 23:57:11,811 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
algo-1-4cuqr_1 | 2019-06-12 23:57:11,813 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
algo-1-4cuqr_1 | 2019-06-12 23:57:11,820 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
algo-1-4cuqr_1 | 2019-06-12 23:57:11,821 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
algo-1-4cuqr_1 | 2019-06-12 23:57:11,825 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
algo-1-4cuqr_1 | 2019-06-12 23:57:11,828 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:11,832 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
algo-1-4cuqr_1 | 2019-06-12 23:57:11,833 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:11,839 sagemaker_sklearn_container.training INFO Invoking user training script.
algo-1-4cuqr_1 | 2019-06-12 23:57:11,842 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
algo-1-4cuqr_1 | 2019-06-12 23:57:11,849 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
algo-1-4cuqr_1 | 2019-06-12 23:57:11,851 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:11,854 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
algo-1-4cuqr_1 | 2019-06-12 23:57:11,857 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:11,863 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/boto3/data/s3/2006-03-01/resources-1.json
algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials DEBUG Looking for credentials via: env
algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials INFO Found credentials in environment variables.
algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/endpoints.json
algo-1-4cuqr_1 | 2019-06-12 23:57:11,871 botocore.hooks DEBUG Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff9c9cafa60>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,875 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/s3/2006-03-01/service-2.json
algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff9c7814400>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08840>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,882 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff9c78141e0>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,883 botocore.args DEBUG The s3 config key is not a dictionary type, ignoring its value of: None
algo-1-4cuqr_1 | 2019-06-12 23:57:11,885 botocore.endpoint DEBUG Setting s3 timeout as (60, 60)
algo-1-4cuqr_1 | 2019-06-12 23:57:11,886 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/_retry.json
algo-1-4cuqr_1 | 2019-06-12 23:57:11,887 botocore.client DEBUG Registering retry handlers for service: s3
algo-1-4cuqr_1 | 2019-06-12 23:57:11,894 botocore.client DEBUG Defaulting to S3 virtual host style addressing with path style addressing fallback.
algo-1-4cuqr_1 | 2019-06-12 23:57:11,895 boto3.resources.factory DEBUG Loading s3:s3
algo-1-4cuqr_1 | 2019-06-12 23:57:11,896 boto3.resources.factory DEBUG Loading s3:Bucket
algo-1-4cuqr_1 | 2019-06-12 23:57:11,897 boto3.resources.model DEBUG Renaming Bucket attribute name
algo-1-4cuqr_1 | 2019-06-12 23:57:11,898 botocore.hooks DEBUG Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08950>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.utils DEBUG Acquiring 0
algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) about to wait for the following futures []
algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) done waiting for dependent futures
algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) with kwargs {'config': <boto3.s3.transfer.TransferConfig object at 0x7ff9c6b5c710>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cef0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8>, 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}
algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.endpoint DEBUG Making request for OperationModel(name=HeadObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1',
'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'HEAD', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.hooks DEBUG Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG Calculating signature using v4 auth.
algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG CanonicalRequest:
algo-1-4cuqr_1 | HEAD
algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
algo-1-4cuqr_1 | x-amz-date:20190612T235711Z
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG StringToSign:
algo-1-4cuqr_1 | AWS4-HMAC-SHA256
algo-1-4cuqr_1 | 20190612T235711Z
algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
algo-1-4cuqr_1 | 3c7d2e923cfbc1df5b252ea8422724d8ecd6c607014af9b27717fcbdf25c94d1
algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.auth DEBUG Signature:
algo-1-4cuqr_1 | 43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f
algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235711Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f'}>
algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443
algo-1-4cuqr_1 | 2019-06-12 23:57:12,024 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "HEAD /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 0
algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': 'F48E7D6B6A526E82', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'cAB/AbGlPTnTc025AlpNhnwIFzC34NmUPjtLJAUp7Tf50k4QwAxC33+RE0jTSrnG7qafBtR38Fs=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
'Wed, 12 Jun 2019 23:57:23 GMT'}
algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response body:
algo-1-4cuqr_1 | b''
algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.retryhandler DEBUG No retry needed.
algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.futures DEBUG Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8> for transfer request: 0.
algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.utils DEBUG Acquiring 0
algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) about to wait for the following futures []
algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) done waiting for dependent futures
algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) with kwargs {'max_attempts': 5, 'bucket': 'sagemaker-ap-northeast-1-130747742019', 'bandwidth_limiter': None, 'io_chunksize': 262144, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'callbacks': [], 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7ff9c6b12cc0>, 'extra_args': {}}
algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.utils DEBUG Releasing acquire 0/None
algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.endpoint DEBUG Making request for OperationModel(name=GetObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1', 'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'GET', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG Calculating signature using v4 auth.
algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG CanonicalRequest:
algo-1-4cuqr_1 | GET
algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
algo-1-4cuqr_1 | x-amz-date:20190612T235712Z
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG StringToSign:
algo-1-4cuqr_1 | AWS4-HMAC-SHA256
algo-1-4cuqr_1 | 20190612T235712Z
algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
algo-1-4cuqr_1 | d5310e2e1112cc81c0cbcd2fdb9f2f8fe645ea026d34a43d2a5cd9ce8cf1b5d3
algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG Signature:
algo-1-4cuqr_1 | 77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4
algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235712Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4'}>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "GET /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 7271
algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': '7250ADF3F5A5A49C', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'WiYMANCqDMYoqe7ZcCVyUJTpmRTWAQ9GRSIeW6JAlxtmugzNooQgP8bb/pnTQ+PBo5seJvLVhoc=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
'Wed, 12 Jun 2019 23:57:23 GMT'}
algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.parsers DEBUG Response body:
algo-1-4cuqr_1 | <botocore.response.StreamingBody object at 0x7ff9c6aba048>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.retryhandler DEBUG No retry needed.
algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'offset': 0, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>}
algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) about to wait for the following futures []
algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) done waiting for dependent futures
algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'final_filename': '/tmp/tmp_f99p1cx/tar_file'}
algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.utils DEBUG Releasing acquire 0/None
algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Module train_model does not provide a setup.py.
algo-1-4cuqr_1 | Generating setup.py
algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating setup.cfg
algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating MANIFEST.in
algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Installing module with the following command:
algo-1-4cuqr_1 | /usr/bin/python3 -m pip install -U .
algo-1-4cuqr_1 | Processing /opt/ml/code
algo-1-4cuqr_1 | Building wheels for collected packages: train-model
algo-1-4cuqr_1 | Running setup.py bdist_wheel for train-model ... done
algo-1-4cuqr_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-d58g4m1m/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
algo-1-4cuqr_1 | Successfully built train-model
algo-1-4cuqr_1 | Installing collected packages: train-model
algo-1-4cuqr_1 | Successfully installed train-model-1.0.0
algo-1-4cuqr_1 | You are using pip version 18.1, however version 19.1.1 is available.
algo-1-4cuqr_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
algo-1-4cuqr_1 | 2019-06-12 23:57:12,983 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
algo-1-4cuqr_1 | 2019-06-12 23:57:12,984 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
algo-1-4cuqr_1 | 2019-06-12 23:57:12,987 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
algo-1-4cuqr_1 | 2019-06-12 23:57:12,988 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
algo-1-4cuqr_1 | 2019-06-12 23:57:12,991 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
algo-1-4cuqr_1 | 2019-06-12 23:57:12,996 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
algo-1-4cuqr_1 | 2019-06-12 23:57:12,998 sagemaker-containers INFO Invoking user script
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | Training Env:
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | {
algo-1-4cuqr_1 | "hosts": [
algo-1-4cuqr_1 | "algo-1-4cuqr"
algo-1-4cuqr_1 | ],
algo-1-4cuqr_1 | "output_dir": "/opt/ml/output",
algo-1-4cuqr_1 | "user_entry_point": "train_model.py",
algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr",
algo-1-4cuqr_1 | "input_data_config": {
algo-1-4cuqr_1 | "train": {
algo-1-4cuqr_1 | "TrainingInputMode": "File"
algo-1-4cuqr_1 | }
algo-1-4cuqr_1 | },
algo-1-4cuqr_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
algo-1-4cuqr_1 | "input_dir": "/opt/ml/input",
algo-1-4cuqr_1 | "output_data_dir": "/opt/ml/output/data",
algo-1-4cuqr_1 | "num_gpus": 0,
algo-1-4cuqr_1 | "log_level": 10,
algo-1-4cuqr_1 | "resource_config": {
algo-1-4cuqr_1 | "hosts": [
algo-1-4cuqr_1 | "algo-1-4cuqr"
algo-1-4cuqr_1 | ],
algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr"
algo-1-4cuqr_1 | },
algo-1-4cuqr_1 | "job_name": "sagemaker-scikit-learn-2019-06-12-23-57-07-460",
algo-1-4cuqr_1 | "additional_framework_parameters": {},
algo-1-4cuqr_1 | "input_config_dir": "/opt/ml/input/config",
algo-1-4cuqr_1 | "network_interface_name": "ethwe",
algo-1-4cuqr_1 | "module_dir": "s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz",
algo-1-4cuqr_1 | "framework_module": "sagemaker_sklearn_container.training:main",
algo-1-4cuqr_1 | "channel_input_dirs": {
algo-1-4cuqr_1 | "train": "/opt/ml/input/data/train"
algo-1-4cuqr_1 | },
algo-1-4cuqr_1 | "module_name": "train_model",
algo-1-4cuqr_1 | "model_dir": "/opt/ml/model",
algo-1-4cuqr_1 | "hyperparameters": {
algo-1-4cuqr_1 | "para_c": 1.0,
algo-1-4cuqr_1 | "para_gamma": 0.01
algo-1-4cuqr_1 | },
algo-1-4cuqr_1 | "num_cpus": 4
algo-1-4cuqr_1 | }
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | Environment variables:
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | PYTHONPATH=/usr/local/bin:/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages
algo-1-4cuqr_1 | SM_MODULE_NAME=train_model
algo-1-4cuqr_1 | SM_LOG_LEVEL=10
algo-1-4cuqr_1 | SM_CHANNELS=["train"]
algo-1-4cuqr_1 | SM_HP_PARA_C=1.0
algo-1-4cuqr_1 | SM_CURRENT_HOST=algo-1-4cuqr
algo-1-4cuqr_1 | SM_FRAMEWORK_PARAMS={}
algo-1-4cuqr_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
algo-1-4cuqr_1 | SM_OUTPUT_DIR=/opt/ml/output
algo-1-4cuqr_1 | SM_HP_PARA_GAMMA=0.01
algo-1-4cuqr_1 | SM_HPS={"para_c":1.0,"para_gamma":0.01}
algo-1-4cuqr_1 | SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main
algo-1-4cuqr_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
algo-1-4cuqr_1 | SM_USER_ARGS=["--para_c","1.0","--para_gamma","0.01"]
algo-1-4cuqr_1 | SM_NUM_CPUS=4
algo-1-4cuqr_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
algo-1-4cuqr_1 | SM_MODULE_DIR=s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
algo-1-4cuqr_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]}
algo-1-4cuqr_1 | SM_HOSTS=["algo-1-4cuqr"]
algo-1-4cuqr_1 | SM_INPUT_DIR=/opt/ml/input
algo-1-4cuqr_1 | SM_CHANNEL_TRAIN=/opt/ml/input/data/train
algo-1-4cuqr_1 | SM_NUM_GPUS=0
algo-1-4cuqr_1 | SM_USER_ENTRY_POINT=train_model.py
algo-1-4cuqr_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1-4cuqr","framework_module":"sagemaker_sklearn_container.training:main","hosts":["algo-1-4cuqr"],"hyperparameters":{"para_c":1.0,"para_gamma":0.01},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","job_name":"sagemaker-scikit-learn-2019-06-12-23-57-07-460","log_level":10,"model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz","module_name":"train_model","network_interface_name":"ethwe","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]},"user_entry_point":"train_model.py"}
algo-1-4cuqr_1 | SM_INPUT_DATA_CONFIG={"train":{"TrainingInputMode":"File"}}
algo-1-4cuqr_1 | SM_NETWORK_INTERFACE_NAME=ethwe
algo-1-4cuqr_1 | SM_MODEL_DIR=/opt/ml/model
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | Invoking script with the following command:
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | /usr/bin/python3 -m train_model --para_c 1.0 --para_gamma 0.01
algo-1-4cuqr_1 |
algo-1-4cuqr_1 |
algo-1-4cuqr_1 | 2019-06-12 23:57:13,528 sagemaker-containers INFO Reporting training SUCCESS
tmpri2jv76o_algo-1-4cuqr_1 exited with code 0
Aborting on container exit...
===== Job Complete =====
  • Created Temp Files
Temp
├─tmpzg964wnz
│ iris_train.csv
│
└─tmpri2jv76o
│ docker-compose.yaml
│ ├─artifacts
├─model
│ model.joblib
│ └─output
└─data
  • docker-compose.yaml
networks:
sagemaker-local:
name: sagemaker-local
services:
algo-1-4cuqr:
command: train
environment:
- AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
- AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
- AWS_REGION=ap-northeast-1
- TRAINING_JOB_NAME=sagemaker-scikit-learn-2019-06-12-23-57-07-460
- HTTP_PROXY=http://username:password@foobar:port
- HTTPS_PROXY=http://username:password@foobar:port
image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
networks:
sagemaker-local:
aliases:
- algo-1-4cuqr
stdin_open: true
tty: true
volumes:
- C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output:/opt/ml/output
- C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\input:/opt/ml/input
- C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output/data:/opt/ml/output/data
- C:\Users\username\AppData\Local\Temp\tmpri2jv76o\model:/opt/ml/model
- C:\Users\username\AppData\Local\Temp\tmpzg964wnz:/opt/ml/input/data/train
version: '2.3'

Deploying

  • Logs
Attaching to tmpdbo8bbod_algo-1-qlg3c_1
algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Starting gunicorn 19.9.0
algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.sock (16)
algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Using worker: gevent
algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [23] [INFO] Booting worker with pid: 23
algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [27] [INFO] Booting worker with pid: 27
algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [29] [INFO] Booting worker with pid: 29
algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [33] [INFO] Booting worker with pid: 33
algo-1-qlg3c_1 | 2019-06-12 23:57:21,219 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)
algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Module train_model does not provide a setup.py.
algo-1-qlg3c_1 | Generating setup.py
algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating setup.cfg
algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating MANIFEST.in
algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Installing module with the following command:
algo-1-qlg3c_1 | /usr/bin/python3 -m pip install -U .
algo-1-qlg3c_1 | Processing /opt/ml/code
algo-1-qlg3c_1 | Building wheels for collected packages: train-model
algo-1-qlg3c_1 | Running setup.py bdist_wheel for train-model ... done
algo-1-qlg3c_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-4x4jzvi4/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
algo-1-qlg3c_1 | Successfully built train-model
algo-1-qlg3c_1 | Installing collected packages: train-model
algo-1-qlg3c_1 | Successfully installed train-model-1.0.0
algo-1-qlg3c_1 | You are using pip version 18.1, however version 19.1.1 is available.
algo-1-qlg3c_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
algo-1-qlg3c_1 | [2019-06-12 23:57:22 +0000] [23] [ERROR] Error handling request /ping
algo-1-qlg3c_1 | Traceback (most recent call last):
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
algo-1-qlg3c_1 | return fn(*args, **kwargs)
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
algo-1-qlg3c_1 | with open(filename, 'rb') as f:
algo-1-qlg3c_1 | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
algo-1-qlg3c_1 |
algo-1-qlg3c_1 | During handling of the above exception, another exception occurred:
algo-1-qlg3c_1 |
algo-1-qlg3c_1 | Traceback (most recent call last):
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
algo-1-qlg3c_1 | self.handle_request(listener_name, req, client, addr)
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
algo-1-qlg3c_1 | addr)
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
algo-1-qlg3c_1 | respiter = self.wsgi(environ, resp.start_response)
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_sklearn_container/serving.py", line 103, in main
algo-1-qlg3c_1 | user_module_transformer.initialize()
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_transformer.py", line 157, in initialize
algo-1-qlg3c_1 | self._model = self._model_fn(_env.model_dir)
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
algo-1-qlg3c_1 | six.reraise(error_class, error_class(e), sys.exc_info()[2])
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
algo-1-qlg3c_1 | raise value.with_traceback(tb)
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
algo-1-qlg3c_1 | return fn(*args, **kwargs)
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
algo-1-qlg3c_1 | with open(filename, 'rb') as f:
algo-1-qlg3c_1 | sagemaker_containers._errors.ClientError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
algo-1-qlg3c_1 | 172.18.0.1 - - [12/Jun/2019:23:57:22 +0000] "GET /ping HTTP/1.1" 500 141 "-" "-"
  • Created Temp Files
Temp
├─tmpgfpx4a40 # Why is it empty
│ └─tmpdbo8bbod
│ docker-compose.yaml
│ └─algo-1-qlg3c
  • docker-compose.yaml
networks:
sagemaker-local:
name: sagemaker-local
services:
algo-1-qlg3c:
command: serve
environment:
- AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
- AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
- HTTP_PROXY=http://username:password@foobar:port
- HTTPS_PROXY=http://username:password@foobar:port
- SAGEMAKER_PROGRAM=train_model.py
- SAGEMAKER_SUBMIT_DIRECTORY=s3://sagemaker-ap-northeast-1-XXXXXXXXXX/sagemaker-scikit-learn-2019-06-12-23-57-07-460/sourcedir.tar.gz
- SAGEMAKER_ENABLE_CLOUDWATCH_METRICS=false
- SAGEMAKER_CONTAINER_LOG_LEVEL=10
- SAGEMAKER_REGION=ap-northeast-1
image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
networks:
sagemaker-local:
aliases:
- algo-1-qlg3c
ports:
- 8080:8080
stdin_open: true
tty: true
volumes:
- C:\Users\username\AppData\Local\Temp\tmpgfpx4a40:/opt/ml/model
version: '2.3'

Exact command to reproduce

sklearn=SKLearn(
entry_point=script_path,
train_instance_type='local',
source_dir='./',
role=role,
py_version='py3',
container_log_level=10,
hyperparameters={'para_c': 1.0, 'para_gamma': 0.01})
sklearn.fit({'train': train_input})
predictor=sklearn.deploy(initial_instance_count=1,
instance_type='local',
env=env)

Cause

I know that the cause is that the tmpgfpx4a40 folder linked to the /opt/ml/model folder is empty, as described in the docker-compose.yaml file. And I think the tmpgfpx4a40 folder needs a model.joblib file.
However, I don't know why the tmpgfpx4a40 folder is empty and I don't know how to avoid it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
       blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
      }
      } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
      })();
      (function(){
      try {
      var __m = "github.com";
      var __re = new RegExp('^' + "github\\.com" + '
      
      Skip to content

      "No such file or directory: '/opt/ml/model/model.joblib'" occured at deploy in the local mode of windows #846

      Description

      @xnaiman

      System Information

      • Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Scikit-Learn
      • Framework Version: 0.20.0 (official sagemaker-scikit-learn-container)
      • Python Version: 3.6
      • CPU or GPU: CPU
      • Python SDK Version: 1.26.0
      • Are you using a custom image: No

      Describe the problem

      No such file or directory: /opt/ml/model/model.joblib occured at deploy in the local mode of windows.
      (Raised after avoiding issue #844 and #845)

      Minimal repro / logs

      Training

      • Logs
      Windows Support for Local Mode is Experimental
      Creating tmpri2jv76o_algo-1-4cuqr_1 ... done
      Attaching to tmpri2jv76o_algo-1-4cuqr_1
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,801 sagemaker-containers INFO Imported framework sagemaker_sklearn_container.training
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,811 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,813 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,820 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,821 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,825 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,828 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,832 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,833 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,839 sagemaker_sklearn_container.training INFO Invoking user training script.
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,842 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,849 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,851 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,854 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,857 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,863 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/boto3/data/s3/2006-03-01/resources-1.json
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials DEBUG Looking for credentials via: env
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials INFO Found credentials in environment variables.
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/endpoints.json
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,871 botocore.hooks DEBUG Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff9c9cafa60>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,875 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/s3/2006-03-01/service-2.json
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff9c7814400>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08840>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,882 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff9c78141e0>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,883 botocore.args DEBUG The s3 config key is not a dictionary type, ignoring its value of: None
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,885 botocore.endpoint DEBUG Setting s3 timeout as (60, 60)
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,886 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/_retry.json
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,887 botocore.client DEBUG Registering retry handlers for service: s3
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,894 botocore.client DEBUG Defaulting to S3 virtual host style addressing with path style addressing fallback.
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,895 boto3.resources.factory DEBUG Loading s3:s3
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,896 boto3.resources.factory DEBUG Loading s3:Bucket
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,897 boto3.resources.model DEBUG Renaming Bucket attribute name
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,898 botocore.hooks DEBUG Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08950>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.utils DEBUG Acquiring 0
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) about to wait for the following futures []
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) done waiting for dependent futures
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) with kwargs {'config': <boto3.s3.transfer.TransferConfig object at 0x7ff9c6b5c710>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cef0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8>, 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.endpoint DEBUG Making request for OperationModel(name=HeadObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1',
      'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'HEAD', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.hooks DEBUG Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG Calculating signature using v4 auth.
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG CanonicalRequest:
      algo-1-4cuqr_1 | HEAD
      algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
      algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      algo-1-4cuqr_1 | x-amz-date:20190612T235711Z
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
      algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG StringToSign:
      algo-1-4cuqr_1 | AWS4-HMAC-SHA256
      algo-1-4cuqr_1 | 20190612T235711Z
      algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
      algo-1-4cuqr_1 | 3c7d2e923cfbc1df5b252ea8422724d8ecd6c607014af9b27717fcbdf25c94d1
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.auth DEBUG Signature:
      algo-1-4cuqr_1 | 43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235711Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f'}>
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
      algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,024 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "HEAD /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 0
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': 'F48E7D6B6A526E82', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'cAB/AbGlPTnTc025AlpNhnwIFzC34NmUPjtLJAUp7Tf50k4QwAxC33+RE0jTSrnG7qafBtR38Fs=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
      'Wed, 12 Jun 2019 23:57:23 GMT'}
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response body:
      algo-1-4cuqr_1 | b''
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.retryhandler DEBUG No retry needed.
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.futures DEBUG Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8> for transfer request: 0.
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.utils DEBUG Acquiring 0
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) about to wait for the following futures []
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) done waiting for dependent futures
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) with kwargs {'max_attempts': 5, 'bucket': 'sagemaker-ap-northeast-1-130747742019', 'bandwidth_limiter': None, 'io_chunksize': 262144, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'callbacks': [], 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7ff9c6b12cc0>, 'extra_args': {}}
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.utils DEBUG Releasing acquire 0/None
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.endpoint DEBUG Making request for OperationModel(name=GetObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1', 'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'GET', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG Calculating signature using v4 auth.
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG CanonicalRequest:
      algo-1-4cuqr_1 | GET
      algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
      algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      algo-1-4cuqr_1 | x-amz-date:20190612T235712Z
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
      algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG StringToSign:
      algo-1-4cuqr_1 | AWS4-HMAC-SHA256
      algo-1-4cuqr_1 | 20190612T235712Z
      algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
      algo-1-4cuqr_1 | d5310e2e1112cc81c0cbcd2fdb9f2f8fe645ea026d34a43d2a5cd9ce8cf1b5d3
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG Signature:
      algo-1-4cuqr_1 | 77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235712Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4'}>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "GET /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 7271
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': '7250ADF3F5A5A49C', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'WiYMANCqDMYoqe7ZcCVyUJTpmRTWAQ9GRSIeW6JAlxtmugzNooQgP8bb/pnTQ+PBo5seJvLVhoc=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
      'Wed, 12 Jun 2019 23:57:23 GMT'}
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.parsers DEBUG Response body:
      algo-1-4cuqr_1 | <botocore.response.StreamingBody object at 0x7ff9c6aba048>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.retryhandler DEBUG No retry needed.
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'offset': 0, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>}
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) about to wait for the following futures []
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) done waiting for dependent futures
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'final_filename': '/tmp/tmp_f99p1cx/tar_file'}
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.utils DEBUG Releasing acquire 0/None
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Module train_model does not provide a setup.py.
      algo-1-4cuqr_1 | Generating setup.py
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating setup.cfg
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating MANIFEST.in
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Installing module with the following command:
      algo-1-4cuqr_1 | /usr/bin/python3 -m pip install -U .
      algo-1-4cuqr_1 | Processing /opt/ml/code
      algo-1-4cuqr_1 | Building wheels for collected packages: train-model
      algo-1-4cuqr_1 | Running setup.py bdist_wheel for train-model ... done
      algo-1-4cuqr_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-d58g4m1m/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
      algo-1-4cuqr_1 | Successfully built train-model
      algo-1-4cuqr_1 | Installing collected packages: train-model
      algo-1-4cuqr_1 | Successfully installed train-model-1.0.0
      algo-1-4cuqr_1 | You are using pip version 18.1, however version 19.1.1 is available.
      algo-1-4cuqr_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,983 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,984 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,987 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,988 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,991 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,996 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
      algo-1-4cuqr_1 | 2019-06-12 23:57:12,998 sagemaker-containers INFO Invoking user script
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | Training Env:
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | {
      algo-1-4cuqr_1 | "hosts": [
      algo-1-4cuqr_1 | "algo-1-4cuqr"
      algo-1-4cuqr_1 | ],
      algo-1-4cuqr_1 | "output_dir": "/opt/ml/output",
      algo-1-4cuqr_1 | "user_entry_point": "train_model.py",
      algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr",
      algo-1-4cuqr_1 | "input_data_config": {
      algo-1-4cuqr_1 | "train": {
      algo-1-4cuqr_1 | "TrainingInputMode": "File"
      algo-1-4cuqr_1 | }
      algo-1-4cuqr_1 | },
      algo-1-4cuqr_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
      algo-1-4cuqr_1 | "input_dir": "/opt/ml/input",
      algo-1-4cuqr_1 | "output_data_dir": "/opt/ml/output/data",
      algo-1-4cuqr_1 | "num_gpus": 0,
      algo-1-4cuqr_1 | "log_level": 10,
      algo-1-4cuqr_1 | "resource_config": {
      algo-1-4cuqr_1 | "hosts": [
      algo-1-4cuqr_1 | "algo-1-4cuqr"
      algo-1-4cuqr_1 | ],
      algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr"
      algo-1-4cuqr_1 | },
      algo-1-4cuqr_1 | "job_name": "sagemaker-scikit-learn-2019-06-12-23-57-07-460",
      algo-1-4cuqr_1 | "additional_framework_parameters": {},
      algo-1-4cuqr_1 | "input_config_dir": "/opt/ml/input/config",
      algo-1-4cuqr_1 | "network_interface_name": "ethwe",
      algo-1-4cuqr_1 | "module_dir": "s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz",
      algo-1-4cuqr_1 | "framework_module": "sagemaker_sklearn_container.training:main",
      algo-1-4cuqr_1 | "channel_input_dirs": {
      algo-1-4cuqr_1 | "train": "/opt/ml/input/data/train"
      algo-1-4cuqr_1 | },
      algo-1-4cuqr_1 | "module_name": "train_model",
      algo-1-4cuqr_1 | "model_dir": "/opt/ml/model",
      algo-1-4cuqr_1 | "hyperparameters": {
      algo-1-4cuqr_1 | "para_c": 1.0,
      algo-1-4cuqr_1 | "para_gamma": 0.01
      algo-1-4cuqr_1 | },
      algo-1-4cuqr_1 | "num_cpus": 4
      algo-1-4cuqr_1 | }
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | Environment variables:
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | PYTHONPATH=/usr/local/bin:/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages
      algo-1-4cuqr_1 | SM_MODULE_NAME=train_model
      algo-1-4cuqr_1 | SM_LOG_LEVEL=10
      algo-1-4cuqr_1 | SM_CHANNELS=["train"]
      algo-1-4cuqr_1 | SM_HP_PARA_C=1.0
      algo-1-4cuqr_1 | SM_CURRENT_HOST=algo-1-4cuqr
      algo-1-4cuqr_1 | SM_FRAMEWORK_PARAMS={}
      algo-1-4cuqr_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
      algo-1-4cuqr_1 | SM_OUTPUT_DIR=/opt/ml/output
      algo-1-4cuqr_1 | SM_HP_PARA_GAMMA=0.01
      algo-1-4cuqr_1 | SM_HPS={"para_c":1.0,"para_gamma":0.01}
      algo-1-4cuqr_1 | SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main
      algo-1-4cuqr_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
      algo-1-4cuqr_1 | SM_USER_ARGS=["--para_c","1.0","--para_gamma","0.01"]
      algo-1-4cuqr_1 | SM_NUM_CPUS=4
      algo-1-4cuqr_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
      algo-1-4cuqr_1 | SM_MODULE_DIR=s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
      algo-1-4cuqr_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]}
      algo-1-4cuqr_1 | SM_HOSTS=["algo-1-4cuqr"]
      algo-1-4cuqr_1 | SM_INPUT_DIR=/opt/ml/input
      algo-1-4cuqr_1 | SM_CHANNEL_TRAIN=/opt/ml/input/data/train
      algo-1-4cuqr_1 | SM_NUM_GPUS=0
      algo-1-4cuqr_1 | SM_USER_ENTRY_POINT=train_model.py
      algo-1-4cuqr_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1-4cuqr","framework_module":"sagemaker_sklearn_container.training:main","hosts":["algo-1-4cuqr"],"hyperparameters":{"para_c":1.0,"para_gamma":0.01},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","job_name":"sagemaker-scikit-learn-2019-06-12-23-57-07-460","log_level":10,"model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz","module_name":"train_model","network_interface_name":"ethwe","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]},"user_entry_point":"train_model.py"}
      algo-1-4cuqr_1 | SM_INPUT_DATA_CONFIG={"train":{"TrainingInputMode":"File"}}
      algo-1-4cuqr_1 | SM_NETWORK_INTERFACE_NAME=ethwe
      algo-1-4cuqr_1 | SM_MODEL_DIR=/opt/ml/model
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | Invoking script with the following command:
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | /usr/bin/python3 -m train_model --para_c 1.0 --para_gamma 0.01
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 |
      algo-1-4cuqr_1 | 2019-06-12 23:57:13,528 sagemaker-containers INFO Reporting training SUCCESS
      tmpri2jv76o_algo-1-4cuqr_1 exited with code 0
      Aborting on container exit...
      ===== Job Complete =====
      
      • Created Temp Files
      Temp
      ├─tmpzg964wnz
      │ iris_train.csv
      │
      └─tmpri2jv76o
      │ docker-compose.yaml
      │ ├─artifacts
      ├─model
      │ model.joblib
      │ └─output
      └─data
      
      • docker-compose.yaml
      networks:
      sagemaker-local:
      name: sagemaker-local
      services:
      algo-1-4cuqr:
      command: train
      environment:
      - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
      - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
      - AWS_REGION=ap-northeast-1
      - TRAINING_JOB_NAME=sagemaker-scikit-learn-2019-06-12-23-57-07-460
      - HTTP_PROXY=http://username:password@foobar:port
      - HTTPS_PROXY=http://username:password@foobar:port
      image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
      networks:
      sagemaker-local:
      aliases:
      - algo-1-4cuqr
      stdin_open: true
      tty: true
      volumes:
      - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output:/opt/ml/output
      - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\input:/opt/ml/input
      - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output/data:/opt/ml/output/data
      - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\model:/opt/ml/model
      - C:\Users\username\AppData\Local\Temp\tmpzg964wnz:/opt/ml/input/data/train
      version: '2.3'
      

      Deploying

      • Logs
      Attaching to tmpdbo8bbod_algo-1-qlg3c_1
      algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Starting gunicorn 19.9.0
      algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.sock (16)
      algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Using worker: gevent
      algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [23] [INFO] Booting worker with pid: 23
      algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [27] [INFO] Booting worker with pid: 27
      algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [29] [INFO] Booting worker with pid: 29
      algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [33] [INFO] Booting worker with pid: 33
      algo-1-qlg3c_1 | 2019-06-12 23:57:21,219 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)
      algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Module train_model does not provide a setup.py.
      algo-1-qlg3c_1 | Generating setup.py
      algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating setup.cfg
      algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating MANIFEST.in
      algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Installing module with the following command:
      algo-1-qlg3c_1 | /usr/bin/python3 -m pip install -U .
      algo-1-qlg3c_1 | Processing /opt/ml/code
      algo-1-qlg3c_1 | Building wheels for collected packages: train-model
      algo-1-qlg3c_1 | Running setup.py bdist_wheel for train-model ... done
      algo-1-qlg3c_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-4x4jzvi4/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
      algo-1-qlg3c_1 | Successfully built train-model
      algo-1-qlg3c_1 | Installing collected packages: train-model
      algo-1-qlg3c_1 | Successfully installed train-model-1.0.0
      algo-1-qlg3c_1 | You are using pip version 18.1, however version 19.1.1 is available.
      algo-1-qlg3c_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
      algo-1-qlg3c_1 | [2019-06-12 23:57:22 +0000] [23] [ERROR] Error handling request /ping
      algo-1-qlg3c_1 | Traceback (most recent call last):
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
      algo-1-qlg3c_1 | return fn(*args, **kwargs)
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
      algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
      algo-1-qlg3c_1 | with open(filename, 'rb') as f:
      algo-1-qlg3c_1 | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
      algo-1-qlg3c_1 |
      algo-1-qlg3c_1 | During handling of the above exception, another exception occurred:
      algo-1-qlg3c_1 |
      algo-1-qlg3c_1 | Traceback (most recent call last):
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
      algo-1-qlg3c_1 | self.handle_request(listener_name, req, client, addr)
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
      algo-1-qlg3c_1 | addr)
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
      algo-1-qlg3c_1 | respiter = self.wsgi(environ, resp.start_response)
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_sklearn_container/serving.py", line 103, in main
      algo-1-qlg3c_1 | user_module_transformer.initialize()
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_transformer.py", line 157, in initialize
      algo-1-qlg3c_1 | self._model = self._model_fn(_env.model_dir)
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
      algo-1-qlg3c_1 | six.reraise(error_class, error_class(e), sys.exc_info()[2])
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
      algo-1-qlg3c_1 | raise value.with_traceback(tb)
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
      algo-1-qlg3c_1 | return fn(*args, **kwargs)
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
      algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
      algo-1-qlg3c_1 | with open(filename, 'rb') as f:
      algo-1-qlg3c_1 | sagemaker_containers._errors.ClientError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
      algo-1-qlg3c_1 | 172.18.0.1 - - [12/Jun/2019:23:57:22 +0000] "GET /ping HTTP/1.1" 500 141 "-" "-"
      
      • Created Temp Files
      Temp
      ├─tmpgfpx4a40 # Why is it empty
      │ └─tmpdbo8bbod
      │ docker-compose.yaml
      │ └─algo-1-qlg3c
      
      • docker-compose.yaml
      networks:
      sagemaker-local:
      name: sagemaker-local
      services:
      algo-1-qlg3c:
      command: serve
      environment:
      - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
      - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
      - HTTP_PROXY=http://username:password@foobar:port
      - HTTPS_PROXY=http://username:password@foobar:port
      - SAGEMAKER_PROGRAM=train_model.py
      - SAGEMAKER_SUBMIT_DIRECTORY=s3://sagemaker-ap-northeast-1-XXXXXXXXXX/sagemaker-scikit-learn-2019-06-12-23-57-07-460/sourcedir.tar.gz
      - SAGEMAKER_ENABLE_CLOUDWATCH_METRICS=false
      - SAGEMAKER_CONTAINER_LOG_LEVEL=10
      - SAGEMAKER_REGION=ap-northeast-1
      image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
      networks:
      sagemaker-local:
      aliases:
      - algo-1-qlg3c
      ports:
      - 8080:8080
      stdin_open: true
      tty: true
      volumes:
      - C:\Users\username\AppData\Local\Temp\tmpgfpx4a40:/opt/ml/model
      version: '2.3'
      

      Exact command to reproduce

      sklearn=SKLearn(
      entry_point=script_path,
      train_instance_type='local',
      source_dir='./',
      role=role,
      py_version='py3',
      container_log_level=10,
      hyperparameters={'para_c': 1.0, 'para_gamma': 0.01})
      sklearn.fit({'train': train_input})
      predictor=sklearn.deploy(initial_instance_count=1,
      instance_type='local',
      env=env)

      Cause

      I know that the cause is that the tmpgfpx4a40 folder linked to the /opt/ml/model folder is empty, as described in the docker-compose.yaml file. And I think the tmpgfpx4a40 folder needs a model.joblib file.
      However, I don't know why the tmpgfpx4a40 folder is empty and I don't know how to avoid it.

      Activity

      Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

      Metadata

      Metadata

      Assignees

      No one assigned

        Type

        No type

        Projects

        No projects

          Milestone

          No milestone

          Relationships

          None yet

          Development

          No branches or pull requests

          Issue actions

          , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
          Skip to content

          "No such file or directory: '/opt/ml/model/model.joblib'" occured at deploy in the local mode of windows #846

          Description

          @xnaiman

          System Information

          • Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Scikit-Learn
          • Framework Version: 0.20.0 (official sagemaker-scikit-learn-container)
          • Python Version: 3.6
          • CPU or GPU: CPU
          • Python SDK Version: 1.26.0
          • Are you using a custom image: No

          Describe the problem

          No such file or directory: /opt/ml/model/model.joblib occured at deploy in the local mode of windows.
          (Raised after avoiding issue #844 and #845)

          Minimal repro / logs

          Training

          • Logs
          Windows Support for Local Mode is Experimental
          Creating tmpri2jv76o_algo-1-4cuqr_1 ... done
          Attaching to tmpri2jv76o_algo-1-4cuqr_1
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,801 sagemaker-containers INFO Imported framework sagemaker_sklearn_container.training
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,811 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,813 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,820 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,821 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,825 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,828 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,832 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,833 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,839 sagemaker_sklearn_container.training INFO Invoking user training script.
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,842 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,849 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,851 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,854 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,857 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,863 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/boto3/data/s3/2006-03-01/resources-1.json
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials DEBUG Looking for credentials via: env
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials INFO Found credentials in environment variables.
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/endpoints.json
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,871 botocore.hooks DEBUG Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff9c9cafa60>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,875 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/s3/2006-03-01/service-2.json
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff9c7814400>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08840>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,882 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff9c78141e0>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,883 botocore.args DEBUG The s3 config key is not a dictionary type, ignoring its value of: None
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,885 botocore.endpoint DEBUG Setting s3 timeout as (60, 60)
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,886 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/_retry.json
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,887 botocore.client DEBUG Registering retry handlers for service: s3
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,894 botocore.client DEBUG Defaulting to S3 virtual host style addressing with path style addressing fallback.
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,895 boto3.resources.factory DEBUG Loading s3:s3
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,896 boto3.resources.factory DEBUG Loading s3:Bucket
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,897 boto3.resources.model DEBUG Renaming Bucket attribute name
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,898 botocore.hooks DEBUG Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08950>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.utils DEBUG Acquiring 0
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) about to wait for the following futures []
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) done waiting for dependent futures
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) with kwargs {'config': <boto3.s3.transfer.TransferConfig object at 0x7ff9c6b5c710>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cef0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8>, 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.endpoint DEBUG Making request for OperationModel(name=HeadObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1',
          'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'HEAD', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.hooks DEBUG Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG Calculating signature using v4 auth.
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG CanonicalRequest:
          algo-1-4cuqr_1 | HEAD
          algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
          algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          algo-1-4cuqr_1 | x-amz-date:20190612T235711Z
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
          algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG StringToSign:
          algo-1-4cuqr_1 | AWS4-HMAC-SHA256
          algo-1-4cuqr_1 | 20190612T235711Z
          algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
          algo-1-4cuqr_1 | 3c7d2e923cfbc1df5b252ea8422724d8ecd6c607014af9b27717fcbdf25c94d1
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.auth DEBUG Signature:
          algo-1-4cuqr_1 | 43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235711Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f'}>
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
          algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,024 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "HEAD /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 0
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': 'F48E7D6B6A526E82', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'cAB/AbGlPTnTc025AlpNhnwIFzC34NmUPjtLJAUp7Tf50k4QwAxC33+RE0jTSrnG7qafBtR38Fs=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
          'Wed, 12 Jun 2019 23:57:23 GMT'}
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response body:
          algo-1-4cuqr_1 | b''
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.retryhandler DEBUG No retry needed.
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.futures DEBUG Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8> for transfer request: 0.
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.utils DEBUG Acquiring 0
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) about to wait for the following futures []
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) done waiting for dependent futures
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) with kwargs {'max_attempts': 5, 'bucket': 'sagemaker-ap-northeast-1-130747742019', 'bandwidth_limiter': None, 'io_chunksize': 262144, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'callbacks': [], 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7ff9c6b12cc0>, 'extra_args': {}}
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.utils DEBUG Releasing acquire 0/None
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.endpoint DEBUG Making request for OperationModel(name=GetObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1', 'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'GET', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG Calculating signature using v4 auth.
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG CanonicalRequest:
          algo-1-4cuqr_1 | GET
          algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
          algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          algo-1-4cuqr_1 | x-amz-date:20190612T235712Z
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
          algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG StringToSign:
          algo-1-4cuqr_1 | AWS4-HMAC-SHA256
          algo-1-4cuqr_1 | 20190612T235712Z
          algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
          algo-1-4cuqr_1 | d5310e2e1112cc81c0cbcd2fdb9f2f8fe645ea026d34a43d2a5cd9ce8cf1b5d3
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG Signature:
          algo-1-4cuqr_1 | 77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235712Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4'}>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "GET /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 7271
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': '7250ADF3F5A5A49C', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'WiYMANCqDMYoqe7ZcCVyUJTpmRTWAQ9GRSIeW6JAlxtmugzNooQgP8bb/pnTQ+PBo5seJvLVhoc=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
          'Wed, 12 Jun 2019 23:57:23 GMT'}
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.parsers DEBUG Response body:
          algo-1-4cuqr_1 | <botocore.response.StreamingBody object at 0x7ff9c6aba048>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.retryhandler DEBUG No retry needed.
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'offset': 0, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>}
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) about to wait for the following futures []
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) done waiting for dependent futures
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'final_filename': '/tmp/tmp_f99p1cx/tar_file'}
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.utils DEBUG Releasing acquire 0/None
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Module train_model does not provide a setup.py.
          algo-1-4cuqr_1 | Generating setup.py
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating setup.cfg
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating MANIFEST.in
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Installing module with the following command:
          algo-1-4cuqr_1 | /usr/bin/python3 -m pip install -U .
          algo-1-4cuqr_1 | Processing /opt/ml/code
          algo-1-4cuqr_1 | Building wheels for collected packages: train-model
          algo-1-4cuqr_1 | Running setup.py bdist_wheel for train-model ... done
          algo-1-4cuqr_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-d58g4m1m/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
          algo-1-4cuqr_1 | Successfully built train-model
          algo-1-4cuqr_1 | Installing collected packages: train-model
          algo-1-4cuqr_1 | Successfully installed train-model-1.0.0
          algo-1-4cuqr_1 | You are using pip version 18.1, however version 19.1.1 is available.
          algo-1-4cuqr_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,983 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,984 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,987 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,988 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,991 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,996 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
          algo-1-4cuqr_1 | 2019-06-12 23:57:12,998 sagemaker-containers INFO Invoking user script
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | Training Env:
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | {
          algo-1-4cuqr_1 | "hosts": [
          algo-1-4cuqr_1 | "algo-1-4cuqr"
          algo-1-4cuqr_1 | ],
          algo-1-4cuqr_1 | "output_dir": "/opt/ml/output",
          algo-1-4cuqr_1 | "user_entry_point": "train_model.py",
          algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr",
          algo-1-4cuqr_1 | "input_data_config": {
          algo-1-4cuqr_1 | "train": {
          algo-1-4cuqr_1 | "TrainingInputMode": "File"
          algo-1-4cuqr_1 | }
          algo-1-4cuqr_1 | },
          algo-1-4cuqr_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
          algo-1-4cuqr_1 | "input_dir": "/opt/ml/input",
          algo-1-4cuqr_1 | "output_data_dir": "/opt/ml/output/data",
          algo-1-4cuqr_1 | "num_gpus": 0,
          algo-1-4cuqr_1 | "log_level": 10,
          algo-1-4cuqr_1 | "resource_config": {
          algo-1-4cuqr_1 | "hosts": [
          algo-1-4cuqr_1 | "algo-1-4cuqr"
          algo-1-4cuqr_1 | ],
          algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr"
          algo-1-4cuqr_1 | },
          algo-1-4cuqr_1 | "job_name": "sagemaker-scikit-learn-2019-06-12-23-57-07-460",
          algo-1-4cuqr_1 | "additional_framework_parameters": {},
          algo-1-4cuqr_1 | "input_config_dir": "/opt/ml/input/config",
          algo-1-4cuqr_1 | "network_interface_name": "ethwe",
          algo-1-4cuqr_1 | "module_dir": "s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz",
          algo-1-4cuqr_1 | "framework_module": "sagemaker_sklearn_container.training:main",
          algo-1-4cuqr_1 | "channel_input_dirs": {
          algo-1-4cuqr_1 | "train": "/opt/ml/input/data/train"
          algo-1-4cuqr_1 | },
          algo-1-4cuqr_1 | "module_name": "train_model",
          algo-1-4cuqr_1 | "model_dir": "/opt/ml/model",
          algo-1-4cuqr_1 | "hyperparameters": {
          algo-1-4cuqr_1 | "para_c": 1.0,
          algo-1-4cuqr_1 | "para_gamma": 0.01
          algo-1-4cuqr_1 | },
          algo-1-4cuqr_1 | "num_cpus": 4
          algo-1-4cuqr_1 | }
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | Environment variables:
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | PYTHONPATH=/usr/local/bin:/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages
          algo-1-4cuqr_1 | SM_MODULE_NAME=train_model
          algo-1-4cuqr_1 | SM_LOG_LEVEL=10
          algo-1-4cuqr_1 | SM_CHANNELS=["train"]
          algo-1-4cuqr_1 | SM_HP_PARA_C=1.0
          algo-1-4cuqr_1 | SM_CURRENT_HOST=algo-1-4cuqr
          algo-1-4cuqr_1 | SM_FRAMEWORK_PARAMS={}
          algo-1-4cuqr_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
          algo-1-4cuqr_1 | SM_OUTPUT_DIR=/opt/ml/output
          algo-1-4cuqr_1 | SM_HP_PARA_GAMMA=0.01
          algo-1-4cuqr_1 | SM_HPS={"para_c":1.0,"para_gamma":0.01}
          algo-1-4cuqr_1 | SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main
          algo-1-4cuqr_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
          algo-1-4cuqr_1 | SM_USER_ARGS=["--para_c","1.0","--para_gamma","0.01"]
          algo-1-4cuqr_1 | SM_NUM_CPUS=4
          algo-1-4cuqr_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
          algo-1-4cuqr_1 | SM_MODULE_DIR=s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
          algo-1-4cuqr_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]}
          algo-1-4cuqr_1 | SM_HOSTS=["algo-1-4cuqr"]
          algo-1-4cuqr_1 | SM_INPUT_DIR=/opt/ml/input
          algo-1-4cuqr_1 | SM_CHANNEL_TRAIN=/opt/ml/input/data/train
          algo-1-4cuqr_1 | SM_NUM_GPUS=0
          algo-1-4cuqr_1 | SM_USER_ENTRY_POINT=train_model.py
          algo-1-4cuqr_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1-4cuqr","framework_module":"sagemaker_sklearn_container.training:main","hosts":["algo-1-4cuqr"],"hyperparameters":{"para_c":1.0,"para_gamma":0.01},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","job_name":"sagemaker-scikit-learn-2019-06-12-23-57-07-460","log_level":10,"model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz","module_name":"train_model","network_interface_name":"ethwe","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]},"user_entry_point":"train_model.py"}
          algo-1-4cuqr_1 | SM_INPUT_DATA_CONFIG={"train":{"TrainingInputMode":"File"}}
          algo-1-4cuqr_1 | SM_NETWORK_INTERFACE_NAME=ethwe
          algo-1-4cuqr_1 | SM_MODEL_DIR=/opt/ml/model
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | Invoking script with the following command:
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | /usr/bin/python3 -m train_model --para_c 1.0 --para_gamma 0.01
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 |
          algo-1-4cuqr_1 | 2019-06-12 23:57:13,528 sagemaker-containers INFO Reporting training SUCCESS
          tmpri2jv76o_algo-1-4cuqr_1 exited with code 0
          Aborting on container exit...
          ===== Job Complete =====
          
          • Created Temp Files
          Temp
          ├─tmpzg964wnz
          │ iris_train.csv
          │
          └─tmpri2jv76o
          │ docker-compose.yaml
          │ ├─artifacts
          ├─model
          │ model.joblib
          │ └─output
          └─data
          
          • docker-compose.yaml
          networks:
          sagemaker-local:
          name: sagemaker-local
          services:
          algo-1-4cuqr:
          command: train
          environment:
          - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
          - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
          - AWS_REGION=ap-northeast-1
          - TRAINING_JOB_NAME=sagemaker-scikit-learn-2019-06-12-23-57-07-460
          - HTTP_PROXY=http://username:password@foobar:port
          - HTTPS_PROXY=http://username:password@foobar:port
          image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
          networks:
          sagemaker-local:
          aliases:
          - algo-1-4cuqr
          stdin_open: true
          tty: true
          volumes:
          - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output:/opt/ml/output
          - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\input:/opt/ml/input
          - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output/data:/opt/ml/output/data
          - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\model:/opt/ml/model
          - C:\Users\username\AppData\Local\Temp\tmpzg964wnz:/opt/ml/input/data/train
          version: '2.3'
          

          Deploying

          • Logs
          Attaching to tmpdbo8bbod_algo-1-qlg3c_1
          algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Starting gunicorn 19.9.0
          algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.sock (16)
          algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Using worker: gevent
          algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [23] [INFO] Booting worker with pid: 23
          algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [27] [INFO] Booting worker with pid: 27
          algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [29] [INFO] Booting worker with pid: 29
          algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [33] [INFO] Booting worker with pid: 33
          algo-1-qlg3c_1 | 2019-06-12 23:57:21,219 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)
          algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Module train_model does not provide a setup.py.
          algo-1-qlg3c_1 | Generating setup.py
          algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating setup.cfg
          algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating MANIFEST.in
          algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Installing module with the following command:
          algo-1-qlg3c_1 | /usr/bin/python3 -m pip install -U .
          algo-1-qlg3c_1 | Processing /opt/ml/code
          algo-1-qlg3c_1 | Building wheels for collected packages: train-model
          algo-1-qlg3c_1 | Running setup.py bdist_wheel for train-model ... done
          algo-1-qlg3c_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-4x4jzvi4/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
          algo-1-qlg3c_1 | Successfully built train-model
          algo-1-qlg3c_1 | Installing collected packages: train-model
          algo-1-qlg3c_1 | Successfully installed train-model-1.0.0
          algo-1-qlg3c_1 | You are using pip version 18.1, however version 19.1.1 is available.
          algo-1-qlg3c_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
          algo-1-qlg3c_1 | [2019-06-12 23:57:22 +0000] [23] [ERROR] Error handling request /ping
          algo-1-qlg3c_1 | Traceback (most recent call last):
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
          algo-1-qlg3c_1 | return fn(*args, **kwargs)
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
          algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
          algo-1-qlg3c_1 | with open(filename, 'rb') as f:
          algo-1-qlg3c_1 | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
          algo-1-qlg3c_1 |
          algo-1-qlg3c_1 | During handling of the above exception, another exception occurred:
          algo-1-qlg3c_1 |
          algo-1-qlg3c_1 | Traceback (most recent call last):
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
          algo-1-qlg3c_1 | self.handle_request(listener_name, req, client, addr)
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
          algo-1-qlg3c_1 | addr)
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
          algo-1-qlg3c_1 | respiter = self.wsgi(environ, resp.start_response)
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_sklearn_container/serving.py", line 103, in main
          algo-1-qlg3c_1 | user_module_transformer.initialize()
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_transformer.py", line 157, in initialize
          algo-1-qlg3c_1 | self._model = self._model_fn(_env.model_dir)
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
          algo-1-qlg3c_1 | six.reraise(error_class, error_class(e), sys.exc_info()[2])
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
          algo-1-qlg3c_1 | raise value.with_traceback(tb)
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
          algo-1-qlg3c_1 | return fn(*args, **kwargs)
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
          algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
          algo-1-qlg3c_1 | with open(filename, 'rb') as f:
          algo-1-qlg3c_1 | sagemaker_containers._errors.ClientError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
          algo-1-qlg3c_1 | 172.18.0.1 - - [12/Jun/2019:23:57:22 +0000] "GET /ping HTTP/1.1" 500 141 "-" "-"
          
          • Created Temp Files
          Temp
          ├─tmpgfpx4a40 # Why is it empty
          │ └─tmpdbo8bbod
          │ docker-compose.yaml
          │ └─algo-1-qlg3c
          
          • docker-compose.yaml
          networks:
          sagemaker-local:
          name: sagemaker-local
          services:
          algo-1-qlg3c:
          command: serve
          environment:
          - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
          - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
          - HTTP_PROXY=http://username:password@foobar:port
          - HTTPS_PROXY=http://username:password@foobar:port
          - SAGEMAKER_PROGRAM=train_model.py
          - SAGEMAKER_SUBMIT_DIRECTORY=s3://sagemaker-ap-northeast-1-XXXXXXXXXX/sagemaker-scikit-learn-2019-06-12-23-57-07-460/sourcedir.tar.gz
          - SAGEMAKER_ENABLE_CLOUDWATCH_METRICS=false
          - SAGEMAKER_CONTAINER_LOG_LEVEL=10
          - SAGEMAKER_REGION=ap-northeast-1
          image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
          networks:
          sagemaker-local:
          aliases:
          - algo-1-qlg3c
          ports:
          - 8080:8080
          stdin_open: true
          tty: true
          volumes:
          - C:\Users\username\AppData\Local\Temp\tmpgfpx4a40:/opt/ml/model
          version: '2.3'
          

          Exact command to reproduce

          sklearn=SKLearn(
          entry_point=script_path,
          train_instance_type='local',
          source_dir='./',
          role=role,
          py_version='py3',
          container_log_level=10,
          hyperparameters={'para_c': 1.0, 'para_gamma': 0.01})
          sklearn.fit({'train': train_input})
          predictor=sklearn.deploy(initial_instance_count=1,
          instance_type='local',
          env=env)

          Cause

          I know that the cause is that the tmpgfpx4a40 folder linked to the /opt/ml/model folder is empty, as described in the docker-compose.yaml file. And I think the tmpgfpx4a40 folder needs a model.joblib file.
          However, I don't know why the tmpgfpx4a40 folder is empty and I don't know how to avoid it.

          Activity

          Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

          Metadata

          Metadata

          Assignees

          No one assigned

            Type

            No type

            Projects

            No projects

              Milestone

              No milestone

              Relationships

              None yet

              Development

              No branches or pull requests

              Issue actions

              , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
              Skip to content

              "No such file or directory: '/opt/ml/model/model.joblib'" occured at deploy in the local mode of windows #846

              Description

              @xnaiman

              System Information

              • Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Scikit-Learn
              • Framework Version: 0.20.0 (official sagemaker-scikit-learn-container)
              • Python Version: 3.6
              • CPU or GPU: CPU
              • Python SDK Version: 1.26.0
              • Are you using a custom image: No

              Describe the problem

              No such file or directory: /opt/ml/model/model.joblib occured at deploy in the local mode of windows.
              (Raised after avoiding issue #844 and #845)

              Minimal repro / logs

              Training

              • Logs
              Windows Support for Local Mode is Experimental
              Creating tmpri2jv76o_algo-1-4cuqr_1 ... done
              Attaching to tmpri2jv76o_algo-1-4cuqr_1
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,801 sagemaker-containers INFO Imported framework sagemaker_sklearn_container.training
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,811 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,813 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,820 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,821 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,825 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,828 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,832 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,833 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,839 sagemaker_sklearn_container.training INFO Invoking user training script.
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,842 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,849 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,851 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,854 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,857 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,863 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/boto3/data/s3/2006-03-01/resources-1.json
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials DEBUG Looking for credentials via: env
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials INFO Found credentials in environment variables.
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/endpoints.json
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,871 botocore.hooks DEBUG Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff9c9cafa60>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,875 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/s3/2006-03-01/service-2.json
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff9c7814400>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08840>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,882 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff9c78141e0>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,883 botocore.args DEBUG The s3 config key is not a dictionary type, ignoring its value of: None
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,885 botocore.endpoint DEBUG Setting s3 timeout as (60, 60)
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,886 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/_retry.json
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,887 botocore.client DEBUG Registering retry handlers for service: s3
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,894 botocore.client DEBUG Defaulting to S3 virtual host style addressing with path style addressing fallback.
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,895 boto3.resources.factory DEBUG Loading s3:s3
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,896 boto3.resources.factory DEBUG Loading s3:Bucket
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,897 boto3.resources.model DEBUG Renaming Bucket attribute name
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,898 botocore.hooks DEBUG Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08950>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.utils DEBUG Acquiring 0
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) about to wait for the following futures []
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) done waiting for dependent futures
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) with kwargs {'config': <boto3.s3.transfer.TransferConfig object at 0x7ff9c6b5c710>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cef0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8>, 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.endpoint DEBUG Making request for OperationModel(name=HeadObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1',
              'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'HEAD', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.hooks DEBUG Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG Calculating signature using v4 auth.
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG CanonicalRequest:
              algo-1-4cuqr_1 | HEAD
              algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
              algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
              algo-1-4cuqr_1 | x-amz-date:20190612T235711Z
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
              algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG StringToSign:
              algo-1-4cuqr_1 | AWS4-HMAC-SHA256
              algo-1-4cuqr_1 | 20190612T235711Z
              algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
              algo-1-4cuqr_1 | 3c7d2e923cfbc1df5b252ea8422724d8ecd6c607014af9b27717fcbdf25c94d1
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.auth DEBUG Signature:
              algo-1-4cuqr_1 | 43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235711Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f'}>
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
              algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,024 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "HEAD /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 0
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': 'F48E7D6B6A526E82', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'cAB/AbGlPTnTc025AlpNhnwIFzC34NmUPjtLJAUp7Tf50k4QwAxC33+RE0jTSrnG7qafBtR38Fs=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
              'Wed, 12 Jun 2019 23:57:23 GMT'}
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response body:
              algo-1-4cuqr_1 | b''
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.retryhandler DEBUG No retry needed.
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.futures DEBUG Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8> for transfer request: 0.
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.utils DEBUG Acquiring 0
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) about to wait for the following futures []
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) done waiting for dependent futures
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) with kwargs {'max_attempts': 5, 'bucket': 'sagemaker-ap-northeast-1-130747742019', 'bandwidth_limiter': None, 'io_chunksize': 262144, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'callbacks': [], 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7ff9c6b12cc0>, 'extra_args': {}}
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.utils DEBUG Releasing acquire 0/None
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.endpoint DEBUG Making request for OperationModel(name=GetObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1', 'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'GET', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG Calculating signature using v4 auth.
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG CanonicalRequest:
              algo-1-4cuqr_1 | GET
              algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
              algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
              algo-1-4cuqr_1 | x-amz-date:20190612T235712Z
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
              algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG StringToSign:
              algo-1-4cuqr_1 | AWS4-HMAC-SHA256
              algo-1-4cuqr_1 | 20190612T235712Z
              algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
              algo-1-4cuqr_1 | d5310e2e1112cc81c0cbcd2fdb9f2f8fe645ea026d34a43d2a5cd9ce8cf1b5d3
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG Signature:
              algo-1-4cuqr_1 | 77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235712Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4'}>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "GET /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 7271
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': '7250ADF3F5A5A49C', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'WiYMANCqDMYoqe7ZcCVyUJTpmRTWAQ9GRSIeW6JAlxtmugzNooQgP8bb/pnTQ+PBo5seJvLVhoc=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
              'Wed, 12 Jun 2019 23:57:23 GMT'}
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.parsers DEBUG Response body:
              algo-1-4cuqr_1 | <botocore.response.StreamingBody object at 0x7ff9c6aba048>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.retryhandler DEBUG No retry needed.
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'offset': 0, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>}
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) about to wait for the following futures []
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) done waiting for dependent futures
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'final_filename': '/tmp/tmp_f99p1cx/tar_file'}
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.utils DEBUG Releasing acquire 0/None
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Module train_model does not provide a setup.py.
              algo-1-4cuqr_1 | Generating setup.py
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating setup.cfg
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating MANIFEST.in
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Installing module with the following command:
              algo-1-4cuqr_1 | /usr/bin/python3 -m pip install -U .
              algo-1-4cuqr_1 | Processing /opt/ml/code
              algo-1-4cuqr_1 | Building wheels for collected packages: train-model
              algo-1-4cuqr_1 | Running setup.py bdist_wheel for train-model ... done
              algo-1-4cuqr_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-d58g4m1m/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
              algo-1-4cuqr_1 | Successfully built train-model
              algo-1-4cuqr_1 | Installing collected packages: train-model
              algo-1-4cuqr_1 | Successfully installed train-model-1.0.0
              algo-1-4cuqr_1 | You are using pip version 18.1, however version 19.1.1 is available.
              algo-1-4cuqr_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,983 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,984 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,987 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,988 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,991 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,996 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
              algo-1-4cuqr_1 | 2019-06-12 23:57:12,998 sagemaker-containers INFO Invoking user script
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | Training Env:
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | {
              algo-1-4cuqr_1 | "hosts": [
              algo-1-4cuqr_1 | "algo-1-4cuqr"
              algo-1-4cuqr_1 | ],
              algo-1-4cuqr_1 | "output_dir": "/opt/ml/output",
              algo-1-4cuqr_1 | "user_entry_point": "train_model.py",
              algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr",
              algo-1-4cuqr_1 | "input_data_config": {
              algo-1-4cuqr_1 | "train": {
              algo-1-4cuqr_1 | "TrainingInputMode": "File"
              algo-1-4cuqr_1 | }
              algo-1-4cuqr_1 | },
              algo-1-4cuqr_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
              algo-1-4cuqr_1 | "input_dir": "/opt/ml/input",
              algo-1-4cuqr_1 | "output_data_dir": "/opt/ml/output/data",
              algo-1-4cuqr_1 | "num_gpus": 0,
              algo-1-4cuqr_1 | "log_level": 10,
              algo-1-4cuqr_1 | "resource_config": {
              algo-1-4cuqr_1 | "hosts": [
              algo-1-4cuqr_1 | "algo-1-4cuqr"
              algo-1-4cuqr_1 | ],
              algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr"
              algo-1-4cuqr_1 | },
              algo-1-4cuqr_1 | "job_name": "sagemaker-scikit-learn-2019-06-12-23-57-07-460",
              algo-1-4cuqr_1 | "additional_framework_parameters": {},
              algo-1-4cuqr_1 | "input_config_dir": "/opt/ml/input/config",
              algo-1-4cuqr_1 | "network_interface_name": "ethwe",
              algo-1-4cuqr_1 | "module_dir": "s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz",
              algo-1-4cuqr_1 | "framework_module": "sagemaker_sklearn_container.training:main",
              algo-1-4cuqr_1 | "channel_input_dirs": {
              algo-1-4cuqr_1 | "train": "/opt/ml/input/data/train"
              algo-1-4cuqr_1 | },
              algo-1-4cuqr_1 | "module_name": "train_model",
              algo-1-4cuqr_1 | "model_dir": "/opt/ml/model",
              algo-1-4cuqr_1 | "hyperparameters": {
              algo-1-4cuqr_1 | "para_c": 1.0,
              algo-1-4cuqr_1 | "para_gamma": 0.01
              algo-1-4cuqr_1 | },
              algo-1-4cuqr_1 | "num_cpus": 4
              algo-1-4cuqr_1 | }
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | Environment variables:
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | PYTHONPATH=/usr/local/bin:/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages
              algo-1-4cuqr_1 | SM_MODULE_NAME=train_model
              algo-1-4cuqr_1 | SM_LOG_LEVEL=10
              algo-1-4cuqr_1 | SM_CHANNELS=["train"]
              algo-1-4cuqr_1 | SM_HP_PARA_C=1.0
              algo-1-4cuqr_1 | SM_CURRENT_HOST=algo-1-4cuqr
              algo-1-4cuqr_1 | SM_FRAMEWORK_PARAMS={}
              algo-1-4cuqr_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
              algo-1-4cuqr_1 | SM_OUTPUT_DIR=/opt/ml/output
              algo-1-4cuqr_1 | SM_HP_PARA_GAMMA=0.01
              algo-1-4cuqr_1 | SM_HPS={"para_c":1.0,"para_gamma":0.01}
              algo-1-4cuqr_1 | SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main
              algo-1-4cuqr_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
              algo-1-4cuqr_1 | SM_USER_ARGS=["--para_c","1.0","--para_gamma","0.01"]
              algo-1-4cuqr_1 | SM_NUM_CPUS=4
              algo-1-4cuqr_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
              algo-1-4cuqr_1 | SM_MODULE_DIR=s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
              algo-1-4cuqr_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]}
              algo-1-4cuqr_1 | SM_HOSTS=["algo-1-4cuqr"]
              algo-1-4cuqr_1 | SM_INPUT_DIR=/opt/ml/input
              algo-1-4cuqr_1 | SM_CHANNEL_TRAIN=/opt/ml/input/data/train
              algo-1-4cuqr_1 | SM_NUM_GPUS=0
              algo-1-4cuqr_1 | SM_USER_ENTRY_POINT=train_model.py
              algo-1-4cuqr_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1-4cuqr","framework_module":"sagemaker_sklearn_container.training:main","hosts":["algo-1-4cuqr"],"hyperparameters":{"para_c":1.0,"para_gamma":0.01},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","job_name":"sagemaker-scikit-learn-2019-06-12-23-57-07-460","log_level":10,"model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz","module_name":"train_model","network_interface_name":"ethwe","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]},"user_entry_point":"train_model.py"}
              algo-1-4cuqr_1 | SM_INPUT_DATA_CONFIG={"train":{"TrainingInputMode":"File"}}
              algo-1-4cuqr_1 | SM_NETWORK_INTERFACE_NAME=ethwe
              algo-1-4cuqr_1 | SM_MODEL_DIR=/opt/ml/model
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | Invoking script with the following command:
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | /usr/bin/python3 -m train_model --para_c 1.0 --para_gamma 0.01
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 |
              algo-1-4cuqr_1 | 2019-06-12 23:57:13,528 sagemaker-containers INFO Reporting training SUCCESS
              tmpri2jv76o_algo-1-4cuqr_1 exited with code 0
              Aborting on container exit...
              ===== Job Complete =====
              
              • Created Temp Files
              Temp
              ├─tmpzg964wnz
              │ iris_train.csv
              │
              └─tmpri2jv76o
              │ docker-compose.yaml
              │ ├─artifacts
              ├─model
              │ model.joblib
              │ └─output
              └─data
              
              • docker-compose.yaml
              networks:
              sagemaker-local:
              name: sagemaker-local
              services:
              algo-1-4cuqr:
              command: train
              environment:
              - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
              - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
              - AWS_REGION=ap-northeast-1
              - TRAINING_JOB_NAME=sagemaker-scikit-learn-2019-06-12-23-57-07-460
              - HTTP_PROXY=http://username:password@foobar:port
              - HTTPS_PROXY=http://username:password@foobar:port
              image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
              networks:
              sagemaker-local:
              aliases:
              - algo-1-4cuqr
              stdin_open: true
              tty: true
              volumes:
              - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output:/opt/ml/output
              - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\input:/opt/ml/input
              - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output/data:/opt/ml/output/data
              - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\model:/opt/ml/model
              - C:\Users\username\AppData\Local\Temp\tmpzg964wnz:/opt/ml/input/data/train
              version: '2.3'
              

              Deploying

              • Logs
              Attaching to tmpdbo8bbod_algo-1-qlg3c_1
              algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Starting gunicorn 19.9.0
              algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.sock (16)
              algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Using worker: gevent
              algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [23] [INFO] Booting worker with pid: 23
              algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [27] [INFO] Booting worker with pid: 27
              algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [29] [INFO] Booting worker with pid: 29
              algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [33] [INFO] Booting worker with pid: 33
              algo-1-qlg3c_1 | 2019-06-12 23:57:21,219 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)
              algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Module train_model does not provide a setup.py.
              algo-1-qlg3c_1 | Generating setup.py
              algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating setup.cfg
              algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating MANIFEST.in
              algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Installing module with the following command:
              algo-1-qlg3c_1 | /usr/bin/python3 -m pip install -U .
              algo-1-qlg3c_1 | Processing /opt/ml/code
              algo-1-qlg3c_1 | Building wheels for collected packages: train-model
              algo-1-qlg3c_1 | Running setup.py bdist_wheel for train-model ... done
              algo-1-qlg3c_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-4x4jzvi4/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
              algo-1-qlg3c_1 | Successfully built train-model
              algo-1-qlg3c_1 | Installing collected packages: train-model
              algo-1-qlg3c_1 | Successfully installed train-model-1.0.0
              algo-1-qlg3c_1 | You are using pip version 18.1, however version 19.1.1 is available.
              algo-1-qlg3c_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
              algo-1-qlg3c_1 | [2019-06-12 23:57:22 +0000] [23] [ERROR] Error handling request /ping
              algo-1-qlg3c_1 | Traceback (most recent call last):
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
              algo-1-qlg3c_1 | return fn(*args, **kwargs)
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
              algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
              algo-1-qlg3c_1 | with open(filename, 'rb') as f:
              algo-1-qlg3c_1 | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
              algo-1-qlg3c_1 |
              algo-1-qlg3c_1 | During handling of the above exception, another exception occurred:
              algo-1-qlg3c_1 |
              algo-1-qlg3c_1 | Traceback (most recent call last):
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
              algo-1-qlg3c_1 | self.handle_request(listener_name, req, client, addr)
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
              algo-1-qlg3c_1 | addr)
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
              algo-1-qlg3c_1 | respiter = self.wsgi(environ, resp.start_response)
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_sklearn_container/serving.py", line 103, in main
              algo-1-qlg3c_1 | user_module_transformer.initialize()
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_transformer.py", line 157, in initialize
              algo-1-qlg3c_1 | self._model = self._model_fn(_env.model_dir)
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
              algo-1-qlg3c_1 | six.reraise(error_class, error_class(e), sys.exc_info()[2])
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
              algo-1-qlg3c_1 | raise value.with_traceback(tb)
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
              algo-1-qlg3c_1 | return fn(*args, **kwargs)
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
              algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
              algo-1-qlg3c_1 | with open(filename, 'rb') as f:
              algo-1-qlg3c_1 | sagemaker_containers._errors.ClientError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
              algo-1-qlg3c_1 | 172.18.0.1 - - [12/Jun/2019:23:57:22 +0000] "GET /ping HTTP/1.1" 500 141 "-" "-"
              
              • Created Temp Files
              Temp
              ├─tmpgfpx4a40 # Why is it empty
              │ └─tmpdbo8bbod
              │ docker-compose.yaml
              │ └─algo-1-qlg3c
              
              • docker-compose.yaml
              networks:
              sagemaker-local:
              name: sagemaker-local
              services:
              algo-1-qlg3c:
              command: serve
              environment:
              - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
              - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
              - HTTP_PROXY=http://username:password@foobar:port
              - HTTPS_PROXY=http://username:password@foobar:port
              - SAGEMAKER_PROGRAM=train_model.py
              - SAGEMAKER_SUBMIT_DIRECTORY=s3://sagemaker-ap-northeast-1-XXXXXXXXXX/sagemaker-scikit-learn-2019-06-12-23-57-07-460/sourcedir.tar.gz
              - SAGEMAKER_ENABLE_CLOUDWATCH_METRICS=false
              - SAGEMAKER_CONTAINER_LOG_LEVEL=10
              - SAGEMAKER_REGION=ap-northeast-1
              image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
              networks:
              sagemaker-local:
              aliases:
              - algo-1-qlg3c
              ports:
              - 8080:8080
              stdin_open: true
              tty: true
              volumes:
              - C:\Users\username\AppData\Local\Temp\tmpgfpx4a40:/opt/ml/model
              version: '2.3'
              

              Exact command to reproduce

              sklearn=SKLearn(
              entry_point=script_path,
              train_instance_type='local',
              source_dir='./',
              role=role,
              py_version='py3',
              container_log_level=10,
              hyperparameters={'para_c': 1.0, 'para_gamma': 0.01})
              sklearn.fit({'train': train_input})
              predictor=sklearn.deploy(initial_instance_count=1,
              instance_type='local',
              env=env)

              Cause

              I know that the cause is that the tmpgfpx4a40 folder linked to the /opt/ml/model folder is empty, as described in the docker-compose.yaml file. And I think the tmpgfpx4a40 folder needs a model.joblib file.
              However, I don't know why the tmpgfpx4a40 folder is empty and I don't know how to avoid it.

              Activity

              Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

              Metadata

              Metadata

              Assignees

              No one assigned

                Type

                No type

                Projects

                No projects

                  Milestone

                  No milestone

                  Relationships

                  None yet

                  Development

                  No branches or pull requests

                  Issue actions

                  , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
                  Skip to content

                  "No such file or directory: '/opt/ml/model/model.joblib'" occured at deploy in the local mode of windows #846

                  Description

                  @xnaiman

                  System Information

                  • Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Scikit-Learn
                  • Framework Version: 0.20.0 (official sagemaker-scikit-learn-container)
                  • Python Version: 3.6
                  • CPU or GPU: CPU
                  • Python SDK Version: 1.26.0
                  • Are you using a custom image: No

                  Describe the problem

                  No such file or directory: /opt/ml/model/model.joblib occured at deploy in the local mode of windows.
                  (Raised after avoiding issue #844 and #845)

                  Minimal repro / logs

                  Training

                  • Logs
                  Windows Support for Local Mode is Experimental
                  Creating tmpri2jv76o_algo-1-4cuqr_1 ... done
                  Attaching to tmpri2jv76o_algo-1-4cuqr_1
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,801 sagemaker-containers INFO Imported framework sagemaker_sklearn_container.training
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,811 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,813 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,820 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,821 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,825 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,828 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,832 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,833 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,839 sagemaker_sklearn_container.training INFO Invoking user training script.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,842 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,849 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,851 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,854 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,857 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,863 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/boto3/data/s3/2006-03-01/resources-1.json
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials DEBUG Looking for credentials via: env
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials INFO Found credentials in environment variables.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/endpoints.json
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,871 botocore.hooks DEBUG Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff9c9cafa60>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,875 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/s3/2006-03-01/service-2.json
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff9c7814400>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08840>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,882 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff9c78141e0>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,883 botocore.args DEBUG The s3 config key is not a dictionary type, ignoring its value of: None
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,885 botocore.endpoint DEBUG Setting s3 timeout as (60, 60)
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,886 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/_retry.json
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,887 botocore.client DEBUG Registering retry handlers for service: s3
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,894 botocore.client DEBUG Defaulting to S3 virtual host style addressing with path style addressing fallback.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,895 boto3.resources.factory DEBUG Loading s3:s3
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,896 boto3.resources.factory DEBUG Loading s3:Bucket
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,897 boto3.resources.model DEBUG Renaming Bucket attribute name
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,898 botocore.hooks DEBUG Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08950>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.utils DEBUG Acquiring 0
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) about to wait for the following futures []
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) done waiting for dependent futures
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) with kwargs {'config': <boto3.s3.transfer.TransferConfig object at 0x7ff9c6b5c710>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cef0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8>, 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.endpoint DEBUG Making request for OperationModel(name=HeadObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1',
                  'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'HEAD', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.hooks DEBUG Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG Calculating signature using v4 auth.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG CanonicalRequest:
                  algo-1-4cuqr_1 | HEAD
                  algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
                  algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                  algo-1-4cuqr_1 | x-amz-date:20190612T235711Z
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
                  algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG StringToSign:
                  algo-1-4cuqr_1 | AWS4-HMAC-SHA256
                  algo-1-4cuqr_1 | 20190612T235711Z
                  algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
                  algo-1-4cuqr_1 | 3c7d2e923cfbc1df5b252ea8422724d8ecd6c607014af9b27717fcbdf25c94d1
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.auth DEBUG Signature:
                  algo-1-4cuqr_1 | 43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235711Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f'}>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
                  algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,024 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "HEAD /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 0
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': 'F48E7D6B6A526E82', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'cAB/AbGlPTnTc025AlpNhnwIFzC34NmUPjtLJAUp7Tf50k4QwAxC33+RE0jTSrnG7qafBtR38Fs=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
                  'Wed, 12 Jun 2019 23:57:23 GMT'}
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response body:
                  algo-1-4cuqr_1 | b''
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.retryhandler DEBUG No retry needed.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.futures DEBUG Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8> for transfer request: 0.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.utils DEBUG Acquiring 0
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) about to wait for the following futures []
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) done waiting for dependent futures
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) with kwargs {'max_attempts': 5, 'bucket': 'sagemaker-ap-northeast-1-130747742019', 'bandwidth_limiter': None, 'io_chunksize': 262144, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'callbacks': [], 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7ff9c6b12cc0>, 'extra_args': {}}
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.utils DEBUG Releasing acquire 0/None
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.endpoint DEBUG Making request for OperationModel(name=GetObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1', 'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'GET', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG Calculating signature using v4 auth.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG CanonicalRequest:
                  algo-1-4cuqr_1 | GET
                  algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
                  algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                  algo-1-4cuqr_1 | x-amz-date:20190612T235712Z
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
                  algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG StringToSign:
                  algo-1-4cuqr_1 | AWS4-HMAC-SHA256
                  algo-1-4cuqr_1 | 20190612T235712Z
                  algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
                  algo-1-4cuqr_1 | d5310e2e1112cc81c0cbcd2fdb9f2f8fe645ea026d34a43d2a5cd9ce8cf1b5d3
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG Signature:
                  algo-1-4cuqr_1 | 77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235712Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4'}>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "GET /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 7271
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': '7250ADF3F5A5A49C', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'WiYMANCqDMYoqe7ZcCVyUJTpmRTWAQ9GRSIeW6JAlxtmugzNooQgP8bb/pnTQ+PBo5seJvLVhoc=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
                  'Wed, 12 Jun 2019 23:57:23 GMT'}
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.parsers DEBUG Response body:
                  algo-1-4cuqr_1 | <botocore.response.StreamingBody object at 0x7ff9c6aba048>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.retryhandler DEBUG No retry needed.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'offset': 0, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>}
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) about to wait for the following futures []
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) done waiting for dependent futures
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'final_filename': '/tmp/tmp_f99p1cx/tar_file'}
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.utils DEBUG Releasing acquire 0/None
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Module train_model does not provide a setup.py.
                  algo-1-4cuqr_1 | Generating setup.py
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating setup.cfg
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating MANIFEST.in
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Installing module with the following command:
                  algo-1-4cuqr_1 | /usr/bin/python3 -m pip install -U .
                  algo-1-4cuqr_1 | Processing /opt/ml/code
                  algo-1-4cuqr_1 | Building wheels for collected packages: train-model
                  algo-1-4cuqr_1 | Running setup.py bdist_wheel for train-model ... done
                  algo-1-4cuqr_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-d58g4m1m/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
                  algo-1-4cuqr_1 | Successfully built train-model
                  algo-1-4cuqr_1 | Installing collected packages: train-model
                  algo-1-4cuqr_1 | Successfully installed train-model-1.0.0
                  algo-1-4cuqr_1 | You are using pip version 18.1, however version 19.1.1 is available.
                  algo-1-4cuqr_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,983 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,984 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,987 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,988 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,991 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,996 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                  algo-1-4cuqr_1 | 2019-06-12 23:57:12,998 sagemaker-containers INFO Invoking user script
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | Training Env:
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | {
                  algo-1-4cuqr_1 | "hosts": [
                  algo-1-4cuqr_1 | "algo-1-4cuqr"
                  algo-1-4cuqr_1 | ],
                  algo-1-4cuqr_1 | "output_dir": "/opt/ml/output",
                  algo-1-4cuqr_1 | "user_entry_point": "train_model.py",
                  algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr",
                  algo-1-4cuqr_1 | "input_data_config": {
                  algo-1-4cuqr_1 | "train": {
                  algo-1-4cuqr_1 | "TrainingInputMode": "File"
                  algo-1-4cuqr_1 | }
                  algo-1-4cuqr_1 | },
                  algo-1-4cuqr_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
                  algo-1-4cuqr_1 | "input_dir": "/opt/ml/input",
                  algo-1-4cuqr_1 | "output_data_dir": "/opt/ml/output/data",
                  algo-1-4cuqr_1 | "num_gpus": 0,
                  algo-1-4cuqr_1 | "log_level": 10,
                  algo-1-4cuqr_1 | "resource_config": {
                  algo-1-4cuqr_1 | "hosts": [
                  algo-1-4cuqr_1 | "algo-1-4cuqr"
                  algo-1-4cuqr_1 | ],
                  algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr"
                  algo-1-4cuqr_1 | },
                  algo-1-4cuqr_1 | "job_name": "sagemaker-scikit-learn-2019-06-12-23-57-07-460",
                  algo-1-4cuqr_1 | "additional_framework_parameters": {},
                  algo-1-4cuqr_1 | "input_config_dir": "/opt/ml/input/config",
                  algo-1-4cuqr_1 | "network_interface_name": "ethwe",
                  algo-1-4cuqr_1 | "module_dir": "s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz",
                  algo-1-4cuqr_1 | "framework_module": "sagemaker_sklearn_container.training:main",
                  algo-1-4cuqr_1 | "channel_input_dirs": {
                  algo-1-4cuqr_1 | "train": "/opt/ml/input/data/train"
                  algo-1-4cuqr_1 | },
                  algo-1-4cuqr_1 | "module_name": "train_model",
                  algo-1-4cuqr_1 | "model_dir": "/opt/ml/model",
                  algo-1-4cuqr_1 | "hyperparameters": {
                  algo-1-4cuqr_1 | "para_c": 1.0,
                  algo-1-4cuqr_1 | "para_gamma": 0.01
                  algo-1-4cuqr_1 | },
                  algo-1-4cuqr_1 | "num_cpus": 4
                  algo-1-4cuqr_1 | }
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | Environment variables:
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | PYTHONPATH=/usr/local/bin:/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages
                  algo-1-4cuqr_1 | SM_MODULE_NAME=train_model
                  algo-1-4cuqr_1 | SM_LOG_LEVEL=10
                  algo-1-4cuqr_1 | SM_CHANNELS=["train"]
                  algo-1-4cuqr_1 | SM_HP_PARA_C=1.0
                  algo-1-4cuqr_1 | SM_CURRENT_HOST=algo-1-4cuqr
                  algo-1-4cuqr_1 | SM_FRAMEWORK_PARAMS={}
                  algo-1-4cuqr_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
                  algo-1-4cuqr_1 | SM_OUTPUT_DIR=/opt/ml/output
                  algo-1-4cuqr_1 | SM_HP_PARA_GAMMA=0.01
                  algo-1-4cuqr_1 | SM_HPS={"para_c":1.0,"para_gamma":0.01}
                  algo-1-4cuqr_1 | SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main
                  algo-1-4cuqr_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
                  algo-1-4cuqr_1 | SM_USER_ARGS=["--para_c","1.0","--para_gamma","0.01"]
                  algo-1-4cuqr_1 | SM_NUM_CPUS=4
                  algo-1-4cuqr_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
                  algo-1-4cuqr_1 | SM_MODULE_DIR=s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                  algo-1-4cuqr_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]}
                  algo-1-4cuqr_1 | SM_HOSTS=["algo-1-4cuqr"]
                  algo-1-4cuqr_1 | SM_INPUT_DIR=/opt/ml/input
                  algo-1-4cuqr_1 | SM_CHANNEL_TRAIN=/opt/ml/input/data/train
                  algo-1-4cuqr_1 | SM_NUM_GPUS=0
                  algo-1-4cuqr_1 | SM_USER_ENTRY_POINT=train_model.py
                  algo-1-4cuqr_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1-4cuqr","framework_module":"sagemaker_sklearn_container.training:main","hosts":["algo-1-4cuqr"],"hyperparameters":{"para_c":1.0,"para_gamma":0.01},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","job_name":"sagemaker-scikit-learn-2019-06-12-23-57-07-460","log_level":10,"model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz","module_name":"train_model","network_interface_name":"ethwe","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]},"user_entry_point":"train_model.py"}
                  algo-1-4cuqr_1 | SM_INPUT_DATA_CONFIG={"train":{"TrainingInputMode":"File"}}
                  algo-1-4cuqr_1 | SM_NETWORK_INTERFACE_NAME=ethwe
                  algo-1-4cuqr_1 | SM_MODEL_DIR=/opt/ml/model
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | Invoking script with the following command:
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | /usr/bin/python3 -m train_model --para_c 1.0 --para_gamma 0.01
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 |
                  algo-1-4cuqr_1 | 2019-06-12 23:57:13,528 sagemaker-containers INFO Reporting training SUCCESS
                  tmpri2jv76o_algo-1-4cuqr_1 exited with code 0
                  Aborting on container exit...
                  ===== Job Complete =====
                  
                  • Created Temp Files
                  Temp
                  ├─tmpzg964wnz
                  │ iris_train.csv
                  │
                  └─tmpri2jv76o
                  │ docker-compose.yaml
                  │ ├─artifacts
                  ├─model
                  │ model.joblib
                  │ └─output
                  └─data
                  
                  • docker-compose.yaml
                  networks:
                  sagemaker-local:
                  name: sagemaker-local
                  services:
                  algo-1-4cuqr:
                  command: train
                  environment:
                  - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
                  - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
                  - AWS_REGION=ap-northeast-1
                  - TRAINING_JOB_NAME=sagemaker-scikit-learn-2019-06-12-23-57-07-460
                  - HTTP_PROXY=http://username:password@foobar:port
                  - HTTPS_PROXY=http://username:password@foobar:port
                  image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
                  networks:
                  sagemaker-local:
                  aliases:
                  - algo-1-4cuqr
                  stdin_open: true
                  tty: true
                  volumes:
                  - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output:/opt/ml/output
                  - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\input:/opt/ml/input
                  - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output/data:/opt/ml/output/data
                  - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\model:/opt/ml/model
                  - C:\Users\username\AppData\Local\Temp\tmpzg964wnz:/opt/ml/input/data/train
                  version: '2.3'
                  

                  Deploying

                  • Logs
                  Attaching to tmpdbo8bbod_algo-1-qlg3c_1
                  algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Starting gunicorn 19.9.0
                  algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.sock (16)
                  algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Using worker: gevent
                  algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [23] [INFO] Booting worker with pid: 23
                  algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [27] [INFO] Booting worker with pid: 27
                  algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [29] [INFO] Booting worker with pid: 29
                  algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [33] [INFO] Booting worker with pid: 33
                  algo-1-qlg3c_1 | 2019-06-12 23:57:21,219 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)
                  algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Module train_model does not provide a setup.py.
                  algo-1-qlg3c_1 | Generating setup.py
                  algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating setup.cfg
                  algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating MANIFEST.in
                  algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Installing module with the following command:
                  algo-1-qlg3c_1 | /usr/bin/python3 -m pip install -U .
                  algo-1-qlg3c_1 | Processing /opt/ml/code
                  algo-1-qlg3c_1 | Building wheels for collected packages: train-model
                  algo-1-qlg3c_1 | Running setup.py bdist_wheel for train-model ... done
                  algo-1-qlg3c_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-4x4jzvi4/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
                  algo-1-qlg3c_1 | Successfully built train-model
                  algo-1-qlg3c_1 | Installing collected packages: train-model
                  algo-1-qlg3c_1 | Successfully installed train-model-1.0.0
                  algo-1-qlg3c_1 | You are using pip version 18.1, however version 19.1.1 is available.
                  algo-1-qlg3c_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
                  algo-1-qlg3c_1 | [2019-06-12 23:57:22 +0000] [23] [ERROR] Error handling request /ping
                  algo-1-qlg3c_1 | Traceback (most recent call last):
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
                  algo-1-qlg3c_1 | return fn(*args, **kwargs)
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
                  algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
                  algo-1-qlg3c_1 | with open(filename, 'rb') as f:
                  algo-1-qlg3c_1 | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
                  algo-1-qlg3c_1 |
                  algo-1-qlg3c_1 | During handling of the above exception, another exception occurred:
                  algo-1-qlg3c_1 |
                  algo-1-qlg3c_1 | Traceback (most recent call last):
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
                  algo-1-qlg3c_1 | self.handle_request(listener_name, req, client, addr)
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
                  algo-1-qlg3c_1 | addr)
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
                  algo-1-qlg3c_1 | respiter = self.wsgi(environ, resp.start_response)
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_sklearn_container/serving.py", line 103, in main
                  algo-1-qlg3c_1 | user_module_transformer.initialize()
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_transformer.py", line 157, in initialize
                  algo-1-qlg3c_1 | self._model = self._model_fn(_env.model_dir)
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
                  algo-1-qlg3c_1 | six.reraise(error_class, error_class(e), sys.exc_info()[2])
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
                  algo-1-qlg3c_1 | raise value.with_traceback(tb)
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
                  algo-1-qlg3c_1 | return fn(*args, **kwargs)
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
                  algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
                  algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
                  algo-1-qlg3c_1 | with open(filename, 'rb') as f:
                  algo-1-qlg3c_1 | sagemaker_containers._errors.ClientError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
                  algo-1-qlg3c_1 | 172.18.0.1 - - [12/Jun/2019:23:57:22 +0000] "GET /ping HTTP/1.1" 500 141 "-" "-"
                  
                  • Created Temp Files
                  Temp
                  ├─tmpgfpx4a40 # Why is it empty
                  │ └─tmpdbo8bbod
                  │ docker-compose.yaml
                  │ └─algo-1-qlg3c
                  
                  • docker-compose.yaml
                  networks:
                  sagemaker-local:
                  name: sagemaker-local
                  services:
                  algo-1-qlg3c:
                  command: serve
                  environment:
                  - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
                  - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
                  - HTTP_PROXY=http://username:password@foobar:port
                  - HTTPS_PROXY=http://username:password@foobar:port
                  - SAGEMAKER_PROGRAM=train_model.py
                  - SAGEMAKER_SUBMIT_DIRECTORY=s3://sagemaker-ap-northeast-1-XXXXXXXXXX/sagemaker-scikit-learn-2019-06-12-23-57-07-460/sourcedir.tar.gz
                  - SAGEMAKER_ENABLE_CLOUDWATCH_METRICS=false
                  - SAGEMAKER_CONTAINER_LOG_LEVEL=10
                  - SAGEMAKER_REGION=ap-northeast-1
                  image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
                  networks:
                  sagemaker-local:
                  aliases:
                  - algo-1-qlg3c
                  ports:
                  - 8080:8080
                  stdin_open: true
                  tty: true
                  volumes:
                  - C:\Users\username\AppData\Local\Temp\tmpgfpx4a40:/opt/ml/model
                  version: '2.3'
                  

                  Exact command to reproduce

                  sklearn=SKLearn(
                  entry_point=script_path,
                  train_instance_type='local',
                  source_dir='./',
                  role=role,
                  py_version='py3',
                  container_log_level=10,
                  hyperparameters={'para_c': 1.0, 'para_gamma': 0.01})
                  sklearn.fit({'train': train_input})
                  predictor=sklearn.deploy(initial_instance_count=1,
                  instance_type='local',
                  env=env)

                  Cause

                  I know that the cause is that the tmpgfpx4a40 folder linked to the /opt/ml/model folder is empty, as described in the docker-compose.yaml file. And I think the tmpgfpx4a40 folder needs a model.joblib file.
                  However, I don't know why the tmpgfpx4a40 folder is empty and I don't know how to avoid it.

                  Activity

                  Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                  Metadata

                  Metadata

                  Assignees

                  No one assigned

                    Type

                    No type

                    Projects

                    No projects

                      Milestone

                      No milestone

                      Relationships

                      None yet

                      Development

                      No branches or pull requests

                      Issue actions

                      , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
                      Skip to content

                      "No such file or directory: '/opt/ml/model/model.joblib'" occured at deploy in the local mode of windows #846

                      Description

                      @xnaiman

                      System Information

                      • Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Scikit-Learn
                      • Framework Version: 0.20.0 (official sagemaker-scikit-learn-container)
                      • Python Version: 3.6
                      • CPU or GPU: CPU
                      • Python SDK Version: 1.26.0
                      • Are you using a custom image: No

                      Describe the problem

                      No such file or directory: /opt/ml/model/model.joblib occured at deploy in the local mode of windows.
                      (Raised after avoiding issue #844 and #845)

                      Minimal repro / logs

                      Training

                      • Logs
                      Windows Support for Local Mode is Experimental
                      Creating tmpri2jv76o_algo-1-4cuqr_1 ... done
                      Attaching to tmpri2jv76o_algo-1-4cuqr_1
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,801 sagemaker-containers INFO Imported framework sagemaker_sklearn_container.training
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,811 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,813 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,820 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,821 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,825 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,828 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,832 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,833 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,839 sagemaker_sklearn_container.training INFO Invoking user training script.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,842 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,849 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,851 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,854 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,857 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,863 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/boto3/data/s3/2006-03-01/resources-1.json
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials DEBUG Looking for credentials via: env
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials INFO Found credentials in environment variables.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/endpoints.json
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,871 botocore.hooks DEBUG Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff9c9cafa60>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,875 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/s3/2006-03-01/service-2.json
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff9c7814400>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08840>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,882 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff9c78141e0>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,883 botocore.args DEBUG The s3 config key is not a dictionary type, ignoring its value of: None
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,885 botocore.endpoint DEBUG Setting s3 timeout as (60, 60)
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,886 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/_retry.json
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,887 botocore.client DEBUG Registering retry handlers for service: s3
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,894 botocore.client DEBUG Defaulting to S3 virtual host style addressing with path style addressing fallback.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,895 boto3.resources.factory DEBUG Loading s3:s3
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,896 boto3.resources.factory DEBUG Loading s3:Bucket
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,897 boto3.resources.model DEBUG Renaming Bucket attribute name
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,898 botocore.hooks DEBUG Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08950>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.utils DEBUG Acquiring 0
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) about to wait for the following futures []
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) done waiting for dependent futures
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) with kwargs {'config': <boto3.s3.transfer.TransferConfig object at 0x7ff9c6b5c710>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cef0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8>, 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.endpoint DEBUG Making request for OperationModel(name=HeadObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1',
                      'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'HEAD', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.hooks DEBUG Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG Calculating signature using v4 auth.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG CanonicalRequest:
                      algo-1-4cuqr_1 | HEAD
                      algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
                      algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                      algo-1-4cuqr_1 | x-amz-date:20190612T235711Z
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
                      algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG StringToSign:
                      algo-1-4cuqr_1 | AWS4-HMAC-SHA256
                      algo-1-4cuqr_1 | 20190612T235711Z
                      algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
                      algo-1-4cuqr_1 | 3c7d2e923cfbc1df5b252ea8422724d8ecd6c607014af9b27717fcbdf25c94d1
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.auth DEBUG Signature:
                      algo-1-4cuqr_1 | 43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235711Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f'}>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
                      algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,024 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "HEAD /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 0
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': 'F48E7D6B6A526E82', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'cAB/AbGlPTnTc025AlpNhnwIFzC34NmUPjtLJAUp7Tf50k4QwAxC33+RE0jTSrnG7qafBtR38Fs=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
                      'Wed, 12 Jun 2019 23:57:23 GMT'}
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response body:
                      algo-1-4cuqr_1 | b''
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.retryhandler DEBUG No retry needed.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.futures DEBUG Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8> for transfer request: 0.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.utils DEBUG Acquiring 0
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) about to wait for the following futures []
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) done waiting for dependent futures
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) with kwargs {'max_attempts': 5, 'bucket': 'sagemaker-ap-northeast-1-130747742019', 'bandwidth_limiter': None, 'io_chunksize': 262144, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'callbacks': [], 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7ff9c6b12cc0>, 'extra_args': {}}
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.utils DEBUG Releasing acquire 0/None
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.endpoint DEBUG Making request for OperationModel(name=GetObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1', 'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'GET', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG Calculating signature using v4 auth.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG CanonicalRequest:
                      algo-1-4cuqr_1 | GET
                      algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
                      algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                      algo-1-4cuqr_1 | x-amz-date:20190612T235712Z
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
                      algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG StringToSign:
                      algo-1-4cuqr_1 | AWS4-HMAC-SHA256
                      algo-1-4cuqr_1 | 20190612T235712Z
                      algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
                      algo-1-4cuqr_1 | d5310e2e1112cc81c0cbcd2fdb9f2f8fe645ea026d34a43d2a5cd9ce8cf1b5d3
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG Signature:
                      algo-1-4cuqr_1 | 77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235712Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4'}>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "GET /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 7271
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': '7250ADF3F5A5A49C', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'WiYMANCqDMYoqe7ZcCVyUJTpmRTWAQ9GRSIeW6JAlxtmugzNooQgP8bb/pnTQ+PBo5seJvLVhoc=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
                      'Wed, 12 Jun 2019 23:57:23 GMT'}
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.parsers DEBUG Response body:
                      algo-1-4cuqr_1 | <botocore.response.StreamingBody object at 0x7ff9c6aba048>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.retryhandler DEBUG No retry needed.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'offset': 0, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>}
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) about to wait for the following futures []
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) done waiting for dependent futures
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'final_filename': '/tmp/tmp_f99p1cx/tar_file'}
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.utils DEBUG Releasing acquire 0/None
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Module train_model does not provide a setup.py.
                      algo-1-4cuqr_1 | Generating setup.py
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating setup.cfg
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating MANIFEST.in
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Installing module with the following command:
                      algo-1-4cuqr_1 | /usr/bin/python3 -m pip install -U .
                      algo-1-4cuqr_1 | Processing /opt/ml/code
                      algo-1-4cuqr_1 | Building wheels for collected packages: train-model
                      algo-1-4cuqr_1 | Running setup.py bdist_wheel for train-model ... done
                      algo-1-4cuqr_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-d58g4m1m/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
                      algo-1-4cuqr_1 | Successfully built train-model
                      algo-1-4cuqr_1 | Installing collected packages: train-model
                      algo-1-4cuqr_1 | Successfully installed train-model-1.0.0
                      algo-1-4cuqr_1 | You are using pip version 18.1, however version 19.1.1 is available.
                      algo-1-4cuqr_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,983 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,984 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,987 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,988 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,991 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,996 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                      algo-1-4cuqr_1 | 2019-06-12 23:57:12,998 sagemaker-containers INFO Invoking user script
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | Training Env:
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | {
                      algo-1-4cuqr_1 | "hosts": [
                      algo-1-4cuqr_1 | "algo-1-4cuqr"
                      algo-1-4cuqr_1 | ],
                      algo-1-4cuqr_1 | "output_dir": "/opt/ml/output",
                      algo-1-4cuqr_1 | "user_entry_point": "train_model.py",
                      algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr",
                      algo-1-4cuqr_1 | "input_data_config": {
                      algo-1-4cuqr_1 | "train": {
                      algo-1-4cuqr_1 | "TrainingInputMode": "File"
                      algo-1-4cuqr_1 | }
                      algo-1-4cuqr_1 | },
                      algo-1-4cuqr_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
                      algo-1-4cuqr_1 | "input_dir": "/opt/ml/input",
                      algo-1-4cuqr_1 | "output_data_dir": "/opt/ml/output/data",
                      algo-1-4cuqr_1 | "num_gpus": 0,
                      algo-1-4cuqr_1 | "log_level": 10,
                      algo-1-4cuqr_1 | "resource_config": {
                      algo-1-4cuqr_1 | "hosts": [
                      algo-1-4cuqr_1 | "algo-1-4cuqr"
                      algo-1-4cuqr_1 | ],
                      algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr"
                      algo-1-4cuqr_1 | },
                      algo-1-4cuqr_1 | "job_name": "sagemaker-scikit-learn-2019-06-12-23-57-07-460",
                      algo-1-4cuqr_1 | "additional_framework_parameters": {},
                      algo-1-4cuqr_1 | "input_config_dir": "/opt/ml/input/config",
                      algo-1-4cuqr_1 | "network_interface_name": "ethwe",
                      algo-1-4cuqr_1 | "module_dir": "s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz",
                      algo-1-4cuqr_1 | "framework_module": "sagemaker_sklearn_container.training:main",
                      algo-1-4cuqr_1 | "channel_input_dirs": {
                      algo-1-4cuqr_1 | "train": "/opt/ml/input/data/train"
                      algo-1-4cuqr_1 | },
                      algo-1-4cuqr_1 | "module_name": "train_model",
                      algo-1-4cuqr_1 | "model_dir": "/opt/ml/model",
                      algo-1-4cuqr_1 | "hyperparameters": {
                      algo-1-4cuqr_1 | "para_c": 1.0,
                      algo-1-4cuqr_1 | "para_gamma": 0.01
                      algo-1-4cuqr_1 | },
                      algo-1-4cuqr_1 | "num_cpus": 4
                      algo-1-4cuqr_1 | }
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | Environment variables:
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | PYTHONPATH=/usr/local/bin:/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages
                      algo-1-4cuqr_1 | SM_MODULE_NAME=train_model
                      algo-1-4cuqr_1 | SM_LOG_LEVEL=10
                      algo-1-4cuqr_1 | SM_CHANNELS=["train"]
                      algo-1-4cuqr_1 | SM_HP_PARA_C=1.0
                      algo-1-4cuqr_1 | SM_CURRENT_HOST=algo-1-4cuqr
                      algo-1-4cuqr_1 | SM_FRAMEWORK_PARAMS={}
                      algo-1-4cuqr_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
                      algo-1-4cuqr_1 | SM_OUTPUT_DIR=/opt/ml/output
                      algo-1-4cuqr_1 | SM_HP_PARA_GAMMA=0.01
                      algo-1-4cuqr_1 | SM_HPS={"para_c":1.0,"para_gamma":0.01}
                      algo-1-4cuqr_1 | SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main
                      algo-1-4cuqr_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
                      algo-1-4cuqr_1 | SM_USER_ARGS=["--para_c","1.0","--para_gamma","0.01"]
                      algo-1-4cuqr_1 | SM_NUM_CPUS=4
                      algo-1-4cuqr_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
                      algo-1-4cuqr_1 | SM_MODULE_DIR=s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                      algo-1-4cuqr_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]}
                      algo-1-4cuqr_1 | SM_HOSTS=["algo-1-4cuqr"]
                      algo-1-4cuqr_1 | SM_INPUT_DIR=/opt/ml/input
                      algo-1-4cuqr_1 | SM_CHANNEL_TRAIN=/opt/ml/input/data/train
                      algo-1-4cuqr_1 | SM_NUM_GPUS=0
                      algo-1-4cuqr_1 | SM_USER_ENTRY_POINT=train_model.py
                      algo-1-4cuqr_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1-4cuqr","framework_module":"sagemaker_sklearn_container.training:main","hosts":["algo-1-4cuqr"],"hyperparameters":{"para_c":1.0,"para_gamma":0.01},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","job_name":"sagemaker-scikit-learn-2019-06-12-23-57-07-460","log_level":10,"model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz","module_name":"train_model","network_interface_name":"ethwe","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]},"user_entry_point":"train_model.py"}
                      algo-1-4cuqr_1 | SM_INPUT_DATA_CONFIG={"train":{"TrainingInputMode":"File"}}
                      algo-1-4cuqr_1 | SM_NETWORK_INTERFACE_NAME=ethwe
                      algo-1-4cuqr_1 | SM_MODEL_DIR=/opt/ml/model
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | Invoking script with the following command:
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | /usr/bin/python3 -m train_model --para_c 1.0 --para_gamma 0.01
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 |
                      algo-1-4cuqr_1 | 2019-06-12 23:57:13,528 sagemaker-containers INFO Reporting training SUCCESS
                      tmpri2jv76o_algo-1-4cuqr_1 exited with code 0
                      Aborting on container exit...
                      ===== Job Complete =====
                      
                      • Created Temp Files
                      Temp
                      ├─tmpzg964wnz
                      │ iris_train.csv
                      │
                      └─tmpri2jv76o
                      │ docker-compose.yaml
                      │ ├─artifacts
                      ├─model
                      │ model.joblib
                      │ └─output
                      └─data
                      
                      • docker-compose.yaml
                      networks:
                      sagemaker-local:
                      name: sagemaker-local
                      services:
                      algo-1-4cuqr:
                      command: train
                      environment:
                      - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
                      - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
                      - AWS_REGION=ap-northeast-1
                      - TRAINING_JOB_NAME=sagemaker-scikit-learn-2019-06-12-23-57-07-460
                      - HTTP_PROXY=http://username:password@foobar:port
                      - HTTPS_PROXY=http://username:password@foobar:port
                      image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
                      networks:
                      sagemaker-local:
                      aliases:
                      - algo-1-4cuqr
                      stdin_open: true
                      tty: true
                      volumes:
                      - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output:/opt/ml/output
                      - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\input:/opt/ml/input
                      - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output/data:/opt/ml/output/data
                      - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\model:/opt/ml/model
                      - C:\Users\username\AppData\Local\Temp\tmpzg964wnz:/opt/ml/input/data/train
                      version: '2.3'
                      

                      Deploying

                      • Logs
                      Attaching to tmpdbo8bbod_algo-1-qlg3c_1
                      algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Starting gunicorn 19.9.0
                      algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.sock (16)
                      algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Using worker: gevent
                      algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [23] [INFO] Booting worker with pid: 23
                      algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [27] [INFO] Booting worker with pid: 27
                      algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [29] [INFO] Booting worker with pid: 29
                      algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [33] [INFO] Booting worker with pid: 33
                      algo-1-qlg3c_1 | 2019-06-12 23:57:21,219 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)
                      algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Module train_model does not provide a setup.py.
                      algo-1-qlg3c_1 | Generating setup.py
                      algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating setup.cfg
                      algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating MANIFEST.in
                      algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Installing module with the following command:
                      algo-1-qlg3c_1 | /usr/bin/python3 -m pip install -U .
                      algo-1-qlg3c_1 | Processing /opt/ml/code
                      algo-1-qlg3c_1 | Building wheels for collected packages: train-model
                      algo-1-qlg3c_1 | Running setup.py bdist_wheel for train-model ... done
                      algo-1-qlg3c_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-4x4jzvi4/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
                      algo-1-qlg3c_1 | Successfully built train-model
                      algo-1-qlg3c_1 | Installing collected packages: train-model
                      algo-1-qlg3c_1 | Successfully installed train-model-1.0.0
                      algo-1-qlg3c_1 | You are using pip version 18.1, however version 19.1.1 is available.
                      algo-1-qlg3c_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
                      algo-1-qlg3c_1 | [2019-06-12 23:57:22 +0000] [23] [ERROR] Error handling request /ping
                      algo-1-qlg3c_1 | Traceback (most recent call last):
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
                      algo-1-qlg3c_1 | return fn(*args, **kwargs)
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
                      algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
                      algo-1-qlg3c_1 | with open(filename, 'rb') as f:
                      algo-1-qlg3c_1 | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
                      algo-1-qlg3c_1 |
                      algo-1-qlg3c_1 | During handling of the above exception, another exception occurred:
                      algo-1-qlg3c_1 |
                      algo-1-qlg3c_1 | Traceback (most recent call last):
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
                      algo-1-qlg3c_1 | self.handle_request(listener_name, req, client, addr)
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
                      algo-1-qlg3c_1 | addr)
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
                      algo-1-qlg3c_1 | respiter = self.wsgi(environ, resp.start_response)
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_sklearn_container/serving.py", line 103, in main
                      algo-1-qlg3c_1 | user_module_transformer.initialize()
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_transformer.py", line 157, in initialize
                      algo-1-qlg3c_1 | self._model = self._model_fn(_env.model_dir)
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
                      algo-1-qlg3c_1 | six.reraise(error_class, error_class(e), sys.exc_info()[2])
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
                      algo-1-qlg3c_1 | raise value.with_traceback(tb)
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
                      algo-1-qlg3c_1 | return fn(*args, **kwargs)
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
                      algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
                      algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
                      algo-1-qlg3c_1 | with open(filename, 'rb') as f:
                      algo-1-qlg3c_1 | sagemaker_containers._errors.ClientError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
                      algo-1-qlg3c_1 | 172.18.0.1 - - [12/Jun/2019:23:57:22 +0000] "GET /ping HTTP/1.1" 500 141 "-" "-"
                      
                      • Created Temp Files
                      Temp
                      ├─tmpgfpx4a40 # Why is it empty
                      │ └─tmpdbo8bbod
                      │ docker-compose.yaml
                      │ └─algo-1-qlg3c
                      
                      • docker-compose.yaml
                      networks:
                      sagemaker-local:
                      name: sagemaker-local
                      services:
                      algo-1-qlg3c:
                      command: serve
                      environment:
                      - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
                      - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
                      - HTTP_PROXY=http://username:password@foobar:port
                      - HTTPS_PROXY=http://username:password@foobar:port
                      - SAGEMAKER_PROGRAM=train_model.py
                      - SAGEMAKER_SUBMIT_DIRECTORY=s3://sagemaker-ap-northeast-1-XXXXXXXXXX/sagemaker-scikit-learn-2019-06-12-23-57-07-460/sourcedir.tar.gz
                      - SAGEMAKER_ENABLE_CLOUDWATCH_METRICS=false
                      - SAGEMAKER_CONTAINER_LOG_LEVEL=10
                      - SAGEMAKER_REGION=ap-northeast-1
                      image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
                      networks:
                      sagemaker-local:
                      aliases:
                      - algo-1-qlg3c
                      ports:
                      - 8080:8080
                      stdin_open: true
                      tty: true
                      volumes:
                      - C:\Users\username\AppData\Local\Temp\tmpgfpx4a40:/opt/ml/model
                      version: '2.3'
                      

                      Exact command to reproduce

                      sklearn=SKLearn(
                      entry_point=script_path,
                      train_instance_type='local',
                      source_dir='./',
                      role=role,
                      py_version='py3',
                      container_log_level=10,
                      hyperparameters={'para_c': 1.0, 'para_gamma': 0.01})
                      sklearn.fit({'train': train_input})
                      predictor=sklearn.deploy(initial_instance_count=1,
                      instance_type='local',
                      env=env)

                      Cause

                      I know that the cause is that the tmpgfpx4a40 folder linked to the /opt/ml/model folder is empty, as described in the docker-compose.yaml file. And I think the tmpgfpx4a40 folder needs a model.joblib file.
                      However, I don't know why the tmpgfpx4a40 folder is empty and I don't know how to avoid it.

                      Activity

                      Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                      Metadata

                      Metadata

                      Assignees

                      No one assigned

                        Type

                        No type

                        Projects

                        No projects

                          Milestone

                          No milestone

                          Relationships

                          None yet

                          Development

                          No branches or pull requests

                          Issue actions

                          , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
                          Skip to content

                          "No such file or directory: '/opt/ml/model/model.joblib'" occured at deploy in the local mode of windows #846

                          Description

                          @xnaiman

                          System Information

                          • Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Scikit-Learn
                          • Framework Version: 0.20.0 (official sagemaker-scikit-learn-container)
                          • Python Version: 3.6
                          • CPU or GPU: CPU
                          • Python SDK Version: 1.26.0
                          • Are you using a custom image: No

                          Describe the problem

                          No such file or directory: /opt/ml/model/model.joblib occured at deploy in the local mode of windows.
                          (Raised after avoiding issue #844 and #845)

                          Minimal repro / logs

                          Training

                          • Logs
                          Windows Support for Local Mode is Experimental
                          Creating tmpri2jv76o_algo-1-4cuqr_1 ... done
                          Attaching to tmpri2jv76o_algo-1-4cuqr_1
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,801 sagemaker-containers INFO Imported framework sagemaker_sklearn_container.training
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,811 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,813 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,820 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,821 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,825 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,828 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,832 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,833 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,839 sagemaker_sklearn_container.training INFO Invoking user training script.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,842 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,849 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,851 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,854 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,857 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,863 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/boto3/data/s3/2006-03-01/resources-1.json
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials DEBUG Looking for credentials via: env
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials INFO Found credentials in environment variables.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/endpoints.json
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,871 botocore.hooks DEBUG Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff9c9cafa60>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,875 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/s3/2006-03-01/service-2.json
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff9c7814400>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08840>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,882 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff9c78141e0>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,883 botocore.args DEBUG The s3 config key is not a dictionary type, ignoring its value of: None
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,885 botocore.endpoint DEBUG Setting s3 timeout as (60, 60)
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,886 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/_retry.json
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,887 botocore.client DEBUG Registering retry handlers for service: s3
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,894 botocore.client DEBUG Defaulting to S3 virtual host style addressing with path style addressing fallback.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,895 boto3.resources.factory DEBUG Loading s3:s3
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,896 boto3.resources.factory DEBUG Loading s3:Bucket
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,897 boto3.resources.model DEBUG Renaming Bucket attribute name
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,898 botocore.hooks DEBUG Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08950>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.utils DEBUG Acquiring 0
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) about to wait for the following futures []
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) done waiting for dependent futures
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) with kwargs {'config': <boto3.s3.transfer.TransferConfig object at 0x7ff9c6b5c710>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cef0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8>, 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.endpoint DEBUG Making request for OperationModel(name=HeadObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1',
                          'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'HEAD', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.hooks DEBUG Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG Calculating signature using v4 auth.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG CanonicalRequest:
                          algo-1-4cuqr_1 | HEAD
                          algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
                          algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                          algo-1-4cuqr_1 | x-amz-date:20190612T235711Z
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
                          algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG StringToSign:
                          algo-1-4cuqr_1 | AWS4-HMAC-SHA256
                          algo-1-4cuqr_1 | 20190612T235711Z
                          algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
                          algo-1-4cuqr_1 | 3c7d2e923cfbc1df5b252ea8422724d8ecd6c607014af9b27717fcbdf25c94d1
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.auth DEBUG Signature:
                          algo-1-4cuqr_1 | 43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235711Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f'}>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
                          algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,024 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "HEAD /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 0
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': 'F48E7D6B6A526E82', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'cAB/AbGlPTnTc025AlpNhnwIFzC34NmUPjtLJAUp7Tf50k4QwAxC33+RE0jTSrnG7qafBtR38Fs=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
                          'Wed, 12 Jun 2019 23:57:23 GMT'}
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response body:
                          algo-1-4cuqr_1 | b''
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.retryhandler DEBUG No retry needed.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.futures DEBUG Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8> for transfer request: 0.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.utils DEBUG Acquiring 0
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) about to wait for the following futures []
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) done waiting for dependent futures
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) with kwargs {'max_attempts': 5, 'bucket': 'sagemaker-ap-northeast-1-130747742019', 'bandwidth_limiter': None, 'io_chunksize': 262144, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'callbacks': [], 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7ff9c6b12cc0>, 'extra_args': {}}
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.utils DEBUG Releasing acquire 0/None
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.endpoint DEBUG Making request for OperationModel(name=GetObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1', 'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'GET', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG Calculating signature using v4 auth.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG CanonicalRequest:
                          algo-1-4cuqr_1 | GET
                          algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
                          algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                          algo-1-4cuqr_1 | x-amz-date:20190612T235712Z
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
                          algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG StringToSign:
                          algo-1-4cuqr_1 | AWS4-HMAC-SHA256
                          algo-1-4cuqr_1 | 20190612T235712Z
                          algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
                          algo-1-4cuqr_1 | d5310e2e1112cc81c0cbcd2fdb9f2f8fe645ea026d34a43d2a5cd9ce8cf1b5d3
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG Signature:
                          algo-1-4cuqr_1 | 77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235712Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4'}>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "GET /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 7271
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': '7250ADF3F5A5A49C', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'WiYMANCqDMYoqe7ZcCVyUJTpmRTWAQ9GRSIeW6JAlxtmugzNooQgP8bb/pnTQ+PBo5seJvLVhoc=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
                          'Wed, 12 Jun 2019 23:57:23 GMT'}
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.parsers DEBUG Response body:
                          algo-1-4cuqr_1 | <botocore.response.StreamingBody object at 0x7ff9c6aba048>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.retryhandler DEBUG No retry needed.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'offset': 0, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>}
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) about to wait for the following futures []
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) done waiting for dependent futures
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'final_filename': '/tmp/tmp_f99p1cx/tar_file'}
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.utils DEBUG Releasing acquire 0/None
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Module train_model does not provide a setup.py.
                          algo-1-4cuqr_1 | Generating setup.py
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating setup.cfg
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating MANIFEST.in
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Installing module with the following command:
                          algo-1-4cuqr_1 | /usr/bin/python3 -m pip install -U .
                          algo-1-4cuqr_1 | Processing /opt/ml/code
                          algo-1-4cuqr_1 | Building wheels for collected packages: train-model
                          algo-1-4cuqr_1 | Running setup.py bdist_wheel for train-model ... done
                          algo-1-4cuqr_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-d58g4m1m/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
                          algo-1-4cuqr_1 | Successfully built train-model
                          algo-1-4cuqr_1 | Installing collected packages: train-model
                          algo-1-4cuqr_1 | Successfully installed train-model-1.0.0
                          algo-1-4cuqr_1 | You are using pip version 18.1, however version 19.1.1 is available.
                          algo-1-4cuqr_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,983 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,984 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,987 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,988 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,991 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,996 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                          algo-1-4cuqr_1 | 2019-06-12 23:57:12,998 sagemaker-containers INFO Invoking user script
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | Training Env:
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | {
                          algo-1-4cuqr_1 | "hosts": [
                          algo-1-4cuqr_1 | "algo-1-4cuqr"
                          algo-1-4cuqr_1 | ],
                          algo-1-4cuqr_1 | "output_dir": "/opt/ml/output",
                          algo-1-4cuqr_1 | "user_entry_point": "train_model.py",
                          algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr",
                          algo-1-4cuqr_1 | "input_data_config": {
                          algo-1-4cuqr_1 | "train": {
                          algo-1-4cuqr_1 | "TrainingInputMode": "File"
                          algo-1-4cuqr_1 | }
                          algo-1-4cuqr_1 | },
                          algo-1-4cuqr_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
                          algo-1-4cuqr_1 | "input_dir": "/opt/ml/input",
                          algo-1-4cuqr_1 | "output_data_dir": "/opt/ml/output/data",
                          algo-1-4cuqr_1 | "num_gpus": 0,
                          algo-1-4cuqr_1 | "log_level": 10,
                          algo-1-4cuqr_1 | "resource_config": {
                          algo-1-4cuqr_1 | "hosts": [
                          algo-1-4cuqr_1 | "algo-1-4cuqr"
                          algo-1-4cuqr_1 | ],
                          algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr"
                          algo-1-4cuqr_1 | },
                          algo-1-4cuqr_1 | "job_name": "sagemaker-scikit-learn-2019-06-12-23-57-07-460",
                          algo-1-4cuqr_1 | "additional_framework_parameters": {},
                          algo-1-4cuqr_1 | "input_config_dir": "/opt/ml/input/config",
                          algo-1-4cuqr_1 | "network_interface_name": "ethwe",
                          algo-1-4cuqr_1 | "module_dir": "s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz",
                          algo-1-4cuqr_1 | "framework_module": "sagemaker_sklearn_container.training:main",
                          algo-1-4cuqr_1 | "channel_input_dirs": {
                          algo-1-4cuqr_1 | "train": "/opt/ml/input/data/train"
                          algo-1-4cuqr_1 | },
                          algo-1-4cuqr_1 | "module_name": "train_model",
                          algo-1-4cuqr_1 | "model_dir": "/opt/ml/model",
                          algo-1-4cuqr_1 | "hyperparameters": {
                          algo-1-4cuqr_1 | "para_c": 1.0,
                          algo-1-4cuqr_1 | "para_gamma": 0.01
                          algo-1-4cuqr_1 | },
                          algo-1-4cuqr_1 | "num_cpus": 4
                          algo-1-4cuqr_1 | }
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | Environment variables:
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | PYTHONPATH=/usr/local/bin:/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages
                          algo-1-4cuqr_1 | SM_MODULE_NAME=train_model
                          algo-1-4cuqr_1 | SM_LOG_LEVEL=10
                          algo-1-4cuqr_1 | SM_CHANNELS=["train"]
                          algo-1-4cuqr_1 | SM_HP_PARA_C=1.0
                          algo-1-4cuqr_1 | SM_CURRENT_HOST=algo-1-4cuqr
                          algo-1-4cuqr_1 | SM_FRAMEWORK_PARAMS={}
                          algo-1-4cuqr_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
                          algo-1-4cuqr_1 | SM_OUTPUT_DIR=/opt/ml/output
                          algo-1-4cuqr_1 | SM_HP_PARA_GAMMA=0.01
                          algo-1-4cuqr_1 | SM_HPS={"para_c":1.0,"para_gamma":0.01}
                          algo-1-4cuqr_1 | SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main
                          algo-1-4cuqr_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
                          algo-1-4cuqr_1 | SM_USER_ARGS=["--para_c","1.0","--para_gamma","0.01"]
                          algo-1-4cuqr_1 | SM_NUM_CPUS=4
                          algo-1-4cuqr_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
                          algo-1-4cuqr_1 | SM_MODULE_DIR=s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                          algo-1-4cuqr_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]}
                          algo-1-4cuqr_1 | SM_HOSTS=["algo-1-4cuqr"]
                          algo-1-4cuqr_1 | SM_INPUT_DIR=/opt/ml/input
                          algo-1-4cuqr_1 | SM_CHANNEL_TRAIN=/opt/ml/input/data/train
                          algo-1-4cuqr_1 | SM_NUM_GPUS=0
                          algo-1-4cuqr_1 | SM_USER_ENTRY_POINT=train_model.py
                          algo-1-4cuqr_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1-4cuqr","framework_module":"sagemaker_sklearn_container.training:main","hosts":["algo-1-4cuqr"],"hyperparameters":{"para_c":1.0,"para_gamma":0.01},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","job_name":"sagemaker-scikit-learn-2019-06-12-23-57-07-460","log_level":10,"model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz","module_name":"train_model","network_interface_name":"ethwe","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]},"user_entry_point":"train_model.py"}
                          algo-1-4cuqr_1 | SM_INPUT_DATA_CONFIG={"train":{"TrainingInputMode":"File"}}
                          algo-1-4cuqr_1 | SM_NETWORK_INTERFACE_NAME=ethwe
                          algo-1-4cuqr_1 | SM_MODEL_DIR=/opt/ml/model
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | Invoking script with the following command:
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | /usr/bin/python3 -m train_model --para_c 1.0 --para_gamma 0.01
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 |
                          algo-1-4cuqr_1 | 2019-06-12 23:57:13,528 sagemaker-containers INFO Reporting training SUCCESS
                          tmpri2jv76o_algo-1-4cuqr_1 exited with code 0
                          Aborting on container exit...
                          ===== Job Complete =====
                          
                          • Created Temp Files
                          Temp
                          ├─tmpzg964wnz
                          │ iris_train.csv
                          │
                          └─tmpri2jv76o
                          │ docker-compose.yaml
                          │ ├─artifacts
                          ├─model
                          │ model.joblib
                          │ └─output
                          └─data
                          
                          • docker-compose.yaml
                          networks:
                          sagemaker-local:
                          name: sagemaker-local
                          services:
                          algo-1-4cuqr:
                          command: train
                          environment:
                          - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
                          - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
                          - AWS_REGION=ap-northeast-1
                          - TRAINING_JOB_NAME=sagemaker-scikit-learn-2019-06-12-23-57-07-460
                          - HTTP_PROXY=http://username:password@foobar:port
                          - HTTPS_PROXY=http://username:password@foobar:port
                          image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
                          networks:
                          sagemaker-local:
                          aliases:
                          - algo-1-4cuqr
                          stdin_open: true
                          tty: true
                          volumes:
                          - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output:/opt/ml/output
                          - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\input:/opt/ml/input
                          - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output/data:/opt/ml/output/data
                          - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\model:/opt/ml/model
                          - C:\Users\username\AppData\Local\Temp\tmpzg964wnz:/opt/ml/input/data/train
                          version: '2.3'
                          

                          Deploying

                          • Logs
                          Attaching to tmpdbo8bbod_algo-1-qlg3c_1
                          algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Starting gunicorn 19.9.0
                          algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.sock (16)
                          algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Using worker: gevent
                          algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [23] [INFO] Booting worker with pid: 23
                          algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [27] [INFO] Booting worker with pid: 27
                          algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [29] [INFO] Booting worker with pid: 29
                          algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [33] [INFO] Booting worker with pid: 33
                          algo-1-qlg3c_1 | 2019-06-12 23:57:21,219 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)
                          algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Module train_model does not provide a setup.py.
                          algo-1-qlg3c_1 | Generating setup.py
                          algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating setup.cfg
                          algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating MANIFEST.in
                          algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Installing module with the following command:
                          algo-1-qlg3c_1 | /usr/bin/python3 -m pip install -U .
                          algo-1-qlg3c_1 | Processing /opt/ml/code
                          algo-1-qlg3c_1 | Building wheels for collected packages: train-model
                          algo-1-qlg3c_1 | Running setup.py bdist_wheel for train-model ... done
                          algo-1-qlg3c_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-4x4jzvi4/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
                          algo-1-qlg3c_1 | Successfully built train-model
                          algo-1-qlg3c_1 | Installing collected packages: train-model
                          algo-1-qlg3c_1 | Successfully installed train-model-1.0.0
                          algo-1-qlg3c_1 | You are using pip version 18.1, however version 19.1.1 is available.
                          algo-1-qlg3c_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
                          algo-1-qlg3c_1 | [2019-06-12 23:57:22 +0000] [23] [ERROR] Error handling request /ping
                          algo-1-qlg3c_1 | Traceback (most recent call last):
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
                          algo-1-qlg3c_1 | return fn(*args, **kwargs)
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
                          algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
                          algo-1-qlg3c_1 | with open(filename, 'rb') as f:
                          algo-1-qlg3c_1 | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
                          algo-1-qlg3c_1 |
                          algo-1-qlg3c_1 | During handling of the above exception, another exception occurred:
                          algo-1-qlg3c_1 |
                          algo-1-qlg3c_1 | Traceback (most recent call last):
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
                          algo-1-qlg3c_1 | self.handle_request(listener_name, req, client, addr)
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
                          algo-1-qlg3c_1 | addr)
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
                          algo-1-qlg3c_1 | respiter = self.wsgi(environ, resp.start_response)
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_sklearn_container/serving.py", line 103, in main
                          algo-1-qlg3c_1 | user_module_transformer.initialize()
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_transformer.py", line 157, in initialize
                          algo-1-qlg3c_1 | self._model = self._model_fn(_env.model_dir)
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
                          algo-1-qlg3c_1 | six.reraise(error_class, error_class(e), sys.exc_info()[2])
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
                          algo-1-qlg3c_1 | raise value.with_traceback(tb)
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
                          algo-1-qlg3c_1 | return fn(*args, **kwargs)
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
                          algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
                          algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
                          algo-1-qlg3c_1 | with open(filename, 'rb') as f:
                          algo-1-qlg3c_1 | sagemaker_containers._errors.ClientError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
                          algo-1-qlg3c_1 | 172.18.0.1 - - [12/Jun/2019:23:57:22 +0000] "GET /ping HTTP/1.1" 500 141 "-" "-"
                          
                          • Created Temp Files
                          Temp
                          ├─tmpgfpx4a40 # Why is it empty
                          │ └─tmpdbo8bbod
                          │ docker-compose.yaml
                          │ └─algo-1-qlg3c
                          
                          • docker-compose.yaml
                          networks:
                          sagemaker-local:
                          name: sagemaker-local
                          services:
                          algo-1-qlg3c:
                          command: serve
                          environment:
                          - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
                          - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
                          - HTTP_PROXY=http://username:password@foobar:port
                          - HTTPS_PROXY=http://username:password@foobar:port
                          - SAGEMAKER_PROGRAM=train_model.py
                          - SAGEMAKER_SUBMIT_DIRECTORY=s3://sagemaker-ap-northeast-1-XXXXXXXXXX/sagemaker-scikit-learn-2019-06-12-23-57-07-460/sourcedir.tar.gz
                          - SAGEMAKER_ENABLE_CLOUDWATCH_METRICS=false
                          - SAGEMAKER_CONTAINER_LOG_LEVEL=10
                          - SAGEMAKER_REGION=ap-northeast-1
                          image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
                          networks:
                          sagemaker-local:
                          aliases:
                          - algo-1-qlg3c
                          ports:
                          - 8080:8080
                          stdin_open: true
                          tty: true
                          volumes:
                          - C:\Users\username\AppData\Local\Temp\tmpgfpx4a40:/opt/ml/model
                          version: '2.3'
                          

                          Exact command to reproduce

                          sklearn=SKLearn(
                          entry_point=script_path,
                          train_instance_type='local',
                          source_dir='./',
                          role=role,
                          py_version='py3',
                          container_log_level=10,
                          hyperparameters={'para_c': 1.0, 'para_gamma': 0.01})
                          sklearn.fit({'train': train_input})
                          predictor=sklearn.deploy(initial_instance_count=1,
                          instance_type='local',
                          env=env)

                          Cause

                          I know that the cause is that the tmpgfpx4a40 folder linked to the /opt/ml/model folder is empty, as described in the docker-compose.yaml file. And I think the tmpgfpx4a40 folder needs a model.joblib file.
                          However, I don't know why the tmpgfpx4a40 folder is empty and I don't know how to avoid it.

                          Activity

                          Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                          Metadata

                          Metadata

                          Assignees

                          No one assigned

                            Type

                            No type

                            Projects

                            No projects

                              Milestone

                              No milestone

                              Relationships

                              None yet

                              Development

                              No branches or pull requests

                              Issue actions

                              , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
                              Skip to content

                              "No such file or directory: '/opt/ml/model/model.joblib'" occured at deploy in the local mode of windows #846

                              Description

                              @xnaiman

                              System Information

                              • Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Scikit-Learn
                              • Framework Version: 0.20.0 (official sagemaker-scikit-learn-container)
                              • Python Version: 3.6
                              • CPU or GPU: CPU
                              • Python SDK Version: 1.26.0
                              • Are you using a custom image: No

                              Describe the problem

                              No such file or directory: /opt/ml/model/model.joblib occured at deploy in the local mode of windows.
                              (Raised after avoiding issue #844 and #845)

                              Minimal repro / logs

                              Training

                              • Logs
                              Windows Support for Local Mode is Experimental
                              Creating tmpri2jv76o_algo-1-4cuqr_1 ... done
                              Attaching to tmpri2jv76o_algo-1-4cuqr_1
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,801 sagemaker-containers INFO Imported framework sagemaker_sklearn_container.training
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,811 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,813 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,820 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,821 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,825 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,826 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,828 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,832 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,833 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,835 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,839 sagemaker_sklearn_container.training INFO Invoking user training script.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,842 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,845 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,848 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,849 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,851 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,854 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,855 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,857 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,863 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/boto3/data/s3/2006-03-01/resources-1.json
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials DEBUG Looking for credentials via: env
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.credentials INFO Found credentials in environment variables.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,866 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/endpoints.json
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,871 botocore.hooks DEBUG Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff9c9cafa60>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,875 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/s3/2006-03-01/service-2.json
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff9c7814400>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,881 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08840>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,882 botocore.hooks DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff9c78141e0>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,883 botocore.args DEBUG The s3 config key is not a dictionary type, ignoring its value of: None
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,885 botocore.endpoint DEBUG Setting s3 timeout as (60, 60)
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,886 botocore.loaders DEBUG Loading JSON file: /usr/local/lib/python3.5/dist-packages/botocore/data/_retry.json
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,887 botocore.client DEBUG Registering retry handlers for service: s3
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,894 botocore.client DEBUG Defaulting to S3 virtual host style addressing with path style addressing fallback.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,895 boto3.resources.factory DEBUG Loading s3:s3
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,896 boto3.resources.factory DEBUG Loading s3:Bucket
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,897 boto3.resources.model DEBUG Renaming Bucket attribute name
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,898 botocore.hooks DEBUG Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7ff9c6d08950>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.utils DEBUG Acquiring 0
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,899 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) about to wait for the following futures []
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) done waiting for dependent futures
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 s3transfer.tasks DEBUG Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}) with kwargs {'config': <boto3.s3.transfer.TransferConfig object at 0x7ff9c6b5c710>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cef0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8>, 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7ff9c6ae4278>}
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,900 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.hooks DEBUG Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,901 botocore.endpoint DEBUG Making request for OperationModel(name=HeadObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1',
                              'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'HEAD', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,902 botocore.hooks DEBUG Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.hooks DEBUG Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,903 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG Calculating signature using v4 auth.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG CanonicalRequest:
                              algo-1-4cuqr_1 | HEAD
                              algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
                              algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                              algo-1-4cuqr_1 | x-amz-date:20190612T235711Z
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
                              algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,904 botocore.auth DEBUG StringToSign:
                              algo-1-4cuqr_1 | AWS4-HMAC-SHA256
                              algo-1-4cuqr_1 | 20190612T235711Z
                              algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
                              algo-1-4cuqr_1 | 3c7d2e923cfbc1df5b252ea8422724d8ecd6c607014af9b27717fcbdf25c94d1
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.auth DEBUG Signature:
                              algo-1-4cuqr_1 | 43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.hooks DEBUG Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,905 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235711Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=43919605f5db884d3d34ed4b5732ace0779cf7c138fbf01b1f1e2b61151ed66f'}>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
                              algo-1-4cuqr_1 | 2019-06-12 23:57:11,906 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,024 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "HEAD /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 0
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': 'F48E7D6B6A526E82', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'cAB/AbGlPTnTc025AlpNhnwIFzC34NmUPjtLJAUp7Tf50k4QwAxC33+RE0jTSrnG7qafBtR38Fs=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
                              'Wed, 12 Jun 2019 23:57:23 GMT'}
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,025 botocore.parsers DEBUG Response body:
                              algo-1-4cuqr_1 | b''
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.retryhandler DEBUG No retry needed.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,026 botocore.hooks DEBUG Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.futures DEBUG Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7ff9c6b5cac8> for transfer request: 0.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,027 s3transfer.utils DEBUG Acquiring 0
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) about to wait for the following futures []
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) done waiting for dependent futures
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.tasks DEBUG Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'sagemaker-ap-northeast-1-130747742019', 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'extra_args': {}}) with kwargs {'max_attempts': 5, 'bucket': 'sagemaker-ap-northeast-1-130747742019', 'bandwidth_limiter': None, 'io_chunksize': 262144, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'callbacks': [], 'key': 'sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'client': <botocore.client.S3 object at 0x7ff9c6b2b048>, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7ff9c6b12cc0>, 'extra_args': {}}
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7ff9c9ccb730>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,028 s3transfer.utils DEBUG Releasing acquire 0/None
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7ff9c9ccb6a8>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7ff9c9ccb2f0>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7ff9c9ccbb70>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.hooks DEBUG Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,029 botocore.endpoint DEBUG Making request for OperationModel(name=GetObject) with params: {'url': 'https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'context': {'client_config': <botocore.config.Config object at 0x7ff9c6b2b518>, 'has_streaming_input': False, 'client_region': 'ap-northeast-1', 'auth_type': None, 'signing': {'bucket': 'sagemaker-ap-northeast-1-130747742019'}}, 'body': b'', 'method': 'GET', 'query_string': {}, 'url_path': '/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz', 'headers': {'User-Agent': 'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource'}}
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7ff9c7659730>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff9c6b2b4a8>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7ff9c6c67390>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7ff9c9ccb1e0>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.hooks DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x7ff9c7914730>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG Checking for DNS compatible bucket for: https://s3.ap-northeast-1.amazonaws.com/sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.utils DEBUG URI updated to: https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG Calculating signature using v4 auth.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,030 botocore.auth DEBUG CanonicalRequest:
                              algo-1-4cuqr_1 | GET
                              algo-1-4cuqr_1 | /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | host:sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com
                              algo-1-4cuqr_1 | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                              algo-1-4cuqr_1 | x-amz-date:20190612T235712Z
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | host;x-amz-content-sha256;x-amz-date
                              algo-1-4cuqr_1 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG StringToSign:
                              algo-1-4cuqr_1 | AWS4-HMAC-SHA256
                              algo-1-4cuqr_1 | 20190612T235712Z
                              algo-1-4cuqr_1 | 20190612/ap-northeast-1/s3/aws4_request
                              algo-1-4cuqr_1 | d5310e2e1112cc81c0cbcd2fdb9f2f8fe645ea026d34a43d2a5cd9ce8cf1b5d3
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.auth DEBUG Signature:
                              algo-1-4cuqr_1 | 77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.hooks DEBUG Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7ff9c76597b8>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 botocore.endpoint DEBUG Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz, headers={'X-Amz-Date': b'20190612T235712Z', 'User-Agent': b'Boto3/1.9.51 Python/3.5.2 Linux/4.9.125-linuxkit Botocore/1.12.51 Resource', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAR44JMG5BUD7WQML6/20190612/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77bc5f5a49de3134943f04f965e28fb747c4caed3e0f466dca439447f71291e4'}>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,031 urllib3.util.retry DEBUG Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 urllib3.connectionpool DEBUG https://sagemaker-ap-northeast-1-130747742019.s3.ap-northeast-1.amazonaws.com:443 "GET /sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz HTTP/1.1" 200 7271
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,066 botocore.parsers DEBUG Response headers: {'ETag': '"2a30f306729c3090cfaa4822ab8854f0"', 'x-amz-request-id': '7250ADF3F5A5A49C', 'Server': 'AmazonS3', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'x-amz-id-2': 'WiYMANCqDMYoqe7ZcCVyUJTpmRTWAQ9GRSIeW6JAlxtmugzNooQgP8bb/pnTQ+PBo5seJvLVhoc=', 'Date': 'Wed, 12 Jun 2019 23:57:25 GMT', 'Content-Length': '7271', 'Last-Modified':
                              'Wed, 12 Jun 2019 23:57:23 GMT'}
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.parsers DEBUG Response body:
                              algo-1-4cuqr_1 | <botocore.response.StreamingBody object at 0x7ff9c6aba048>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,067 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7ff9c6b2bf60>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.retryhandler DEBUG No retry needed.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,068 botocore.hooks DEBUG Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff9c6b2b5f8>>
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,076 s3transfer.tasks DEBUG Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'offset': 0, 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>}
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) about to wait for the following futures []
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) done waiting for dependent futures
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.tasks DEBUG Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/tmp_f99p1cx/tar_file'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7ff9c6b12cf8>, 'osutil': <s3transfer.utils.OSUtils object at 0x7ff9c6b5c8d0>, 'final_filename': '/tmp/tmp_f99p1cx/tar_file'}
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,077 s3transfer.utils DEBUG Releasing acquire 0/None
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Module train_model does not provide a setup.py.
                              algo-1-4cuqr_1 | Generating setup.py
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating setup.cfg
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Generating MANIFEST.in
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,084 sagemaker-containers INFO Installing module with the following command:
                              algo-1-4cuqr_1 | /usr/bin/python3 -m pip install -U .
                              algo-1-4cuqr_1 | Processing /opt/ml/code
                              algo-1-4cuqr_1 | Building wheels for collected packages: train-model
                              algo-1-4cuqr_1 | Running setup.py bdist_wheel for train-model ... done
                              algo-1-4cuqr_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-d58g4m1m/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
                              algo-1-4cuqr_1 | Successfully built train-model
                              algo-1-4cuqr_1 | Installing collected packages: train-model
                              algo-1-4cuqr_1 | Successfully installed train-model-1.0.0
                              algo-1-4cuqr_1 | You are using pip version 18.1, however version 19.1.1 is available.
                              algo-1-4cuqr_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,983 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,984 botocore.hooks DEBUG Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,987 botocore.hooks DEBUG Changing event name from before-call.apigateway to before-call.api-gateway
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,988 botocore.hooks DEBUG Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,990 botocore.hooks DEBUG Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,991 botocore.hooks DEBUG Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,995 botocore.hooks DEBUG Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,996 botocore.hooks DEBUG Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
                              algo-1-4cuqr_1 | 2019-06-12 23:57:12,998 sagemaker-containers INFO Invoking user script
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | Training Env:
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | {
                              algo-1-4cuqr_1 | "hosts": [
                              algo-1-4cuqr_1 | "algo-1-4cuqr"
                              algo-1-4cuqr_1 | ],
                              algo-1-4cuqr_1 | "output_dir": "/opt/ml/output",
                              algo-1-4cuqr_1 | "user_entry_point": "train_model.py",
                              algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr",
                              algo-1-4cuqr_1 | "input_data_config": {
                              algo-1-4cuqr_1 | "train": {
                              algo-1-4cuqr_1 | "TrainingInputMode": "File"
                              algo-1-4cuqr_1 | }
                              algo-1-4cuqr_1 | },
                              algo-1-4cuqr_1 | "output_intermediate_dir": "/opt/ml/output/intermediate",
                              algo-1-4cuqr_1 | "input_dir": "/opt/ml/input",
                              algo-1-4cuqr_1 | "output_data_dir": "/opt/ml/output/data",
                              algo-1-4cuqr_1 | "num_gpus": 0,
                              algo-1-4cuqr_1 | "log_level": 10,
                              algo-1-4cuqr_1 | "resource_config": {
                              algo-1-4cuqr_1 | "hosts": [
                              algo-1-4cuqr_1 | "algo-1-4cuqr"
                              algo-1-4cuqr_1 | ],
                              algo-1-4cuqr_1 | "current_host": "algo-1-4cuqr"
                              algo-1-4cuqr_1 | },
                              algo-1-4cuqr_1 | "job_name": "sagemaker-scikit-learn-2019-06-12-23-57-07-460",
                              algo-1-4cuqr_1 | "additional_framework_parameters": {},
                              algo-1-4cuqr_1 | "input_config_dir": "/opt/ml/input/config",
                              algo-1-4cuqr_1 | "network_interface_name": "ethwe",
                              algo-1-4cuqr_1 | "module_dir": "s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz",
                              algo-1-4cuqr_1 | "framework_module": "sagemaker_sklearn_container.training:main",
                              algo-1-4cuqr_1 | "channel_input_dirs": {
                              algo-1-4cuqr_1 | "train": "/opt/ml/input/data/train"
                              algo-1-4cuqr_1 | },
                              algo-1-4cuqr_1 | "module_name": "train_model",
                              algo-1-4cuqr_1 | "model_dir": "/opt/ml/model",
                              algo-1-4cuqr_1 | "hyperparameters": {
                              algo-1-4cuqr_1 | "para_c": 1.0,
                              algo-1-4cuqr_1 | "para_gamma": 0.01
                              algo-1-4cuqr_1 | },
                              algo-1-4cuqr_1 | "num_cpus": 4
                              algo-1-4cuqr_1 | }
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | Environment variables:
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | PYTHONPATH=/usr/local/bin:/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages
                              algo-1-4cuqr_1 | SM_MODULE_NAME=train_model
                              algo-1-4cuqr_1 | SM_LOG_LEVEL=10
                              algo-1-4cuqr_1 | SM_CHANNELS=["train"]
                              algo-1-4cuqr_1 | SM_HP_PARA_C=1.0
                              algo-1-4cuqr_1 | SM_CURRENT_HOST=algo-1-4cuqr
                              algo-1-4cuqr_1 | SM_FRAMEWORK_PARAMS={}
                              algo-1-4cuqr_1 | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
                              algo-1-4cuqr_1 | SM_OUTPUT_DIR=/opt/ml/output
                              algo-1-4cuqr_1 | SM_HP_PARA_GAMMA=0.01
                              algo-1-4cuqr_1 | SM_HPS={"para_c":1.0,"para_gamma":0.01}
                              algo-1-4cuqr_1 | SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main
                              algo-1-4cuqr_1 | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
                              algo-1-4cuqr_1 | SM_USER_ARGS=["--para_c","1.0","--para_gamma","0.01"]
                              algo-1-4cuqr_1 | SM_NUM_CPUS=4
                              algo-1-4cuqr_1 | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
                              algo-1-4cuqr_1 | SM_MODULE_DIR=s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz
                              algo-1-4cuqr_1 | SM_RESOURCE_CONFIG={"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]}
                              algo-1-4cuqr_1 | SM_HOSTS=["algo-1-4cuqr"]
                              algo-1-4cuqr_1 | SM_INPUT_DIR=/opt/ml/input
                              algo-1-4cuqr_1 | SM_CHANNEL_TRAIN=/opt/ml/input/data/train
                              algo-1-4cuqr_1 | SM_NUM_GPUS=0
                              algo-1-4cuqr_1 | SM_USER_ENTRY_POINT=train_model.py
                              algo-1-4cuqr_1 | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1-4cuqr","framework_module":"sagemaker_sklearn_container.training:main","hosts":["algo-1-4cuqr"],"hyperparameters":{"para_c":1.0,"para_gamma":0.01},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","job_name":"sagemaker-scikit-learn-2019-06-12-23-57-07-460","log_level":10,"model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-ap-northeast-1-130747742019/sagemaker-scikit-learn-2019-06-12-23-57-07-460/source/sourcedir.tar.gz","module_name":"train_model","network_interface_name":"ethwe","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-4cuqr","hosts":["algo-1-4cuqr"]},"user_entry_point":"train_model.py"}
                              algo-1-4cuqr_1 | SM_INPUT_DATA_CONFIG={"train":{"TrainingInputMode":"File"}}
                              algo-1-4cuqr_1 | SM_NETWORK_INTERFACE_NAME=ethwe
                              algo-1-4cuqr_1 | SM_MODEL_DIR=/opt/ml/model
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | Invoking script with the following command:
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | /usr/bin/python3 -m train_model --para_c 1.0 --para_gamma 0.01
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 |
                              algo-1-4cuqr_1 | 2019-06-12 23:57:13,528 sagemaker-containers INFO Reporting training SUCCESS
                              tmpri2jv76o_algo-1-4cuqr_1 exited with code 0
                              Aborting on container exit...
                              ===== Job Complete =====
                              
                              • Created Temp Files
                              Temp
                              ├─tmpzg964wnz
                              │ iris_train.csv
                              │
                              └─tmpri2jv76o
                              │ docker-compose.yaml
                              │ ├─artifacts
                              ├─model
                              │ model.joblib
                              │ └─output
                              └─data
                              
                              • docker-compose.yaml
                              networks:
                              sagemaker-local:
                              name: sagemaker-local
                              services:
                              algo-1-4cuqr:
                              command: train
                              environment:
                              - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
                              - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
                              - AWS_REGION=ap-northeast-1
                              - TRAINING_JOB_NAME=sagemaker-scikit-learn-2019-06-12-23-57-07-460
                              - HTTP_PROXY=http://username:password@foobar:port
                              - HTTPS_PROXY=http://username:password@foobar:port
                              image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
                              networks:
                              sagemaker-local:
                              aliases:
                              - algo-1-4cuqr
                              stdin_open: true
                              tty: true
                              volumes:
                              - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output:/opt/ml/output
                              - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\input:/opt/ml/input
                              - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\algo-1-4cuqr\output/data:/opt/ml/output/data
                              - C:\Users\username\AppData\Local\Temp\tmpri2jv76o\model:/opt/ml/model
                              - C:\Users\username\AppData\Local\Temp\tmpzg964wnz:/opt/ml/input/data/train
                              version: '2.3'
                              

                              Deploying

                              • Logs
                              Attaching to tmpdbo8bbod_algo-1-qlg3c_1
                              algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Starting gunicorn 19.9.0
                              algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.sock (16)
                              algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [16] [INFO] Using worker: gevent
                              algo-1-qlg3c_1 | [2019-06-12 23:57:20 +0000] [23] [INFO] Booting worker with pid: 23
                              algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [27] [INFO] Booting worker with pid: 27
                              algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [29] [INFO] Booting worker with pid: 29
                              algo-1-qlg3c_1 | [2019-06-12 23:57:21 +0000] [33] [INFO] Booting worker with pid: 33
                              algo-1-qlg3c_1 | 2019-06-12 23:57:21,219 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)
                              algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Module train_model does not provide a setup.py.
                              algo-1-qlg3c_1 | Generating setup.py
                              algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating setup.cfg
                              algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Generating MANIFEST.in
                              algo-1-qlg3c_1 | 2019-06-12 23:57:21,465 INFO - sagemaker-containers - Installing module with the following command:
                              algo-1-qlg3c_1 | /usr/bin/python3 -m pip install -U .
                              algo-1-qlg3c_1 | Processing /opt/ml/code
                              algo-1-qlg3c_1 | Building wheels for collected packages: train-model
                              algo-1-qlg3c_1 | Running setup.py bdist_wheel for train-model ... done
                              algo-1-qlg3c_1 | Stored in directory: /tmp/pip-ephem-wheel-cache-4x4jzvi4/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3
                              algo-1-qlg3c_1 | Successfully built train-model
                              algo-1-qlg3c_1 | Installing collected packages: train-model
                              algo-1-qlg3c_1 | Successfully installed train-model-1.0.0
                              algo-1-qlg3c_1 | You are using pip version 18.1, however version 19.1.1 is available.
                              algo-1-qlg3c_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
                              algo-1-qlg3c_1 | [2019-06-12 23:57:22 +0000] [23] [ERROR] Error handling request /ping
                              algo-1-qlg3c_1 | Traceback (most recent call last):
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
                              algo-1-qlg3c_1 | return fn(*args, **kwargs)
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
                              algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
                              algo-1-qlg3c_1 | with open(filename, 'rb') as f:
                              algo-1-qlg3c_1 | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
                              algo-1-qlg3c_1 |
                              algo-1-qlg3c_1 | During handling of the above exception, another exception occurred:
                              algo-1-qlg3c_1 |
                              algo-1-qlg3c_1 | Traceback (most recent call last):
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
                              algo-1-qlg3c_1 | self.handle_request(listener_name, req, client, addr)
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
                              algo-1-qlg3c_1 | addr)
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
                              algo-1-qlg3c_1 | respiter = self.wsgi(environ, resp.start_response)
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_sklearn_container/serving.py", line 103, in main
                              algo-1-qlg3c_1 | user_module_transformer.initialize()
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_transformer.py", line 157, in initialize
                              algo-1-qlg3c_1 | self._model = self._model_fn(_env.model_dir)
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
                              algo-1-qlg3c_1 | six.reraise(error_class, error_class(e), sys.exc_info()[2])
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
                              algo-1-qlg3c_1 | raise value.with_traceback(tb)
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sagemaker_containers/_functions.py", line 84, in wrapper
                              algo-1-qlg3c_1 | return fn(*args, **kwargs)
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/train_model.py", line 53, in model_fn
                              algo-1-qlg3c_1 | model = joblib.load(os.path.join(model_dir, MODEL_NAME))
                              algo-1-qlg3c_1 | File "/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 590, in load
                              algo-1-qlg3c_1 | with open(filename, 'rb') as f:
                              algo-1-qlg3c_1 | sagemaker_containers._errors.ClientError: [Errno 2] No such file or directory: '/opt/ml/model/model.joblib'
                              algo-1-qlg3c_1 | 172.18.0.1 - - [12/Jun/2019:23:57:22 +0000] "GET /ping HTTP/1.1" 500 141 "-" "-"
                              
                              • Created Temp Files
                              Temp
                              ├─tmpgfpx4a40 # Why is it empty
                              │ └─tmpdbo8bbod
                              │ docker-compose.yaml
                              │ └─algo-1-qlg3c
                              
                              • docker-compose.yaml
                              networks:
                              sagemaker-local:
                              name: sagemaker-local
                              services:
                              algo-1-qlg3c:
                              command: serve
                              environment:
                              - AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
                              - AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX
                              - HTTP_PROXY=http://username:password@foobar:port
                              - HTTPS_PROXY=http://username:password@foobar:port
                              - SAGEMAKER_PROGRAM=train_model.py
                              - SAGEMAKER_SUBMIT_DIRECTORY=s3://sagemaker-ap-northeast-1-XXXXXXXXXX/sagemaker-scikit-learn-2019-06-12-23-57-07-460/sourcedir.tar.gz
                              - SAGEMAKER_ENABLE_CLOUDWATCH_METRICS=false
                              - SAGEMAKER_CONTAINER_LOG_LEVEL=10
                              - SAGEMAKER_REGION=ap-northeast-1
                              image: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
                              networks:
                              sagemaker-local:
                              aliases:
                              - algo-1-qlg3c
                              ports:
                              - 8080:8080
                              stdin_open: true
                              tty: true
                              volumes:
                              - C:\Users\username\AppData\Local\Temp\tmpgfpx4a40:/opt/ml/model
                              version: '2.3'
                              

                              Exact command to reproduce

                              sklearn=SKLearn(
                              entry_point=script_path,
                              train_instance_type='local',
                              source_dir='./',
                              role=role,
                              py_version='py3',
                              container_log_level=10,
                              hyperparameters={'para_c': 1.0, 'para_gamma': 0.01})
                              sklearn.fit({'train': train_input})
                              predictor=sklearn.deploy(initial_instance_count=1,
                              instance_type='local',
                              env=env)

                              Cause

                              I know that the cause is that the tmpgfpx4a40 folder linked to the /opt/ml/model folder is empty, as described in the docker-compose.yaml file. And I think the tmpgfpx4a40 folder needs a model.joblib file.
                              However, I don't know why the tmpgfpx4a40 folder is empty and I don't know how to avoid it.

                              Activity

                              Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                              Metadata

                              Metadata

                              Assignees

                              No one assigned

                                Type

                                No type

                                Projects

                                No projects

                                  Milestone

                                  No milestone

                                  Relationships

                                  None yet

                                  Development

                                  No branches or pull requests

                                  Issue actions