AWS Integration Pack

The StackStorm AWS integration pack supplies action integration for numerous AWS services.

Prerequisites

AWS and Stackstorm, up and running.

Setup

Install AWS pack on StackStorm

1. Install the AWS pack:

   # Install AWS
   st2 pack install aws
   # Check it
   st2 action list -p aws
   

2. Copy the example configuration in aws.yaml.example to /opt/stackstorm/configs/aws.yaml and edit as required. It must contain:

• region - Region where AWS commands will be executed
• aws_access_key_id - Access key
• aws_secret_access_key_id - Secret Access key

You can generate the access key and secret access key by following these directions:

http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html#Using_CreateAccessKey

If you would like to use the IAM role assigned to the instance stackstorm is running set the key and secret to None and set the region.

---
region: "us-east-1"aws_access_key_id: Noneaws_secret_access_key: Nonest2_user_data: ""

In previous versions there was a 'setup' object within the config, which has been deprecated. This will break the configuration where iam roles are being used:

• service_notifications_sensor.host - Listen host for the HTTP interface.
• service_notifications_sensor.port - Listen port for the HTTP interface.
• service_notifications_sensor.path - Path where the events need to be sent.

Note : When modifying the configuration in /opt/stackstorm/configs/ please remember to tell StackStorm to load these new values by running st2ctl reload --register-configs

st2_user_data

Optionally, you can add the UserData with a set of provisioning instructions to be used when using the ec2_run_instances action.

Note that the ec2 boto3 SDK requires the parameter UserData, for the aws.ec2_run_instances action. This was previosly user_data.

Put your user_data file somewhere accessible by the StackStorm user, and use the st2_user_data config option to set it. This file/script will be used for all invocations of the ec2_run_instances action:

st2_user_data: "/full/path/to/file"

Alternatively, user data can be specified inline:

st2_user_data: | #!/bin/bash echo "I'm EC2 user data: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html"

Actions

Prior to installation of the aws pack, you can get the list of available actions here:

https://github.com/StackStorm-Exchange/stackstorm-aws/tree/master/actions

Once you have installed the aws pack, you can get a list of actions in the AWS pack using:

st2 action list -p aws

To get information on a specific action, please run:

root@2e1d15fd5d07:/# st2 action get aws.route53_list_hosted_zones
+-------------+--------------------------------------------------------------+
| Property | Value |
+-------------+--------------------------------------------------------------+
| id | 594bfa4ed5d05c0b7e504803 |
| uid | action:aws:route53_list_hosted_zones |
| ref | aws.route53_list_hosted_zones |
| pack | aws |
| name | route53_list_hosted_zones |
| enabled | True |
| entry_point | run.py |
| runner_type | run-python |
| parameters | { |
| | "DelegationSetId": { |
| | "type": "string", |
| | "description": "If you"re using reusable delegation |
| | sets and you want to list all of the hosted zones that are |
| | associated with a reusable delegation set, specify the ID of |
| | that reusable delegation set. " |
| | }, |
| | "headers": { |
| | "type": "string" |
| | }, |
| | "MaxItems": { |
| | "type": "string", |
| | "description": "(Optional) The maximum number of |
| | hosted zones to be included in the response body for this |
| | request. If you have more than maxitems hosted zones, the |
| | value of the IsTruncated element in the response is true, |
| | and the value of the NextMarker element is the hosted zone |
| | ID of the first hosted zone in the next group of maxitems |
| | hosted zones." |
| | }, |
| | "Marker": { |
| | "type": "string", |
| | "description": "(Optional) If you have more hosted |
| | zones than the value of maxitems, ListHostedZones returns |
| | only the first maxitems hosted zones. To get the next group |
| | of maxitems hosted zones, submit another request to |
| | ListHostedZones. For the value of marker, specify the value |
| | of the NextMarker element that was returned in the previous |
| | response. Hosted zones are listed in the order in which they |
| | were created." |
| | }, |
| | "module_path": { |
| | "default": "boto3", |
| | "type": "string", |
| | "immutable": true |
| | }, |
| | "action": { |
| | "default": "list_hosted_zones", |
| | "type": "string", |
| | "immutable": true |
| | }, |
| | "cls": { |
| | "default": "route53", |
| | "type": "string" |
| | } |
| | } |
| notify | |
| tags | |
+-------------+--------------------------------------------------------------+

Since this action does not take any required parameters and won't create any resources that will cost us anything, let's run it and see what it returns.

$ st2 run aws.route53_list_hosted_zones
.
id: 594cc1c2d5d05c1185bd51e5
status: succeeded
parameters: None
result:
exit_code: 0
result:
- HostedZones:
- CallerReference: 63FC852C-8B5F-12F9-8945-BC8FF413430A
Config:
PrivateZone: false
Id: /hostedzone/Z2MDOUVGZHYDZ7
Name: example.com.
ResourceRecordSetCount: 8
IsTruncated: false
MaxItems: '100'
ResponseMetadata:
HTTPHeaders:
content-length: '464'
content-type: text/xml
date: Fri, 23 Jun 2017 07:22:42 GMT
x-amzn-requestid: cf583b3d-57f4-11e7-a217-4b0c7596d765
HTTPStatusCode: 200
RequestId: cf583b3d-57f4-11e7-a217-4b0c7596d765
RetryAttempts: 0
stderr: ''
stdout: ''

To regenerate the actions used by this pack, run the following:

source /opt/stackstorm/virtualenvs/aws/bin/activate
cd /opt/stackstorm/packs/aws/etc/st2packgen
python st2packgen.py -d /opt/stackstorm/packs/aws/actions
st2ctl reload --register-actions

st2packgen.py will overwrite any existing actions at /opt/stackstorm/packs/aws/actions.

Sensors

ServiceNotificationsSensor

This sensor exposes a HTTP interface and listens for service event notifications which are delivered via AWS SNS service using HTTP(s) endpoint.

Currently it supports event notifications generated by the S3 service - http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

Keep in mind that this sensor doesn't implement any kind of authentication. This means that if the sensor is not behind a proxy which implements authentication and is accessible to the outside world, you need to configure path setting to include a random secret which is only known to the AWS SNS service.

aws.service_notification

This trigger is emitted for every service event notification.

Example trigger payload:

{
"source": "aws:s3",
"region": "us-west-2",
"name": "ObjectCreated:Put",
"timestamp": 123456789,
"response_elements": {
"x-amz-id-2": "blahid//4U+rk=",
"x-amz-request-id": "5FF5BB6EDE3631F8"
},
"request_parameters": {
"sourceIPAddress": "127.0.0.1"
},
"payload": {
"configurationId": "snsnotificationforput",
"object": {
"eTag": "5dfd7f29bce6d94dc5c73553f269659b",
"key": "myfile.tar.gz",
"size": 178428
},
"bucket": {
"ownerIdentity": {
"principalId": "BLAHBLAH"
},
"name": "testbucket33333",
"arn": "arn:aws:s3:::testbucket33333"
},
"s3SchemaVersion": "1.0"
}
}

source, region, name and timestamp attributes are included with all the events, but the format and values inside the payload attribute different across different services and event types.

For a list of supported S3 event types see http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#supported-notification-event-types

AWS SQS sensor

This is generic SQS Sensor using boto3 api to fetch messages from SQS queue. After receiving a message it's content is passed as payload to a trigger 'aws.sqs_new_message'

This sensor can be configured either by using aws.yaml or by creating following values in datastore:

• aws.input_queues (list queues as comma separated string: first_queue,second_queue)
• aws.aws_access_key_id
• aws.aws_secret_access_key
• aws.region
• aws.max_number_of_messages (must be between 1 - 10)

For configuration in aws.yaml with config like this

aws_access_key_id:
aws_access_key_id:
region:
sqs_sensor:
input_queues:
- first_queue
- second_queue (queue list can be url or queue name)sqs_other:
max_number_of_messages: 1

If any value exist in datastore it will be taken instead of any value in aws.yaml

aws.sqs_new_message

This trigger is emitted when a single message is received from a queue.

{
"queue": "first_sqs_queue",
"body": "example message body"
}

Troubleshooting

If there are connection errors in st2sensorcontainer.log such as: 2020-02-06 22:04:05,202 140016844996688 INFO connectionpool [-] Starting new HTTP connection (1): 169.254.169.254. It's likely a result of turning on the sensor without copying the aws.yaml over to /opt/stackstorm/configs/aws.yaml