s3sync

s3 文件同步服务

应用部署实例较多,在不开通 sftp,ftp,且新增实例不能访问 git 仓库的情况下,为了让文件能自动同步,特编写本程序.

文件上传到 s3 之后,会触发一个 SQS 事件,本应用从 SQS 中获取消息,从 s3 中复制文件到本地

运行: s3sync --config=<配置文件>

处理流程

s3sync 定时执行,从 指定的消息队列读取消息,对得到的消息进行处理.

从 sqs 中得到的 key 的样式如下:

test-folder/folder-01/folder-02/peccancy.html
test-folder/folder-01/folder-02/left.png

得到上述的 key 之后，需要跟配置文件中的 sync->[key] 做最大化的匹配,匹配成功则将当前 key 指向的值复制到 sync->[value] 指定的目录中,完全匹配的前缀会被忽略。

例如:

有配置 test-folder/folder-01/ => /srv/web-pages/html/

从 sqs 中读取到的 object key`` 为 test-folder/folder-01/folder-02/peccancy.html`

则 test-folder/folder-01/folder-02/peccancy.html 中匹配的前缀 test-folder/folder-01/ 忽略,变成 folder-02/peccancy.html, 然后将 object key 的文件复制到本地的 /srv/web-pages/html/folder-02/peccancy.html 目录中.

注: 从 sqs 中得到的 object key 不包含 s3 bucket name

配置文件样式

{
"queue":"transport-dev",
"sync":[
"test-folder/": "/tmp/xxx/local-store-dir",
"test-folder/folder-01/": "/tmp/srv/local-store-dir",
"a/": "b",
"domain.com/path1/path2/": "/srv/web-pages/html/CmbcRecharge",
"cc/": "b"
]
}

文件上传事件

{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "cn-north-1",
"eventTime": "2017-07-14T09:16:28.877Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:AIDAPMPV34ZZL6EU6GYYW"
},
"requestParameters": {
"sourceIPAddress": "1.119.128.6"
},
"responseElements": {
"x-amz-request-id": "84143A72250ABA0B",
"x-amz-id-2": "kj6un32mldro+auadL6f+OWw86fH6vjoGljot0mAoWV7VGu1yyJ+dBAh6mWius+H"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "EventForTransport",
"bucket": {
"name": "transport-dev",
"ownerIdentity": {
"principalId": "AWS:xxxxxxx"
},
"arn": "arn:aws:s3:::transport-dev"
},
"object": {
"key": "plink.exe",
"size": 545880,
"eTag": "528248ae133191c591ec6d12732f2cfd",
"sequencer": "0059688BECBD741D89"
}
}
}
]
}

文件删除事件

{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "cn-north-1",
"eventTime": "2017-07-14T09:06:40.490Z",
"eventName": "ObjectRemoved:Delete",
"userIdentity": {
"principalId": "AWS:AIDAPMPV34ZZL6EU6GYYW"
},
"requestParameters": {
"sourceIPAddress": "54.222.13.7"
},
"responseElements": {
"x-amz-request-id": "8ADFAFDA4AAA0F4A",
"x-amz-id-2": "GDKw54pQliHWDHkQQMP1QMJg9Y/pYSZGAZeBawwO2hUOTaklO9Ro6Bpzwhyxj0xNqs9ColpmFiQ="
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "EventForTransport",
"bucket": {
"name": "transport-dev",
"ownerIdentity": {
"principalId": "AWS:xxxxxxx"
},
"arn": "arn:aws:s3:::transport-dev"
},
"object": {
"key": "xxxxx.html",
"sequencer": "00596889A0793F359F"
}
}
}
]
}

创建目录事件

{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "cn-north-1",
"eventTime": "2017-07-14T09:19:24.046Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:AIDAPMPV34ZZL6EU6GYYW"
},
"requestParameters": {
"sourceIPAddress": "54.222.11.3"
},
"responseElements": {
"x-amz-request-id": "198665B125B981BF",
"x-amz-id-2": "ba+y9Ir7Na+gAE8lVhX1FRioEbbmA8W3IpaaC2c6K5INJ6mBRTA+uT6JeF5JkuqMcyfqic7/V1U="
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "EventForTransport",
"bucket": {
"name": "transport-dev",
"ownerIdentity": {
"principalId": "AWS:xxxxxxx"
},
"arn": "arn:aws:s3:::transport-dev"
},
"object": {
"key": "test-folder/",
"size": 0,
"eTag": "d41d8cd98f00b204e9800998ecf8427e",
"sequencer": "0059688C9BF506F8D3"
}
}
}
]
}

S3 事件配置

在 s3 bucket 中配置 event

Event

• ObjectDelete(All)
• ObjectCreate (All)
• Type SQS [queue name]

SQS Policy

{
"Version": "2012-10-17",
"Id": "arn:aws-cn:sqs:cn-north-1:xxxxxx:ms-transport-dev/SQSDefaultPolicy",
"Statement": [
{
"Sid": "Sid1500023033663",
"Effect": "Allow",
"Principal": "*",
"Action": "SQS:SendMessage",
"Resource": "arn:aws-cn:sqs:cn-north-1:xxxxxx:ms-transport-dev",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws-cn:s3:*:*:transport-dev"
}
}
}
]
}

部署运行

在需要从 s3 中同步文件的 ec2 实例上运行本应用，可用 supervisor 作为守护进行运行或 cron 定时执行

ec2 实例所需权限

1. 从配置文件中指定的消息队列中获取消息
2. 从配置文件中指定的消息队列中删除消息
3. 从触发事件的 s3 中读取对象
4. 运行本程序的用户能读写配置文件中 sync 配置段中的本地存储路径

建议

在 s3 event 中，根据不同的 prefix 来将事件分发到不同的 sqs 中，在不同的实例上执行本程序，可避免匹配不到消息，导致同步速度慢的问题。