s3sync

Golang utility for syncing between s3 and local

Migration guide

• v2

Usage

Use New to create a manager, and Sync function syncs between s3 and local filesystem.

import (
"github.com/aws/aws-sdk-go-v2/config""github.com/seqsense/s3sync/v2"
)
funcmain() {
ctx:=context.TODO()
// Load AWS configcfg, err:=config.LoadDefaultConfig(ctx, config.WithRegion("ap-northeast-1"))
iferr!=nil {
panic(err)
}
syncManager:=s3sync.New(cfg)
// Sync from s3 to localsyncManager.Sync(ctx, "s3://yourbucket/path/to/dir", "local/path/to/dir")
// Sync from local to s3syncManager.Sync(ctx, "local/path/to/dir", "s3://yourbucket/path/to/dir")
// Sync from s3 to s3syncManager.Sync(ctx, "s3://yourbucket/path/to/dir", "s3://anotherbucket/path/to/dir")
}

Sets the custom logger

You can set your custom logger.

import"github.com/seqsense/s3sync/v2"...s3sync.SetLogger(&CustomLogger{})
...

The logger needs to implement Logf methods. See the godoc for details.

Sets up the parallelism

You can configure the number of parallel jobs for sync. Default is 16.

s3sync.New(cfg, s3sync.WithParallel(16)) // This is the same as default.
s3sync.New(cfg, s3sync.WithParallel(1)) // You can sync one by one.

License

Apache 2.0 License. See LICENSE.