Concurrent s3 parts upload by barnabehvrd · Pull Request #205 · pelican/wings

barnabehvrd · 2026-08-11T20:00:53Z

Following PR pelican/panel#2510 on the Panel, this PR makes Wings upload S3 backup parts concurrently instead of one after another.
The number of parts in flight is bounded by the max_concurrent_uploads value returned by the Panel alongside the part URLs, and the upload stops as soon as any part fails permanently.

Uploading parts in parallel was not possible before: every part read from the same shared io.Reader, so concurrent reads would have interleaved and produced a corrupted archive.

Each part now gets its own io.SectionReader over the backup file, which is safe for concurrent use and lets a failed part be retried from its correct offset. Parts are uploaded through an errgroup whose limit comes from the max_concurrent_uploads value returned by the Panel alongside the part URLs. Results are written to a pre-allocated slice so the part list stays ordered regardless of completion order.

If the Panel does not include max_concurrent_uploads in its response (older Panel version) or sends an invalid value, Wings falls back to 1, preserving the previous sequential behavior. Upgrading Wings without upgrading the Panel is therefore safe.

This also fixes the retry path, which could never work: the request was built once outside the backoff loop, so a retry replayed a body that had already been consumed and sent an empty payload with a mismatched Content-Length.

The request is now rebuilt on every attempt and the section is rewound first.

The backoff's elapsed-time bound has been removed as well, since a single part can take longer than the previous one-minute budget, which meant a second attempt never happened. The two-hour request timeout and the retry count are unchanged.

Idle connections are now kept in the pool between parts instead of being renegotiated, and response bodies are drained so connections can be reused.

The results are as follows:

Concurrency	Duration	Throughput	Speedup
1	45.2 s	23.2 MB/s	baseline
5	5.37 s	195 MB/s	8.4x
10	4.31 s	244 MB/s	10.5x
20	3.46 s	303 MB/s	13.1x

Note :
These results were obtained on a fresh GCP E2 instance (2vCPU / 4GB of RAM / roughly 3-4 Gbps bandwith) for a bakcup of ~ 1000 MiB saved at BackBlaze B2. All parts were 50MiB except the last one (just a few KiB)
The benchmark backup only contains ~20 parts, which caps the benefit of higher concurrency levels: at 20 concurrent uploads, half the backup is in flight at once, so the marginal gain flattens. On real-world backups (10-50+ GiB, hundreds of parts), a default of 10 keeps the upload pipeline full throughout the transfer without hammering the S3 endpoint.

Most of the gain is already there at the lower concurrency levels, and the highest setting only buys a little more while getting close to the link's capacity. The sequential baseline is also the median of several runs that varied widely: with a single connection the total time depends entirely on how that one connection happens to behave, and one stalled part goes straight into the total. Concurrent uploads absorb that, so the win is in predictability as much as in raw throughput.

Summary by CodeRabbit

New Features
- Added support for reporting the maximum number of concurrent backup uploads.
- Backup uploads now run concurrently with a controlled limit, improving throughput.
- Added automatic retries for temporary upload failures, including rate limiting responses.
Reliability
- Uploads now respond more effectively to cancellation and retry individual parts when needed.
- Improved connection handling helps maintain reliable multipart uploads.

coderabbitai · 2026-08-11T20:01:34Z

📝 Walkthrough

Walkthrough

The upload response now reports the maximum concurrent upload count. S3 multipart uploads now run with bounded concurrency, independent file sections, context cancellation, dedicated HTTP transport settings, and expanded retry handling.

Changes

Backup upload flow

Layer / File(s)	Summary
Upload concurrency contract `remote/types.go`	`BackupRemoteUploadResponse` now includes `MaxConcurrentUploads` serialized as `max_concurrent_uploads`.
Concurrent S3 multipart execution `server/backup/backup_s3.go`	Multipart uploads use bounded `errgroup` workers and independent `io.SectionReader` instances. Requests use the provided context, rebuild bodies on retries, drain response bodies, retry HTTP 429 and 5xx responses, and detect wrapped backoff errors with `errors.As`.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested reviewers:quintenqvd0

Sequence Diagram(s)

sequenceDiagram
participant BackupUploader
participant osFile as *os.File
participant errgroup
participant HTTPClient
participant S3Endpoint
BackupUploader->>osFile: Read file size
BackupUploader->>HTTPClient: Retrieve upload URLs with context
BackupUploader->>errgroup: Start bounded upload workers
errgroup->>osFile: Read independent io.SectionReader parts
errgroup->>HTTPClient: Send multipart part request
HTTPClient->>S3Endpoint: Upload part
S3Endpoint-->>HTTPClient: Return response
HTTPClient->>S3Endpoint: Retry HTTP 429 or 5xx response

Poem

I’m a rabbit with uploads to make,
Many small parts hop in the wake.
Workers bound, retries grow wise,
Drained responses keep connections alive.
One field now tells how many may fly.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: concurrent S3 part uploads.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

barnabehvrd · 2026-08-11T20:00:53Z