Skip to content

[IMP] Make the network retries configurable, with jitter - #3

Merged
hugosantosred merged 1 commit into
masterfrom
imp-retry-backoff-jitter
Aug 31, 2026
Merged

hugosantosred merged 1 commit into
masterfrom
imp-retry-backoff-jitter

Conversation

@hugosantosred

Copy link
Copy Markdown
Member

GitHub rate limits the unauthenticated git operations per IP address. Once the limit is reached it answers with an HTTP 401, which git reports as "could not read Username for 'https://github.com'". A CI job aggregating ~100 repositories from a shared runner hits it regularly, and the 3 retries within 35s of the previous defaults were not enough to get past it.

  • Raise the defaults to 4 retries, 15s delay and a 0.2 jitter, so the waits are 15s, 30s, 60s and 120s instead of 5s, 10s and 20s.
  • Add the jitter, to keep concurrent aggregations from retrying in lockstep against a rate limiting remote, only to get limited together again.
  • Add --retry-max, --retry-delay, --retry-backoff-factor and --retry-jitter, taking precedence over the retry section of the configuration file. It was only configurable per repository in the yaml file, which is not practical in a CI environment with a hundred of them.
  • Merge the retry section with the defaults, so setting one parameter no longer drops the others, and raise a ConfigException on unknown ones.
  • Document all of it in the README, including the fact that the merge check done while loading the configuration file is not retried on purpose.

GitHub rate limits the unauthenticated git operations per IP address. Once
the limit is reached it answers with an HTTP 401, which git reports as
"could not read Username for 'https://github.com'". A CI job aggregating
~100 repositories from a shared runner hits it regularly, and the 3 retries
within 35s of the previous defaults were not enough to get past it.

* Raise the defaults to 4 retries, 15s delay and a 0.2 jitter, so the waits
  are 15s, 30s, 60s and 120s instead of 5s, 10s and 20s.
* Add the jitter, to keep concurrent aggregations from retrying in lockstep
  against a rate limiting remote, only to get limited together again.
* Add --retry-max, --retry-delay, --retry-backoff-factor and --retry-jitter,
  taking precedence over the `retry` section of the configuration file. It
  was only configurable per repository in the yaml file, which is not
  practical in a CI environment with a hundred of them.
* Merge the `retry` section with the defaults, so setting one parameter no
  longer drops the others, and raise a ConfigException on unknown ones.
* Document all of it in the README, including the fact that the merge check
  done while loading the configuration file is not retried on purpose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hugosantosred
hugosantosred merged commit 93280bc into master Aug 31, 2026
0 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant