[IMP] Make the network retries configurable, with jitter - #3
Merged
Merged
Conversation
GitHub rate limits the unauthenticated git operations per IP address. Once the limit is reached it answers with an HTTP 401, which git reports as "could not read Username for 'https://github.com'". A CI job aggregating ~100 repositories from a shared runner hits it regularly, and the 3 retries within 35s of the previous defaults were not enough to get past it. * Raise the defaults to 4 retries, 15s delay and a 0.2 jitter, so the waits are 15s, 30s, 60s and 120s instead of 5s, 10s and 20s. * Add the jitter, to keep concurrent aggregations from retrying in lockstep against a rate limiting remote, only to get limited together again. * Add --retry-max, --retry-delay, --retry-backoff-factor and --retry-jitter, taking precedence over the `retry` section of the configuration file. It was only configurable per repository in the yaml file, which is not practical in a CI environment with a hundred of them. * Merge the `retry` section with the defaults, so setting one parameter no longer drops the others, and raise a ConfigException on unknown ones. * Document all of it in the README, including the fact that the merge check done while loading the configuration file is not retried on purpose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub rate limits the unauthenticated git operations per IP address. Once the limit is reached it answers with an HTTP 401, which git reports as "could not read Username for 'https://github.com'". A CI job aggregating ~100 repositories from a shared runner hits it regularly, and the 3 retries within 35s of the previous defaults were not enough to get past it.
retrysection of the configuration file. It was only configurable per repository in the yaml file, which is not practical in a CI environment with a hundred of them.retrysection with the defaults, so setting one parameter no longer drops the others, and raise a ConfigException on unknown ones.