proxy.config.http.cache.open_read_retry_time default value impractical

[traeak]

The default value for proxy.config.http.cache.open_read_retry_time is 10ms.

While running synthetic load testing simulating a thundering herd against a rotating manifest file I added debug into HttpCacheSM to printout time when read retries occurred. On a broadwell xeon with nvme drives I was consistently seeing more than 900ms between these calls. However under these conditions I was seeing at most 2-3 calls being made before the read lock was acquired.

[bryancall]

We need more information to determine what we should do here. @traeak will investigate more.

[c-taylor]

I agree with this: 10ms default is at odds with other ATS defaults, and have seen this behaviour also.

Hypothesis:

• Tightly time coupled thundering herd will naturally cause read lock failures
• ATS read lock failures will reschedule (10ms default)
• I suspect 10ms + (reschedule delay) will have 'expected' effect

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.