Investigate/test fetcher retries

[jku]

I'm expecting urllib3 to retry 3 times on common http failures when using urllib3fetcher

There's an interesting failure in sigstore-probers where this does not seem to happen:

Traceback (most recent call last):
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 754, in _error_catcher
 yield
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 879, in _raw_read
 data = self._fp_read(amt, read1=read1) if not fp_closed else b""
 ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 862, in _fp_read
 return self._fp.read(amt) if amt is not None else self._fp.read()
 ~~~~~~~~~~~~~^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/http/client.py", line 479, in read
 s = self.fp.read(amt)
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/socket.py", line 719, in readinto
 return self._sock.recv_into(b)
 ~~~~~~~~~~~~~~~~~~~~^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/ssl.py", line 1304, in recv_into
 return self.read(nbytes, buffer)
 ~~~~~~~~~^^^^^^^^^^^^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/ssl.py", line 1138, in read
 return self._sslobj.read(len, buffer)
 ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/opt/hostedtoolcache/Python/3.13.4/x64/bin/tuf-on-ci-test-client", line 8, in <module>
 sys.exit(client())
 ~~~~~~^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
 return self.main(*args, **kwargs)
 ~~~~~~~~~^^^^^^^^^^^^^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1082, in main
 rv = self.invoke(ctx)
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
 return ctx.invoke(self.callback, **ctx.params)
 ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 788, in invoke
 return __callback(*args, **kwargs)
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf_on_ci/client.py", line 84, in client
 updater.refresh()
 ~~~~~~~~~~~~~~~^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 170, in refresh
 self._load_timestamp()
 ~~~~~~~~~~~~~~~~~~~~^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 425, in _load_timestamp
 data = self._download_metadata(
 Timestamp.type, self.config.timestamp_max_length
 )
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 313, in _download_metadata
 return self._fetcher.download_bytes(url, length)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/fetcher.py", line 139, in download_bytes
 with self.download_file(url, max_length) as dl_file:
 ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/contextlib.py", line 141, in __enter__
 return next(self.gen)
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/fetcher.py", line 102, in download_file
 for chunk in chunks:
 ^^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/urllib3_fetcher.py", line 105, in _chunks
 yield from response.stream(self.chunk_size)
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 1066, in stream
 data = self.read(amt=amt, decode_content=decode_content)
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 955, in read
 data = self._raw_read(amt)
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 878, in _raw_read
 with self._error_catcher():
 ~~~~~~~~~~~~~~~~~~~^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/contextlib.py", line 162, in __exit__
 self.gen.throw(value)
 ~~~~~~~~~~~~~~^^^^^^^
 File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 759, in _error_catcher
 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e # type: ignore[arg-type]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='tuf-repo-cdn.sigstore.dev', port=443): Read timed out.

[jku]

possibly in the streaming case only the connection establishment uses retries and the actual streaming does not?

[ishaanxgupta]

Hi @jku can I work on this one?

[jku]

feel free if you have ideas -- my assumption above was that

• urllib3 does 3 retries by default if GET fails
• but it seems like in streaming mode this only applies to the initial connection: if a read fails mid-stream, there are no retries

ngclient does use streaming. A modest number of retries would be nice but I would also like to avoid adding complicated code like manually restarting failed requests: so it may also be that there is no useful improvement to make here.