gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555)

[serhiy-storchaka]

Disallow all access in urllib.robotparser if the robots.txt file is unreachable due to server or network errors.

• Issue: robotparser reads empty robots.txt file as "all denied" #79638

[serhiy-storchaka]

I am puzzled. Tests on Android and iOS fail because they modify urllib.requests._opener. But there is an explicit cleanup code for it. The only difference is that it is now a class cleanup instead of a separate test cleanup, and the server and the thread are ran at the class level (for performance).

[serhiy-storchaka]

It turned out that the explicit check is not needed, because the current code passes the required tests. It may be needed in future, when we change the code (for example, to allow reloading of robots.txt).