[3.14] gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555)

[3.14] gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555) #256199