[3.14] gh-79638: Restore "Treat an unreachable robots.txt as disallow all" (GH-152525)

[3.14] gh-79638: Restore "Treat an unreachable robots.txt as disallow all" (GH-152525) #275307