robotparser reads empty robots.txt file as "all denied"

[larsfuse]

BPO	35457
Nosy	@terryjreedy, @berkerpeksag, @tirkarthi, @andreburgaud

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-12-11.09:30:47.872>
labels = ['3.8', 'type-feature', 'library']
title = 'robotparser reads empty robots.txt file as "all denied"'
updated_at = <Date 2020-01-02.15:45:55.440>
user = 'https://bugs.python.org/larsfuse'

bugs.python.org fields:

activity = <Date 2020-01-02.15:45:55.440>
actor = 'gallicrooster'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-12-11.09:30:47.872>
creator = 'larsfuse'
dependencies = []
files = []
hgrepos = []
issue_num = 35457
keywords = []
message_count = 6.0
messages = ['331595', '331870', '331963', '359180', '359185', '359202']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'berker.peksag', 'xtreak', 'larsfuse', 'gallicrooster']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35457'
versions = ['Python 3.8']

Linked PRs

• gh-79638: Treat an unreachable robots.txt as "disallow all" #138555
• [3.14] gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555) #149512
• [3.13] gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555) #149513
• gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555) #149541
• gh-79638: Test other HTTP error codes besides 403 in test_robotparser #149569
• [3.15] gh-79638: Test other HTTP error codes besides 403 in test_robotparser (GH-149569) #149580
• [3.14] gh-79638: Test other HTTP error codes besides 403 in test_robotparser (GH-149569) #149581
• [3.13] gh-79638: Test other HTTP error codes besides 403 in test_robotparser (GH-149569) #149582
• gh-79638: Restore "Treat an unreachable robots.txt as disallow all" #152525
• [3.15] gh-79638: Restore "Treat an unreachable robots.txt as disallow all" (GH-152525) #153107
• [3.14] gh-79638: Restore "Treat an unreachable robots.txt as disallow all" (GH-152525) #153108
• [3.13] gh-79638: Restore "Treat an unreachable robots.txt as disallow all" (GH-152525) #153109

[larsfuse]

The standard (http://www.robotstxt.org/robotstxt.html) says:

To allow all robots complete access:
User-agent: *
Disallow:
(or just create an empty "/robots.txt" file, or don't use one at all)

Here I give python an empty file:
$ curl http://10.223.68.186/robots.txt
$

Code:

rp = robotparser.RobotFileParser()
print (robotsurl)
rp.set_url(robotsurl)
rp.read()
print( "fetch /", rp.can_fetch(useragent = "*", url = "/"))
print( "fetch /admin", rp.can_fetch(useragent = "*", url = "/admin"))

Result:

$ ./test.py
http://10.223.68.186/robots.txt
('fetch /', False)
('fetch /admin', False)

And the result is, robotparser thinks the site is blocked.

[terryjreedy]

https://docs.python.org/2.7/library/robotparser.html#module-robotparser
and
https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
refers users, for file structure, to http://www.robotstxt.org/orig.html.
This says nothing about the effect of an empty file, so I don't see this as a bug. Even if it was, I would be dubious about reversing the effect without a deprecation notice first, and definitely not in 2.7.

I would propose instead that the doc be changed to refer to the new file, with more and better examples, but add a note that robotparser interprets empty files as 'block all' rather than 'allow all'.

Try bringing this up on python-ideas.

[larsfuse]

(...) refers users, for file structure, to http://www.robotstxt.org/orig.html. This says nothing about the effect of an empty file, so I don't see this as a bug.

That is incorrect. From that url you can find:

The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.

So this is definitely a bug.

[andreburgaud]

Hi,

Is this ticket still relevant for Python 3.8?

While running some tests with an empty robotstxt file I realized that it was returning "ALLOWED" for any path (as per the current draft of the Robots Exclusion Protocol: https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1 ")

Code:

from urllib import robotparser

robots_url = "file:///tmp/empty.txt"

rp = robotparser.RobotFileParser()
print(robots_url)
rp.set_url(robots_url)
rp.read()
print( "fetch /", rp.can_fetch(useragent = "*", url = "/"))
print( "fetch /admin", rp.can_fetch(useragent = "*", url = "/admin"))

Output:

$ cat /tmp/empty.txt
$ python -V
Python 3.8.1
$ python test_robot3.py
file:///tmp/empty.txt
fetch / True
fetch /admin True

[tirkarthi]

There is a behavior change. parse() sets the modified time and unless the modified time is set the can_fetch method returns false. In Python 2 the parse method was called only when the file is non-empty [0] but in Python 3 it's always called though the file is empty [1] . The change was done with 1afc169 to always read parse and then in 122541b modified function was always called during parse thus setting the modified_time to return True from can_fetch in the end.

I think the behavior of robotparser for empty file was undefined allowing these changes and it will be good to have a test for this behavior.

[0]

cpython/Lib/robotparser.py

Lines 66 to 67 in f82e59a

	elif self.errcode == 200 and lines:
	self.parse(lines)

[1]

cpython/Lib/urllib/robotparser.py

Lines 69 to 70 in 149175c

	raw = f.read()
	self.parse(raw.decode("utf-8").splitlines())

[andreburgaud]

Thanks @XTreak for providing some clarification on this behavior! I can write some tests to cover this behavior, assuming that we agree that an empty file means "unlimited access". This was worded as such in the old internet draft from 1996 (section 3.2.1 in https://www.robotstxt.org/norobots-rfc.txt). The current draft is more ambiguous with "If no group satisfies either condition, or no groups are present at all, no rules apply." https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1

https://www.robotstxt.org/robotstxt.html clearly states that an empty file gives full access, but I'm getting lost in figuring out which is the official spec at the moment :-)