isDisallowed() returns true for / despite no matching Disallow rule in robots.txt

[seveun]

Hi,

I'm encountering a possible issue with the way isDisallowed() behaves when parsing the robots.txt file from https://www.natureetdecouvertes.com/robots.txt.

Context
I'm checking if crawling the homepage / is allowed for a standard browser User-Agent like:

swift
Copier
Modifier
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36
When I run:

js
Copier
Modifier
const isAllowed = robotsTxt.isAllowed('https://www.natureetdecouvertes.com/', userAgent);
const isDisallowed = robotsTxt.isDisallowed('https://www.natureetdecouvertes.com/', userAgent);
I get:

js
Copier
Modifier
isAllowed === undefined
isDisallowed === true
However, the robots.txt does not contain any rule explicitly disallowing /. There is no Disallow: /, and the default behavior according to the Robots Exclusion Protocol (RFC 9309) is to allow access to / unless explicitly blocked.

Expected behavior
isDisallowed('/') should return false, and isAllowed('/') should return true (or at least not undefined if a fallback to User-agent: * applies).

Notes
The user-agent I’m testing is not listed in any specific User-agent: group, so the fallback to User-agent: * should apply.

There is no Disallow: / or any wildcard rule that matches only /.

Can you confirm if this is intended behavior? Otherwise, it seems like a parsing bug or fallback handling issue.

Thanks!

[samclarke]

Thanks for reporting!

I've just tried to reproduce this but it's giving the expected result. Could you see if I'm doing anything differently:

const contents = '...robots.txt contents...';
const userAgent = 'SomeBot';
const robots = robotsParser('https://www.natureetdecouvertes.com/robots.txt', contents);

const isAllowed = robots.isAllowed('https://www.natureetdecouvertes.com/', userAgent);
const isDisallowed = robots.isDisallowed('https://www.natureetdecouvertes.com/', userAgent);

console.log(`IsAllowed: ${isAllowed}`); // outputs true
console.log(`IsDisallowed: ${isDisallowed}`); // outputs false

Working demo: https://jsbin.com/yakeruduru/edit?html,console

Only thing I can think of, is the host and port of the URL being passed to robotsParser() the same as the ones being checked? If not, isAllowed() will return undefined as the robots.txt file won't apply to the given URL.

[AnandChowdhary]

I reproduced the isAllowed() === undefined / isDisallowed() === true inconsistency for invalid/out-of-scope URLs and opened #45 to make isDisallowed() preserve the same undefined result instead of coercing it to true.