RobotsFile.isAllowed returns false for allowed routes

[Sajito]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/utils

Issue description

Load a robots.txt containing only disallow rules and check any non matching url. It should return true, but returns false.

That is because the underlying robots-parser package returns undefined for urls, which are not part of the robots.txt. In the RobotsFile class undefined is converted to false which is wrong. robots.txt is used to define exclusion rules, therefore non matching urls should be allowed.

Either undefined should be converted to true or the wrapping method should also return undefined.

Code sample

const robots = `
User-agent: *
Disallow: /private
`;
const robots = RobotsFile.from('https://example.com', robots);

robots.isAllowed('https://example.com/allowed'); // returns false, should return true

Package version

3.9.2

Node.js version

v21.7.3

Operating system

No response

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[janbuchar]

Hello and thank you for the bug report. Since you already identified the cause, are you interested in submitting a PR?

[Sajito]

Sure, will do later. I'd go with returning undefined when the underlying solution returns undefined, as it would be less disruptive. Most people tend to do short checks like if (!robots.isAllowed(...)), therefore it wouldn't change existing behavior.

[janbuchar]

Sure, will do later. I'd go with returning undefined when the underlying solution returns undefined, as it would be less disruptive. Most people tend to do short checks like if (!robots.isAllowed(...)), therefore it wouldn't change existing behavior.

That sounds less than optimal to me - 1. it does not make the code sample in your bug report work and 2. it makes the return type harder to understand - does boolean | undefined mean true/false/don't know? Or something else?

If undefined means "robots.txt does not say anything about the URL", I'd say isAllowed should return true - just like you wrote in the code sample.