Skip to content

fix: return true when robots.isAllowed returns undefined#2439

Merged
janbuchar merged 2 commits intoapify:masterfrom
Sajito:master
May 3, 2024
Merged

fix: return true when robots.isAllowed returns undefined#2439
janbuchar merged 2 commits intoapify:masterfrom
Sajito:master

Conversation

@Sajito
Copy link
Contributor

@Sajito Sajito commented Apr 30, 2024

undefined means that there is no explicit rule for the requested route. No rules means no disallow, therefore it's allowed.

Fixes #2437

`undefined` means that there is no explicit rule for the requested route.
No rules means no disallow, therefore it's allowed.
@janbuchar janbuchar self-assigned this May 2, 2024
Co-authored-by: Jan Buchar <Teyras@gmail.com>
@janbuchar janbuchar self-requested a review May 3, 2024 06:54
Copy link
Contributor

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, thank you very much for the code!

@janbuchar janbuchar merged commit 6f541f8 into apify:master May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RobotsFile.isAllowed returns false for allowed routes

2 participants