Skip to content

Conversation

@Mews
Copy link
Collaborator

@Mews Mews commented Jun 16, 2024

This pr closes issue #13

Changes

  • Added url_regex argument to the Spider class.
  • Urls then get matched against this pattern inside the Spider.crawl method. If they don't match the pattern, they are skipped.
  • Also added a test case for this feature, in test_url_regex.

Please let me know if any changes need to be made to this pr.

@indrajithi
Copy link
Collaborator

@Mews Can you make a pull from master. The CI fails because of some outdated workflow in master.

@Mews
Copy link
Collaborator Author

Mews commented Jun 16, 2024

I don't think that was the issue, it was because of the import re line, it should be before requests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants