-
Notifications
You must be signed in to change notification settings - Fork 11
Open
Labels
Description
Because of #19 , the type hint for Spider.crawl_result broke, and it was temporarily replaced with Dict[str, Dict[str, Any]].
This should be fixed to actually reflect the contents of crawl_result, which has the following format:
crawl_result = {
"url1":{
"urls":["some url", "some other url", ...],
"body": "the html of the page"
},
"url2":{
"urls":["some url", "some other url", ...],
"body": "the html of the page"
},
}Where body is only present if the include_body argument is set to True, and as such might not always be present.
See #19 for previous discussions about this.
You can verify the type hint is working if the mypy checks pass.