Skip to content

Fix crawl_result type hint #21

@Mews

Description

@Mews

Because of #19 , the type hint for Spider.crawl_result broke, and it was temporarily replaced with Dict[str, Dict[str, Any]].
This should be fixed to actually reflect the contents of crawl_result, which has the following format:

crawl_result = {
    "url1":{
        "urls":["some url", "some other url", ...],
        "body": "the html of the page"
    },
    "url2":{
        "urls":["some url", "some other url", ...],
        "body": "the html of the page"
    },
}

Where body is only present if the include_body argument is set to True, and as such might not always be present.
See #19 for previous discussions about this.
You can verify the type hint is working if the mypy checks pass.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions