Crawl requests and candidates

Crawl requests

Crawl requests are used to specify which URLs the crawler should visit.

Create an instance

Use:

• CrawlRequestBuilder: to create a customized crawl request
• createDefault: to create a crawl request with the default configuration

Set the priority of the request

The priority determines the order in which requests are processed. Requests with higher priority will be processed earlier. Negative values represent low priority.

Set the metadata associated with the request

This data can later be retrieved in the callbacks.

Crawl candidates

Crawl candidates represent crawl requests which have been processed by the crawler.