Skip to content

Sending patterns and urls to the instance does not mean they will be crawled? #87

Open
@scaraliu

Description

@scaraliu

When sending the allowed patterns and the URL list to the instance/index, does not start the crawl process on that list. I had to use the OpenSearchServer\Crawler\Web\Crawl() request to actually start the crawl process on that list of URLs. Just injecting the urls and setting the allowed patterns doesn't guarantee that those URL will be processed as configured in the running index.

I thought just by injecting urls and setting them as allowed patterns, the instance will process them.
So, to index some urls I have to do as follows:

  • Inject url list OpenSearchServer\Document\Put();
  • Inject those URLs as allowed path/s OpenSearchServer\Crawler\Web\Patterns\Inclusion\Insert();
  • and also to do a crawl request on each URL OpenSearchServer\Crawler\Web\Crawl();

Am I missing something?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions