Current crawl speed is: # items/min

[jdubs1994]

Hey, newer to elixir and very new to Crawly. I'm trying to figure out what's going on. I have all the crawling logic working and I have the data I want to save to an Ecto Model(currently the variable head is the data I'd be saving). I'm just kinda confused on why Crawly keeps outputting the current crawl speed. Is this process supposed to end at some point or is it crawling every 1 minute? I don't know why it's logging crawl speed each minute, but it only outputs what I need once? Do I need to kill the process or something? Ideally, id like to crawl the site every x amount of time. I tried starting the same process again but it errors saying spider already started. So if the spider is already started and the process never ends, how do I get updated data from scraping the website again?

Thanks in advance!

  @impl Crawly.Spider
  def parse_item(response) do
     {:ok, document} = Floki.parse_document(response.body)

     post_links = document
     |> Floki.find("a.title")
     |> Floki.attribute("href")
     [head | _tail] = post_links
     IO.puts(head)


   %Crawly.ParsedItem{:items => [head], :requests => []}
  end

[nuno84]

Probably very late, but for future record, you must stop the crawler or set a limit for it to stop using for instance:
closespider_timeout or closespider_itemcount
More info here:
configuration