Skip to content

Scrapy template failure can pass tests #309

@Pijukatel

Description

@Pijukatel

In situation where scrapy template explodes and produces no results, template test passes. It is because test framework checks only for return code of template script and the scrapy code from template swallows all the exceptions. So even in case of such critical failure the template script returns status code 0

Adapt the test(or the template?) to make it fail is such scenarios.

Error example:

Unhandled error in Deferred:
[twisted] CRITICAL Unhandled error in Deferred:

Traceback (most recent call last):
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\twisted\internet\defer.py", line 2017, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\scrapy\crawler.py", line 154, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\twisted\internet\defer.py", line 2017, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\scrapy\core\engine.py", line 393, in open_spider
    if d := scheduler.open(spider):
  File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\apify\scrapy\scheduler.py", line 59, in open
    self._rq = nested_event_loop.run_until_complete(open_queue())
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 623, in run_until_complete
    self._check_running()
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 585, in _check_running
    raise RuntimeError(
builtins.RuntimeError: Cannot run the event loop while another loop is running

No results example:

[scrapy.statscollectors] INFO  Dumping Scrapy stats:
{'items_per_minute': None,
 'log_count/ERROR': 2,
 'log_count/INFO': 8,
 'log_count/WARNING': 1,
 'responses_per_minute': None} 

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions