-
Notifications
You must be signed in to change notification settings - Fork 31
Open
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Description
In situation where scrapy template explodes and produces no results, template test passes. It is because test framework checks only for return code of template script and the scrapy code from template swallows all the exceptions. So even in case of such critical failure the template script returns status code 0
Adapt the test(or the template?) to make it fail is such scenarios.
Error example:
Unhandled error in Deferred:
[twisted] CRITICAL Unhandled error in Deferred:
Traceback (most recent call last):
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\twisted\internet\defer.py", line 2017, in _inlineCallbacks
result = context.run(gen.send, result)
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\scrapy\crawler.py", line 154, in crawl
yield self.engine.open_spider(self.spider, start_requests)
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\twisted\internet\defer.py", line 2017, in _inlineCallbacks
result = context.run(gen.send, result)
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\scrapy\core\engine.py", line 393, in open_spider
if d := scheduler.open(spider):
File "C:\Users\RUNNER~1\AppData\Local\Temp\python-scrapyz9IcFM\.venv\lib\site-packages\apify\scrapy\scheduler.py", line 59, in open
self._rq = nested_event_loop.run_until_complete(open_queue())
File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 623, in run_until_complete
self._check_running()
File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 585, in _check_running
raise RuntimeError(
builtins.RuntimeError: Cannot run the event loop while another loop is running
No results example:
[scrapy.statscollectors] INFO Dumping Scrapy stats:
{'items_per_minute': None,
'log_count/ERROR': 2,
'log_count/INFO': 8,
'log_count/WARNING': 1,
'responses_per_minute': None}
Metadata
Metadata
Assignees
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.