Browser breaks with new tab on empty content response

[d-balaskas]

Hello world!

I am building a common Spider that crawls sites and the contained request.
I use scrapy-playwright to load websites first and get the requests that are sent.

I noticed that when I parse urls that have no content on body the execution freezes and playwright's browser shows empty tab.
To be clear reproduction of the problem is when parse a url that has the following condition as true:

response_body_text = await response.text()
response_body_text  == ''

For the urls that this condition is false spider works perfectly!

For the reproduction, I have a quite common configuration with:

CrawlerProcess({
        ...
        # Playwright settings
        'DOWNLOAD_HANDLERS': {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        'TWISTED_REACTOR': "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        'PLAYWRIGHT_BROWSER_TYPE': 'chromium',
        'PLAYWRIGHT_MAX_PAGES_PER_CONTEXT': 10,
        'PLAYWRIGHT_LAUNCH_OPTIONS': {
            'headless': True,
        }        
})

and on each scrapy.Request() I pass the following meta:

{
            "playwright": True
}

Has anybody else come up with this issue?

Thank you all!

[elacuesta]

I don't know how to attempt to reproduce. I assume response is a Playwright response given Response.text (Scrapy's TextResponse.text is an attribute, not a coroutine), however Playwright responses are not directly exposed by scrapy-playwright. Are you maybe connecting to the response event with playwright_page_event_handlers?

Please share code that causes the issue, not only configuration settings. Also include logs that show the error.

[d-balaskas]

The 'await response.text()' was in a response handler that used to try catch the error but removed the whole handler.
Helped me identify that if avoid parsing this type of urls script would finish normally.

A sample reproduction is the following. It is required to fill the start_urls with a request which will have empty response and status 200.

import scrapy

from scrapy.spiders import SitemapSpider
from scrapy.crawler import CrawlerProcess
from scrapy.http import Response
from typing import Any


class MySpider(SitemapSpider):
    name = "MySpider"
    forbiden_calls = set()

    def __init__(self, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.start_urls = []
        self.scrapy_meta = {
            "playwright": True
        }

    def start_requests(self):
        for url in self.start_urls:
            print("Start URL", url)
            yield scrapy.Request(url, self.parse, errback=self.onError, meta=self.scrapy_meta)

    def parse(self, response: Response) -> Any:
        print("Parsing", response.url, response.body, response.status)
                
    def onError(self, failure):
        print(f"Failed to response parsing {failure.request.url}: {failure.value}")

def main():
    process = CrawlerProcess({
        'DOWNLOAD_HANDLERS': {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        'PLAYWRIGHT_LAUNCH_OPTIONS': {
            'headless': False,
        }        
    })

    process.crawl(MySpider)
    process.start()

if __name__ == '__main__':
    main()

[elacuesta]

I cannot reproduce.

$ scrapy version -v
Scrapy       : 2.11.1
lxml         : 5.1.0.0
libxml2      : 2.12.3
cssselect    : 1.2.0
parsel       : 1.8.1
w3lib        : 2.1.2
Twisted      : 23.10.0
Python       : 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
pyOpenSSL    : 24.0.0 (OpenSSL 3.2.1 30 Jan 2024)
cryptography : 42.0.5
Platform     : Linux-6.5.0-35-generic-x86_64-with-glibc2.35

$ python -c "import scrapy_playwright; print(scrapy_playwright.__version__)"
0.0.37

$ playwright --version               
Version 1.44.0

Running a minimal Flask application that returns an empty 200 response:

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return ""

$ curl -v http://127.0.0.1:5000
*   Trying 127.0.0.1:5000...
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:5000
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: Werkzeug/3.0.3 Python/3.10.12
< Date: Thu, 04 Jul 2024 18:08:21 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: close
< 
* Closing connection 0

$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get("http://127.0.0.1:5000")
>>> r.text, r.status_code
('', 200)
>>> 

I modified your code in two ways:

• added "http://127.0.0.1:5000" to self.start_urls
• added "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor" to the crawler settings

This is the result (notice there's some additional wrapping HTML, this is expected because the Scrapy response is the result of Page.content instead of Response.text):

...
2024-07-04 14:28:55 [scrapy.core.engine] INFO: Spider opened
2024-07-04 14:28:55 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-04 14:28:55 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2024-07-04 14:28:55 [scrapy-playwright] INFO: Starting download handler
2024-07-04 14:28:55 [scrapy-playwright] INFO: Starting download handler
Start URL http://127.0.0.1:5000/
2024-07-04 14:29:00 [scrapy-playwright] INFO: Launching browser chromium
2024-07-04 14:29:00 [scrapy-playwright] INFO: Browser chromium launched
2024-07-04 14:29:00 [scrapy-playwright] DEBUG: Browser context started: 'default' (persistent=False, remote=False)
2024-07-04 14:29:00 [scrapy-playwright] DEBUG: [Context=default] New page created, page count is 1 (1 for all contexts)
2024-07-04 14:29:00 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://127.0.0.1:5000/> (resource type: document)
2024-07-04 14:29:00 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://127.0.0.1:5000/>
2024-07-04 14:29:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://127.0.0.1:5000/> (referer: None) ['playwright']
Parsing http://127.0.0.1:5000/ b'<html><head></head><body></body></html>' 200
2024-07-04 14:29:01 [scrapy.core.engine] INFO: Closing spider (finished)
...

Given that the TWISTED_REACTOR was not set I suspect you're running on Windows, is this correct? Please share your software versions as explained here.

[d-balaskas]

Thank you for your cooperation and the detailed answer. I appreciate it.

Your reproduction works on my example too, probably the endpoint dont return the same format. I have already tested the implementation with TWISTED_REACTOR and without but always get the same result. Freezing browser.

By the way the output of curl on the url that trying to crawl is:

StatusCode        : 204
StatusDescription : No Content
Content           : {}
RawContent        : HTTP/1.1 204 No Content
                              Connection: keep-alive
                              ...
RawContentLength  : 0

Scrapy logs don't show any information but stop on

DEBUG: [Context=default] Response: <204 ...>

[elacuesta]

I see, the 204 status changes things. Running standalone Playwright and pointing to https://httpbin.org/status/204:

import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as pw:
        browser = await pw.chromium.launch()
        page = await browser.new_page()
        # same behavior for all values of "wait_until"
        await page.goto("https://httpbin.org/status/204", wait_until="networkidle")
        await browser.close()

asyncio.run(main())

I get this:

playwright._impl._errors.Error: Page.goto: net::ERR_ABORTED at https://httpbin.org/status/204
Call log:
navigating to "https://httpbin.org/status/204", waiting until "networkidle"

This is causing this condition (added in #228 to identify binary downloads) to be False, so the exception is not raised and the spider blocks right after, waiting indefinitely for a download that never happens.

I'll need to determine a better way to identify the download scenario so it doesn't interfere with this. For now I'd recommend you to switch to firefox or webkit as a possible workaround, though knowing that the request will still fail and trigger the errback: this is expected as it also happens with standalone Playwright.

[elacuesta]

For the record, I've opened microsoft/playwright#31570 upstream, I would not expect the browser to fail on 204 responses.