scrapy-splash recursive crawl using CrawlSpider not working

[dijadev]

Hi !

I have integrated scrapy-splash in my CrawlSpider process_request in rules like this:

 def process_request(self,request):
        request.meta['splash']={
            'args': {
                # set rendering arguments here
                'html': 1,
            }
        }
        return request

The problem is that the crawl renders just urls in the first depth,
I wonder also how can I get response even with bad http code or redirected reponse;

Thanks in advance,

[wattsin]

I also have this issue.

NORMAL REQUEST - it will follow the rules and Follow=True
yield scrapy.Request(url, callback=self.parse, dont_filter=True, errback=self.errback_httpbin)

USING SPLASH - it will only visit the first url
yield scrapy.Request(url, callback=self.parse, dont_filter=True, errback=self.errback_httpbin, meta={
'splash': {
'endpoint': 'render.html',
'args': { 'wait': 0.5 }
}
})

[dijadev]

Has someone found the solution ?

[wattsin]

i have not. unfortunately  On Fri, Jan 27, 2017 at 1:10 PM -0500, "dijadev" <notifications@github.com> wrote: Has someone found the solution ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[amirj]

I have the same problem, any solution?

[wattsin]

Negative.

[brianherbert]

+1 over here. Encountering the same issue as described by @wattsin.

[dwj1324]

I also got the same issue here today and found that CrawlSpider do a response type check in _requests_to_follow function

def _requests_to_follow(self, response):
    if not isinstance(response, HtmlResponse):
        return
    ...

However responses generated by Splash would be SplashTextResponse or SplashJsonResponse. That check caused splash response won't have any requests to follow.

[komuher]

+1

+1

[hieu-n]

@dwj1324

I tried to debug my spider with PyCharm and set a breakpoint at if not isinstance(response, HtmlResponse):. That code was never reached when SplashRequest was used instead of scrapy.Request.

What worked for me is to add this to the callback parsing function:

def parse_item(self, response):
    """Parse response into item also create new requests."""

    page = RescrapItem()
    ...
    yield page

    if isinstance(response, (HtmlResponse, SplashTextResponse)):
        seen = set()
        for n, rule in enumerate(self._rules):
            links = [lnk for lnk in rule.link_extractor.extract_links(response)
                     if lnk not in seen]
            if links and rule.process_links:
                links = rule.process_links(links)
            for link in links:
                seen.add(link)
                r = SplashRequest(url=link.url, callback=self._response_downloaded, 
                                              args=SPLASH_RENDER_ARGS)
                r.meta.update(rule=rule, link_text=link.text)
                yield rule.process_request(r)

[NingLu]

+1, any update for this issue?

[NingLu]

@hieu-n i use the code you paste here, and change splash request to request since i need to use the header, but it doesn't work, the spider still crawl the first depth content, any suggestion will be appreciated

[hieu-n]

@NingLu I haven't touched scrapy for a while. In your case, what I would do is to set a few breakpoints and step through your code and the scrapy's code. Good luck!

[Goles]

+1 any updates here?