Debugging tools

[mcshaz]

First thanks for developing what seems quite an intuitive and clean API

I am trying to scrape a page just so I can look for a second hand car with a few more tools at my disposal for searching than those offered by the web UI. However the application keeps freezing after logging:

 Base url:  https://www.carsales.com.au/cars/used [...]

I can scrape the demo page, and have tried with and without a ScrollToEnd action, but to no avail.

Are there tools for digging deaper - e.g. outputting what headers & html the http response contains?

The full code with various stuff commented out temporarily in order to see if it would work is

    var address = "https://www.carsales.com.au/cars/used/toyota/corolla/queensland-state/hatch-bodystyle/";

    var engine = await new ScraperEngineBuilder()
        .GetWithBrowser(new[] { address }, actions => actions
            .ScrollToEnd()
            .Build())
        //.Paginate(".page-link.next", ".pagination")
        .FollowWithBrowser(".js-encode-search")
        .Parse(new()
        {
            new("title", ".details-title h1"),
            /* new("info", ".csn-badge-title"),
            new("price", ".price"),
            new("odometer", ".key-details-item:nth-child(1) .key-details-item-title"),
            new("transmission", ".key-details-item:nth-child(3) .key-details-item-title"),
            new("powertrain", ".key-details-item:nth-child(4) .key-details-item-title"),
            new("location", ".profile-content .subBodyCls") */
        })
        .WriteToCsvFile(Path.Combine(SpecialDirectories.MyDocuments, "output.csv"), true)
        // .PageCrawlLimit(10)
        // .WithParallelismDegree(30)
        .LogToConsole()
        .BuildAsync();
    await engine.RunAsync();

Thank you

[mcshaz]

Additional - playing in node, I found pupateer will basically only open the page if it is not in headless mode. Is there a way to set this default in webreaper?