system diagnostics tool

[razorxl]

thank you guys for this great tool! I have seen the latest update about the doctor feature, just wondering how to use it, I can't find the example or tutorial in nowhere. my application becoming really slow after comprehensive crawling. this would help. thx verymuch!

[unclecode]

@razorxl Thanks for your kind words. May I know please what "Docker features" you are referring to? And for your information, the new version, 0.3.74, the scraping process became very efficient - it's around 100 milliseconds. Above that, there are two important components that came to the game (this year): one is an executor, which is very efficient to use whatever resources are available adaptively to crawl multiple URLs at the same time. The second one is a scraper that gets the website and uses graph search algorithms and extracts everything to all the layers of that website. If you give me a little bit more information about the way you do it, I may be able to help you reach the efficiency that you want. Thanks again.

[aravindkarnam]

@razorxl Closing this issue due to inactivity. I believe you are talking about crawl4ai-doctor command in cli. Is it? If yes, please start this discussion in Forums