This is a helper for complex and authenticated web scrapings that listens to all requests made by the browser while getting the content of a website. It makes it easier to view and navigate through the requests and responses made by the browser as well as the authentication process.
- Change the callback function in the
main.pyfile in order to navigate to the website page you want to scrape. - View the requests and responses made by the browser inside the txt files in
tmp/logsfolder, they follow the same order as they were made.
pip install selenium-wirepython main.py