Python Web Graph

Wraps over graphviz, uses bs4 to parse and httpx for requests. Takes advantage of Pythons built in robots.txt parser. Very respectful to servers by default. Main file can easily be modified for experimentation.

How to Run

Clone repo into new folder:

git clone https://github.com/zachjesus/web-graph.git .

Set up virtual environment:

python3 -m venv venv

Active venv

source venv/bin/activate  # Linux / macOS
venv\Scripts\activate     # Windows

Install requirements

 pip install -r requirements.txt

Configure main.py, settings are graphviz's, see docs (or use default):

python3 main.py

Result will be placed in a folder called directory named "output" in the folder where the code is installed. By default, the web graph made graph is an svg image like the one above.