This code can help you parse important for you content from different sites you want.
With the main_url file, you can extract all the necessary internal and external links from the desired website and get them in a .txt file in the current directory. And with the main file, you can simply copy the links and get all the necessary text content in the form of .txt files in the current directory.