Replacing gotor with httpx and other major changes #307
+549
−814
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KingAkeem
added
New Feature
Breaking Change
Major Upgrade
Hacktoberfest
KingAkeem
changed the title
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request contains some major changes.
Many of the flags have been removed and incorporated into the tree so that they are always parsed. This improves performance since we can parse the existing HTML while going through nodes and reduces the number of flags needed to gather information.
These flags include:
-m(all emails are retrieved whenever a tree is created)-p(all phone numbers are retrieved whenever a tree is created)candcAll(classification is always done for all nodes)Some flags that were not previously being used have been removed.
These flags include:
eSome flags were redundant or not very useful.
These flags include:
--download(data can be saved as a JSON file or text tree with the--saveflag), other formats can be added to this flag.--gather(this is basically download and has been replaced by--saveas well)Some new flags have been added.
Includes:
v(turns on verbose logging which sets the logger todebuglevel)disable-socks5(constructs HTTP client without SOCKS5 proxy)gotorhas been completely removed and replaced withhttpx.None of the tests were working or actually being run, a new test suite needs to be developed. The code has been written in a more modular fashion which should help with this task.
Most of the code now has types added to the function definitions.
The version is now read directly from the
pyproject.tomlso it will update as the project gets updated usingpoetry.Multiple arguments can be used in tandem such as save, visualize or info now.