|
1 | 1 | # LeetCode-AC-Code-Crawler
|
2 | 2 |
|
3 |
| -#### Origin |
4 |
| -I've been solving LeetCode problems recently, and when I realized that why not put my solutions on github, I found it time consuming to copy and paste hundreds of problems that I solved, so I decided to write a crawler instead. |
| 3 | +### Origin |
5 | 4 |
|
6 |
| -At the begining, I found a useful dependency in python called ```BeautifulSoup```, but it seems that it is not able to deal with dynamic website, which means that if some elements shown in a website is controlled by javascript. After serching for a while, I found ```selenium```, which automatically controll website like a human, but it seems that it is unefficient to use selenium to parse data in html, so I combine BeautifulSoup and selenium to finish this project. |
| 5 | +It is time consuming to copy and paste hundreds of problems that I solved on LeetCode, so I decided to write a crawler instead. |
| 6 | + |
| 7 | +At the begining, I found a useful dependency in python called ```BeautifulSoup```, but it seems that it is not able to deal with dynamic website, which means that if some elements shown in a website is controlled by javascript. After searching for a while, I found ```selenium```, which automatically controll website like a human, but it seems that it is unefficient to use selenium to parse data in html, so I combine BeautifulSoup and selenium to finish this project. |
| 8 | + |
| 9 | +### Install Dependencies |
| 10 | + |
| 11 | +``` |
| 12 | +# python setup.py install |
| 13 | +``` |
| 14 | +or |
7 | 15 |
|
8 |
| -#### Install Dependencies |
9 | 16 | ```
|
10 |
| -# pip install bs4 |
11 |
| -# pip install beautifulsoup |
12 |
| -# pip install selenium |
| 17 | +# python3 setup.py install |
13 | 18 | ```
|
14 | 19 |
|
15 |
| -#### Install ChromeDriver |
16 |
| -This project use [Chromedriver](http://chromedriver.chromium.org/) in selenium, before running this project, you have to add the downloaded executable file to PATH enviroment variable. |
| 20 | +### Install ChromeDriver |
| 21 | + |
| 22 | +This project use [Chromedriver](http://chromedriver.chromium.org/) in selenium, before running this project, you have to add the downloaded executable file to PATH enviroment variable, or if your python version is greater than 3.5, than you can enable chromedriver auto install by setting `true` in `["Chromedriver"]["AutoInstall"]` in `conf.json`. |
| 23 | + |
| 24 | +Note: [Chromedriver-autoinstaller](https://github.com/yeongbin-jo/python-chromedriver-autoinstaller) is a open source project, it is indeed convenient, but still some limitations, and if you having issue using it, maybe consider manually download Chromedriver by yourself. |
| 25 | + |
| 26 | +### Setting |
17 | 27 |
|
18 |
| -#### Setting |
19 | 28 | You have to modify variables in `conf.json`.
|
20 | 29 | ```json
|
21 | 30 | {
|
22 | 31 | "Username": "YOUR_USERNAME",
|
23 | 32 | "Password": "YOUR_PASSWORD",
|
24 | 33 | "OutputDir": "THE_ABSOLUTE_PATH_THAT_YOU_WANT_TO_SAVE_FILES",
|
25 |
| - "ChromedriverPath": "THE_ABSOLUTE_PATH_OF_CHROMEDRIVER", |
| 34 | + "Chromedriver": { |
| 35 | + "AutoInstall": false, |
| 36 | + "Path": "THE_ABSOLUTE_PATH_OF_CHROMEDRIVER" |
| 37 | + }, |
26 | 38 | "Headless": true,
|
27 | 39 | "Overwrite": false
|
28 | 40 | }
|
|
0 commit comments