search volume + advanced serps data

Author: diije

README.md

@@ -48,46 +48,101 @@ Remember to switch it back when things get real ;)
 
 ## Usage
 
-### 1. Post tasks
+### SERPs
+
+Get results from Google on a list of keywords.  
+
+#### 1. Post tasks
 
 Put the list of keywords in a text file (one request per line), then run:  
 
-    $ python tasks_post.py --input yourfile.txt  
+    $ python serps_post.py --input yourfile.txt  
 
 The script will send the keywords to the API by batches of 100 requests, and write a report containing the `task_id` for each keyword.  
 
 The `input` argument is required. Other options are available:  
 - `config`: configuration file to use (default: `config.ini`)  
 - `input`: input file with the list of requests (required)  
-- `output`: output basename for the report (default: `keyword-requests`)  
+- `output`: output basename for the report (default: `serps-tasks`)  
 - `language_code`: language code for the requests (default: `fr`)  
 - `location_code`: location code for requests (default: "2250" for France, get other codes on <https://api.dataforseo.com/v3/serp/google/locations>)  
 - `nb_results`: number of results to ask for (default: `10`)  
 - `device`: choice between `desktop` (default) and `mobile`  
+- `pixels`: activate to get the number of pixels from the top of the page to the result (this will double requests costs). You'll need to use the `--advanced` option when getting the results.  
 - `priority`: choose the priority queue between `low` (default) and `high` (note: `high` priority requests will be charged more by DataForSEO)  
 - `batch`: number of requests to send for each batch, between 1 and 100 (default: `100`)  
 - `delay`: delay in seconds between batches of requests (default: `10`)  
 - `sep`: CSV separator for `output`(default: `;`)  
 
 
-### 2. See if tasks are ready
+#### 2. See if tasks are ready
 
 This step is not required, but will show you how many of your requests have been processed and are available for download.  
 Simply run:  
 
-    $ python tasks_ready.py  
+    $ python serps_ready.py  
 
 Note that the API will only show you 1000 available results at most, but there might be more not shown.  
 
-### 3. Get results
+#### 3. Get results
 
 Once you've posted your requests, it usually takes only a few instants for the API to process.  
 To get the results, run:  
 
-    $ python tasks_get.py  
+    $ python serps_get.py  
 
 The script will request the API to check for available results, then download the data for each done task and repeat while there are available results.  
 
+Available options:  
+- `config`: configuration file to use (default: `config.ini`)  
+- `output`: output basename for the results (default: `serps-results`)  
+- `advanced`: get advanced details, such as presence of image, video, ratings, sitelinks, ... or position of the result in pixels (default: `False`).  
+- `delay`: delay in seconds between batches of requests (default: `10`)  
+
+
+### Keywords
+
+Get keywords data from Google AdWords API: search volume, cpc, competition, categories.  
+
+#### 1. Post tasks  
+
+Put the list of keywords in a text file (one request per line), then run:  
+
+    $ python keywords_data_post.py --input yourfile.txt  
+
+The script will send the keywords to the API by batches of 100 requests, and write a report containing the `task_id` for each keyword.  
+
+The `input` argument is required. Other options are available:  
+- `config`: configuration file to use (default: `config.ini`)  
+- `input`: input file with the list of requests (required)  
+- `output`: output basename for the report (default: `keywords-data-tasks`)  
+- `language_code`: language code for the requests (default: `fr`)  
+- `location_code`: location code for requests (default: "2250" for France, get other codes on <https://api.dataforseo.com/v3/keywords_data/google/locations>)  
+- `batch`: number of keywords to include for each batch, between 1 and 700 (default: `700`). You will be charged the same amount per batch, no matter what number of keywords you include (so better stay with 700!).  
+- `delay`: delay in seconds between batches of requests (default: `1`)  
+- `sep`: CSV separator for `output`(default: `;`)  
+
+#### 2. See if tasks are ready  
+
+This step is not required, but will show you how many of your requests have been processed and are available for download.  
+Simply run:  
+
+    $ python keywords_data_ready.py  
+
+#### 3. Get results  
+
+Once you've posted your requests, it usually takes only a few instants for the API to process.  
+To get the results, run:  
+
+    $ python keywords_data_get.py  
+
+The script will request the API to check for available results, then download the data for each done task and repeat while there are available results.  
+
+Available options:  
+- `config`: configuration file to use (default: `config.ini`)  
+- `output`: output basename for the results (default: `keywords-data-results`)  
+- `delay`: delay in seconds between batches of requests (default: `10`)  
+
 ## Contributing
 
 If you wish to contribute to this repository or to report an issue, please do this [on GitLab](https://gitlab.com/databulle/dataforseo-serp-api-python-client).  

keywords_data_get.py

@@ -0,0 +1,86 @@
+####
+## DATAFORSEO GOOGLE KEYWORDS API
+##
+## Fetches available results from API.
+####
+
+import csv
+import configparser
+import argparse
+import time
+import datetime
+from client import RestClient
+import json
+
+if __name__ == '__main__': 
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', default="config.ini",
+            type=str, help='Global config file (default: "config.ini")')
+    parser.add_argument('--output', default="keywords-data-results",
+            type=str, help='Output basename (default: "keywords-data-results")')
+    parser.add_argument('--delay', default=1,
+            type=float, help='Delay in seconds between batches of requests (default: 1)')
+    args = parser.parse_args()
+
+    conf = configparser.ConfigParser()
+    conf.read(args.config)
+    user = conf['general']['user']
+    password = conf['general']['password']
+
+    # Output headers
+    fields=['keyword','location_code','language_code','search_partners','search_volume','cpc','competition','categories','monthly_searches']
+    # Output name
+    timestr = time.strftime("%Y%m%d-%H%M%S")
+    tag = args.output + "-" + timestr
+    filename = tag + ".csv"
+
+    with open(filename,'w',newline='') as file:
+        writer = csv.DictWriter(file, fieldnames=fields, delimiter=";")
+        writer.writeheader()
+        file.close()
+
+    client = RestClient(user,password)
+
+    # While there are results, request the next batch
+    next_batch = True
+    while next_batch:
+        response = client.get("/v3/keywords_data/google/search_volume/tasks_ready")
+        if response['status_code'] == 20000:    
+            tasks_available = response["tasks"][0]["result_count"]
+            print("{} tasks available".format(tasks_available))
+            if tasks_available < 1:
+                next_batch = False
+
+            for task in response["tasks"]:
+                if (task['result'] and (len(task['result']) > 0)):
+                    for result_task_info in task['result']:
+                        if(result_task_info['endpoint']):
+                            res = client.get(result_task_info['endpoint'])
+                            
+                            for t in res["tasks"]:
+                                if (t['result'] and (len(t['result']) > 0)):
+                                    for k in t['result']:
+                                        data = dict()
+                                        data["keyword"] = k["keyword"]
+                                        data["location_code"] = k["location_code"]
+                                        data["language_code"] = k["language_code"]
+                                        data["search_partners"] = k["search_partners"]
+                                        data["search_volume"] = k["search_volume"]
+                                        data["cpc"] = k["cpc"]
+                                        data["competition"] = k["competition"]
+                                        data["categories"] = json.dumps(k["categories"])
+                                        monthly_searches = []
+                                        for m in sorted(k["monthly_searches"], key=lambda x: (x['year'], x['month'])):
+                                            monthly_searches. append(m["search_volume"])
+                                        data["monthly_searches"] = json.dumps(monthly_searches)
+                                        
+                                        with open(filename,'a',newline='') as file:
+                                            writer = csv.DictWriter(file, fieldnames=fields, delimiter=";")
+                                            writer.writerow(data)
+                                            file.close()
+                            
+            print("Batch done.")
+            time.sleep(args.delay)
+        else:
+            next_batch = False
+            print("error. Code: %d Message: %s" % (response["status_code"], response["status_message"])) 

keywords_data_post.py

@@ -0,0 +1,105 @@
+####
+## DATAFORSEO GOOGLE KEYWORDS API
+##
+## Posts keywords search volume requests.  
+####
+
+import csv
+import configparser
+import argparse
+import time
+import datetime
+from client import RestClient
+
+def range_limited_int_type(arg):
+    """ Type function for argparse - an int within some predefined bounds """
+    try:
+        f = int(arg)
+    except ValueError:    
+        raise argparse.ArgumentTypeError("Must be an integer")
+    if f < 1 or f > 700:
+        raise argparse.ArgumentTypeError("Argument must be < " + str(1) + " and > " + str(700))
+    return f
+
+if __name__ == '__main__': 
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', default="config.ini",
+            type=str, help='Global config file (default: "config.ini").')
+    parser.add_argument('--input', required=True,
+            type=str, help='List of keywords to request.')
+    parser.add_argument('--output', default="keywords-data-tasks",
+            type=str, help='Output basename (default: "keywords-data-tasks").')
+    parser.add_argument('--language_code', default="fr",
+            type=str, help='Language code for requests (default: "fr")')
+    parser.add_argument('--location_code', default=2250,
+            type=int, help='Location code for requests (default: "2250" for France, get other codes on <https://api.dataforseo.com/v3/serp/google/locations>)')
+    parser.add_argument('--batch', default=700,
+            type=range_limited_int_type, help='Max number of tasks per batch. Max 700. Each batch costs 0.05$ (default: 700).')
+    parser.add_argument('--delay', default=1,
+            type=float, help='Delay in seconds between batches of requests (default: 1).')
+    parser.add_argument('--sep', default=";",
+            type=str, help='CSV file separator (default: ";").')
+    args = parser.parse_args()
+
+
+        # Read list of requests
+    with open(args.input,'r') as file:
+        kws = list()
+        for line in file.readlines():
+            kws.append(str.strip(line))
+        file.close()
+    # Output headers
+    fields=['id','status','tag','nb_requests','first_kw','last_kw']
+    # Output name
+    timestr = time.strftime("%Y%m%d-%H%M%S")
+    tag = args.output + "-" + timestr
+    filename = tag + ".csv"
+
+    conf = configparser.ConfigParser()
+    conf.read(args.config)
+    user = conf['general']['user']
+    password = conf['general']['password']
+
+    with open(filename,'w',newline='') as file:
+        writer = csv.DictWriter(file,fieldnames=fields,delimiter=args.sep)
+        writer.writeheader()
+        file.close()
+
+        client = RestClient(user, password)
+        i = 0
+        j = args.batch
+
+        # Cut the kws list in batches
+        while j < len(kws)+args.batch:
+            post_data = {}
+            post_data[len(post_data)] = dict(
+                language_code = args.language_code,
+                location_code = args.location_code,
+                keywords = kws[i:j],
+                tag = tag,
+            )
+
+
+            response = client.post("/v3/keywords_data/google/search_volume/task_post", post_data)
+            if response["status_code"] == 20000:
+                for task in response["tasks"]:
+                    data = dict()
+                    data["nb_requests"] = len(task["data"]["keywords"])
+                    data["first_kw"] = task["data"]["keywords"][0]
+                    data["last_kw"] = task["data"]["keywords"][-1]
+                    data["status"] = task["status_message"]
+                    data["id"] = task["id"]
+                    data["tag"] = task["data"]["tag"]
+                    with open(filename,'a',newline='') as file:
+                        writer = csv.DictWriter(file,fieldnames=fields,delimiter=args.sep)
+                        writer.writerow(data)
+                        file.close()
+                print("Batch {} done.".format(int((i/args.batch) + 1)))
+            else:
+                print("error. Code: %d Message: %s" % (response["status_code"], response["status_message"]))
+
+            i = j
+            j += args.batch
+            time.sleep(args.delay)
+
+        file.close()

keywords_data_ready.py

@@ -0,0 +1,30 @@
+####
+## DATAFORSEO GOOGLE KEYWORDS API
+##
+## Shows number of tasks ready for download.
+####
+
+import configparser
+import argparse
+from client import RestClient
+
+
+if __name__ == '__main__': 
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', default="config.ini",
+            type=str, help='Global config file (default: "config.ini")')
+    args = parser.parse_args()
+
+    conf = configparser.ConfigParser()
+    conf.read(args.config)
+    user = conf['general']['user']
+    password = conf['general']['password']
+
+    client = RestClient(user,password)
+
+    response = client.get("/v3/keywords_data/google/search_volume/tasks_ready")
+    if response["status_code"] == 20000:
+        tasks_available = response["tasks"][0]["result_count"]
+        print("{} tasks available".format(tasks_available))
+    else:
+        print("error. Code: %d Message: %s" % (response["status_code"], response["status_message"]))