go-bulkrequest

---

Description

go-bulkrequest is a simple library for fetching and parsing data from multiple URLs/endpoints.

Project is developed in go.

Every bulk request is highly configurable, for example:

• You can use proxy connection instead of direct connection, for every proxy, requests will be parallelized.
• You can set cookies/headers/user-agent for every request.
• You can provide custom parser for every request.

Installation

go get github.com/jodua/go-bulkrequest/bulkrequest

Usage

Importing library

import "github.com/jodua/go-bulkrequest/bulkrequest"

Creating bulk request builder

bulkRequest := bulkrequest.NewBulkRequest()

Customizing bulk request

Builder methods

SetBaseUrl - sets base URL for all requests.

Params:

• baseUrl string - baseUrl

bulkRequest.SetBaseUrl("https://example.com")

SetUrls - sets list of URL suffixes for all requests.

Params:

• urls []string - list of URL suffixes

Following example will create requests for URLs:

• Base URL + Urls[0]
• Base URL + Urls[1]
• ...
• Base URL + Urls[len(Urls)-1]

urlList := []string{"1", "2", "3", "4", "5"}
bulkRequest.SetUrls(urlList)

SetTimeout - sets timeout after which request will be aborted.

Params:

• timeout time.Duration - duration of timeout

bulkRequest.SetTimeout(time.Second * 10)

AddHeader - adds header to all requests.

Params:

• key string - header key
• value string - header value

bulkRequest.AddHeader("X-Header", "Value")

AddCookie - adds cookie to all requests.

Params:

• name string - header key
• value string - header value

bulkRequest.AddCookie("Cookie", "Value")

SetDelayConfig - sets delay configuration for all requests.

Params:

• delayConfig *datatypes.DelayConfig - delay configuration

Delay config is a struct that contains delay configuration for all requests. It consists of:

• DelayMin time.Duration - minimum delay between requests
• DelayMax time.Duration - maximum delay between requests

When fetching data from multiple URLs, requests will be delayed between DelayMin and DelayMax time.

import "github.com/jodua/go-bulkrequest/datatypes"

delayConfig := datatypes.DelayConfig{
    DelayMin: time.Second * 1,
    DelayMax: time.Second * 2,
}

bulkRequest.SetDelayConfig(&delayConfig)

SetProxyConfig - sets proxy configuration for all requests. If proxy configuration is not set, requests will be made directly.

Params:

• proxyConfig *datatypes.ProxyConfig - proxy configuration

Proxy config is a struct that contains proxy configuration for all requests. It consists of:

• ProxyList []string - list of proxies
• RequestsPerProxy - number of requests that will be sent through each proxy

import "github.com/jodua/go-bulkrequest/datatypes"

proxyList := []string{"http://proxy1:1231", "http://proxy2:1111"}

proxyConfig := datatypes.ProxyConfig{
    ProxyList:        proxyList,
    RequestsPerProxy: 5,
}

bulkRequest.SetProxyConfig(&delayConfig)

SetUserAgentConfig - sets user agent configuration for all requests.

Params:

• userAgentConfig *datatypes.UserAgentConfig - user agent configuration

User agent config is a struct that contains user agent configuration for all requests. It consists of:

• UserAgentList []string - list of user agents

import "github.com/jodua/go-bulkrequest/datatypes" 

userAgentList := []string{"UserAgent1", "UserAgent2"}

userAgentConfig := datatypes.UserAgentConfig{
    UserAgentList: userAgentList,
}

bulkRequest.SetUserAgentConfig(&userAgentConfig)

SetParser - sets parser for all requests.

Params:

• parser *jsonparser.JSONParser - pointer to parser object

Example parser can be found in github.com/jodua/go-bulkrequest/JSONParser/schemas package.

var parser = schemas.JSONPlaceholderTodoParser

bulkRequest.SetParser(&parser)

Building and executing bulk request

br := bulkRequest.Build()

fetch, err := br.Fetch()
if err != nil {
    // handle error
}
log.Println(fetch)

Full example can be found in main.go file.

Building JSON parser

JSONParser struct consists of:

• JSONSchema any - pointer to JSON schema object
• ConvertFunction func(any,string) any - function that converts JSON data to desired format, second parameter is request URL
• ValidatorFunction func(any) error - function that validates JSON data
• Output any - pointer to output struct that will be filled with data
• Name string - name of parser

Example parser can be found in github.com/jodua/go-bulkrequest/JSONParser/schemas package.

Issues

File issues through Issues tab.

License

MIT License