Datagenerator is a Go program that generates UK customer data suitable for demonstration and testing. The key features are:
- Should run for any environment that supports Go + PostgreSQL - there are no other dependancies. Tested on Windows and Ubuntu servers
- PostgeSQL 9.6 back end
- Generates customers and optionally contacts and invoice level transactions
- Options set in simple text
inilike environment file - Generate anything from 1,000 to 500,000 customers
- Support for anything from a multi branch/region structure to a single national distribution point
- Up to 28 branches organised in up to 7 regions
- Postcodes return valid addreses so can be used with mapping applications
- Shape of data is correct - for example a Chester customer account will have a telephone number starting with the correct dial code. The data looks 'okay' when demonstrating
- Control over style and frequency of names generated by editing
company_type.csv - Control over ratio of trade to cash retail accounts
- Includes area, rep and industry type mappings
A quick browse through datagenerator.env shows what may be set or tweaked. The *.csv and *.txt files in the data directory are the source of the generated data.
There are ~50 settings in the datagenerator.env file. A subset of the settings are shown below:
table_prefix = dg
csv_prefix=
default_count =10000
data_directory = C:\\Users\\rcoll\\goProjects\\src\\github.com\\HamptonNorth\\datagenerator\\data\\
# uncomment ONE of the following
#target_structure=national
#head_office_branch_id = 1
#target_structure=regional
#region_id_list=1,2,3,4,5,6,7,8
target_structure=branch
branch_id_list=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28
#branch_id_list=1,2,3,4,5,6
If moving from Widows to Linux or visa versa, check the data directory pointed to in datagenaertor.env
On linux ensure path to *.csv and *.txt data files has suitable permissions for postgres. /tmp/something works fine as a data directory
When entering dates to PostgreSQL from *.csv or *.txt files, as convention, enter 17 Feb 2015 as 20150212 (This is the ISO 8601 concatenated date format)
On Windows ensure the data directory allows the PostgreSQL user to read/write files
Ensure the PostgreSQL extension ltree is enabled/installed. Execute CREATE EXTNSION ltree; or run query select COUNT(*) from pg_available_extensions WHERE name LIKE 'ltree%' which should give a count of 4
On i7 Windows 10 laptop, 16GB memeory with SSD - 10,000 customers, 15,000 Contacts and 280,000 aged debt transactions took ~55 seconds to generate.
On i5 Ubuntu 17.04 NUC bax with 8GB and SSD - - 10,000 customers, 15,000 Contacts and 280,000 aged debt transactions took ~70 seconds to generate.
Repeated for 100,000 customers reuslts in 148,000 contacts and 2.75 million open item transactions. The timinings were 12m 30 secs and 16 m 30 secs respectively. This gives rates of 7.5 secs/thou and 9.9 secs/thou respectively
Say 6 to 10 seconds per 1,000 customers with contacts and a full year of open item debtor transactions