This is a utility to make use of golangs sql package and it's streaming capabilities, for exporting large result sets out of an RDBMS and into a text file.
Currently, it supports MySQL and Postgres drivers, which means any database that supports those drivers should work. For example, RedShift works with the Postgres driver, and MemSQL works with the MySQL driver.
If there are unsupported drivers, or issues with "compatible" drivers, please file an issue, or better yet open a PR to address the problem.
SqlToCSV supports a number of flags to direct how it works
- d : The (d)atabase adapter to use
- c : The (c)onnection string to use
- q : The (q)uery to use
- m : The deli(m)eter to use: 'comma' or 'tab'. Defaults to 'comma'
One key feature of SqlToCSV is the ability to obfuscate data if needed. You might need
to anonymize certain fields when exporting, for the purpose of giving the data to a
third party. The -o flag will let you specify which fields get this treatment.
When obfuscating, SqlToCSV will track the original value, and replace it with an auto-incrementing number (future processes for obfuscation to better match the original field type are coming). Each occurance of the same underlying value will get the same obfuscated id
To run it, call from bash like so:
sqltocsv -d "mysql" -c "root@tcp(localhost:3306)/mydb" -q "SELECT * FROM table" > outfile.csv- Obfuscation should be more fleshed out - generate values that match the original type, option to not use the same value, etc.
Apache v2 - See LICENSE