Handle byte order marks (BOMs) in CSV files

[simonw]

I often find sqlite-utils insert ... --csv creates a first column with a weird character at the start of it - which it turns out is the UTF-8 BOM. Fix that.

[simonw]

https://stackoverflow.com/a/44573867/6083 says:

There is no reason to check if a BOM exists or not, utf-8-sig manages that for you and behaves exactly as utf-8 if the BOM does not exist

[simonw]

The other option is to check if the file starts with codecs.BOM_UTF8 - which is b'\xef\xbb\xbf'.

[simonw]

I needed to find some CSV files on my computer with a BOM at the beginning - I figured out this recipe:

% rg -U -E none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .

TIL here: https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep