Add bulk operations utilities

[KevinJBoyer]

Ticket

n/a

Changes

• Add bulk_ops.py, which exposes a bulk_upsert function for efficiently upserting large amounts of data into the database

Context for reviewers

• Projects frequently need to read in large amounts of data into the database from external sources such as CSV files. This utility provides a flexible way of doing so efficiently.
• I'm not familiar with the platform's approach to abstracting away the underlying database -- the code here is Postgres specific and uses the psycopg library. Feedback on how to adapt the code here to the platform's approach is welcome/appreciated.

Testing

make test args="tests/src/db/test_bulk_ops.py"

[KevinJBoyer]

Would love to know if there's a better way of doing this. I also considered:

    db_client = db.PostgresDBClient()
    conn = db_client._engine.raw_connection()

but accessing _engine directly did not feel appropriate (and doesn't solve for the type issue in any case)

[lorenyu]

hmm, not sure

[chouinar]

We could consider adding a raw_connection() method to the client class which does what you suggested. For the docs, mention that unless you're trying to do something very low level (ie. in psycopg) you'll almost never actually want to use it.

[KevinJBoyer]

I added this and included a comment with that context -- LMK what you think!

[lorenyu]

hmm, not sure

[lorenyu]

nit: it'd be nice to have the test case do a combination of inserts and updates rather than just inserts and updates separately

[KevinJBoyer]

Added -- one round of inserts, then a second round of combo insert + updates

[chouinar]

This is probably a very niche edge case, but what would happen if two temp tables were created with the same name by different processes? Does that cause any issues, or does them being in the transactions entirely shield them?

[KevinJBoyer]

Great question, I tested it locally and it looks like the transaction isolation works like you'd expect. Here's the SQL I ran:

CREATE TEMP TABLE test (id INT) ON COMMIT DROP;
SELECT * FROM test;

-- In a separate connection!
BEGIN;
CREATE TEMP TABLE test (other INT) ON COMMIT DROP;
SELECT * FROM test;
COMMIT;

-- Back in the original connection
COMMIT;

[chouinar]

We could consider adding a raw_connection() method to the client class which does what you suggested. For the docs, mention that unless you're trying to do something very low level (ie. in psycopg) you'll almost never actually want to use it.

[lorenyu]

looks great. just a nit on the test, don't feel too strongly about it though