In Which We Flush Buffers After Finishing Writing to Them

[waltaskew]

I'd written some code like

with tempfile.NamedTemporaryFile() as temp:
    civis.io.civis_to_file(file_id, temp)
    df = pandas.read_csv(temp.name)

with a bit of extra logic for read_csv-ing in chunks to keep from going out of memory.
Because temp wasn't being flushed, I would occasionally get errors from unexpected end of files.

I'm not sure if that's expected behavior or not (if I'm passing the buffer to civis_to_file should I be expected to flush it myself?) but flushing them civis_to_file-side seems harmless at worst and bug-avoiding at best.

@mheilman

[mheilman]

it seems like this should be in _civis_to_file, no?

[waltaskew]

it seems like this should be in _civis_to_file, no?

The with open(buf, 'wb') as f: bit ensures the flush happens when the user passes a file path. It's only when they pass a buffer that we need the flush, although it wouldn't hurt to have an extra flush.

[mheilman]

should we also rename _civis_to_file to _civis_to_stream or something?

[waltaskew]

should we also rename _civis_to_file to _civis_to_stream or something?

I think of files in the unix sense of anything you can read & write rather than filesystem files, so I don't have an issue with _civis_to_file as a name. I wouldn't object to _civis_to_stream, though.

[mheilman]

please add a note in the changelog

[mheilman]

lgtm, thanks