is it possible to force caching of tables to disk?

[ophiry]

The scenario is reading a parquet file from disk, doing some transformations, and caching results.
I want to cache the results to disk, to avoid the memory overhead.

from what I understand, using simple ibis.read_parquet will cache results to memory.
I tried using duckdb, after calling cache I see a new table was added, but the db file size reamins very small - so I assume it didn't help.

is there another way to force caching to disk?

[gforsyth]

Hey @ophiry ! I'm assuming you are using DuckDB here?

Two things:

1. If you want to force the contents of your parquet file into the on-disk DuckDB database, pass the result of read_parquet to create_table -- that will persist that file in the local db:

import ibis

con = ibis.duckdb.connect("mydb.db")
t = con.read_parquet('test.parquet')
con.create_table("persistent", obj=t)

2. DuckDB is very, very good at keeping memory usage low and you shouldn't have to think about memory pressure as much as with e.g. pandas