Compressor panics with: output buffer sized too small

[adsharma]

What happened?

fsst compressor panics when compressing a small partition inside a 400MB database with 16GB memory limit in duckdb. Always happens on the same date partition.

Steps to reproduce

When I run:

https://github.com/Ladybug-Memory/opencode-pgembed-migrate

uv run migrate_opencode.py # migrates sqlite -> postgres and optionally copies older date partitions to vortex

I get this stacktrace + panic:

https://gist.github.com/adsharma/13bccd98761e662028adf0c579a2a879

Environment

Using duckdb 1.4.4 + vortex extension

Additional context

No response

[a10y]

Hey @adsharma thanks for filing this. There is currently a limitation where our compressor doesn't handle individual string values > 16MB very well.

I'm working on a fix to prevent the panic in this case. You might need to wait until the next DuckDB release next week to get it.