🐛 BUG: Import an D1 exported DB cannot be imported in local - HashIndex detected hash table inconsistency

[lakano]

Which Cloudflare product(s) does this pertain to?

D1

What versions are you using?

3.108.0

What operating system and version are you using?

Arch Linux

Please provide a link to a minimal reproduction

No response

Describe the Bug

Hi,

After an export of our D1 database on remote ( wrangler d1 export dbname --remote --output export.sql ), I'm trying to import it on local ( wrangler d1 execute --local --file export.sql ), but I got a lot of errors.

After theses long list of errors I also got a generic « ✘ [ERROR] internal error », and none of the INSERT request are executed.
The file contain ~3000 INSERT with each request could expand until 12000 characters each (JSON content), if I split the files in multiples files of ~200 INSERT, I could be able to import all the file, wittout the « ✘ [ERROR] internal error » but still with the repeated errors « HashIndex detected hash table inconsistency ».

Please provide any relevant error logs

kj/table.c++:57: error: HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.
stack: ; kj::getStackTrace() = /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2b4cc7d /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2b4868c /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2b4b0ab /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2277fc0 /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@39d1109 /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@39d0edd /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@38358f5

[mrankine]

I am also seeing this, after upgrading from 3.103.2 to 3.109.1.

[petebacondarwin]

Would it be possible to provide an import file that causes this error so that we can investigate?

[lakano]

Unfortunately I'm not allowed by the company. I'll try to find time to create a dumb requests of JSON fields with a lot of the content, and duplicate this query to reach the same massive SQL file, to see if I can reproduce it and then to attach it here to help you to debug.
I'm pretty sure it's caused by some buffer / limit problems.

[mrankine]

Here's an import file which causes many HashIndex errors. Multiple INSERTs of the same values don't appear to trigger the error, at least in some quick testing.

hash_errors.sql.txt

[emily-shen]

@mrankine I've just tried to execute that file and it seems to work fine without issues :(

I'm testing with a clean new database, wondering if you're running this on a local db with stuff in it already, or if there are any other details of your setup that i might need to reproduce this?