Fix ptrmap data corruption with pre-initialize autovacuum database

[ddwalias]

Reproduce:

1. Use SQLite to initialize an auto-vacuum DB (to produce a live pointer-map page at page 2):

sqlite3 /tmp/av_corruption.db "PRAGMA auto_vacuum=FULL; VACUUM; IF NOT EXISTS __sim_autovacuum_seed__(id INTEGER PRIMARY KEY, value TEXT); INSERT OR IGNORE INTO __sim_autovacuum_seed__ VALUES (1,'seed');"

2. Let Turso mutate it:

tursodb /tmp/av_corruption.db "CREATE TABLE t(a);" \ for i in $(seq 1 2000); do target/debug/tursodb /tmp/av_corruption.db "INSERT INTO t VALUES ($i);"; done

3. Back in SQLite, run PRAGMA integrity_check; -> it reports Tree … Failed to read ptrmap key=….. However, Turso’s own PRAGMA integrity_check; still prints ok, so the engine doesn’t notice the damage.

Why the simulator misses it:

The deterministic runs always start from Turso-built databases, and never import a file produced by SQLite, so this path is never exercised.

Reference: #3895

[ddwalias]

The cause seem to be when Turso allocates leaf or overflow pages, it never writes the corresponding pointer‑map entries.

I attempted to fix this issue with this commit, but seems like I introduced a new bug in the process. For this PR, I will focus only on updating the simulator. The fix will be on another PR.

[penberg]

Hey @ddwalias! Good catch! I am disabling autovacuum by default in #3899. Let's have other people review this PR too, but assuming the improvement is good, this is eligible for the data corruption challenge reward, although functionality is disabled now.

[ddwalias]

I managed to make the integration test passes, however, the simulator is still failing, likely from another issue

[penberg]

This pull request has been marked as stale due to inactivity. It will be closed in 7 days if no further activity occurs.

[penberg]

This pull request has been closed due to inactivity. Please feel free to reopen it if you have further updates.

[turso-bot]

Please review @jussisaurio

[LeMikaelF]

Similar to #3830

[LeMikaelF]

@ddwalias we've decided to award you the 800$ Turso Challenge reward, since you did patch the simulator to demonstrate
data corruption.

However, after careful consideration, we chose to go a different route for fixing this bug. Here is our preferred
solution:

1. Patch Turso to write a 1-page db to disk when the autovacuum mode is
   changed (Write the DB file to disk when the header is modified #4459). This mimics SQLite's behaviour.
2. Rely on the simulator's existing pragma auto_vacuum support to reproduce the bug.
3. Open a DB in read-only mode if it's auto-vacuumed (Open database in readonly mode if it is an autovacuum database and the experimental autovacuum feature is not enabled #4457). This should
   have been the current behaviour, since the intent of the --experimental-autovacuum flag was to hide a feature that
   wasn't production ready.

You're more than welcome to help us implement these fixes.

As for the ptrmap fix in the b-tree, it's definitely something we want to implement, see the tracking
issue #2290. Unfortunately, it looks like there's some merge conflicts that
need to be addressed. Apologies for the delay in responding to this PR. You're welcome to open a separate PR with these
changes. If you do so, they should also be tested more extensively. I suggest using a fuzz test. There's already a few
in the code base that you could use as a model.

[glommer]

/tip $800 @ddwalias

[algora-pbc]

Please visit Algora to complete your tip via Stripe.

[algora-pbc]

🎉🎈 @ddwalias has been awarded $800 by Turso Database! 🎈🎊