[11.x] Add ability to configure SQLite busy_timeout, journal_mode, and synchronous pragmas

[bakerkretzmar]

This PR adds optional busy_timeout, journal_mode, and synchronous configuration options to allow setting the corresponding pragmas on SQLite database connections. Together, these three configuration options can improve SQLite's performance enormously.

Background

This SQLite configuration:

PRAGMA busy_timeout = 5000;
PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;

is becoming the widely recommended default for modern apps because it is orders of magnitude faster than the default setup and far less prone to SQLITE_BUSY errors.

• journal_mode configures how transactions are implemented. The default setting, DELETE, uses a rollback journal. Since SQLite 3.7 a WAL mode is available that uses a write ahead log, which is significantly faster and allows concurrent database reads and writes in more scenarios.
• busy_timeout configures how long SQLite will wait when a table is locked before returning a SQLITE_BUSY error. The default is 0, which means any attempted concurrent writes will fail immediately. Setting it to a positive integer will make SQLite wait that many milliseconds if the database is locked, which in most cases means that operations will complete with a negligible delay instead of erroring.
• synchronous configures how often SQLite flushes the contents of the database to the disk. The default, FULL, writes data to disk more or less immediately when it changes. The NORMAL mode syncs data when necessary but less often, and in combination with WAL mode is much faster, fully consistent, and highly durable. NORMAL is SQLite's recommended setting when used with WAL mode.

Rails and Django both currently support some or all of this functionality. They both have a timeout configuration option, Rails 7.1 defaults to WAL mode and NORMAL, and Django 5.1 (releasing in August) is going to add configuration fields for setting transaction modes and other custom pragmas:

• https://guides.rubyonrails.org/configuring.html#configuring-an-sqlite3-database
• https://github.com/rails/rails/blob/cacf96ecd053f0a3d64c7bf478dac63f98e6aa6d/activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb#L81
• https://docs.djangoproject.com/en/5.0/ref/databases/#database-is-locked-errors
• https://docs.djangoproject.com/en/dev/ref/databases/#setting-pragma-options

More info:

• https://www.sqlite.org/wal.html
• https://www.sqlite.org/pragma.html#pragma_busy_timeout
• https://www.sqlite.org/pragma.html#pragma_journal_mode
• https://www.sqlite.org/pragma.html#pragma_synchronous
• https://kerkour.com/sqlite-for-servers
• https://fractaledmind.github.io/2024/04/15/sqlite-on-rails-the-how-and-why-of-optimal-performance/
• https://www.bigbinary.com/blog/rails-7-1-comes-with-an-optimized-default-sqlite3-adapter-connection-configuration
• https://gcollazo.com/optimal-sqlite-settings-for-django/

Implementation

I based this on the existing code for setting the foreign_keys pragma, it works exactly the same way except that the new options accept custom values instead of just being on or off.

busy_timeout and synchronous are per-connection settings and have to be set every time Laravel connects to a SQLite database (like the existing foreign_keys pragma), so it makes sense to store them in the config like foreign_keys. journal_mode is a persistent setting and technically only has to be set once, but the implementation is much simpler and more consistent if we just set it on every connection too.

The new configuration options all default to null so that existing apps will be completely unaffected by this change.

Alternatives

How I'm currently doing this:

// in AppServiceProvider.php

public function boot(): void
{
    Event::listen(function (ConnectionEstablished $event) {
        if ($event->connection instanceof SQLiteConnection) {
            $event->connection->statement(<<<SQL
                PRAGMA busy_timeout = 5000;
                PRAGMA journal_mode = WAL;
                PRAGMA synchronous = NORMAL;
                SQL);
        }
    });
}

Future Scope

I'm going to work on a PR for immediate transactions too, which also combine really well with these settings but will have to be implemented differently.

[osbre]

Excuse my ignorance, but aren't we supposed to set journal_mode once for the database file rather than configuring it on every connection? Doesn't SQLite persist it?

[bakerkretzmar]

Excuse my ignorance, but aren't we supposed to set journal_mode once for the database file rather than configuring it on every connection? Doesn't SQLite persist it?

Yeah good question, I addressed that in my notes above. We can just set it once but I don't know if we're necessarily "supposed to." I'm assuming that the overhead of setting it to the same value repeatedly is completely negligible but I can do some more research, I haven't seen any mention of that being an issue and it's what Rails currently does. I figured keeping the code here simple and consistent was more important.

[bakerkretzmar]

And actually it's only wal and delete modes that persist, none of the others, so technically if you're setting it to anything else (highly unlikely but possible) it is required for every new connection.

[kratos-digital]

@bakerkretzmar Am I missing something?

'sqlite' => [
    'driver' => 'sqlite',
    'url' => env('DB_URL'),
    'database' => env('DB_DATABASE', database_path('database.sqlite')),
    'prefix' => '',
    'foreign_key_constraints' => env('DB_FOREIGN_KEYS', true),
    'busy_timeout' => 5000,
    'journal_mode' => 'WAL',
    'synchronous' => 'NORMAL',
],

I run migrations, check the mode and it returns delete.

dump(\DB::connection('sqlite')->select('PRAGMA journal_mode')[0]->journal_mode);

EDIT: Forget it, someone cached my config locally

[kikky7]

I am getting error General error: 11 database disk image is malformed when running migrate:fresh using this:

'busy_timeout' => 5000,
'journal_mode' => 'WAL',
'synchronous' => 'NORMAL',

When I comment those settings, no error. Error appears when running migrate:fresh on existing database, when it's empty it runs and creates tables. So I need to run migrate:fresh twice to refresh migrations.

How is this merged without testing @taylorotwell @bakerkretzmar