Sqlite "Access violation" while simple upsert loop with transaction.

[foriequal0]

Envorinment

• OS: Windows 10
• Rust: stable 1.55.0 MSVC
• sqlx: { version = "0.5.7", features = ["runtime-tokio-rustls", "sqlite"] }

Following code crashes while run.

use std::time::Instant;
use anyhow::Result;
use sqlx::{Connection, ConnectOptions};
use sqlx::sqlite::SqliteConnectOptions;
use rand;
use rand::Rng;

#[tokio::main]
async fn main() -> Result<()> {
    let mut conn = SqliteConnectOptions::new()
        .filename(":memory:")
        .connect().await?;

    sqlx::query(r#"
    CREATE TABLE kv (k PRIMARY KEY, v);
    CREATE INDEX idx_kv ON kv (v);
    "#).execute(&mut conn).await?;

    let mut rng = rand::thread_rng();
    for i in 0..1_000_000 {
        if i % 1_000 == 0{
            println!("{}", i);
        }
        let key = rng.gen_range(0..1_000);
        let value = rng.gen_range(0..1_000);
        let mut tx = conn.begin().await?;

        let exists = sqlx::query("SELECT 1 FROM kv WHERE k = ?")
            .bind(key)
            .fetch_optional(&mut tx)
            .await?;
        if exists.is_some() {
            sqlx::query("UPDATE kv SET v = ? WHERE k = ?")
                .bind(value)
                .bind(key)
                .execute(&mut tx)
                .await?;
        } else {
            sqlx::query("INSERT INTO kv(k, v) VALUES (?, ?)")
                .bind(key)
                .bind(value)
                .execute(&mut tx)
                .await?;
        }
        tx.commit().await?;
    }
    Ok(())
}

log:

...
176000
177000
178000
179000
Exception: Exception 0xc0000005 encountered at address 0x7ff6caea661f: Access violation reading location 0x10000003e

stacktrace:

sqlite3DbMallocRawNN(sqlite3 *,unsigned long long) sqlite3.c:28224
sqlite3DbMallocRaw(sqlite3 *,unsigned long long) sqlite3.c:28191
sqlite3VdbeMemGrow(sqlite3_value *,int,int) sqlite3.c:76723
sqlite3VdbeMemClearAndResize(sqlite3_value *,int) sqlite3.c:76765
allocateCursor(Vdbe *,int,int,int,unsigned char) sqlite3.c:86156
sqlite3VdbeExec(Vdbe *) sqlite3.c:89797
sqlite3Step(Vdbe *) sqlite3.c:84331
sqlite3_step(sqlite3_stmt *) sqlite3.c:84389
sqlx_core::sqlite::statement::worker::impl$0::new::closure$0(closure$0) worker.rs:51
std::sys_common::backtrace::__rust_begin_short_backtrace<sqlx_core::sqlite::statement::worker::impl$0::new::closure$0,tuple$<> >(closure$0) backtrace.rs:125
std::thread::impl$0::spawn_unchecked::closure$0::closure$0<sqlx_core::sqlite::statement::worker::impl$0::new::closure$0,tuple$<> >(closure$0) mod.rs:476
std::panic::impl$30::call_once<tuple$<>,std::thread::impl$0::spawn_unchecked::closure$0::closure$0>(AssertUnwindSafe<std::thread::impl$0::spawn_unchecked::closure$0::closure$0>) panic.rs:347
std::panicking::try::do_call<std::panic::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked::closure$0::closure$0>,tuple$<> >(unsigned char *) panicking.rs:401
_$LT$core..ptr..unique..Unique$LT$T$GT$$u20$as$u20$core..convert..From$LT$$RF$mut$u20$T$GT$$GT$::from::hd0746f89f6d41b16 0x00007ff6cacfc483
std::panicking::try<tuple$<>,std::panic::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked::closure$0::closure$0> >(AssertUnwindSafe<std::thread::impl$0::spawn_unchecked::closure$0::closure$0>) panicking.rs:365
std::panic::catch_unwind<std::panic::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked::closure$0::closure$0>,tuple$<> >(AssertUnwindSafe<std::thread::impl$0::spawn_unchecked::closure$0::closure$0>) panic.rs:434
std::thread::impl$0::spawn_unchecked::closure$0<sqlx_core::sqlite::statement::worker::impl$0::new::closure$0,tuple$<> >(closure$0) mod.rs:475
core::ops::function::FnOnce::call_once<std::thread::impl$0::spawn_unchecked::closure$0,tuple$<> >(std::thread::impl$0::spawn_unchecked::closure$0 *) function.rs:227
std::sys::windows::thread::impl$0::new::thread_start() thread.rs:57
BaseThreadInitThunk 0x00007ffc15717034
RtlUserThreadStart 0x00007ffc17482651

It doesn't crash if I run it without the transaction.

[abonander]

@foriequal0 can you reproduce this with 0.5.9? we recently put out some fixes to do with SQLite.

[foriequal0]

It still happens with 0.5.9.

[Dhole]

I can reproduce this under Linux. I'm actually investigating a similar segfault that happens in a project I'm working on, but haven't managed to reproduce it in a small test.

I've run the exact same code from the first message in this issue (in a test, with #[async_std::test]) with the following environment:

• OS: Arch Linux x86_64 5.12.0-rc1-1-mainline
• Rust: 1.57.0-nightly (497ee321a 2021-09-09)
• sqlx: { version = "0.5.9", features = ["sqlite", "chrono", "runtime-async-std-native-tls"] }

I get segfaults at two different points. The first one is like the one in the first message:

Thread 17 "utils::test::te" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffdedf6640 (LWP 3410913)]
0x0000555555a739fd in sqlite3DbMallocRawNN (db=0x7fffb0000cf8, n=432) at sqlite3/sqlite3.c:28224
28224	    db->lookaside.pFree = pBuf->pNext;
(gdb) bt
#0  0x0000555555a739fd in sqlite3DbMallocRawNN (db=0x7fffb0000cf8, n=432) at sqlite3/sqlite3.c:28224
#1  0x0000555555a738b4 in sqlite3DbMallocRaw (db=0x7fffb0000cf8, n=432) at sqlite3/sqlite3.c:28191
#2  0x0000555555aa2cb8 in sqlite3VdbeMemGrow (pMem=0x7ffff000e1c8, n=432, bPreserve=0) at sqlite3/sqlite3.c:76723
#3  0x0000555555aa2dda in sqlite3VdbeMemClearAndResize (pMem=0x7ffff000e1c8, szNew=432) at sqlite3/sqlite3.c:76765
#4  0x0000555555aafcc9 in allocateCursor (p=0x7ffff000d998, iCur=2, nField=2, iDb=0, eCurType=0 '\000') at sqlite3/sqlite3.c:86156
#5  0x0000555555ab54fd in sqlite3VdbeExec (p=0x7ffff000d998) at sqlite3/sqlite3.c:89797
#6  0x0000555555aada66 in sqlite3Step (p=0x7ffff000d998) at sqlite3/sqlite3.c:84331
#7  0x0000555555aadcbc in sqlite3_step (pStmt=0x7ffff000d998) at sqlite3/sqlite3.c:84388
#8  0x000055555595c4b3 in sqlx_core::sqlite::statement::worker::{impl#0}::new::{closure#0} () at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/sqlite/statement/worker.rs:54
#9  0x0000555555962588 in std::sys_common::backtrace::__rust_begin_short_backtrace<sqlx_core::sqlite::statement::worker::{impl#0}::new::{closure#0}, ()> (f=...) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/sys_common/backtrace.rs:125
#10 0x0000555555985b66 in std::thread::{impl#0}::spawn_unchecked::{closure#0}::{closure#0}<sqlx_core::sqlite::statement::worker::{impl#0}::new::{closure#0}, ()> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/thread/mod.rs:481
#11 0x00005555559546c9 in core::panic::unwind_safe::{impl#23}::call_once<(), std::thread::{impl#0}::spawn_unchecked::{closure#0}::{closure#0}> (self=..., _args=()) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/panic/unwind_safe.rs:271
#12 0x00005555559d52ba in std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked::{closure#0}::{closure#0}>, ()> (data=0x7fffdedf5978) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/panicking.rs:403
#13 0x00005555559db72b in __rust_try ()
#14 0x00005555559d51fa in std::panicking::try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked::{closure#0}::{closure#0}>> (f=...) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/panicking.rs:367
#15 0x0000555555995e29 in std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked::{closure#0}::{closure#0}>, ()> (f=...) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/panic.rs:129
#16 0x0000555555985981 in std::thread::{impl#0}::spawn_unchecked::{closure#0}<sqlx_core::sqlite::statement::worker::{impl#0}::new::{closure#0}, ()> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/thread/mod.rs:480
#17 0x00005555559793ef in core::ops::function::FnOnce::call_once<std::thread::{impl#0}::spawn_unchecked::{closure#0}, ()> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/ops/function.rs:227
#18 0x0000555555d62783 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/alloc/src/boxed.rs:1636
#19 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/alloc/src/boxed.rs:1636
#20 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:106
#21 0x00007ffff77c2259 in start_thread () from /usr/lib/libpthread.so.0
#22 0x00007ffff759e5e3 in clone () from /usr/lib/libc.so.6

The second one is longer. I've redacted the beginning of the stack trace:

#0  0x0000555555aa6f44 in vdbeFreeOpArray (db=0x7fffb400a0e8, aOp=0x300000001, nOp=2130968576) at sqlite3/sqlite3.c:79551
#1  0x0000555555aaa2ab in sqlite3VdbeClearObject (db=0x7fffb400a0e8, p=0x7fffb400aef8) at sqlite3/sqlite3.c:81816
#2  0x0000555555aaa3b4 in sqlite3VdbeDelete (p=0x7fffb400aef8) at sqlite3/sqlite3.c:81857
#3  0x0000555555aaa167 in sqlite3VdbeFinalize (p=0x7fffb400aef8) at sqlite3/sqlite3.c:81762
#4  0x0000555555aacc1f in sqlite3_finalize (pStmt=0x7fffb400aef8) at sqlite3/sqlite3.c:83738
#5  0x0000555555973f72 in sqlx_core::sqlite::statement::handle::{impl#3}::drop (self=0x7ffff00052b0) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/sqlite/statement/handle.rs:325
#6  0x000055555597f6db in core::ptr::drop_in_place<sqlx_core::sqlite::statement::handle::StatementHandle> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/ptr/mod.rs:188
#7  0x0000555555989c94 in alloc::sync::Arc<sqlx_core::sqlite::statement::handle::StatementHandle>::drop_slow<sqlx_core::sqlite::statement::handle::StatementHandle> (self=0x7ffff4c7d208) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/alloc/src/sync.rs:1065
#8  0x000055555598b8a3 in alloc::sync::{impl#27}::drop<sqlx_core::sqlite::statement::handle::StatementHandle> (self=0x7ffff4c7d208) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/alloc/src/sync.rs:1625
#9  0x000055555598041b in core::ptr::drop_in_place<alloc::sync::Arc<sqlx_core::sqlite::statement::handle::StatementHandle>> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/ptr/mod.rs:188
#10 0x00005555559cead0 in sqlx_core::sqlite::statement::virtual::{impl#1}::drop (self=0x7ffff4c871a0) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/sqlite/statement/virtual.rs:200
#11 0x000055555597f787 in core::ptr::drop_in_place<sqlx_core::sqlite::statement::virtual::VirtualStatement> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/ptr/mod.rs:188
#12 0x00005555559799fb in core::ptr::drop_in_place<core::option::Option<sqlx_core::sqlite::statement::virtual::VirtualStatement>> () at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/ptr/mod.rs:188
#13 0x00005555559a983e in sqlx_core::sqlite::connection::executor::prepare::{generator#0} () at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/sqlite/connection/executor.rs:27
#14 0x00005555559d3945 in core::future::from_generator::{impl#1}::poll<sqlx_core::sqlite::connection::executor::prepare::{generator#0}> (self=..., cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/mod.rs:80
#15 0x00005555559ab906 in sqlx_core::sqlite::connection::executor::{impl#2}::fetch_many::{closure#0}::{generator#0}<&str> () at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/sqlite/connection/executor.rs:113
#16 0x00005555559d3df2 in core::future::from_generator::{impl#1}::poll<sqlx_core::sqlite::connection::executor::{impl#2}::fetch_many::{closure#0}::{generator#0}> (self=..., cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/mod.rs:80
#17 0x00005555559bc335 in sqlx_core::ext::async_stream::{impl#0}::new::{generator#0}<either::Either<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::sqlite::row::SqliteRow>, sqlx_core::sqlite::connection::executor::{impl#2}::fetch_many::{closure#0}, core::future::from_generator::GenFuture<sqlx_core::sqlite::connection::executor::{impl#2}::fetch_many::{closure#0}::{generator#0}>> () at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/ext/async_stream.rs:28
#18 0x00005555559d2b02 in core::future::from_generator::{impl#1}::poll<sqlx_core::ext::async_stream::{impl#0}::new::{generator#0}> (self=..., cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/mod.rs:80
#19 0x0000555555954338 in futures_util::future::future::fuse::{impl#3}::poll<core::future::from_generator::GenFuture<sqlx_core::ext::async_stream::{impl#0}::new::{generator#0}>> (self=..., cx=0x7ffff4c86470)
    at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/fuse.rs:86
#20 0x000055555595389f in core::future::future::{impl#1}::poll<alloc::boxed::Box<(dyn core::future::future::Future, Output=core::result::Result<(), sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>> (self=..., cx=0x7ffff4c86470)
    at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/future.rs:119
#21 0x0000555555952b43 in core::future::future::{impl#0}::poll<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future, Output=core::result::Result<(), sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>>> (self=..., cx=0x7ffff4c86470)
    at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/future.rs:107
#22 0x00005555559bcced in sqlx_core::ext::async_stream::{impl#1}::poll_next<either::Either<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::sqlite::row::SqliteRow>> (self=..., cx=0x7ffff4c86470)
    at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/ext/async_stream.rs:51
#23 0x0000555555953b99 in futures_core::stream::{impl#1}::poll_next<alloc::boxed::Box<(dyn futures_core::stream::Stream, Item=core::result::Result<either::Either<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::sqlite::row::SqliteRow>, sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>> (self=..., cx=0x7ffff4c86470) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/stream.rs:120
#24 0x000055555595111b in futures_core::stream::{impl#4}::try_poll_next<core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream, Item=core::result::Result<either::Either<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::sqlite::row::SqliteRow>, sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>>, either::Either<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::sqlite::row::SqliteRow>, sqlx_core::error::Error> (self=..., cx=0x7ffff4c86470)
    at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/stream.rs:196
#25 0x00005555559789a2 in futures_util::stream::try_stream::try_filter_map::{impl#3}::poll_next<core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream, Item=core::result::Result<either::Either<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::sqlite::row::SqliteRow>, sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>>, core::future::from_generator::GenFuture<sqlx_core::executor::Executor::execute_many::{closure#0}::{generator#0}>, sqlx_core::executor::Executor::execute_many::{closure#0}, sqlx_core::sqlite::query_result::SqliteQueryResult>
    (self=..., cx=0x7ffff4c86470) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/stream/try_stream/try_filter_map.rs:76
#26 0x0000555555953c29 in futures_core::stream::{impl#1}::poll_next<alloc::boxed::Box<(dyn futures_core::stream::Stream, Item=core::result::Result<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>> (self=..., 
    cx=0x7ffff4c86470) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/stream.rs:120
#27 0x00005555559510eb in futures_core::stream::{impl#4}::try_poll_next<core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream, Item=core::result::Result<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>>, sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::error::Error> (self=..., cx=0x7ffff4c86470) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/stream.rs:196
#28 0x000055555596c2ae in futures_util::stream::try_stream::try_collect::{impl#2}::poll<core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream, Item=core::result::Result<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>>, sqlx_core::sqlite::query_result::SqliteQueryResult> (self=..., cx=0x7ffff4c86470) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/stream/try_stream/try_collect.rs:46
#29 0x000055555595380f in core::future::future::{impl#1}::poll<alloc::boxed::Box<(dyn core::future::future::Future, Output=core::result::Result<sqlx_core::sqlite::query_result::SqliteQueryResult, sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>> (self=..., 
    cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/future.rs:119
#30 0x00005555559afaf2 in sqlx_core::sqlite::transaction::{impl#0}::commit::{generator#0} () at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/sqlite/transaction.rs:36
#31 0x00005555559d3002 in core::future::from_generator::{impl#1}::poll<sqlx_core::sqlite::transaction::{impl#0}::commit::{generator#0}> (self=..., cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/mod.rs:80
#32 0x000055555595389f in core::future::future::{impl#1}::poll<alloc::boxed::Box<(dyn core::future::future::Future, Output=core::result::Result<(), sqlx_core::error::Error>> + core::marker::Send), alloc::alloc::Global>> (self=..., cx=0x7ffff4c86470)
    at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/future.rs:119
#33 0x00005555559a8e0d in sqlx_core::transaction::{impl#0}::commit::{generator#0}<sqlx_core::sqlite::database::Sqlite> () at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/sqlx-core-0.5.9/src/transaction.rs:82
#34 0x00005555559d3633 in core::future::from_generator::{impl#1}::poll<sqlx_core::transaction::{impl#0}::commit::{generator#0}> (self=..., cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/mod.rs:80
#35 0x000055555565f849 in backend::utils::test::test_sqlx2::{generator#0} () at src/utils.rs:153
#36 0x0000555555644731 in core::future::from_generator::{impl#1}::poll<backend::utils::test::test_sqlx2::{generator#0}> (self=..., cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/mod.rs:80
#37 0x000055555564f6d3 in async_std::task::builder::{impl#1}::poll::{closure#0}<core::future::from_generator::GenFuture<backend::utils::test::test_sqlx2::{generator#0}>> () at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.10.0/src/task/builder.rs:199
#38 0x00005555556a299d in async_std::task::task_locals_wrapper::{impl#0}::set_current::{closure#0}<async_std::task::builder::{impl#1}::poll::{closure#0}, core::task::poll::Poll<core::result::Result<(), sqlx_core::error::Error>>> (current=0x7ffff4c954e0)
    at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.10.0/src/task/task_locals_wrapper.rs:60
#39 0x000055555567bce2 in std::thread::local::LocalKey<core::cell::Cell<*const async_std::task::task_locals_wrapper::TaskLocalsWrapper>>::try_with<core::cell::Cell<*const async_std::task::task_locals_wrapper::TaskLocalsWrapper>, async_std::task::task_locals_wrapper::{impl#0}::set_current::{closure#0}, core::task::poll::Poll<core::result::Result<(), sqlx_core::error::Error>>> (self=0x555555f6b7c8, f=...) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/thread/local.rs:399
#40 0x000055555567aedc in std::thread::local::LocalKey<core::cell::Cell<*const async_std::task::task_locals_wrapper::TaskLocalsWrapper>>::with<core::cell::Cell<*const async_std::task::task_locals_wrapper::TaskLocalsWrapper>, async_std::task::task_locals_wrapper::{impl#0}::set_current::{closure#0}, core::task::poll::Poll<core::result::Result<(), sqlx_core::error::Error>>> (self=0x555555f6b7c8, f=...) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/thread/local.rs:375
#41 0x00005555556a258f in async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current<async_std::task::builder::{impl#1}::poll::{closure#0}, core::task::poll::Poll<core::result::Result<(), sqlx_core::error::Error>>> (task=0x7ffff4c87100, f=...)
    at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.10.0/src/task/task_locals_wrapper.rs:55
#42 0x000055555564f522 in async_std::task::builder::{impl#1}::poll<core::future::from_generator::GenFuture<backend::utils::test::test_sqlx2::{generator#0}>> (self=..., cx=0x7ffff4c86470) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.10.0/src/task/builder.rs:197
#43 0x00005555556a9e66 in futures_lite::future::{impl#12}::poll<core::result::Result<(), sqlx_core::error::Error>, async_std::task::builder::SupportTaskLocals<core::future::from_generator::GenFuture<backend::utils::test::test_sqlx2::{generator#0}>>, core::future::from_generator::GenFuture<async_executor::{impl#4}::run::{generator#0}::{generator#0}>> (self=..., cx=0x7ffff4c86470) at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-lite-1.12.0/src/future.rs:526
#44 0x0000555555657562 in async_executor::{impl#4}::run::{generator#0}<core::result::Result<(), sqlx_core::error::Error>, async_std::task::builder::SupportTaskLocals<core::future::from_generator::GenFuture<backend::utils::test::test_sqlx2::{generator#0}>>> ()
    at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/async-executor-1.4.1/src/lib.rs:242
#45 0x00005555556459c1 in core::future::from_generator::{impl#1}::poll<async_executor::{impl#4}::run::{generator#0}> (self=..., cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/mod.rs:80
#46 0x0000555555656990 in async_executor::{impl#9}::run::{generator#0}<core::result::Result<(), sqlx_core::error::Error>, async_std::task::builder::SupportTaskLocals<core::future::from_generator::GenFuture<backend::utils::test::test_sqlx2::{generator#0}>>> ()
    at /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/async-executor-1.4.1/src/lib.rs:447
#47 0x0000555555644931 in core::future::from_generator::{impl#1}::poll<async_executor::{impl#9}::run::{generator#0}> (self=..., cx=0x7ffff4c86470) at /rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/core/src/future/mod.rs:80
#48 0x00005555556a3f7a in async_io::driver::block_on<core::result::Result<(), sqlx_core::error::Error>, core::future::from_generator::GenFuture<async_executor::{impl#9}::run::{generator#0}>> (future=...)

[Dhole]

Hey I think I found the bug in sqlx.

The sqlite connection is being opened with SQLITE_OPEN_NOMUTEX:

Multi-thread. In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads.

So the database connection shouldn't be used simultaneously in two or more threads. I've seen that the ConnectionHandle doesn't implement the Sync trait, so apparently all should be good... But I'm afraid that's not the whole picture.

I believe the segmentation fault here happens because sqlite is using the Lookaside memory allocator https://sqlite.org/malloc.html#lookaside for the statements, which is found inside the database connection, and there can be unsafe concurrent operations on this structure happening in different threads. The first one is when sqlite3_prepare_v3 is being called, which takes a connection pointer. This one will try to allocate the statement into the db's connection lookaside memory allocator. The other one is in sqlite3_finalize which can free the statement from the db's connection lookaside memory allocator. See sqlite source to verify how the statement contains a pointer to the db.

Since the StatementHandle doesn't contain the ConnectionHandle, and is Sync, operations on the StatementHandle can happen concurrently to operations on the ConnectionHandle, but sometimes this is problematic.

Since the method sqlite3_finalize can touch the database (at least when the Lookaside memory allocator is used this happens, there may be other cases), then I think there are two possible solutions:
A. If SQLITE_OPEN_NOMUTEX is desired, then we must guarantee that sqlite3_finalize is called always in the same thread that operates on the ConnectionHandle which created that statement.
B. Use SQLITE_OPEN_FULLMUTEX to allow complete thread-safety.

[madadam]

I think I encountered the same issue. Here is a much simpler reproduction case:

#[sqlx_macros::test]
async fn successive_empty_transactions_dont_segfault() {
    let mut conn = SqliteConnection::connect(":memory:").await.unwrap();

    for _ in 0..10000 {
        let tx = conn.begin().await.unwrap();
        tx.commit().await.unwrap();
    }
}

Note I can only reproduce the crash with --release for some reason.

[Dhole]

As a temporary solution until this bug is fixed, you can now set the sqlite connection to be opened in SQLITE_OPEN_FULLMUTEX mode like this:

    let mut opts = SqliteConnectOptions::from_str(url)?
        .serialized(true);
    Ok(SqlitePool::connect_with(opts).await?)

This option setting was introduced recently and is not yet available in a release, so you'd need to pull the crate from github. For example, in your Cargo.toml you can add:

[patch.crates-io]
sqlx = { git = "https://github.com/launchbadge/sqlx", rev = "b419bf529823241a7fd598628919ab451fd7d136" }

[abonander]

We probably don't need SQLITE_OPEN_FULLMUTEX as that's a bit overkill, but we do need some sort of mutual exclusion when potentially calling SQLite functions concurrently. The likely culprit here is that we're calling sqlite3_reset on the main thread while sqlite3_step is still executing on the worker thread, so we need to make sure to always wait for the latter to finish before executing on a new statement.

On the other hand, I'm wondering if we should just have the worker thread manage the connection directly, never calling any SQLite functions outside of it, and then we essentially interact with it like any other client/server database model, with only high-level commands being sent between threads. We're already talking about doing that more or less in #1175 since the SQLite driver in its current incarnation is so slow anyways.

[Dhole]

We probably don't need SQLITE_OPEN_FULLMUTEX as that's a bit overkill, but we do need some sort of mutual exclusion when potentially calling SQLite functions concurrently. The likely culprit here is that we're calling sqlite3_reset on the main thread while sqlite3_step is still executing on the worker thread, so we need to make sure to always wait for the latter to finish before executing on a new statement.

The only instance of sqlite3_reset I see is in the worker thread.

sqlx/sqlx-core/src/sqlite/statement/worker.rs

Line 66 in 8299687

unsafe { sqlite3_reset(statement.as_ptr()) };

As I stated before, I believe the data race that causes this segfault is between sqlite3_prepare_v3 and sqlite3_finalize

On the other hand, I'm wondering if we should just have the worker thread manage the connection directly, never calling any SQLite functions outside of it, and then we essentially interact with it like any other client/server database model, with only high-level commands being sent between threads. We're already talking about doing that more or less in #1175 since the SQLite driver in its current incarnation is so slow anyways.

I believe this is the right approach to achieve a safe sqlite wrapper with SQLITE_OPEN_NOMUTEX.

[abonander]

Would you folks mind trying the changes I have in #1551 and see if the access violations still happen or not?

[Dhole]

Would you folks mind trying the changes I have in #1551 and see if the access violations still happen or not?

I've run the test code from the first message 5 times using #1551 and all runs completed successfully without any segmentation fault :) (on Linux x86_64)

@abonander thanks for working on this!

[madadam]

I can confirm that #1551 fixes the issue for me as well 👍