fix(server): mutex calls to sqlitex

[Joibel]

Fixes #13154
Fixes #13140

Motivation

zombiezen/go-sqlite is not thread safe when used through a single connection. The current code is provably racing (run the server with -race and a few workflows being run) and it will tell you this if you argo list via the server a few times. Unless it crashes first.

Modifications

This change doesn't attempt to move to a multiple connection model, it's a minimal change to stop the server crashing all the time, by mutexing the use of the sql connection.

Verification

Running v3.5.7

Inject 200 example/dag-diamond.yamls into your cluster. Run argo list -s <server> to list the workflows. It will crash within a few attempts, usually

With this change it didn't crash in 200 attempts at argo list.

-race build of the server doesn't complain any more either.

[agilgur5]

cc @jiachengxu for review

[jiachengxu]

The change looks good to me, thanks for the work @Joibel!

[Joibel]

@agilgur5, @tczhao, @isubasinghe, @terrytangyuan - could someone take a look at this one so we can prep a 3.5.8 with it in?

[sarabala1979]

good catch @Joibel

[agilgur5]

Nice work figuring out the root cause @Joibel!
Since we hadn't found any nil pointers in our own code, I was suspecting it might be within the SQLite library, although I definitely did not expect that it wasn't thread-safe. Those GoDoc comments really belong in the README of the library IMO 😕

Will you or @jiachengxu be taking up the full fix with thread pooling?

so we can prep a 3.5.8 with it in?

Would like to get a fix for #13177 in as well and then release it ASAP

[agilgur5]

Will you or @jiachengxu be taking up the full fix with thread pooling?

Wrote up a follow-up issue to track this in #13638