Skip to content

bpo-45040: Simplify sqlite3 transaction control functions #28019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 19, 2021
Merged

bpo-45040: Simplify sqlite3 transaction control functions #28019

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 21 additions & 47 deletions Modules/_sqlite/connection.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,43 +396,29 @@ static PyObject *
pysqlite_connection_commit_impl(pysqlite_Connection *self)
/*[clinic end generated code: output=3da45579e89407f2 input=39c12c04dda276a8]*/
{
int rc;
sqlite3_stmt* statement;

if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) {
return NULL;
}

if (!sqlite3_get_autocommit(self->db)) {
int rc;

Py_BEGIN_ALLOW_THREADS
sqlite3_stmt *statement;
rc = sqlite3_prepare_v2(self->db, "COMMIT", 7, &statement, NULL);
Py_END_ALLOW_THREADS
if (rc != SQLITE_OK) {
_pysqlite_seterror(self->state, self->db);
goto error;
if (rc == SQLITE_OK) {
(void)sqlite3_step(statement);
rc = sqlite3_finalize(statement);
}

rc = pysqlite_step(statement);
if (rc != SQLITE_DONE) {
_pysqlite_seterror(self->state, self->db);
}

Py_BEGIN_ALLOW_THREADS
rc = sqlite3_finalize(statement);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems that sqlite3_finalize was protected with the GIL previously and the check for PyErr_Occurred makes me thing that some callback or similar could end calling Python code. Could you confirm if this is indeed safe?

Copy link
Contributor Author

@erlend-aasland erlend-aasland Aug 30, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, sqlite3_finalize was called outside of the GIL previously, and it still is. Previously, we would release/acquire GIL before/after every SQLite API call; now we release, do all SQLite API calls, then acquire.

Previously

  1. allow threads (unlock GIL), prepare statement, disallow threads (lock GIL)
  2. raise exception and bail on error
  3. unlock GIL, execute SQL statement (sqlite3_step), lock GIL
  4. raise exception and "fall through" on error
  5. unlock GIL, finalise statement (release resources, etc.), lock GIL
  6. raise exception, unless sqlite3_step already did, on error => this is the extra PyErr_Occurred

Since sqlite3_finalize will pass on the error set by sqlite3_step, there is no need to check for errors after stepping. We only need to do it after finalising the statement.

Now

  1. unlock GIL, create, execute, and finalise SQL statement, lock GIL
  2. raise exception and bail on error

Copy link
Member

@pablogsal pablogsal Aug 30, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can sqlite3_finalize raise Python errors on its own (via cleanups, callbacks or otherwise)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not the way we currently use the SQLite API. If we'd added SQLite profile support, sqlite3_finalize (and other API's) would invoke our _trace_callback() C callback with argument type set to SQLITE_TRACE_PROFILE. We could add an assert(type != SQLITE_TRACE_PROFILE) and a comment about this to _trace_callback().

Py_END_ALLOW_THREADS
if (rc != SQLITE_OK && !PyErr_Occurred()) {
_pysqlite_seterror(self->state, self->db);
}

if (rc != SQLITE_OK) {
(void)_pysqlite_seterror(self->state, self->db);
return NULL;
}
}

error:
if (PyErr_Occurred()) {
return NULL;
} else {
Py_RETURN_NONE;
}
Py_RETURN_NONE;
}

/*[clinic input]
Expand All @@ -445,44 +431,32 @@ static PyObject *
pysqlite_connection_rollback_impl(pysqlite_Connection *self)
/*[clinic end generated code: output=b66fa0d43e7ef305 input=12d4e8d068942830]*/
{
int rc;
sqlite3_stmt* statement;

if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) {
return NULL;
}

if (!sqlite3_get_autocommit(self->db)) {
pysqlite_do_all_statements(self);

int rc;

Py_BEGIN_ALLOW_THREADS
sqlite3_stmt *statement;
rc = sqlite3_prepare_v2(self->db, "ROLLBACK", 9, &statement, NULL);
Py_END_ALLOW_THREADS
if (rc != SQLITE_OK) {
_pysqlite_seterror(self->state, self->db);
goto error;
}

rc = pysqlite_step(statement);
if (rc != SQLITE_DONE) {
_pysqlite_seterror(self->state, self->db);
if (rc == SQLITE_OK) {
(void)sqlite3_step(statement);
rc = sqlite3_finalize(statement);
}

Py_BEGIN_ALLOW_THREADS
rc = sqlite3_finalize(statement);
Py_END_ALLOW_THREADS
if (rc != SQLITE_OK && !PyErr_Occurred()) {
_pysqlite_seterror(self->state, self->db);

if (rc != SQLITE_OK) {
(void)_pysqlite_seterror(self->state, self->db);
return NULL;
}

}

error:
if (PyErr_Occurred()) {
return NULL;
} else {
Py_RETURN_NONE;
}
Py_RETURN_NONE;
}

static int
Expand Down
23 changes: 7 additions & 16 deletions Modules/_sqlite/cursor.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -431,31 +431,22 @@ static int
begin_transaction(pysqlite_Connection *self)
{
int rc;
sqlite3_stmt *statement;

Py_BEGIN_ALLOW_THREADS
sqlite3_stmt *statement;
rc = sqlite3_prepare_v2(self->db, self->begin_statement, -1, &statement,
NULL);
Py_END_ALLOW_THREADS

if (rc != SQLITE_OK) {
_pysqlite_seterror(self->state, self->db);
goto error;
if (rc == SQLITE_OK) {
(void)sqlite3_step(statement);
rc = sqlite3_finalize(statement);
}

Py_BEGIN_ALLOW_THREADS
sqlite3_step(statement);
rc = sqlite3_finalize(statement);
Py_END_ALLOW_THREADS

if (rc != SQLITE_OK && !PyErr_Occurred()) {
_pysqlite_seterror(self->state, self->db);
}

error:
if (PyErr_Occurred()) {
if (rc != SQLITE_OK) {
(void)_pysqlite_seterror(self->state, self->db);
return -1;
}

return 0;
}

Expand Down