Allow incremental I/O to blobs in sqlite3

[jimminter]

BPO	24905
Nosy	@berkerpeksag, @serhiy-storchaka, @palaviv, @eamanu, @erlend-aasland, @nightlark
PRs	bpo-24905: Support BLOB incremental I/O in sqlite module #271 gh-69093: Add incremental I/O to blobs support in sqlite3 #30356 gh-69093: Support basic incremental I/O to blobs in sqlite3 #30680
Files	blob.patch blob2.patch blob3.patch blob4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-08-21.08:05:21.911>
labels = ['extension-modules', 'type-feature', '3.8']
title = 'Allow incremental I/O to blobs in sqlite3'
updated_at = <Date 2022-01-20.12:42:10.486>
user = 'https://bugs.python.org/jimminter'

bugs.python.org fields:

activity = <Date 2022-01-20.12:42:10.486>
actor = 'erlendaasland'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2015-08-21.08:05:21.911>
creator = 'jim_minter'
dependencies = []
files = ['41798', '44142', '46531', '46549']
hgrepos = []
issue_num = 24905
keywords = ['patch']
message_count = 28.0
messages = ['248945', '259262', '259520', '272873', '272883', '273052', '286957', '287040', '287065', '287081', '287162', '288515', '288517', '288519', '288724', '288725', '297205', '315450', '329457', '372225', '399813', '409494', '409580', '409584', '409591', '409592', '410920', '411021']
nosy_count = 9.0
nosy_names = ['ghaering', 'berker.peksag', 'serhiy.storchaka', 'jim_minter', 'waldhol', 'palaviv', 'eamanu', 'erlendaasland', 'rmast']
pr_nums = ['271', '30356', '30680']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24905'
versions = ['Python 3.8']

[jimminter]

SQLite supports incremental I/O to blobs, i.e. the capability to stream reads and writes to blobs without having to load the entire blob into memory first. See https://www.sqlite.org/c3ref/blob_open.html for more details on the C API.

It'd be nice if it were possible to do this in Python using sqlite3 (it is already possible with apsw).

[palaviv]

I opened a pull request for blob support in the pysqlite github repository:
ghaering/pysqlite#93

I will do the needed changes for python3 and will post a patch soon.

[palaviv]

I did the needed changes for the pull request at pysqlite for porting to python3. I will continue updating the patch if there will be changes in pysqlite and vice versa.

[palaviv]

Pinging as mentioned in the devguide.

[serhiy-storchaka]

Added comments on Rietveld. The patch contains typos and violates PEP-7 and PEP-12. And there are questions about API.

[palaviv]

Thanks for the review Serhiy. Attached is the updated patch after the changes.

[palaviv]

Pinging again. I think this would be a great enhancement to the sqlite module.

[serhiy-storchaka]

Sorry Aviv, I just forget about this issue.

Added new comments on Rietveld. Many lines in sqlite3.rst still are too long.

It would be worth to ask other developers about wanted interface.

[palaviv]

Thanks for the CR Serhiy. Attached is a new patch after the fixes from the CR.

What other developers should I ask? The interface is file like and is the same as apsw.

[serhiy-storchaka]

apsw has different order of arguments for blobopen(), all arguments are mandatory, and the parameter for read/write mode is called "writeable" rather than "readonly". This part of API needs to be discussed. Ask for discussion on mailing lists: Python-Ideas and maybe Python-List.

[palaviv]

Uploading patch after fixes from berker CR.

The blob_open API can can have the following options:

1. The table, column and row must be mandatory parameters.
2. The read/write permissions can have the following options:
   a. No default (mandatory parameter).
   b. default read-only
   c. default write-only
3. The dbname can be without a default of "main" and then it will be a mandatory parameter.

I don't think that there is enough differences between the possible API's to justify sending a message to the mailing lists.

[palaviv]

I opened a PR in github. I tagged some other developers if that will not start a discussion on the API I will post in the python-ideas.