bpo-24905: Support BLOB incremental I/O in sqlite module

[palaviv]

This PR adds support in BLOB incremental I/O at the sqlite module. As asked by @serhiy-storchaka and @berkerpeksag I will try to get some more developers to give their input on the wanted API. I am tagging some people that are active in the ghaering/pysqlite and rogerbinns/apsw.
@ghaering, @rianhunter, @rogerbinns, @phdru. Please look at the PR and give your notes.

https://bugs.python.org/issue24905

[rogerbinns]

The APSW doc for reference is at https://rogerbinns.github.io/apsw/blob.html

Does having len make sense? Files don't have that method. It is also confusing - should len return the size from the current seek offset?

The documentation should make clearer that you cannot change the size of a blob, and mention zeroblob as the means to make a blob in a query without having to fill it in.

It may be worth mentioning that another approach is to store large data in a file, and only store the filename in the database. (This comes up on the sqlite-users mailing list quite a lot.)

[phdru]

I cannot remember I ever was in need to read/write a part of a BLOB; it was always "all or nothing" for me. So I never used BLOB APIs; instead I always SELECT/INSERT/UPDATE BLOB columns; in Postgres they are not even BLOB columns — I always use BYTEA type.

So I'm -0 on exposing BLOB API for SQLite.

[rogerbinns]

@phdru SQLite is the same with regular queries: you can only read or write blobs in their entirety. That for example means that if you store a 25MB blob then you must read or write 25MB at once.

SQLite has the "incremental blob" API for accessing just portions of blobs. The motivation comes from "Lite" in the name - developers use SQLite because it is lighter weight (amongst other reasons). DBAPI doesn't specify incremental blob I/O so only developers intending to use SQLite directly and not another database would use it. Should they be able to?

[phdru]

-0 from me means: I don't care and if there will be such an API I'm not gonna use it. That's all.

[palaviv]

Thanks for the input @rogerbinns.

Does having len make sense? Files don't have that method. It is also confusing - should len return the size from the current seek offset?

What is the difference between implementing __len__ to the method length APSW blob has?

[rogerbinns]

@palaviv there is no difference between the value returned by len and length or similar methods. It is however very uncommon to have a len method on file like objects - I couldn't find an example of any! For example StringIO is closest and has no len. Hence my recommendation to avoid len in favour of another method name.

[serhiy-storchaka]

The mmap.mmap() object is an example of file-like object supporting len().

[rogerbinns]

@serhiy-storchaka good example. They don't document it though, and there is a size() method although it is returning something slightly different. There also seems to be a correlation between types that have len and those that can you can array access.

In any event my recommendation is to avoid breaking new ground with a len method since that seems not to be normal practise for this kind of thing that provides a file like interface.

[palaviv]

I actually think that we should use __len__ as by the definition this is the length of the object. The Blob object is a representation of the BLOB and that is the BLOB length.

[serhiy-storchaka]

Pull request conversation is purposed for discussing the code.

It would be better to continue the design discussion on the bug tracker or mailing list.

[palaviv]

@serhiy-storchaka I have implemented the sequence protocol but I have a few questions:

1. Do I need both PySequenceMethods.sq_item, PySequenceMethods.sq_ass_item and PyMappingMethods.mp_subscript, PyMappingMethods.mp_ass_subscript.
2. I can't make __contains__ work. Could you point me to how fix that?

[palaviv]

I think that the contains operation should not be supported for blobs. As blobs can be very large looking for a subset of bytes inside them will be a very inefficient process in memory or in compute.

[auvipy]

3.8

[palaviv]

Done

[auvipy]

3.8

[palaviv]

Done

[auvipy]

you can retarget this for py 3.9

[matrixise]

Hi @palaviv

Would you be interested to upgrade your PR to the last master?

Thank you

[auvipy]

Retarget for python 3.9

[auvipy]

3.9

[eamanu]

Hi @palaviv There is some plan with this PR?
Was open 3 years ago. Are you still interest on this patch?
If yes, could you fix the conflict. please?

[palaviv]

Hi @eamanu,
This patch exist since January 2016 and I kind of given up on it ever going into CPython as there is no core developer that works on sqlite. I would recommend you to use apsw that support this feature. In case any core developer would be in interested in working on this I would gladly fix any needed changes.

[simonw]

I'd love to see this land in Python. I think there's a strong case for it: SQLite lets you store up to 2GB of data in a BLOB, and reading an entire 2GB value into memory at once isn't nearly as pleasant as reading it incrementally, which is what this would let us do.

[palaviv]

Thanks for the review @berkerpeksag. I have made the requested changes; please review again.

[bedevere-bot]

Thanks for making the requested changes!

@berkerpeksag: please review the changes made to this pull request.

[nightlark]

Other than rebasing (due new conflicts arising over time), is there anything that can be done to help move this PR along?

(@palaviv do you want to do the rebase? if you'd like or are too busy I can do the rebase, though I'd need to open a new PR since I don't think I can modify yours)

[erlend-aasland]

@nightlark, @palaviv: Here' a short list from the top-of-my head of what is needed to rebase this onto main:

• the test suite has been normalized; we now use snake case test_foo_bar method names
• Argument Clinic
• use heap types iso. static types
• exception types are accessed through the (temporary) global state; for Connection objects, it's available through self->state

If you want to try to land this, Ryan, please give Aviv a week or so to respond before opening a new PR :)

[nightlark]

@erlend-aasland Okay — I think I understand how to use argument clinic. Is there a guide to what iso. static types (or heap types)? Is the iso. a prefix for the types or an abbreviation? If it’s an abbreviation maybe that’s the missing a search term I should be using to find relevant resources.

[erlend-aasland]

@erlend-aasland Okay — I think I understand how to use argument clinic.

Great! AC is nice once you get into it. Feel free to ask if you get stuck :)

Is there a guide to what iso. static types (or heap types)?

There's some info in the docs, but you can also check the PR's that converted the existing types:

• bpo-41861: Convert _sqlite3 CursorType and ConnectionType to heap types #22478
• bpo-41861: Convert _sqlite3 RowType and StatementType to heap types #22444
• bpo-41861: Convert sqlite3 PrepareProtocolType to heap type #22428
• bpo-41861: Convert sqlite3 cache and node types to heap types #22417
• bpo-42972: Fully implement GC protocol for sqlite3 heap types #26104, bpo-42972: Fix sqlite3 traverse/clear #26452 (see these PR's for proper GC support)

Don't hesitate to ask if you need more pointers.

Is the iso. a prefix for the types or an abbreviation?

Sorry, it's an abbreviation: instead of :) I have the bad habit of using it too much.

[erlend-aasland]

Regarding heap types, take a look at Victor's blog: https://vstinner.github.io/isolate-subinterpreters.html

[erlend-aasland]

@nightlark Are you still working on this PR?

[CoolCat467]

It's 2022 now woooo

[dholth]

+1

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[JelleZijlstra]

I just merged #30680, a simplified version of this PR. Blobs will be in Python 3.11.