Skip to content

RFC: Add SQLiteStore specification#4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
auxym wants to merge 12 commits into
base:main
Choose a base branch
from
Open

RFC: Add SQLiteStore specification #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
062dc35
Add spec (1st draft)
auxym Jul 10, 2026
3d39c73
Spec clarifications and add created-by metadata
auxym Jul 11, 2026
ce6433d
Various additions and modifications to the proposed spec
auxym Jul 18, 2026
c089d3d
Edits for clarity
auxym Jul 19, 2026
93885d6
Add requirement of valid keys
auxym Jul 19, 2026
704b600
Edit key comparison requirement
auxym Jul 20, 2026
dcbaf90
Remove Canonical URI section (empty)
auxym Jul 25, 2026
15fcc6b
Add NOT NULL constraint on k PRIMARY KEY columns
auxym Jul 25, 2026
a2cbcf3
Add constraint on timestamp format
auxym Jul 31, 2026
52b7ba0
Edits reflecting recent comment
auxym Jul 31, 2026
4558814
Typo created_at -> modified_at
auxym Jul 31, 2026
4d8f9cb
Add appendix to spec with example SQL statements
auxym Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

- Support for key deletion, overwriting, and partial value writes.
- Full ACID guarantees provided by SQLite.
- High availability of SQLite implementations across programming languages
and environments.

@jbmsjbmsJul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.

@newvillenewvilleJul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?

@jbmsjbmsJul 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.

@auxymauxymJul 13, 2026

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.

For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).

Example usage:

```python
Expand All@@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
RFC: Add SQLiteStore specification by auxym · Pull Request #4 · auxym/zarr-sqlite-python · GitHub
Skip to content
RFC: Add SQLiteStore specification#4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

auxym
wants to merge
12
commits into

base:main
Choose a base branch

from

 Open


RFC: Add SQLiteStore specification
#4
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
062dc35

Add spec (1st draft)


auxym Jul 10, 2026
3d39c73

Spec clarifications and add created-by metadata


auxym Jul 11, 2026
ce6433d

Various additions and modifications to the proposed spec


auxym Jul 18, 2026
c089d3d

Edits for clarity


auxym Jul 19, 2026
93885d6

Add requirement of valid keys


auxym Jul 19, 2026
704b600

Edit key comparison requirement


auxym Jul 20, 2026
dcbaf90

Remove Canonical URI section (empty)


auxym Jul 25, 2026
15fcc6b

Add NOT NULL constraint on k PRIMARY KEY columns


auxym Jul 25, 2026
a2cbcf3

Add constraint on timestamp format


auxym Jul 31, 2026
52b7ba0

Edits reflecting recent comment


auxym Jul 31, 2026
4558814

Typo created_at -> modified_at


auxym Jul 31, 2026
4d8f9cb

Add appendix to spec with example SQL statements


auxym Aug 5, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


13 changes: 13 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

 - Support for key deletion, overwriting, and partial value writes.
 - Full ACID guarantees provided by SQLite.
 - High availability of SQLite implementations across programming languages
 and environments.

@jbmsjbmsJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.
@newvillenewvilleJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?
@jbmsjbmsJul 12, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.
@auxymauxymJul 13, 2026

Copy link


Copy Markdown

OwnerAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.
For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).
Example usage:

```python
Expand All@@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
RFC: Add SQLiteStore specification by auxym · Pull Request #4 · auxym/zarr-sqlite-python · GitHub
Skip to content
RFC: Add SQLiteStore specification#4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

auxym
wants to merge
12
commits into

base:main
Choose a base branch

from

 Open


RFC: Add SQLiteStore specification
#4
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
062dc35

Add spec (1st draft)


auxym Jul 10, 2026
3d39c73

Spec clarifications and add created-by metadata


auxym Jul 11, 2026
ce6433d

Various additions and modifications to the proposed spec


auxym Jul 18, 2026
c089d3d

Edits for clarity


auxym Jul 19, 2026
93885d6

Add requirement of valid keys


auxym Jul 19, 2026
704b600

Edit key comparison requirement


auxym Jul 20, 2026
dcbaf90

Remove Canonical URI section (empty)


auxym Jul 25, 2026
15fcc6b

Add NOT NULL constraint on k PRIMARY KEY columns


auxym Jul 25, 2026
a2cbcf3

Add constraint on timestamp format


auxym Jul 31, 2026
52b7ba0

Edits reflecting recent comment


auxym Jul 31, 2026
4558814

Typo created_at -> modified_at


auxym Jul 31, 2026
4d8f9cb

Add appendix to spec with example SQL statements


auxym Aug 5, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


13 changes: 13 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

 - Support for key deletion, overwriting, and partial value writes.
 - Full ACID guarantees provided by SQLite.
 - High availability of SQLite implementations across programming languages
 and environments.

@jbmsjbmsJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.
@newvillenewvilleJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?
@jbmsjbmsJul 12, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.
@auxymauxymJul 13, 2026

Copy link


Copy Markdown

OwnerAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.
For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).
Example usage:

```python
Expand All@@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
RFC: Add SQLiteStore specification by auxym · Pull Request #4 · auxym/zarr-sqlite-python · GitHub
Skip to content
RFC: Add SQLiteStore specification#4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

auxym
wants to merge
12
commits into

base:main
Choose a base branch

from

 Open


RFC: Add SQLiteStore specification
#4
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
062dc35

Add spec (1st draft)


auxym Jul 10, 2026
3d39c73

Spec clarifications and add created-by metadata


auxym Jul 11, 2026
ce6433d

Various additions and modifications to the proposed spec


auxym Jul 18, 2026
c089d3d

Edits for clarity


auxym Jul 19, 2026
93885d6

Add requirement of valid keys


auxym Jul 19, 2026
704b600

Edit key comparison requirement


auxym Jul 20, 2026
dcbaf90

Remove Canonical URI section (empty)


auxym Jul 25, 2026
15fcc6b

Add NOT NULL constraint on k PRIMARY KEY columns


auxym Jul 25, 2026
a2cbcf3

Add constraint on timestamp format


auxym Jul 31, 2026
52b7ba0

Edits reflecting recent comment


auxym Jul 31, 2026
4558814

Typo created_at -> modified_at


auxym Jul 31, 2026
4d8f9cb

Add appendix to spec with example SQL statements


auxym Aug 5, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


13 changes: 13 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

 - Support for key deletion, overwriting, and partial value writes.
 - Full ACID guarantees provided by SQLite.
 - High availability of SQLite implementations across programming languages
 and environments.

@jbmsjbmsJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.
@newvillenewvilleJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?
@jbmsjbmsJul 12, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.
@auxymauxymJul 13, 2026

Copy link


Copy Markdown

OwnerAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.
For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).
Example usage:

```python
Expand All@@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
RFC: Add SQLiteStore specification by auxym · Pull Request #4 · auxym/zarr-sqlite-python · GitHub
Skip to content
RFC: Add SQLiteStore specification#4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

auxym
wants to merge
12
commits into

base:main
Choose a base branch

from

 Open


RFC: Add SQLiteStore specification
#4
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
062dc35

Add spec (1st draft)


auxym Jul 10, 2026
3d39c73

Spec clarifications and add created-by metadata


auxym Jul 11, 2026
ce6433d

Various additions and modifications to the proposed spec


auxym Jul 18, 2026
c089d3d

Edits for clarity


auxym Jul 19, 2026
93885d6

Add requirement of valid keys


auxym Jul 19, 2026
704b600

Edit key comparison requirement


auxym Jul 20, 2026
dcbaf90

Remove Canonical URI section (empty)


auxym Jul 25, 2026
15fcc6b

Add NOT NULL constraint on k PRIMARY KEY columns


auxym Jul 25, 2026
a2cbcf3

Add constraint on timestamp format


auxym Jul 31, 2026
52b7ba0

Edits reflecting recent comment


auxym Jul 31, 2026
4558814

Typo created_at -> modified_at


auxym Jul 31, 2026
4d8f9cb

Add appendix to spec with example SQL statements


auxym Aug 5, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


13 changes: 13 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

 - Support for key deletion, overwriting, and partial value writes.
 - Full ACID guarantees provided by SQLite.
 - High availability of SQLite implementations across programming languages
 and environments.

@jbmsjbmsJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.
@newvillenewvilleJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?
@jbmsjbmsJul 12, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.
@auxymauxymJul 13, 2026

Copy link


Copy Markdown

OwnerAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.
For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).
Example usage:

```python
Expand All@@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
RFC: Add SQLiteStore specification by auxym · Pull Request #4 · auxym/zarr-sqlite-python · GitHub
Skip to content
RFC: Add SQLiteStore specification#4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

auxym
wants to merge
12
commits into

base:main
Choose a base branch

from

 Open


RFC: Add SQLiteStore specification
#4
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
062dc35

Add spec (1st draft)


auxym Jul 10, 2026
3d39c73

Spec clarifications and add created-by metadata


auxym Jul 11, 2026
ce6433d

Various additions and modifications to the proposed spec


auxym Jul 18, 2026
c089d3d

Edits for clarity


auxym Jul 19, 2026
93885d6

Add requirement of valid keys


auxym Jul 19, 2026
704b600

Edit key comparison requirement


auxym Jul 20, 2026
dcbaf90

Remove Canonical URI section (empty)


auxym Jul 25, 2026
15fcc6b

Add NOT NULL constraint on k PRIMARY KEY columns


auxym Jul 25, 2026
a2cbcf3

Add constraint on timestamp format


auxym Jul 31, 2026
52b7ba0

Edits reflecting recent comment


auxym Jul 31, 2026
4558814

Typo created_at -> modified_at


auxym Jul 31, 2026
4d8f9cb

Add appendix to spec with example SQL statements


auxym Aug 5, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


13 changes: 13 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

 - Support for key deletion, overwriting, and partial value writes.
 - Full ACID guarantees provided by SQLite.
 - High availability of SQLite implementations across programming languages
 and environments.

@jbmsjbmsJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.
@newvillenewvilleJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?
@jbmsjbmsJul 12, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.
@auxymauxymJul 13, 2026

Copy link


Copy Markdown

OwnerAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.
For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).
Example usage:

```python
Expand All@@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
RFC: Add SQLiteStore specification by auxym · Pull Request #4 · auxym/zarr-sqlite-python · GitHub
Skip to content
RFC: Add SQLiteStore specification#4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

auxym
wants to merge
12
commits into

base:main
Choose a base branch

from

 Open


RFC: Add SQLiteStore specification
#4
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
062dc35

Add spec (1st draft)


auxym Jul 10, 2026
3d39c73

Spec clarifications and add created-by metadata


auxym Jul 11, 2026
ce6433d

Various additions and modifications to the proposed spec


auxym Jul 18, 2026
c089d3d

Edits for clarity


auxym Jul 19, 2026
93885d6

Add requirement of valid keys


auxym Jul 19, 2026
704b600

Edit key comparison requirement


auxym Jul 20, 2026
dcbaf90

Remove Canonical URI section (empty)


auxym Jul 25, 2026
15fcc6b

Add NOT NULL constraint on k PRIMARY KEY columns


auxym Jul 25, 2026
a2cbcf3

Add constraint on timestamp format


auxym Jul 31, 2026
52b7ba0

Edits reflecting recent comment


auxym Jul 31, 2026
4558814

Typo created_at -> modified_at


auxym Jul 31, 2026
4d8f9cb

Add appendix to spec with example SQL statements


auxym Aug 5, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


13 changes: 13 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

 - Support for key deletion, overwriting, and partial value writes.
 - Full ACID guarantees provided by SQLite.
 - High availability of SQLite implementations across programming languages
 and environments.

@jbmsjbmsJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.
@newvillenewvilleJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?
@jbmsjbmsJul 12, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.
@auxymauxymJul 13, 2026

Copy link


Copy Markdown

OwnerAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.
For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).
Example usage:

```python
Expand All@@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
RFC: Add SQLiteStore specification by auxym · Pull Request #4 · auxym/zarr-sqlite-python · GitHub
Skip to content
RFC: Add SQLiteStore specification#4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

auxym
wants to merge
12
commits into

base:main
Choose a base branch

from

 Open


RFC: Add SQLiteStore specification
#4
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
062dc35

Add spec (1st draft)


auxym Jul 10, 2026
3d39c73

Spec clarifications and add created-by metadata


auxym Jul 11, 2026
ce6433d

Various additions and modifications to the proposed spec


auxym Jul 18, 2026
c089d3d

Edits for clarity


auxym Jul 19, 2026
93885d6

Add requirement of valid keys


auxym Jul 19, 2026
704b600

Edit key comparison requirement


auxym Jul 20, 2026
dcbaf90

Remove Canonical URI section (empty)


auxym Jul 25, 2026
15fcc6b

Add NOT NULL constraint on k PRIMARY KEY columns


auxym Jul 25, 2026
a2cbcf3

Add constraint on timestamp format


auxym Jul 31, 2026
52b7ba0

Edits reflecting recent comment


auxym Jul 31, 2026
4558814

Typo created_at -> modified_at


auxym Jul 31, 2026
4d8f9cb

Add appendix to spec with example SQL statements


auxym Aug 5, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


13 changes: 13 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

 - Support for key deletion, overwriting, and partial value writes.
 - Full ACID guarantees provided by SQLite.
 - High availability of SQLite implementations across programming languages
 and environments.

@jbmsjbmsJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.
@newvillenewvilleJul 11, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?
@jbmsjbmsJul 12, 2026

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.
@auxymauxymJul 13, 2026

Copy link


Copy Markdown

OwnerAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.
For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).
Example usage:

```python
Expand All@@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading