docs(PLAN): spec within-file chunk concurrency#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CrispStrobe merged 1 commit into from
Jun 29, 2026
Merged

docs(PLAN): spec within-file chunk concurrency #8

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions PLAN.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to:

- Sync engine, file versioning, GUI, cross-account migration
- Workspaces (`workspaces-list/use/unset`)

---

# Performance: within-file chunk concurrency (added 2026-06-29)

File-level (batch) concurrency ALREADY exists: `cli.py` uses a
`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/
`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the
transfer of a SINGLE large file. Two opportunities, ported from the
filen-python / filen-dart "bounded chunk concurrency" work (read those repos'
PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first).

Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption
is inherently sequential — like a running hash), and **S3** storage (multipart
parts on upload; a single presigned object on download).

## Step A — parallel multipart part uploads ⬜ TODO
`services/drive.py` → `_perform_network_upload` (~line 185): the sequential
`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part
(`encryptor.update`), updates `sha`, and PUTs it via
`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`)
before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart;
smaller files are a single PUT — keep them sequential.

Approach (mirror filen Step 1):
- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN
ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then
hands `(part_index, urls[part_index], ciphertext)` to a bounded
`ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag /
`DriveService` attr).
- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size
the list and assign BY INDEX (parts finish out of order).
- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))`
before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live.
- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber,
then `api.finish_upload(...)` unchanged.

CONSTRAINTS — do NOT just flip the loop:
1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer
(keystream + content hash are order-dependent). Only the PUT is parallel.
2. `parts_manifest` ordered by PartNumber regardless of completion order.
3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough).
4. On part failure, surface after join; let the existing checkpoint/resume path
(`create_upload_checkpoint`, `services/drive.py:898`) handle restart.

Win is the single-large-file case (batch is already file-parallel).

## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag)
`services/drive.py` → `download_file` (~line 1616): today one
`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL
(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to
disk. S3 presigned GETs honor HTTP `Range`.

Approach:
- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)`
(sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool.
- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or
`os.pwrite`); pre-size with `f.truncate(file_size)`.
- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is
`initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic,
bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned
offset. Then each range decrypts independently. Verify the existing content-hash.

CONSTRAINTS:
1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned.
2. Bound bytes-in-flight (`_mem_acquire`).
3. Fall back to the sequential single-stream path on 200-not-206 or small files.

## Tests (unit hermetic + live)
Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered;
seekable-CTR decrypt reproduces plaintext; small files stay sequential.
Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up):
- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline
- [ ] interrupted multipart upload resumes (checkpoint) and completes
- [ ] ranged download of a large file is byte-exact + faster than single-stream
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
docs(PLAN): spec within-file chunk concurrency#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

CrispStrobe
merged 1 commit into

from

Jun 29, 2026
 Merged


docs(PLAN): spec within-file chunk concurrency
#8
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


75 changes: 75 additions & 0 deletions
PLAN.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to:

- Sync engine, file versioning, GUI, cross-account migration
- Workspaces (`workspaces-list/use/unset`)

---

# Performance: within-file chunk concurrency (added 2026-06-29)

File-level (batch) concurrency ALREADY exists: `cli.py` uses a
`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/
`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the
transfer of a SINGLE large file. Two opportunities, ported from the
filen-python / filen-dart "bounded chunk concurrency" work (read those repos'
PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first).

Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption
is inherently sequential — like a running hash), and **S3** storage (multipart
parts on upload; a single presigned object on download).

## Step A — parallel multipart part uploads ⬜ TODO
`services/drive.py` → `_perform_network_upload` (~line 185): the sequential
`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part
(`encryptor.update`), updates `sha`, and PUTs it via
`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`)
before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart;
smaller files are a single PUT — keep them sequential.

Approach (mirror filen Step 1):
- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN
 ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then
 hands `(part_index, urls[part_index], ciphertext)` to a bounded
 `ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag /
 `DriveService` attr).
- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size
 the list and assign BY INDEX (parts finish out of order).
- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))`
 before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live.
- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber,
 then `api.finish_upload(...)` unchanged.

CONSTRAINTS — do NOT just flip the loop:
1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer
 (keystream + content hash are order-dependent). Only the PUT is parallel.
2. `parts_manifest` ordered by PartNumber regardless of completion order.
3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough).
4. On part failure, surface after join; let the existing checkpoint/resume path
 (`create_upload_checkpoint`, `services/drive.py:898`) handle restart.

Win is the single-large-file case (batch is already file-parallel).

## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag)
`services/drive.py` → `download_file` (~line 1616): today one
`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL
(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to
disk. S3 presigned GETs honor HTTP `Range`.

Approach:
- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)`
 (sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool.
- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or
 `os.pwrite`); pre-size with `f.truncate(file_size)`.
- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is
 `initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic,
 bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned
 offset. Then each range decrypts independently. Verify the existing content-hash.

CONSTRAINTS:
1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned.
2. Bound bytes-in-flight (`_mem_acquire`).
3. Fall back to the sequential single-stream path on 200-not-206 or small files.

## Tests (unit hermetic + live)
Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered;
seekable-CTR decrypt reproduces plaintext; small files stay sequential.
Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up):
- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline
- [ ] interrupted multipart upload resumes (checkpoint) and completes
- [ ] ranged download of a large file is byte-exact + faster than single-stream

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
docs(PLAN): spec within-file chunk concurrency#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

CrispStrobe
merged 1 commit into

from

Jun 29, 2026
 Merged


docs(PLAN): spec within-file chunk concurrency
#8
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


75 changes: 75 additions & 0 deletions
PLAN.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to:

- Sync engine, file versioning, GUI, cross-account migration
- Workspaces (`workspaces-list/use/unset`)

---

# Performance: within-file chunk concurrency (added 2026-06-29)

File-level (batch) concurrency ALREADY exists: `cli.py` uses a
`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/
`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the
transfer of a SINGLE large file. Two opportunities, ported from the
filen-python / filen-dart "bounded chunk concurrency" work (read those repos'
PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first).

Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption
is inherently sequential — like a running hash), and **S3** storage (multipart
parts on upload; a single presigned object on download).

## Step A — parallel multipart part uploads ⬜ TODO
`services/drive.py` → `_perform_network_upload` (~line 185): the sequential
`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part
(`encryptor.update`), updates `sha`, and PUTs it via
`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`)
before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart;
smaller files are a single PUT — keep them sequential.

Approach (mirror filen Step 1):
- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN
 ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then
 hands `(part_index, urls[part_index], ciphertext)` to a bounded
 `ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag /
 `DriveService` attr).
- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size
 the list and assign BY INDEX (parts finish out of order).
- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))`
 before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live.
- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber,
 then `api.finish_upload(...)` unchanged.

CONSTRAINTS — do NOT just flip the loop:
1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer
 (keystream + content hash are order-dependent). Only the PUT is parallel.
2. `parts_manifest` ordered by PartNumber regardless of completion order.
3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough).
4. On part failure, surface after join; let the existing checkpoint/resume path
 (`create_upload_checkpoint`, `services/drive.py:898`) handle restart.

Win is the single-large-file case (batch is already file-parallel).

## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag)
`services/drive.py` → `download_file` (~line 1616): today one
`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL
(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to
disk. S3 presigned GETs honor HTTP `Range`.

Approach:
- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)`
 (sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool.
- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or
 `os.pwrite`); pre-size with `f.truncate(file_size)`.
- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is
 `initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic,
 bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned
 offset. Then each range decrypts independently. Verify the existing content-hash.

CONSTRAINTS:
1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned.
2. Bound bytes-in-flight (`_mem_acquire`).
3. Fall back to the sequential single-stream path on 200-not-206 or small files.

## Tests (unit hermetic + live)
Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered;
seekable-CTR decrypt reproduces plaintext; small files stay sequential.
Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up):
- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline
- [ ] interrupted multipart upload resumes (checkpoint) and completes
- [ ] ranged download of a large file is byte-exact + faster than single-stream

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
docs(PLAN): spec within-file chunk concurrency#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

CrispStrobe
merged 1 commit into

from

Jun 29, 2026
 Merged


docs(PLAN): spec within-file chunk concurrency
#8
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


75 changes: 75 additions & 0 deletions
PLAN.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to:

- Sync engine, file versioning, GUI, cross-account migration
- Workspaces (`workspaces-list/use/unset`)

---

# Performance: within-file chunk concurrency (added 2026-06-29)

File-level (batch) concurrency ALREADY exists: `cli.py` uses a
`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/
`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the
transfer of a SINGLE large file. Two opportunities, ported from the
filen-python / filen-dart "bounded chunk concurrency" work (read those repos'
PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first).

Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption
is inherently sequential — like a running hash), and **S3** storage (multipart
parts on upload; a single presigned object on download).

## Step A — parallel multipart part uploads ⬜ TODO
`services/drive.py` → `_perform_network_upload` (~line 185): the sequential
`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part
(`encryptor.update`), updates `sha`, and PUTs it via
`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`)
before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart;
smaller files are a single PUT — keep them sequential.

Approach (mirror filen Step 1):
- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN
 ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then
 hands `(part_index, urls[part_index], ciphertext)` to a bounded
 `ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag /
 `DriveService` attr).
- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size
 the list and assign BY INDEX (parts finish out of order).
- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))`
 before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live.
- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber,
 then `api.finish_upload(...)` unchanged.

CONSTRAINTS — do NOT just flip the loop:
1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer
 (keystream + content hash are order-dependent). Only the PUT is parallel.
2. `parts_manifest` ordered by PartNumber regardless of completion order.
3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough).
4. On part failure, surface after join; let the existing checkpoint/resume path
 (`create_upload_checkpoint`, `services/drive.py:898`) handle restart.

Win is the single-large-file case (batch is already file-parallel).

## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag)
`services/drive.py` → `download_file` (~line 1616): today one
`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL
(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to
disk. S3 presigned GETs honor HTTP `Range`.

Approach:
- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)`
 (sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool.
- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or
 `os.pwrite`); pre-size with `f.truncate(file_size)`.
- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is
 `initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic,
 bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned
 offset. Then each range decrypts independently. Verify the existing content-hash.

CONSTRAINTS:
1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned.
2. Bound bytes-in-flight (`_mem_acquire`).
3. Fall back to the sequential single-stream path on 200-not-206 or small files.

## Tests (unit hermetic + live)
Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered;
seekable-CTR decrypt reproduces plaintext; small files stay sequential.
Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up):
- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline
- [ ] interrupted multipart upload resumes (checkpoint) and completes
- [ ] ranged download of a large file is byte-exact + faster than single-stream

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
docs(PLAN): spec within-file chunk concurrency#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

CrispStrobe
merged 1 commit into

from

Jun 29, 2026
 Merged


docs(PLAN): spec within-file chunk concurrency
#8
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


75 changes: 75 additions & 0 deletions
PLAN.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to:

- Sync engine, file versioning, GUI, cross-account migration
- Workspaces (`workspaces-list/use/unset`)

---

# Performance: within-file chunk concurrency (added 2026-06-29)

File-level (batch) concurrency ALREADY exists: `cli.py` uses a
`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/
`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the
transfer of a SINGLE large file. Two opportunities, ported from the
filen-python / filen-dart "bounded chunk concurrency" work (read those repos'
PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first).

Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption
is inherently sequential — like a running hash), and **S3** storage (multipart
parts on upload; a single presigned object on download).

## Step A — parallel multipart part uploads ⬜ TODO
`services/drive.py` → `_perform_network_upload` (~line 185): the sequential
`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part
(`encryptor.update`), updates `sha`, and PUTs it via
`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`)
before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart;
smaller files are a single PUT — keep them sequential.

Approach (mirror filen Step 1):
- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN
 ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then
 hands `(part_index, urls[part_index], ciphertext)` to a bounded
 `ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag /
 `DriveService` attr).
- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size
 the list and assign BY INDEX (parts finish out of order).
- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))`
 before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live.
- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber,
 then `api.finish_upload(...)` unchanged.

CONSTRAINTS — do NOT just flip the loop:
1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer
 (keystream + content hash are order-dependent). Only the PUT is parallel.
2. `parts_manifest` ordered by PartNumber regardless of completion order.
3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough).
4. On part failure, surface after join; let the existing checkpoint/resume path
 (`create_upload_checkpoint`, `services/drive.py:898`) handle restart.

Win is the single-large-file case (batch is already file-parallel).

## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag)
`services/drive.py` → `download_file` (~line 1616): today one
`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL
(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to
disk. S3 presigned GETs honor HTTP `Range`.

Approach:
- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)`
 (sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool.
- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or
 `os.pwrite`); pre-size with `f.truncate(file_size)`.
- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is
 `initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic,
 bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned
 offset. Then each range decrypts independently. Verify the existing content-hash.

CONSTRAINTS:
1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned.
2. Bound bytes-in-flight (`_mem_acquire`).
3. Fall back to the sequential single-stream path on 200-not-206 or small files.

## Tests (unit hermetic + live)
Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered;
seekable-CTR decrypt reproduces plaintext; small files stay sequential.
Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up):
- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline
- [ ] interrupted multipart upload resumes (checkpoint) and completes
- [ ] ranged download of a large file is byte-exact + faster than single-stream

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
docs(PLAN): spec within-file chunk concurrency#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

CrispStrobe
merged 1 commit into

from

Jun 29, 2026
 Merged


docs(PLAN): spec within-file chunk concurrency
#8
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


75 changes: 75 additions & 0 deletions
PLAN.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to:

- Sync engine, file versioning, GUI, cross-account migration
- Workspaces (`workspaces-list/use/unset`)

---

# Performance: within-file chunk concurrency (added 2026-06-29)

File-level (batch) concurrency ALREADY exists: `cli.py` uses a
`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/
`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the
transfer of a SINGLE large file. Two opportunities, ported from the
filen-python / filen-dart "bounded chunk concurrency" work (read those repos'
PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first).

Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption
is inherently sequential — like a running hash), and **S3** storage (multipart
parts on upload; a single presigned object on download).

## Step A — parallel multipart part uploads ⬜ TODO
`services/drive.py` → `_perform_network_upload` (~line 185): the sequential
`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part
(`encryptor.update`), updates `sha`, and PUTs it via
`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`)
before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart;
smaller files are a single PUT — keep them sequential.

Approach (mirror filen Step 1):
- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN
 ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then
 hands `(part_index, urls[part_index], ciphertext)` to a bounded
 `ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag /
 `DriveService` attr).
- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size
 the list and assign BY INDEX (parts finish out of order).
- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))`
 before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live.
- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber,
 then `api.finish_upload(...)` unchanged.

CONSTRAINTS — do NOT just flip the loop:
1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer
 (keystream + content hash are order-dependent). Only the PUT is parallel.
2. `parts_manifest` ordered by PartNumber regardless of completion order.
3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough).
4. On part failure, surface after join; let the existing checkpoint/resume path
 (`create_upload_checkpoint`, `services/drive.py:898`) handle restart.

Win is the single-large-file case (batch is already file-parallel).

## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag)
`services/drive.py` → `download_file` (~line 1616): today one
`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL
(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to
disk. S3 presigned GETs honor HTTP `Range`.

Approach:
- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)`
 (sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool.
- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or
 `os.pwrite`); pre-size with `f.truncate(file_size)`.
- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is
 `initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic,
 bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned
 offset. Then each range decrypts independently. Verify the existing content-hash.

CONSTRAINTS:
1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned.
2. Bound bytes-in-flight (`_mem_acquire`).
3. Fall back to the sequential single-stream path on 200-not-206 or small files.

## Tests (unit hermetic + live)
Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered;
seekable-CTR decrypt reproduces plaintext; small files stay sequential.
Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up):
- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline
- [ ] interrupted multipart upload resumes (checkpoint) and completes
- [ ] ranged download of a large file is byte-exact + faster than single-stream

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
docs(PLAN): spec within-file chunk concurrency#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

CrispStrobe
merged 1 commit into

from

Jun 29, 2026
 Merged


docs(PLAN): spec within-file chunk concurrency
#8
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


75 changes: 75 additions & 0 deletions
PLAN.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to:

- Sync engine, file versioning, GUI, cross-account migration
- Workspaces (`workspaces-list/use/unset`)

---

# Performance: within-file chunk concurrency (added 2026-06-29)

File-level (batch) concurrency ALREADY exists: `cli.py` uses a
`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/
`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the
transfer of a SINGLE large file. Two opportunities, ported from the
filen-python / filen-dart "bounded chunk concurrency" work (read those repos'
PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first).

Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption
is inherently sequential — like a running hash), and **S3** storage (multipart
parts on upload; a single presigned object on download).

## Step A — parallel multipart part uploads ⬜ TODO
`services/drive.py` → `_perform_network_upload` (~line 185): the sequential
`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part
(`encryptor.update`), updates `sha`, and PUTs it via
`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`)
before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart;
smaller files are a single PUT — keep them sequential.

Approach (mirror filen Step 1):
- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN
 ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then
 hands `(part_index, urls[part_index], ciphertext)` to a bounded
 `ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag /
 `DriveService` attr).
- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size
 the list and assign BY INDEX (parts finish out of order).
- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))`
 before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live.
- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber,
 then `api.finish_upload(...)` unchanged.

CONSTRAINTS — do NOT just flip the loop:
1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer
 (keystream + content hash are order-dependent). Only the PUT is parallel.
2. `parts_manifest` ordered by PartNumber regardless of completion order.
3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough).
4. On part failure, surface after join; let the existing checkpoint/resume path
 (`create_upload_checkpoint`, `services/drive.py:898`) handle restart.

Win is the single-large-file case (batch is already file-parallel).

## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag)
`services/drive.py` → `download_file` (~line 1616): today one
`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL
(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to
disk. S3 presigned GETs honor HTTP `Range`.

Approach:
- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)`
 (sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool.
- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or
 `os.pwrite`); pre-size with `f.truncate(file_size)`.
- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is
 `initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic,
 bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned
 offset. Then each range decrypts independently. Verify the existing content-hash.

CONSTRAINTS:
1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned.
2. Bound bytes-in-flight (`_mem_acquire`).
3. Fall back to the sequential single-stream path on 200-not-206 or small files.

## Tests (unit hermetic + live)
Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered;
seekable-CTR decrypt reproduces plaintext; small files stay sequential.
Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up):
- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline
- [ ] interrupted multipart upload resumes (checkpoint) and completes
- [ ] ranged download of a large file is byte-exact + faster than single-stream

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
docs(PLAN): spec within-file chunk concurrency#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

CrispStrobe
merged 1 commit into

from

Jun 29, 2026
 Merged


docs(PLAN): spec within-file chunk concurrency
#8
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


75 changes: 75 additions & 0 deletions
PLAN.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,3 +51,78 @@ to become queryable). The retry pattern is applied to:

- Sync engine, file versioning, GUI, cross-account migration
- Workspaces (`workspaces-list/use/unset`)

---

# Performance: within-file chunk concurrency (added 2026-06-29)

File-level (batch) concurrency ALREADY exists: `cli.py` uses a
`ThreadPoolExecutor` + `--workers`, gated by `DriveService._mem_acquire`/
`_mem_release` (`services/drive.py:121,147`). What is still SEQUENTIAL is the
transfer of a SINGLE large file. Two opportunities, ported from the
filen-python / filen-dart "bounded chunk concurrency" work (read those repos'
PLAN.md "Performance" section + filen-dart `LEARNINGS.md` first).

Internxt differs from Filen: **AES-CTR, one continuous keystream** (encryption
is inherently sequential — like a running hash), and **S3** storage (multipart
parts on upload; a single presigned object on download).

## Step A — parallel multipart part uploads ⬜ TODO
`services/drive.py` → `_perform_network_upload` (~line 185): the sequential
`for part_index in range(parts):` loop (~238) that encrypts a 30 MB part
(`encryptor.update`), updates `sha`, and PUTs it via
`_put_with_retry(lambda: api.upload_part(urls[i], ciphertext))` (`utils/api.py:423`)
before the next. Only files ≥ `MULTIPART_MIN_SIZE` (100 MiB) use multipart;
smaller files are a single PUT — keep them sequential.

Approach (mirror filen Step 1):
- A SEQUENTIAL producer reads each part, runs `encryptor.update(plaintext)` IN
 ORDER (CTR keystream is stateful — CANNOT parallelize) and updates `sha`, then
 hands `(part_index, urls[part_index], ciphertext)` to a bounded
 `ThreadPoolExecutor(max_workers=N)` (N=4 default; add a `--chunk-workers` flag /
 `DriveService` attr).
- Each worker PUTs and writes its ETag to `parts_manifest[part_index]` — pre-size
 the list and assign BY INDEX (parts finish out of order).
- Bound bytes-in-flight with the EXISTING gate: `self._mem_acquire(len(ciphertext))`
 before dispatch, `self._mem_release(...)` in the worker `finally`. N×30 MB live.
- After join: assert `bytes_done == file_size`, manifest is ordered by PartNumber,
 then `api.finish_upload(...)` unchanged.

CONSTRAINTS — do NOT just flip the loop:
1. CTR `encryptor.update` + `sha.update` stay strictly sequential in the producer
 (keystream + content hash are order-dependent). Only the PUT is parallel.
2. `parts_manifest` ordered by PartNumber regardless of completion order.
3. Bound by BYTES in flight via `_mem_acquire` (30 MB parts — count is not enough).
4. On part failure, surface after join; let the existing checkpoint/resume path
 (`create_upload_checkpoint`, `services/drive.py:898`) handle restart.

Win is the single-large-file case (batch is already file-parallel).

## Step B — parallel ranged downloads ⬜ TODO (riskier; gate behind a flag)
`services/drive.py` → `download_file` (~line 1616): today one
`api.download_stream(url)` (`utils/api.py:461`) over a single presigned S3 URL
(`api.get_download_links(...)['shards'][0]['url']`), streamed + CTR-decrypted to
disk. S3 presigned GETs honor HTTP `Range`.

Approach:
- Split into N aligned ranges (e.g. 30 MB). Add `api.download_range(url, start, end)`
 (sends `Range: bytes=a-b`; expect 206). Fetch ranges with a bounded pool.
- Write each range at its offset (`f.seek(off); f.write(...)` under a lock, or
 `os.pwrite`); pre-size with `f.truncate(file_size)`.
- HARD PART — seekable CTR decrypt: AES-CTR's counter for byte offset O is
 `initial_counter + O//16`. Add `crypto.new_download_decryptor_at(mnemonic,
 bucket_id, file_index_hex, offset)` that inits the counter for a 16-B-aligned
 offset. Then each range decrypts independently. Verify the existing content-hash.

CONSTRAINTS:
1. Range starts MUST be AES-block-aligned (16 B); 30 MB is aligned.
2. Bound bytes-in-flight (`_mem_acquire`).
3. Fall back to the sequential single-stream path on 200-not-206 or small files.

## Tests (unit hermetic + live)
Unit (mock `utils/api.py` PUT/GET): assert peak in-flight ≤ N; manifest ordered;
seekable-CTR decrypt reproduces plaintext; small files stay sequential.
Live (mirror `tests/test_live_smoke.py` skip gate; confine to a temp folder, clean up):
- [ ] ≥100 MB upload: byte-exact round-trip + faster than the sequential baseline
- [ ] interrupted multipart upload resumes (checkpoint) and completes
- [ ] ranged download of a large file is byte-exact + faster than single-stream