Uh oh!
There was an error while loading. Please reload this page.
Rewrite Arrow.jl internals for 3.0 - #609
Draft
quinnj wants to merge 313 commits into
Draft
Conversation
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Return the exact lifecycle state to open when the winning closer is interrupted while waiting for active guards. This keeps the region usable and allows a later close to release it exactly once. Co-Authored-By: Codex <codex@openai.com>
Clarify that same-size mutation can invalidate cached semantic validation for borrowed Julia buffers, while resizing can also invalidate their pointers. Co-Authored-By: Codex <codex@openai.com>
Detect Arrow 0.17's Message-level experimental compression marker before record or dictionary body decoding. This prevents length-prefixed compressed bytes from being exposed as ordinary values. Co-Authored-By: Codex <codex@openai.com>
State that mapped and foreign backing bytes must stay alive and unchanged while Core uses them or their cached validation certificates. External mutation cannot be detected by the prove-out. Co-Authored-By: Codex <codex@openai.com>
Reject unknown flag bits and flags attached to layouts where their semantics do not apply. Failed imports still release both moved lifetimes exactly once. Co-Authored-By: Codex <codex@openai.com>
Document five adversarial findings, their dispositions, scope decisions, and final validation. Update the README review index for round eight. Co-Authored-By: Codex <codex@openai.com>
Construct imported owners without a finalizer, move the source under an interruption-safe handoff, and arm the copied owner only after ownership transfers. Failed post-move setup releases the copied producer callback exactly once. Co-Authored-By: Codex <codex@openai.com>
Keep the claim, recursive release, and completion transaction interrupt-safe. Roll failed claims back to LIVE and retry idempotent descendant progress before the void C callback returns. Co-Authored-By: Codex <codex@openai.com>
Keep each root registered while one reaper owns its cleanup. Record native frees and source-pin releases as progress so interrupted cleanup can resume without leaks or double frees. Co-Authored-By: Codex <codex@openai.com>
Install release and cleanup rollback handlers before their state claims. Preallocated claim slots let task-delivered exceptions restore LIVE or clear the cleanup claim at the exact post-mutation boundary. Co-Authored-By: Codex <codex@openai.com>
Use the source ArrowArray release field as the authoritative move marker. Task cancellation immediately after the move store now releases only the copied owner and cannot strand the producer callback. Co-Authored-By: Codex <codex@openai.com>
Document the three C Data interruption and ownership findings, their dispositions, scope decisions, and final validation evidence. Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Retry interrupted mapping release at both constructor and finalizer ownership boundaries. Check munmap failures before publishing the mapping as released. Co-Authored-By: Codex <codex@openai.com>
Record pins directly in their root and keep stable registry keys across schema and array publication. Make private and multi-root cleanup retry interrupted work without rereading freed C structs. Co-Authored-By: Codex <codex@openai.com>
Publish the aggregate counter, node state, and public release callback as one rollbackable transaction. Transfer the claim under the registry lock so a post-commit exception cannot touch reaped control memory. Co-Authored-By: Codex <codex@openai.com>
Defer interruption across allocation and ledger registration. Free an unregistered allocation exactly once when the handoff fails. Co-Authored-By: Codex <codex@openai.com>
Defer interruption from mmap through owner construction, retry claim rollback, and make close-claim rollback no-escape. Add focused failures after mmap and during rollback. Co-Authored-By: Codex <codex@openai.com>
Carry a Julia-side commit token through the C callback retry loop. Once a root can be reaped, later exceptions return without reading or calling its freed raw pointer. Also clear local malloc ownership inside retryable cleanup so deferred interruption cannot repeat a completed free. Co-Authored-By: Codex <codex@openai.com>
Make guard, close, mmap, and manual-finalizer rollback handoffs retry interruption without stealing a later owner claim. Co-Authored-By: Codex <codex@openai.com>
Keep one persistent moved-array struct for producer callbacks and retry C void releases until they publish release=NULL. Install schema cleanup before owner construction and route schema-finally failures through moved-owner cleanup. Co-Authored-By: Codex <codex@openai.com>
Keep the public pull frame responsible for its claim and speculative index. Roll back a batch advance before releasing the single-puller gate on interruption. Co-Authored-By: Codex <codex@openai.com>
Keep the constructed region reachable behind a cleanup handler until finalizer registration and the public constructor return handoff complete. Co-Authored-By: Codex <codex@openai.com>
Do not let a second interruption escape while the constructor is still the only owner of a release callback. Co-Authored-By: Codex <codex@openai.com>
Document six lifecycle and ownership findings, their dispositions, scope decisions, and final validation. Update the README review index through round ten. Co-Authored-By: Codex <codex@openai.com>
State the committed ownership and cursor handoffs that roll back or retain a cleanup owner. Bound the remaining guarantee to Julia safepoints, safe retries, finalizer-backed owners, and consumer-released C exports. Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nt worker pool, no GCS claim - `SourceFile` requires `sourcelength` to return an `Integer` (a Float, String, or `nothing` is a ValidationError, pinned through `Arrow.Table`) - the range-read worker pool follows the repository guidelines: the shared request counter is an `@atomic` field on a mutable `_SpanQueue` (no `Threads.Atomic`), each worker is `errormonitor(Threads.@Spawn …)`, and the per-worker loop is its own function - the CloudStore extension claims S3 and Azure Blob Storage only: the registered CloudStore releases (1.6–1.8) have no GCS `Object` path Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e extension) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Arrow.jl test cell, the monorepo job, the docs job and the release verification failed at precompile because registered Tables.jl has no `Tables.Scan`. The root Project's `[sources]` now pins Tables to the `jq/scan` branch (read by Pkg from Julia 1.11, including the release verification's Julia 1.12); the 1.10 test cells, the monorepo env and the docs env add it explicitly alongside the local ArrowStrings develop. The docs job installs Julia through setup-julia like the other jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Complete the final correctness, API, documentation, test, CI, conformance, and release-administration pass for the Arrow 3 rewrite. Track the exact reviewed Tables.Scan prerequisite and add focused regressions for its current API and semantics.
Keep ArrowTypes minimum-version coverage on Ubuntu while excluding only the macOS Apple Silicon cells, for which Julia 1.0 has no binary.
Process root and subpackage coverage from their own source directories so the ArrowTypes Julia 1.0 lane never parses Arrow 3 source syntax.
Restore ArrowTypes as a facade dependency and compatibility boundary. Lower and lift custom values recursively, preserve extension metadata, synthesize dense unions, and keep extension filters in the public value domain. Add cross-version, downstream, nested-layout, partition, scan, and malformed-input coverage. Update the migration, release, CI, and conformance guidance for ArrowTypes 2.4 and ArrowStrings 1.0.
… release files - bench/: the 2.x leg (bench_2x.jl, env2x/) is gone; the driver times this package and PyArrow - docs/dev/research-flatbuffers-cdata.md removed - dev/release/README.md and verify_rc.sh are back at main's version Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- `release!` is the one release verb across the API: `release!(::Table)`, `release!(::Stream)`, `release!(::OwnerRegion)`, and the C-data owners. Arrow exports it (and imports ArrowCore's generic so cdata.jl extends it) - ForeignOwner has one verb and one contract: `release!(owner)` revokes every region over the import through the shared cell (later access is an InvalidStateException), then the cell's action runs the producer release exactly once — the raw post-release undefined behaviour is gone - `ArrowStrings.ArrowStringVector` is `ArrowStrings.StringVector` Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`_applyscan` now consumes a `Tables.Scan` exactly through `_ScanSink` (src/scan.jl), on both the in-memory and the byte-range paths: - the filter is evaluated per decoded batch by the generic evaluator (`Tables.filtermask` over the decode set), so Arrow and the executor share one three-valued semantics by construction - `offset`/`limit` compose over the qualifying rows with saturating arithmetic; without a filter the metadata window still skips whole batches, with one decoding stops the moment the window is full - selection and renames are applied at column construction, in selection order; the residual is empty except for type overrides - the battery pins the consumed residual, filter+limit in one pass, and that a filled window stops decoding before a corrupt later batch Tables.jl's `jq/scan` branch moved: `Tables.bind` is `Tables.resolve` (the `BoundScan` carries the name-normalized filter, so Arrow's own resolver is gone), `All()` is the projection identity, and `coleq`/`colne`/`in_` are `colcmp(==, …)`/`colcmp(!=, …)`/`colin` — adapted in src, tests, and manual. Docs: the DESIGN note describes the implemented pushdown (no stages); manual, core-README and README follow; the core-README "Interruption contract" section is removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unify scan planning and column construction around bounded, single-pass seams. Restore ArrowTypes lowering and lifting, remove JSON3 integration, deepen conformance and deterministic fuzz coverage, update documentation, and prepare ArrowStrings 1.0. Preserve empty-scan schema metadata and charge every package-owned allocation.
Tables.jl 1.14.0 ships Tables.Scan, so the temporary source override on the reviewed scan-branch commit is gone: the package, CI workflows, fuzz workflow, conformance image, and docs all resolve Tables from the registry, and the compat floor is 1.14. The released Scan represents "keep every column" as Tables.All() instead of nothing, so the resolver work model drops its now-impossible nothing branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- apply the review round's comment audit: delete rewrite-process narration, re-attach misplaced docstrings, tighten wordy passages, and add missing invariant comments across the IPC reader, FlatBuffers runtime, C data interface, ArrowTypes, and ArrowStrings - narrow ArrowTypes' Union-branch conversion catch to conversion errors so InterruptException propagates - refresh README's development instructions and the dev design doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- accept vtables that understate a wider field's extent, as the reference verifier does: Arrow 2.x's builder deduplicated vtables by slot offsets, so every 2.x file with a dict-encoded, metadata-carrying field (all CategoricalArrays columns) was refused; 2.x-written fixtures pin the fix - surface hostile metadata as ValidationError everywhere in the validation tiers instead of leaking InexactError/OverflowError - guard Int32 offset overflow in the core string and list builders with a clear error - dedup DictEncode pools structurally before construction: identity- keyed pools degraded to O(rows) work for mutable category values - synthesize hidden dictionary slots through the value field, resolving the writer's synthesize/placeholder contradiction - remove dead dictionary construction entry points and apply the comment audit to both files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ArrowTimeZonesExt restores Arrow 2.x ZonedDateTime behavior when
TimeZones.jl is loaded: second/millisecond timezone-declared
timestamps read as ZonedDateTime, round-trip through retained
rewrites, lower exactly in scan filters, and fresh single-zone
ZonedDateTime columns write as timezone-declared millisecond
timestamps
- Tables.Scan select type overrides rebuild composite rows: a
`:col => NamedTuple{...}` (or Vector{...}) target reads Struct and
list columns as typed rows
- Tables.schema returns a typed schema at or below 256 fields so
Tables.rowtable works again; wider schemas stay stored
- restore Arrow.getmetadata(table) over the DataAPI metadata interface
and export ArrowTypes, the one Arrow 2.x export
- add a PrecompileTools workload: first write+read drops ~15.5s -> ~0.2s
- make the ArrowTypes write path's storage lowering monomorphic: custom
logical column writes drop from ~2 allocations per element to O(1)
per column
- harden the Symbol-guard child against JIT interning of Julia IR and
Base local-variable names, which the precompile workload resurfaced
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>- extract the IPC writers' shared loop into IPCWriteState with beginwrite!/writebatch!/finishwrite!; the eager writers drive it and stay byte-identical - Arrow.Writer(sink; file=true) publishes each written table's batches immediately for both formats; the first table fixes the schema and later tables construct against it through the retained path - Arrow.append(sink, table) extends an existing IPC stream in place after validating it, reusing content-identical dictionary pools and emitting replacements when the stream declares the feature - streams whose schema has dictionary fields now always declare DictionaryReplacement, and a codec always declares CompressedBody, so 3.0-written streams stay appendable - restore tobuffer, the curried write form, and accept-and-warn for the removed 2.x writer keywords; document the migration surface Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Arrow.jl keeps the full version/OS/thread matrix; ArrowTypes.jl and ArrowStrings.jl run one minimum-Julia Ubuntu job each - the Arrow.jl matrix already exercises both packages' code everywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nanoarrow, spec-faithfully, refuses any stream declaring a feature it does not support, so the preemptive DictionaryReplacement declaration on every dict-bearing stream broke ours->nanoarrow oracle round-trips for streams containing no replacement at all. - the eager stream writer declares exactly what its batches use, as before the incremental-writer change - Arrow.Writer gains dictreplacement=false: pass true (stream format only) to declare DictionaryReplacement up front so later writes and appends may replace pools; the default writer refuses a changed pool with a message naming the kwarg, and content-identical pools still reuse the emitted dictionary with no declaration - the 2.x null-slot-pool migration recipe becomes one rewrite through a replacement-declaring writer; docs and battery pins updated Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StringVector (and the new BytesVector) gain a Val(:trusted) constructor that
skips per-payload validation for builders that produced every payload from
bounds they already checked, where re-validating doubles construction cost.
ArrowBytes/BytesVector are the BinaryView counterparts of
ArrowString/StringVector over the same 16-byte payload: allocation-free
access, equality/hash agreeing with Vector{UInt8}, and materialize.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Apply JuliaFormatter 2.12.4 to the latest ArrowStrings implementation and test edits so the repository's Format job is deterministic.
davidanthoff
commented
Sep 1, 2026
Contributor
Very cool! One question: could ArrowCore.jl be a real standalone, dependency-free package, similar to how ArrowStrings.jl is split out already? From reading the code, it looks like ArrowCore.jl could contain what it has at the moment in this branch, plus cdata.jl, and the uncompressed IPC read/write path (with the vendored FlatBuffers/Meta layer). And the Tables.jl facade, ArrowTypes lowering, and the compression codecs could then stay in Arrow.jl on top of it. As far as I can tell this split already exists at the code level: ArrowCore only imports Mmap, cdata.jl only uses ArrowCore, and neither ipc_read.jl nor ipc_write.jl references Tables at all — the codecs only appear at the buffer (de)compression step. So ArrowCore.jl would depend on Base and stdlibs only: no binary deps (Lz4_jll/Zstd_jll), no Tables.jl. The two places where the package boundary doesn't quite match the code today seem small: EnumX (but Meta is generated, so the generator could emit self-contained enum modules instead), and compression (which could be a plain dispatch seam — ArrowCore.jl defines compress/decompress generics and errors helpfully on compressed buffers, Arrow.jl keeps its hard codec deps and implements them; so using Arrow behaves exactly as it does now. The reason I'm asking: we could use this in the Julia VS Code extension. The REPL process could use ArrowCore.jl and ArrowStrings.jl to hand large tables to the extension's interactive table viewer — write an uncompressed IPC file, and the TypeScript side reads it with Arrow JS, with the file-format footer giving the viewer random access to just the visible row range. That would be really nice! But the way we ship code that loads into the user's REPL process rules out anything with a binary dependency. And we also couldn't use anything that pulls in Tables.jl, because that would pin the Tables.jl version users see in their REPL to whatever we ship. So this could only work if there is a core package without those deps. (I'd also have some uses for this in Queryverse, but the VS Code one probably the more interesting one) And, as all ideas these days, heavily Claude helped ;) |
quinnj
commented
Sep 4, 2026
MemberAuthor
I think it's close to that, but currently we still have the new ArrowStrings.jl package which provides the native offset + string view representation. We want that as a stand-alone package because I want to be able to use it as a dep for CSV.jl to do direct parse into that representation, I.e. we can parse a csv file directly into a zero-copy arrow representation. So not quite zero-dependency. I'm also not sure just ArrowCore.jl would be quite enough to have arrow JS be able to read easily. Currently we have the CData & IPC layers on top of the core buffers to actually fulfill those interfaces. I do have another project/data format I've been working on that would be a much cleaner/simpler binary table format with cross-language implementations. I've been slowly iterating on it, but it could potentially be a really good fit for this. |
davidanthoff
commented
Sep 4, 2026
Contributor
Yeah, I also think it is super close. I think the ArrowStrings.jl would not be a problem, i.e. if ArrowCore.jl were to depend on ArrowStrings.jl that would work (we use a trick for that kind of situation a lot in the extension). The main thing is really that the compression stuff would have to not be in there, because we can't handle the binary dependencies that are pulled in. And yes, the cdata and ipc layers would have to move into ArrowCore.jl as well. Claude thinks that would be easy, but who knows ;) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tables.Scanonce, keep storage and public-domain plans separate, preserve stable empty-result schemas, and support sparse byte-range reads with footer statisticsDependency
JuliaData/Tables.jl#380 is merged and released as Tables.jl 1.14.0. Commit dd8c272 removes the temporary source and CI pins, raises the Tables.jl compatibility floor to 1.14, and resolves Tables from the General registry everywhere (package, CI workflows, conformance image, docs).
Breaking changes
This is the Arrow.jl 3.0 rewrite. Important changes include:
Arrow.TableandArrow.Streamreturn materialized Julia vectors instead of lazy Arrow vector wrappers.Arrow.writeis eager and whole-buffer.Arrow.Writerand IPC-stream append are reimplemented with fixed-schema semantics; the curried write andtobuffercompatibility forms remain. Obsolete writer tuning keywords are ignored with warnings.Arrow.write(io, table)writes file format by default; usefile=falsefor stream format.ArrowTypesstays exported only as an Arrow 2.x compatibility exception. Packages that define mappings should depend on andimport ArrowTypesdirectly.Arrow.ArrowTypesremains as a qualified compatibility binding.Arrow.Fresh declared heterogeneous Julia Union columns are written as canonical dense Arrow Unions. A retained Union rewrite fails closed when materialization has lost its original route. Nested retained dictionary pools follow the same fail-closed rule.
See
CHANGELOG.mdanddocs/src/migration.mdfor the complete list and migration guidance.Validation
67e17cfpasses 115/115 GitHub checks across the push, pull-request, and Conformance workflows, including Windows Julia latest9878699passes 59/59 GitHub checks across the push, pull-request, and Conformance workflows9878699passes the complete root package suite in a fresh Julia 1.12.7 environment with registered Tables 1.14.0git diff --checkand release-script syntax checks passRelease preparation
@JuliaRegistrator register subdir=src/ArrowStringsstep3.0.0-DEVuntil the rewrite is merged and ArrowStrings 1.0 is registered and releasedCo-authored by Codex