ARROW-67: C++ metadata flatbuffer serialization and data movement to memory maps by wesm · Pull Request #28 · apache/arrow

wesm · 2016-03-17T07:38:04Z

Several things here:

Add Google flatbuffers dependency
Flatbuffers IDL draft in collaboration with @jacques-n and @StevenMPhillips
Add Schema wrapper in Cython
arrow::Schema conversion to/from flatbuffer representation
Remove unneeded physical layout types from type.h
Refactor ListType to be a nested type with a single child
Implement shared memory round-trip for numeric row batches
mmap-based shared memory interface and MemorySource abstract API

Quite a bit of judicious code cleaning and consolidation as part of this. For example, List types are now internally equivalent to a nested type with 1 named child field (versus a struct, which can have any number of child fields).

Associated JIRAs: ARROW-48, ARROW-57, ARROW-58

wesm · 2016-03-17T07:44:01Z

FWIW, I have only good things to say about Flatbuffers (https://github.com/google/flatbuffers) so far. It's only metadata (serializing data itself with flatbuffers would not be appropriate for several reasons), but it's fast and most importantly lightweight (compared with the alternatives, e.g. Thrift).

xhochy · 2016-03-17T08:24:25Z

Use PATH instead of DIRECTORY here to support CMake 2.8

thank you, fixed

wesm · 2016-03-19T05:50:41Z

I'm calling a checkpoint here (before the patch gets much larger) as I've got simple integer row batches round-tripping successfully to a memory map. Sorry this got so big; lot of scaffolding needed to make this all work.

@StevenMPhillips I wouldn't mind a review of the code in adapter.h/adapter.cc to see if the logic jives with your thinking about the Java side.

I'll make a pass at incorporating feedback from #16. There's a long list of follow ups to incorporate all aspects of the Arrow spec into the IPC code, so I'll create lots of follow up JIRAs to break things into more bite-sized pieces.

wesm · 2016-03-19T18:50:50Z

Incorporated code review comments from #16.

emkornfield · 2016-03-21T01:49:58Z

it looks like this might conflict with #29 as well

…instantiation

wesm · 2016-03-23T01:12:39Z

- bool Equals(const Array& arr) const;
- virtual bool Equals(const std::shared_ptr<Array>& arr) const;
+ bool EqualsExact(const Array& arr) const;
+ virtual bool Equals(const std::shared_ptr<Array>& arr) const = 0;


Clang revealed that this (plus the downstream PrimitiveArrayImpl templates) was causing some bad behavior, so we can review in the future how to have the cleanest code for the proliferation of array containers while avoiding unwanted template instantiation (I admit I don't know enough about this to make a judgment

wesm · 2016-03-23T01:44:29Z

+1, thank you all

julienledem · 2016-03-30T18:33:25Z

+ List,
+ Tuple,
+ Union,
+ JSONScalar


If this is an alias for Union<Int, ...> it should probably not be in this type union.
Does flatbuff have constant literals?

Author: Wes McKinney <wes@cloudera.com> Closesapache#28 from wesm/PARQUET-439 and squashes the following commits: 7c0712b [Wes McKinney] PARQUET-439: Conform copyright headers to ASF requirements Change-Id: I23c1c61346cc4d617b2124e0a6ad208a24aeb4c5

This PR enables tests for `ARROW_COMPUTE`, `ARROW_DATASET`, `ARROW_FILESYSTEM`, `ARROW_HDFS`, `ARROW_ORC`, and `ARROW_IPC` (default on). #7131 enabled a minimal set of tests as a starting point. I confirmed that these tests pass locally with the current master. In the current TravisCI environment, we cannot see this result due to a lot of error messages in `arrow-utility-test`. ``` $ git log | head -1 commit ed5f534 % ctest ... Start 1: arrow-array-test 1/51 Test #1: arrow-array-test ..................... Passed 4.62 sec Start 2: arrow-buffer-test 2/51 Test #2: arrow-buffer-test .................... Passed 0.14 sec Start 3: arrow-extension-type-test 3/51 Test #3: arrow-extension-type-test ............ Passed 0.12 sec Start 4: arrow-misc-test 4/51 Test #4: arrow-misc-test ...................... Passed 0.14 sec Start 5: arrow-public-api-test 5/51 Test #5: arrow-public-api-test ................ Passed 0.12 sec Start 6: arrow-scalar-test 6/51 Test #6: arrow-scalar-test .................... Passed 0.13 sec Start 7: arrow-type-test 7/51 Test #7: arrow-type-test ...................... Passed 0.14 sec Start 8: arrow-table-test 8/51 Test #8: arrow-table-test ..................... Passed 0.13 sec Start 9: arrow-tensor-test 9/51 Test #9: arrow-tensor-test .................... Passed 0.13 sec Start 10: arrow-sparse-tensor-test 10/51 Test #10: arrow-sparse-tensor-test ............. Passed 0.16 sec Start 11: arrow-stl-test 11/51 Test #11: arrow-stl-test ....................... Passed 0.12 sec Start 12: arrow-concatenate-test 12/51 Test #12: arrow-concatenate-test ............... Passed 0.53 sec Start 13: arrow-diff-test 13/51 Test #13: arrow-diff-test ...................... Passed 1.45 sec Start 14: arrow-c-bridge-test 14/51 Test #14: arrow-c-bridge-test .................. Passed 0.18 sec Start 15: arrow-io-buffered-test 15/51 Test #15: arrow-io-buffered-test ............... Passed 0.20 sec Start 16: arrow-io-compressed-test 16/51 Test #16: arrow-io-compressed-test ............. Passed 3.48 sec Start 17: arrow-io-file-test 17/51 Test #17: arrow-io-file-test ................... Passed 0.74 sec Start 18: arrow-io-hdfs-test 18/51 Test #18: arrow-io-hdfs-test ................... Passed 0.12 sec Start 19: arrow-io-memory-test 19/51 Test #19: arrow-io-memory-test ................. Passed 2.77 sec Start 20: arrow-utility-test 20/51 Test #20: arrow-utility-test ...................***Failed 5.65 sec Start 21: arrow-threading-utility-test 21/51 Test #21: arrow-threading-utility-test ......... Passed 1.34 sec Start 22: arrow-compute-compute-test 22/51 Test #22: arrow-compute-compute-test ........... Passed 0.13 sec Start 23: arrow-compute-boolean-test 23/51 Test #23: arrow-compute-boolean-test ........... Passed 0.15 sec Start 24: arrow-compute-cast-test 24/51 Test #24: arrow-compute-cast-test .............. Passed 0.22 sec Start 25: arrow-compute-hash-test 25/51 Test #25: arrow-compute-hash-test .............. Passed 2.61 sec Start 26: arrow-compute-isin-test 26/51 Test #26: arrow-compute-isin-test .............. Passed 0.81 sec Start 27: arrow-compute-match-test 27/51 Test #27: arrow-compute-match-test ............. Passed 0.40 sec Start 28: arrow-compute-sort-to-indices-test 28/51 Test #28: arrow-compute-sort-to-indices-test ... Passed 3.33 sec Start 29: arrow-compute-nth-to-indices-test 29/51 Test #29: arrow-compute-nth-to-indices-test .... Passed 1.51 sec Start 30: arrow-compute-util-internal-test 30/51 Test #30: arrow-compute-util-internal-test ..... Passed 0.13 sec Start 31: arrow-compute-add-test 31/51 Test #31: arrow-compute-add-test ............... Passed 0.12 sec Start 32: arrow-compute-aggregate-test 32/51 Test #32: arrow-compute-aggregate-test ......... Passed 14.70 sec Start 33: arrow-compute-compare-test 33/51 Test #33: arrow-compute-compare-test ........... Passed 7.96 sec Start 34: arrow-compute-take-test 34/51 Test #34: arrow-compute-take-test .............. Passed 4.80 sec Start 35: arrow-compute-filter-test 35/51 Test #35: arrow-compute-filter-test ............ Passed 8.23 sec Start 36: arrow-dataset-dataset-test 36/51 Test #36: arrow-dataset-dataset-test ........... Passed 0.25 sec Start 37: arrow-dataset-discovery-test 37/51 Test #37: arrow-dataset-discovery-test ......... Passed 0.13 sec Start 38: arrow-dataset-file-ipc-test 38/51 Test #38: arrow-dataset-file-ipc-test .......... Passed 0.21 sec Start 39: arrow-dataset-file-test 39/51 Test #39: arrow-dataset-file-test .............. Passed 0.12 sec Start 40: arrow-dataset-filter-test 40/51 Test #40: arrow-dataset-filter-test ............ Passed 0.16 sec Start 41: arrow-dataset-partition-test 41/51 Test #41: arrow-dataset-partition-test ......... Passed 0.13 sec Start 42: arrow-dataset-scanner-test 42/51 Test #42: arrow-dataset-scanner-test ........... Passed 0.20 sec Start 43: arrow-filesystem-test 43/51 Test #43: arrow-filesystem-test ................ Passed 1.62 sec Start 44: arrow-hdfs-test 44/51 Test #44: arrow-hdfs-test ...................... Passed 0.13 sec Start 45: arrow-feather-test 45/51 Test #45: arrow-feather-test ................... Passed 0.91 sec Start 46: arrow-ipc-read-write-test 46/51 Test #46: arrow-ipc-read-write-test ............ Passed 5.77 sec Start 47: arrow-ipc-json-simple-test 47/51 Test #47: arrow-ipc-json-simple-test ........... Passed 0.16 sec Start 48: arrow-ipc-json-test 48/51 Test #48: arrow-ipc-json-test .................. Passed 0.27 sec Start 49: arrow-json-integration-test 49/51 Test #49: arrow-json-integration-test .......... Passed 0.13 sec Start 50: arrow-json-test 50/51 Test #50: arrow-json-test ...................... Passed 0.26 sec Start 51: arrow-orc-adapter-test 51/51 Test #51: arrow-orc-adapter-test ............... Passed 1.92 sec 98% tests passed, 1 tests failed out of 51 Label Time Summary: arrow-tests = 27.38 sec (27 tests) arrow_compute = 45.11 sec (14 tests) arrow_dataset = 1.21 sec (7 tests) arrow_ipc = 6.20 sec (3 tests) unittest = 79.91 sec (51 tests) Total Test time (real) = 79.99 sec The following tests FAILED: 20 - arrow-utility-test (Failed) Errors while running CTest ``` Closes#7142 from kiszk/ARROW-8754 Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

* apacheGH-36209: [Java] Upgrade Netty due to security vulnerability (apache#36211) Upgrading Netty dependency due to CVE GHSA-6mjq-h674-j845 This also requires a patch to arrow-memory Upgrading Netty, gRPC and Protobuf dependencies Existing tests No **This PR contains a "Critical Fix".** netty-handler SniHandler 16MB allocation The SniHandler can allocate up to 16MB of heap for each channel during the TLS handshake. When the handler or the channel does not have an idle timeout, it can be used to make a TCP server using the SniHandler to allocate 16MB of heap. GHSA-6mjq-h674-j845 * Closes: apache#36209 Authored-by: Bryan Cutler <cutlerb@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com> * Restore jackson version. * Use local based ccache for Mac build instead of sscache. --------- Signed-off-by: David Li <li.davidm96@gmail.com> Co-authored-by: Bryan Cutler <cutlerb@gmail.com>

- Added NullHandling test with 12 comprehensive test cases - Tests IS NULL and IS VALID predicates with statistics - Tests three-valued logic behavior with NULL values - Tests IS NULL with all-null stripe (finds all NULLs) - Tests IS VALID with all-null stripe (skips NULL stripe) - Tests IS NULL AND comparison (always FALSE - NULLs don't satisfy comparisons) - Tests IS NULL OR comparison (finds NULLs plus matching values) - Tests IS VALID AND comparison (finds only valid values) - Tests NOT IS NULL equivalence to IS VALID - Tests NOT IS VALID equivalence to IS NULL - Tests comparison operators with NULL values (don't match NULLs) - Tests equality with NULL values (x = value doesn't match NULL) - Tests range queries with NULL stripes - Tests complex NULL predicates with AND/OR combinations - Tests NOT NULL combined with comparisons - Uses 3-stripe test file with all-null middle stripe Verified: All 12 test cases correctly implement three-valued logic and handle NULL values according to SQL semantics (comparisons with NULL yield UNKNOWN, not TRUE or FALSE). Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

xhochy reviewed Mar 17, 2016
View reviewed changes

wesm mentioned this pull request Mar 17, 2016
ARROW-24: C++: Implement a logical Table container type #16
Closed

wesm force-pushed the cpp-ipc-draft branch from 09d6213 to 043319aCompare March 17, 2016 22:04

wesm changed the title ~~ARROW-67: [WIP] C++ metadata and data movement to memory maps~~ARROW-67: C++ metadata and data movement to memory mapsMar 19, 2016

wesm mentioned this pull request Mar 21, 2016
ARROW-28: Adding google's benchmark library to the toolchain #29
Closed

emkornfield reviewed Mar 21, 2016
View reviewed changes

Comment threadcpp/setup_build_env.sh Outdated
emkornfieldMar 21, 2016
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it looks like this might conflict with #29 as well

wesm force-pushed the cpp-ipc-draft branch from 98c1d7f to 6fa6319Compare March 23, 2016 00:19

Create explicit PrimitiveArray subclasses to avoid unwanted template …
5e841f7
…instantiation

wesm reviewed Mar 23, 2016
View reviewed changes

Add NullArray type now that Array is virtual, fix pyarrow build
0cef7ea

wesm changed the title ~~ARROW-67: C++ metadata and data movement to memory maps~~ARROW-67: C++ metadata flatbuffer serialization and data movement to memory mapsMar 23, 2016

asfgit closed this in 65db0daMar 23, 2016

wesm deleted the cpp-ipc-draft branch March 23, 2016 01:45

julienledem reviewed Mar 30, 2016
View reviewed changes

guyuqi mentioned this pull request Nov 22, 2018
ARROW-3849: [C++] Leverage Armv8 crc32 extension instructions to accelerate the hash computation for Arm64 #3010
Closed

brills mentioned this pull request Jun 4, 2019
importing pyarrow and Tensorflow crashes #4472
Closed

paleolimbot mentioned this pull request Jan 28, 2023
[R] Crash on MacOS (x86) when running tests with homebrew apache-arrow also installed #33903
Closed

jorisvandenbossche mentioned this pull request Oct 6, 2023
GH-38068: [C++][CI] Fixing Parquet unittest arrow_reader_writer_test.cc compile #38069
Merged

prniii mentioned this pull request Jan 12, 2024
[Python] macOS Segfault on Import, Both arm64 and x86_64 #37010
Open

jonkeane mentioned this pull request May 11, 2025
[C++][R]: gcc-UBSAN errors on CRAN #46394
Closed

paddyroddy referenced this pull request in rok/arrow Jul 19, 2025
[pre-commit.ci] pre-commit autoupdate (#28)
d71693a

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 20, 2026
Mark Task#11as complete (apache#28)
ba1fbd1

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 20, 2026
Mark Taskapache#28as complete (apache#67)
758db5e

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 24, 2026
Mark Task#11as complete (apache#28)
4caafd0

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 24, 2026
Mark Taskapache#28as complete (apache#67)
c4ca020

Navigation Menu

Uh oh!

ARROW-67: C++ metadata flatbuffer serialization and data movement to memory maps - #28

ARROW-67: C++ metadata flatbuffer serialization and data movement to memory maps#28
wesm wants to merge 3 commits into
apache:masterfrom
wesm:cpp-ipc-draft

wesm commented Mar 17, 2016

Uh oh!

wesm commented Mar 17, 2016

Uh oh!

xhochyMar 17, 2016

Uh oh!

wesmMar 17, 2016

Uh oh!

wesm commented Mar 19, 2016

Uh oh!

wesm commented Mar 19, 2016

Uh oh!

emkornfieldMar 21, 2016

Uh oh!

wesmMar 23, 2016

Uh oh!

wesm commented Mar 23, 2016

Uh oh!

julienledemMar 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

wesm commented Mar 17, 2016

Uh oh!

wesm commented Mar 17, 2016

Uh oh!

xhochyMar 17, 2016

Choose a reason for hiding this comment

Uh oh!

wesmMar 17, 2016

Choose a reason for hiding this comment

Uh oh!

wesm commented Mar 19, 2016

Uh oh!

wesm commented Mar 19, 2016

Uh oh!

emkornfieldMar 21, 2016

Choose a reason for hiding this comment

Uh oh!

wesmMar 23, 2016

Choose a reason for hiding this comment

Uh oh!

wesm commented Mar 23, 2016

Uh oh!

julienledemMar 30, 2016

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants