Skip to content

ARROW-37: [C++ / Python] Implement BooleanArray and BooleanBuilder. Handle Python built-in bool - #40

Closed
wesm wants to merge 8 commits into
apache:masterfrom
wesm:ARROW-37
Closed

ARROW-37: [C++ / Python] Implement BooleanArray and BooleanBuilder. Handle Python built-in bool#40
wesm wants to merge 8 commits into
apache:masterfrom
wesm:ARROW-37

Conversation

@wesm

@wesmwesm commented Mar 25, 2016

Copy link
Copy Markdown
Member

This patch implements arrow::Type::BOOL in bit-packed form.

ARROW-59 (Support for boolean in Python lists) is also addressed here.

@wesmwesm changed the title ARROW-37: [C++] Implement BooleanArray and BooleanBuilderARROW-37: [C++ / Python] Implement BooleanArray and BooleanBuilder. Handle Python built-in boolMar 25, 2016
@wesm

wesm commented Mar 25, 2016

Copy link
Copy Markdown
MemberAuthor

Here's what this looks like in Python note:

In [9]: arr=pyarrow.from_pylist([True, False, None])
In [10]: arrOut[10]: <pyarrow.array.BooleanArrayobjectat0x7f8db43566d8>
[
True,
False,
NA
]
In [11]: arr[0]
Out[11]: True

@wesm
wesm deleted the ARROW-37 branch March 25, 2016 23:52
@wesm

wesm commented Mar 25, 2016

Copy link
Copy Markdown
MemberAuthor

Reviews welcome. I need this to move on to a patch to add conversion between pandas.DataFrame and Arrow tables

wesm added a commit to wesm/arrow that referenced this pull request Sep 8, 2018
@wesm@julienledem
PARQUET-455: Fix OS X / Clang compiler warnings
5e91cff
There actually was a legitimate bug fixed here for malformed Parquet files, but we are not yet in a position to write a decent test for it until PARQUET-497. I will make a note on that JIRA.
I also set our Travis CI build to fail on future compiler warnings.
This also closesapache#15.
Author: Wes McKinney <wes@cloudera.com>
Closesapache#40 from wesm/PARQUET-455 and squashes the following commits:
a348063 [Wes McKinney] Compiler warnings fail the build
271d71e [Wes McKinney] Fix OS X / Clang compiler warnings
Change-Id: I8b89f98506f59bb5b207b6cf1b0f2ea5d17c2585
@guyuqiguyuqi mentioned this pull request Nov 22, 2018
Closed
kou pushed a commit that referenced this pull request May 10, 2020
@kiszk@kou
ARROW-8754: [C++][CI] Enable additional tests on s390x
392cb9e
This PR enables tests for `ARROW_COMPUTE`, `ARROW_DATASET`, `ARROW_FILESYSTEM`, `ARROW_HDFS`, `ARROW_ORC`, and `ARROW_IPC` (default on). #7131 enabled a minimal set of tests as a starting point.
I confirmed that these tests pass locally with the current master. In the current TravisCI environment, we cannot see this result due to a lot of error messages in `arrow-utility-test`.
```
$ git log | head -1
commit ed5f534
% ctest
...
Start 1: arrow-array-test
1/51 Test #1: arrow-array-test ..................... Passed 4.62 sec
Start 2: arrow-buffer-test
2/51 Test #2: arrow-buffer-test .................... Passed 0.14 sec
Start 3: arrow-extension-type-test
3/51 Test #3: arrow-extension-type-test ............ Passed 0.12 sec
Start 4: arrow-misc-test
4/51 Test #4: arrow-misc-test ...................... Passed 0.14 sec
Start 5: arrow-public-api-test
5/51 Test #5: arrow-public-api-test ................ Passed 0.12 sec
Start 6: arrow-scalar-test
6/51 Test #6: arrow-scalar-test .................... Passed 0.13 sec
Start 7: arrow-type-test
7/51 Test #7: arrow-type-test ...................... Passed 0.14 sec
Start 8: arrow-table-test
8/51 Test #8: arrow-table-test ..................... Passed 0.13 sec
Start 9: arrow-tensor-test
9/51 Test #9: arrow-tensor-test .................... Passed 0.13 sec
Start 10: arrow-sparse-tensor-test
10/51 Test #10: arrow-sparse-tensor-test ............. Passed 0.16 sec
Start 11: arrow-stl-test
11/51 Test #11: arrow-stl-test ....................... Passed 0.12 sec
Start 12: arrow-concatenate-test
12/51 Test #12: arrow-concatenate-test ............... Passed 0.53 sec
Start 13: arrow-diff-test
13/51 Test #13: arrow-diff-test ...................... Passed 1.45 sec
Start 14: arrow-c-bridge-test
14/51 Test #14: arrow-c-bridge-test .................. Passed 0.18 sec
Start 15: arrow-io-buffered-test
15/51 Test #15: arrow-io-buffered-test ............... Passed 0.20 sec
Start 16: arrow-io-compressed-test
16/51 Test #16: arrow-io-compressed-test ............. Passed 3.48 sec
Start 17: arrow-io-file-test
17/51 Test #17: arrow-io-file-test ................... Passed 0.74 sec
Start 18: arrow-io-hdfs-test
18/51 Test #18: arrow-io-hdfs-test ................... Passed 0.12 sec
Start 19: arrow-io-memory-test
19/51 Test #19: arrow-io-memory-test ................. Passed 2.77 sec
Start 20: arrow-utility-test
20/51 Test #20: arrow-utility-test ...................***Failed 5.65 sec
Start 21: arrow-threading-utility-test
21/51 Test #21: arrow-threading-utility-test ......... Passed 1.34 sec
Start 22: arrow-compute-compute-test
22/51 Test #22: arrow-compute-compute-test ........... Passed 0.13 sec
Start 23: arrow-compute-boolean-test
23/51 Test #23: arrow-compute-boolean-test ........... Passed 0.15 sec
Start 24: arrow-compute-cast-test
24/51 Test #24: arrow-compute-cast-test .............. Passed 0.22 sec
Start 25: arrow-compute-hash-test
25/51 Test #25: arrow-compute-hash-test .............. Passed 2.61 sec
Start 26: arrow-compute-isin-test
26/51 Test #26: arrow-compute-isin-test .............. Passed 0.81 sec
Start 27: arrow-compute-match-test
27/51 Test #27: arrow-compute-match-test ............. Passed 0.40 sec
Start 28: arrow-compute-sort-to-indices-test
28/51 Test #28: arrow-compute-sort-to-indices-test ... Passed 3.33 sec
Start 29: arrow-compute-nth-to-indices-test
29/51 Test #29: arrow-compute-nth-to-indices-test .... Passed 1.51 sec
Start 30: arrow-compute-util-internal-test
30/51 Test #30: arrow-compute-util-internal-test ..... Passed 0.13 sec
Start 31: arrow-compute-add-test
31/51 Test #31: arrow-compute-add-test ............... Passed 0.12 sec
Start 32: arrow-compute-aggregate-test
32/51 Test #32: arrow-compute-aggregate-test ......... Passed 14.70 sec
Start 33: arrow-compute-compare-test
33/51 Test #33: arrow-compute-compare-test ........... Passed 7.96 sec
Start 34: arrow-compute-take-test
34/51 Test #34: arrow-compute-take-test .............. Passed 4.80 sec
Start 35: arrow-compute-filter-test
35/51 Test #35: arrow-compute-filter-test ............ Passed 8.23 sec
Start 36: arrow-dataset-dataset-test
36/51 Test #36: arrow-dataset-dataset-test ........... Passed 0.25 sec
Start 37: arrow-dataset-discovery-test
37/51 Test #37: arrow-dataset-discovery-test ......... Passed 0.13 sec
Start 38: arrow-dataset-file-ipc-test
38/51 Test #38: arrow-dataset-file-ipc-test .......... Passed 0.21 sec
Start 39: arrow-dataset-file-test
39/51 Test #39: arrow-dataset-file-test .............. Passed 0.12 sec
Start 40: arrow-dataset-filter-test
40/51 Test #40: arrow-dataset-filter-test ............ Passed 0.16 sec
Start 41: arrow-dataset-partition-test
41/51 Test #41: arrow-dataset-partition-test ......... Passed 0.13 sec
Start 42: arrow-dataset-scanner-test
42/51 Test #42: arrow-dataset-scanner-test ........... Passed 0.20 sec
Start 43: arrow-filesystem-test
43/51 Test #43: arrow-filesystem-test ................ Passed 1.62 sec
Start 44: arrow-hdfs-test
44/51 Test #44: arrow-hdfs-test ...................... Passed 0.13 sec
Start 45: arrow-feather-test
45/51 Test #45: arrow-feather-test ................... Passed 0.91 sec
Start 46: arrow-ipc-read-write-test
46/51 Test #46: arrow-ipc-read-write-test ............ Passed 5.77 sec
Start 47: arrow-ipc-json-simple-test
47/51 Test #47: arrow-ipc-json-simple-test ........... Passed 0.16 sec
Start 48: arrow-ipc-json-test
48/51 Test #48: arrow-ipc-json-test .................. Passed 0.27 sec
Start 49: arrow-json-integration-test
49/51 Test #49: arrow-json-integration-test .......... Passed 0.13 sec
Start 50: arrow-json-test
50/51 Test #50: arrow-json-test ...................... Passed 0.26 sec
Start 51: arrow-orc-adapter-test
51/51 Test #51: arrow-orc-adapter-test ............... Passed 1.92 sec
98% tests passed, 1 tests failed out of 51
Label Time Summary:
arrow-tests = 27.38 sec (27 tests)
arrow_compute = 45.11 sec (14 tests)
arrow_dataset = 1.21 sec (7 tests)
arrow_ipc = 6.20 sec (3 tests)
unittest = 79.91 sec (51 tests)
Total Test time (real) = 79.99 sec
The following tests FAILED:
20 - arrow-utility-test (Failed)
Errors while running CTest
```
Closes#7142 from kiszk/ARROW-8754
Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
zhouyuan added a commit to zhouyuan/arrow that referenced this pull request May 15, 2020
@zhouyuan
Merge pull requestapache#40from zhouyuan/rename_dataset_proto_file
7fed75a
[C++] rename proto file for dataset
jayhomn-bitquill referenced this pull request in Bit-Quill/arrow Aug 10, 2022
@escobargabriel
Merge pull request#40from rafael-telles/fix-day-and-year-interval-f…
7f1eeec
…ormat
Change the behavior to return the correct format for day and year intervals.
@paleolimbotpaleolimbot mentioned this pull request Jan 28, 2023
Closed
@prniiiprniii mentioned this pull request Jan 12, 2024
Open
@jonkeanejonkeane mentioned this pull request May 11, 2025
Closed
cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 20, 2026
@cbb330@claude
Task#15: Implement TryCountRows for count optimization (apache#40)
d175fa0
- Implements fast-path row counting using stripe statistics
- Returns count directly if predicate can be fully evaluated from metadata
- Falls back to full scan (returns nullopt) if statistics insufficient
Algorithm:
1. No field refs (e.g., WHERE 1=1): sum all stripe row counts
2. With predicates: evaluate against each stripe via TestStripes
- literal(false): exclude stripe from count
- literal(true): include stripe row count
- partial match: return nullopt (need full scan)
Mirrors Parquet's TryCountRows (file_parquet.cc:986-1004)
Use case: COUNT(*) WHERE x > 1000 on file with strong statistics
can avoid reading data entirely if statistics prove stripes match/exclude.
Verified: Logic matches Parquet pattern, uses correct ORC APIs
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 24, 2026
@cbb330@claude
Task#15: Implement TryCountRows for count optimization (apache#40)
3d4ea4e
- Implements fast-path row counting using stripe statistics
- Returns count directly if predicate can be fully evaluated from metadata
- Falls back to full scan (returns nullopt) if statistics insufficient
Algorithm:
1. No field refs (e.g., WHERE 1=1): sum all stripe row counts
2. With predicates: evaluate against each stripe via TestStripes
- literal(false): exclude stripe from count
- literal(true): include stripe row count
- partial match: return nullopt (need full scan)
Mirrors Parquet's TryCountRows (file_parquet.cc:986-1004)
Use case: COUNT(*) WHERE x > 1000 on file with strong statistics
can avoid reading data entirely if statistics prove stripes match/exclude.
Verified: Logic matches Parquet pattern, uses correct ORC APIs
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@wesm