ARROW-70: Add adapt 'lite' DCHECK macros from Kudu as also used in Parquet by wesm · Pull Request #33 · apache/arrow

wesm · 2016-03-23T01:53:02Z

Also added a null pointer DCHECK to show that it works. cc @emkornfield

emkornfield · 2016-03-23T04:25:46Z

 namespace ipc {

 static bool IsPrimitive(const DataType* type) {
+ DCHECK(type != nullptr);


Do you prefer this or DCHECK(type)? For checks like this I think I prefer this without explicit comparison, but its not a strong preference.

I don't have a strong preference either.

emkornfield · 2016-03-23T04:26:01Z

Thank you looks good.

wesm · 2016-03-23T17:58:59Z

+1, thank you.

Validating the input schema and expressions during the projector build.

We were using `--verbose=4` which catches only egregious style errors. `--verbose=2` with a few warning types suppressed is much more reasonable. This patch updates the cmake configuration and fixes the existing style errors to get the build passing. Author: Wes McKinney <wes@cloudera.com> Closesapache#33 from wesm/PARQUET-496 and squashes the following commits: de674d4 [Wes McKinney] Use --verbose=2 for cpplint and fix outstanding style issues Change-Id: Ief9ece0d44653cd80fd611317c5c0642485a6790

Validating the input schema and expressions during the projector build.

This PR enables tests for `ARROW_COMPUTE`, `ARROW_DATASET`, `ARROW_FILESYSTEM`, `ARROW_HDFS`, `ARROW_ORC`, and `ARROW_IPC` (default on). #7131 enabled a minimal set of tests as a starting point. I confirmed that these tests pass locally with the current master. In the current TravisCI environment, we cannot see this result due to a lot of error messages in `arrow-utility-test`. ``` $ git log | head -1 commit ed5f534 % ctest ... Start 1: arrow-array-test 1/51 Test #1: arrow-array-test ..................... Passed 4.62 sec Start 2: arrow-buffer-test 2/51 Test #2: arrow-buffer-test .................... Passed 0.14 sec Start 3: arrow-extension-type-test 3/51 Test #3: arrow-extension-type-test ............ Passed 0.12 sec Start 4: arrow-misc-test 4/51 Test #4: arrow-misc-test ...................... Passed 0.14 sec Start 5: arrow-public-api-test 5/51 Test #5: arrow-public-api-test ................ Passed 0.12 sec Start 6: arrow-scalar-test 6/51 Test #6: arrow-scalar-test .................... Passed 0.13 sec Start 7: arrow-type-test 7/51 Test #7: arrow-type-test ...................... Passed 0.14 sec Start 8: arrow-table-test 8/51 Test #8: arrow-table-test ..................... Passed 0.13 sec Start 9: arrow-tensor-test 9/51 Test #9: arrow-tensor-test .................... Passed 0.13 sec Start 10: arrow-sparse-tensor-test 10/51 Test #10: arrow-sparse-tensor-test ............. Passed 0.16 sec Start 11: arrow-stl-test 11/51 Test #11: arrow-stl-test ....................... Passed 0.12 sec Start 12: arrow-concatenate-test 12/51 Test #12: arrow-concatenate-test ............... Passed 0.53 sec Start 13: arrow-diff-test 13/51 Test #13: arrow-diff-test ...................... Passed 1.45 sec Start 14: arrow-c-bridge-test 14/51 Test #14: arrow-c-bridge-test .................. Passed 0.18 sec Start 15: arrow-io-buffered-test 15/51 Test #15: arrow-io-buffered-test ............... Passed 0.20 sec Start 16: arrow-io-compressed-test 16/51 Test #16: arrow-io-compressed-test ............. Passed 3.48 sec Start 17: arrow-io-file-test 17/51 Test #17: arrow-io-file-test ................... Passed 0.74 sec Start 18: arrow-io-hdfs-test 18/51 Test #18: arrow-io-hdfs-test ................... Passed 0.12 sec Start 19: arrow-io-memory-test 19/51 Test #19: arrow-io-memory-test ................. Passed 2.77 sec Start 20: arrow-utility-test 20/51 Test #20: arrow-utility-test ...................***Failed 5.65 sec Start 21: arrow-threading-utility-test 21/51 Test #21: arrow-threading-utility-test ......... Passed 1.34 sec Start 22: arrow-compute-compute-test 22/51 Test #22: arrow-compute-compute-test ........... Passed 0.13 sec Start 23: arrow-compute-boolean-test 23/51 Test #23: arrow-compute-boolean-test ........... Passed 0.15 sec Start 24: arrow-compute-cast-test 24/51 Test #24: arrow-compute-cast-test .............. Passed 0.22 sec Start 25: arrow-compute-hash-test 25/51 Test #25: arrow-compute-hash-test .............. Passed 2.61 sec Start 26: arrow-compute-isin-test 26/51 Test #26: arrow-compute-isin-test .............. Passed 0.81 sec Start 27: arrow-compute-match-test 27/51 Test #27: arrow-compute-match-test ............. Passed 0.40 sec Start 28: arrow-compute-sort-to-indices-test 28/51 Test #28: arrow-compute-sort-to-indices-test ... Passed 3.33 sec Start 29: arrow-compute-nth-to-indices-test 29/51 Test #29: arrow-compute-nth-to-indices-test .... Passed 1.51 sec Start 30: arrow-compute-util-internal-test 30/51 Test #30: arrow-compute-util-internal-test ..... Passed 0.13 sec Start 31: arrow-compute-add-test 31/51 Test #31: arrow-compute-add-test ............... Passed 0.12 sec Start 32: arrow-compute-aggregate-test 32/51 Test #32: arrow-compute-aggregate-test ......... Passed 14.70 sec Start 33: arrow-compute-compare-test 33/51 Test #33: arrow-compute-compare-test ........... Passed 7.96 sec Start 34: arrow-compute-take-test 34/51 Test #34: arrow-compute-take-test .............. Passed 4.80 sec Start 35: arrow-compute-filter-test 35/51 Test #35: arrow-compute-filter-test ............ Passed 8.23 sec Start 36: arrow-dataset-dataset-test 36/51 Test #36: arrow-dataset-dataset-test ........... Passed 0.25 sec Start 37: arrow-dataset-discovery-test 37/51 Test #37: arrow-dataset-discovery-test ......... Passed 0.13 sec Start 38: arrow-dataset-file-ipc-test 38/51 Test #38: arrow-dataset-file-ipc-test .......... Passed 0.21 sec Start 39: arrow-dataset-file-test 39/51 Test #39: arrow-dataset-file-test .............. Passed 0.12 sec Start 40: arrow-dataset-filter-test 40/51 Test #40: arrow-dataset-filter-test ............ Passed 0.16 sec Start 41: arrow-dataset-partition-test 41/51 Test #41: arrow-dataset-partition-test ......... Passed 0.13 sec Start 42: arrow-dataset-scanner-test 42/51 Test #42: arrow-dataset-scanner-test ........... Passed 0.20 sec Start 43: arrow-filesystem-test 43/51 Test #43: arrow-filesystem-test ................ Passed 1.62 sec Start 44: arrow-hdfs-test 44/51 Test #44: arrow-hdfs-test ...................... Passed 0.13 sec Start 45: arrow-feather-test 45/51 Test #45: arrow-feather-test ................... Passed 0.91 sec Start 46: arrow-ipc-read-write-test 46/51 Test #46: arrow-ipc-read-write-test ............ Passed 5.77 sec Start 47: arrow-ipc-json-simple-test 47/51 Test #47: arrow-ipc-json-simple-test ........... Passed 0.16 sec Start 48: arrow-ipc-json-test 48/51 Test #48: arrow-ipc-json-test .................. Passed 0.27 sec Start 49: arrow-json-integration-test 49/51 Test #49: arrow-json-integration-test .......... Passed 0.13 sec Start 50: arrow-json-test 50/51 Test #50: arrow-json-test ...................... Passed 0.26 sec Start 51: arrow-orc-adapter-test 51/51 Test #51: arrow-orc-adapter-test ............... Passed 1.92 sec 98% tests passed, 1 tests failed out of 51 Label Time Summary: arrow-tests = 27.38 sec (27 tests) arrow_compute = 45.11 sec (14 tests) arrow_dataset = 1.21 sec (7 tests) arrow_ipc = 6.20 sec (3 tests) unittest = 79.91 sec (51 tests) Total Test time (real) = 79.99 sec The following tests FAILED: 20 - arrow-utility-test (Failed) Errors while running CTest ``` Closes#7142 from kiszk/ARROW-8754 Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

Add support for ";" as parameter divider

…e#33) Bumps org.apache.orc:orc-core from 1.9.5 to 2.0.3. Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Main entry point for stripe filtering with predicate pushdown. Mirrors Parquet's FilterRowGroups pattern. Implementation: 1. Ensure complete metadata is loaded (file, manifest, statistics cache) 2. Call TestStripes to evaluate predicate against stripe statistics 3. Filter results to include only stripes where: - Predicate is satisfiable (not literal(false)) - Stripe is non-empty (num_rows > 0) 4. Return vector of selected stripe indices Stripes are skipped if: - The predicate simplifies to literal(false) given statistics - The stripe contains zero rows This function is called by: - ScanBatchesAsync (for scan optimization) - Subset (for fragment splitting) - TryCountRows (for count optimization) Verified: Mirrors cpp/src/arrow/dataset/file_parquet.cc FilterRowGroups (lines 918-931) Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

- Extended ORC adapter to extract string/binary statistics - Added truncation-aware handling in predicate evaluation - Added comprehensive string/binary tests (5 test scenarios) Implementation details: - ORC STRING/VARCHAR/CHAR types now return min/max statistics - ORC BINARY type returns min/max as BinaryScalar - Truncation awareness: ORC may truncate long strings for space - Conservative approach: treats all string stats as potentially truncated - Min is prefix (lower bound), max is incremented prefix (upper bound) Tests cover: - String basic filtering (range-based stripe skipping) - String equality (conservative with truncation) - String range filtering (compound predicates) - Empty string handling - Binary data filtering Truncation handling: - Current limitation: liborc doesn't expose truncation flags - Conservative approach: statistics used as-is (ORC truncation is conservative) - Min truncation: cannot definitively prove exclusion - Max truncation: still valid upper bound - Future: expose truncation flags when liborc API supports it Verified: arrow_orc target builds successfully Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

…Float and String support (apache#82)

Reverts PRs apache#76-88 which added: - Task apache#32: Float32/float64 predicate pushdown - Task apache#33: String/binary predicate pushdown - Task apache#34: Timestamp/date predicate pushdown - Task apache#35: Nested type tests Tasks 32-35 are now marked as pending for future implementation.

Main entry point for stripe filtering with predicate pushdown. Mirrors Parquet's FilterRowGroups pattern. Implementation: 1. Ensure complete metadata is loaded (file, manifest, statistics cache) 2. Call TestStripes to evaluate predicate against stripe statistics 3. Filter results to include only stripes where: - Predicate is satisfiable (not literal(false)) - Stripe is non-empty (num_rows > 0) 4. Return vector of selected stripe indices Stripes are skipped if: - The predicate simplifies to literal(false) given statistics - The stripe contains zero rows This function is called by: - ScanBatchesAsync (for scan optimization) - Subset (for fragment splitting) - TryCountRows (for count optimization) Verified: Mirrors cpp/src/arrow/dataset/file_parquet.cc FilterRowGroups (lines 918-931) Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

- Extended ORC adapter to extract string/binary statistics - Added truncation-aware handling in predicate evaluation - Added comprehensive string/binary tests (5 test scenarios) Implementation details: - ORC STRING/VARCHAR/CHAR types now return min/max statistics - ORC BINARY type returns min/max as BinaryScalar - Truncation awareness: ORC may truncate long strings for space - Conservative approach: treats all string stats as potentially truncated - Min is prefix (lower bound), max is incremented prefix (upper bound) Tests cover: - String basic filtering (range-based stripe skipping) - String equality (conservative with truncation) - String range filtering (compound predicates) - Empty string handling - Binary data filtering Truncation handling: - Current limitation: liborc doesn't expose truncation flags - Conservative approach: statistics used as-is (ORC truncation is conservative) - Min truncation: cannot definitively prove exclusion - Max truncation: still valid upper bound - Future: expose truncation flags when liborc API supports it Verified: arrow_orc target builds successfully Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

…Float and String support (apache#82)

Reverts PRs apache#76-88 which added: - Task apache#32: Float32/float64 predicate pushdown - Task apache#33: String/binary predicate pushdown - Task apache#34: Timestamp/date predicate pushdown - Task apache#35: Nested type tests Tasks 32-35 are now marked as pending for future implementation.

Add adapt 'lite' DCHECK macros from Kudu as also used in Parquet
258d77b

emkornfield reviewed Mar 23, 2016
View reviewed changes

asfgit closed this in a4002c6Mar 23, 2016

wesm deleted the ARROW-70 branch March 23, 2016 18:00

robertnishihara mentioned this pull request Jun 4, 2016
replace fatal logging with check and dcheck ray-project/ray-legacy#84
Closed

praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Aug 30, 2018
GDV-55: [C++] Added validation to projector build. (apache#33)
bff714c
Validating the input schema and expressions during the projector build.

praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Aug 30, 2018
GDV-55: [C++] Added validation to projector build. (apache#33)
ebb73ad
Validating the input schema and expressions during the projector build.

praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Aug 30, 2018
GDV-55: [C++] Added validation to projector build. (apache#33)
58e04bc
Validating the input schema and expressions during the projector build.

praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Aug 30, 2018
GDV-55: [C++] Added validation to projector build. (apache#33)
4adbc86
Validating the input schema and expressions during the projector build.

praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Sep 4, 2018
GDV-55: [C++] Added validation to projector build. (apache#33)
f0ef530
Validating the input schema and expressions during the projector build.

praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Sep 10, 2018
GDV-55: [C++] Added validation to projector build. (apache#33)
857bf57
Validating the input schema and expressions during the projector build.

praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Sep 10, 2018
GDV-55: [C++] Added validation to projector build. (apache#33)
30eab61
Validating the input schema and expressions during the projector build.

guyuqi mentioned this pull request Nov 22, 2018
ARROW-3849: [C++] Leverage Armv8 crc32 extension instructions to accelerate the hash computation for Arm64 #3010
Closed

brills mentioned this pull request Jun 4, 2019
importing pyarrow and Tensorflow crashes #4472
Closed

zhztheplayer added a commit to zhztheplayer/arrow-1 that referenced this pull request Sep 23, 2021
Unsafe Serialization: Null count is always set to 0 (apache#33)
a11e628

zhztheplayer added a commit to zhztheplayer/arrow-1 that referenced this pull request Feb 8, 2022
Unsafe Serialization: Null count is always set to 0 (apache#33)
32ffa5c

zhztheplayer added a commit to zhztheplayer/arrow-1 that referenced this pull request Mar 3, 2022
Unsafe Serialization: Null count is always set to 0 (apache#33)
b8c8d86

rui-mo pushed a commit to rui-mo/arrow-1 that referenced this pull request Mar 23, 2022
Unsafe Serialization: Null count is always set to 0 (apache#33)
587a920

zhouyuan pushed a commit to zhouyuan/arrow that referenced this pull request Apr 26, 2022
Unsafe Serialization: Null count is always set to 0 (apache#33)
3cd059b

jayhomn-bitquill referenced this pull request in Bit-Quill/arrow Aug 10, 2022
Merge pull request#33from rafael-telles/parse-connection-string
f24ef45
Add support for ";" as parameter divider

jonkeane mentioned this pull request May 11, 2025
[C++][R]: gcc-UBSAN errors on CRAN #46394
Closed

paddyroddy referenced this pull request in rok/arrow Jul 19, 2025
chore: update LICENSE date (#33)
b2790a0

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 20, 2026
Mark Taskapache#33as complete (apache#81)
713c035

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 20, 2026
Session 17 final progress: completed Tasksapache#32andapache#33- …
783f658
…Float and String support (apache#82)

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 24, 2026
Mark Taskapache#33as complete (apache#81)
06c3b59

cbb330 added a commit to cbb330/arrow that referenced this pull request Feb 24, 2026
Session 17 final progress: completed Tasksapache#32andapache#33- …
de0ec84
…Float and String support (apache#82)

Navigation Menu

Uh oh!

ARROW-70: Add adapt 'lite' DCHECK macros from Kudu as also used in Parquet - #33

ARROW-70: Add adapt 'lite' DCHECK macros from Kudu as also used in Parquet#33
wesm wants to merge 1 commit into
apache:masterfrom
wesm:ARROW-70

wesm commented Mar 23, 2016

Uh oh!

emkornfieldMar 23, 2016

Uh oh!

wesmMar 23, 2016

Uh oh!

emkornfield commented Mar 23, 2016

Uh oh!

wesm commented Mar 23, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wesm commented Mar 23, 2016

Uh oh!

emkornfieldMar 23, 2016

Choose a reason for hiding this comment

Uh oh!

wesmMar 23, 2016

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Mar 23, 2016

Uh oh!

wesm commented Mar 23, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants