ORC-1813: [C++] Fix has_null forward compatibility by suxiaogang223 · Pull Request #2082 · apache/orc

suxiaogang223 · 2024-12-10T16:49:34Z

What changes were proposed in this pull request?

close: #2079
relate pr: #2055
Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation.
The Java implementation includes the following logic:

if (stats.hasHasNull()) {
hasNull = stats.getHasNull();
} else {
hasNull = true;
}

In contrast, the C++ implementation directly uses the has_null value without any fallback logic:

ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) {
stats_.setNumberOfValues(pb.number_of_values());
stats_.setHasNull(pb.has_null());
}

Why are the changes needed?

We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly.
This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(

How was this patch tested?

I have tested this using Doris external pipeline:
apache/doris#45104
apache/doris-thirdparty#259

Was this patch authored or co-authored using generative AI tooling?

No

dongjoon-hyun · 2024-12-10T21:18:43Z

Thank you for making a PR, @suxiaogang223 .

suxiaogang223 · 2024-12-11T09:01:06Z

I have fix some orc c++ uts because the orc file orc_index_int_string.orc is written by old orc version which don't write the has_null statistics.

wgtmac · 2024-12-11T14:31:06Z

Thanks for fixing this! Could you create a JIRA issue? We use JIRA instead of GitHub issue for tracking purpose.

suxiaogang223 · 2024-12-11T15:10:39Z

Thanks for fixing this! Could you create a JIRA issue? We use JIRA instead of GitHub issue for tracking purpose.

How to create jira? it seems that your jira system cannot be used :(

wgtmac · 2024-12-11T15:29:23Z

You may request a JIRA account via: https://selfserve.apache.org/jira-account.html

It was open for registration. Unfortunately it has been disabled due to spam :(

dongjoon-hyun · 2024-12-11T16:31:37Z

Thank you for fling JIRA and update the PR title, @suxiaogang223 .

BTW, does it affect all ORC versions like 1.9.x and 1.8.x? The JIRA issue seems to be reported on Apache ORC 2.0.x only.

dongjoon-hyun · 2024-12-11T16:32:30Z

For now, I set the milestone, 2.0.4, based on Affected Versions of ORC-1813.

ffacs

LGTM. Thank you @suxiaogang223 @wgtmac @dongjoon-hyun ~

wgtmac

+1

dongjoon-hyun · 2024-12-12T15:49:10Z

Thank you all, @suxiaogang223 , @wgtmac , @ffacs !

Merged to main.

Could you make a backporting PR to branch-2.0? There was a conflict.

close: apache#2079 relate pr: apache#2055 Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation. The Java implementation includes the following logic: ```java if (stats.hasHasNull()) { hasNull = stats.getHasNull(); } else { hasNull = true; } ``` In contrast, the C++ implementation directly uses the has_null value without any fallback logic: ```c++ ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) { stats_.setNumberOfValues(pb.number_of_values()); stats_.setHasNull(pb.has_null()); } ``` We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly. **This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(** I have tested this using [Doris](https://github.com/apache/doris) external pipeline: apache/doris#45104 apache/doris-thirdparty#259 No Closesapache#2082 from suxiaogang223/fix_has_null. Authored-by: Socrates <suxiaogang223@icloud.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

suxiaogang223 · 2024-12-16T09:54:34Z

Thank you all, @suxiaogang223 , @wgtmac , @ffacs !
Merged to main.
Could you make a backporting PR to branch-2.0? There was a conflict.

There is a conflict fixed pr : #2086

close: #2079 relate pr: #2055 Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation. The Java implementation includes the following logic: ```java if (stats.hasHasNull()) { hasNull = stats.getHasNull(); } else { hasNull = true; } ``` In contrast, the C++ implementation directly uses the has_null value without any fallback logic: ```c++ ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) { stats_.setNumberOfValues(pb.number_of_values()); stats_.setHasNull(pb.has_null()); } ``` We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly. **This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(** I have tested this using [Doris](https://github.com/apache/doris) external pipeline: apache/doris#45104 apache/doris-thirdparty#259 No Closes#2082 from suxiaogang223/fix_has_null. Authored-by: Socrates <suxiaogang223icloud.com> ### What changes were proposed in this pull request? ### Why are the changes needed? ### How was this patch tested? ### Was this patch authored or co-authored using generative AI tooling? Closes#2086 from suxiaogang223/cherry_pick_fix_has_null. Authored-by: Socrates <suxiaogang223@icloud.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2024-12-20T17:42:05Z

For the record, the following landed to branch-2.0.

ORC-1813: [C++] Fix has_null forward compatibility #2086

fix has_null forward compatibility
3fd6f51

github-actionsBot added the CPP label Dec 10, 2024

suxiaogang223 added 3 commits December 11, 2024 12:30

fix test
3d1beb0

fix format
5e0fb88

fix test again :(
85be057

ffacs requested changes Dec 11, 2024
View reviewed changes

Comment threadc++/src/sargs/PredicateLeaf.cc Outdated

rename function in camel case
d09a785

wgtmac reviewed Dec 11, 2024
View reviewed changes

Comment threadc++/src/sargs/PredicateLeaf.cc Outdated
Comment threadc++/src/sargs/PredicateLeaf.cc Outdated
Comment threadc++/src/sargs/PredicateLeaf.cc
Comment threadc++/test/TestPredicateLeaf.cc Outdated

suxiaogang223 changed the title ~~fix has_null forward compatibility~~ORC-1813: [C++] fix has_null forward compatibilityDec 11, 2024

dongjoon-hyun added this to the 2.0.4 milestone Dec 11, 2024

fix
2271efe

ffacs approved these changes Dec 12, 2024
View reviewed changes

wgtmac changed the title ~~ORC-1813: [C++] fix has_null forward compatibility~~ORC-1813: [C++] Fix has_null forward compatibilityDec 12, 2024

wgtmac approved these changes Dec 12, 2024
View reviewed changes

dongjoon-hyun approved these changes Dec 12, 2024
View reviewed changes

dongjoon-hyun closed this in da5bda5Dec 12, 2024

suxiaogang223 deleted the fix_has_null branch December 12, 2024 15:57

suxiaogang223 mentioned this pull request Dec 16, 2024
ORC-1813: [C++] Fix has_null forward compatibility #2086
Closed

dongjoon-hyun modified the milestones: 2.0.4, 2.1.0 Dec 20, 2024

Uh oh!

Conversation

suxiaogang223 commented Dec 10, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun commented Dec 10, 2024

Uh oh!

suxiaogang223 commented Dec 11, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wgtmac commented Dec 11, 2024

Uh oh!

suxiaogang223 commented Dec 11, 2024

Uh oh!

wgtmac commented Dec 11, 2024

Uh oh!

dongjoon-hyun commented Dec 11, 2024

Uh oh!

dongjoon-hyun commented Dec 11, 2024

Uh oh!

ffacs left a comment

Choose a reason for hiding this comment

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Dec 12, 2024

Uh oh!

suxiaogang223 commented Dec 16, 2024

Uh oh!

dongjoon-hyun commented Dec 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

suxiaogang223 commented Dec 10, 2024•
edited
Loading

suxiaogang223 commented Dec 11, 2024•
edited
Loading