HIVE-24316. Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to upgrade Apache ORC from 1.5.6 to 1.5.8.

Why are the changes needed?

This will bring eleven bug fixes.

• ORC 1.5.7: https://issues.apache.org/jira/projects/ORC/versions/12345702
• ORC 1.5.8: https://issues.apache.org/jira/projects/ORC/versions/12346462

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CI with the existing test cases.

[pgaref]

Thanks for the patch @dongjoon-hyun -- can you please reopen the PR as I dont see the pre-commit test results at all (I guess they were never triggered)

[dongjoon-hyun]

Thanks, @pgaref . I closed and reopened this.

[dongjoon-hyun]

Hi, @sunchao .
There is no way to trigger the real CI until now?

[sunchao]

No. There is no jenkins file in branch-3.1 so there's no way to run CI at the moment. We'd have to do something similar to #1398 to enable that.

[dongjoon-hyun]

Do you think you can do that for the Apache Hive community?

[sunchao]

yeah I can help on that - I think it won't be too difficult after going through the process for branch-2.3.

[sunchao]

opened #1626

[dongjoon-hyun]

Thank you so much, @sunchao!

[sunchao]

@dongjoon-hyun can you re-trigger CI perhaps with an empty commit?

[dongjoon-hyun]

Sure!

[sunchao]

LGTM. All test failures are existing and there's no new failures.

[dongjoon-hyun]

Thank you for your review and approval, @sunchao !

[sunchao]

@pgaref let me know if you want to take another look before I merge it. Thanks.

[pgaref]

Hey @sunchao the orc version bump should be safe so patch LGTM -- I see we have some test failures with the new CI, is that expected, or something to look at?

Cheers

[sunchao]

Thanks @pgaref . Yes those are existing test failures in the branch that are probably worth taking a look. Not sure whether they are flaky or real issues.

[sunchao]

Merged. Thanks @dongjoon-hyun

[dongjoon-hyun]

Thank you, @sunchao and @pgaref !

[dongjoon-hyun]

Could you resolve HIVE-24316 please, @sunchao ?

[sunchao]

it's done

[dongjoon-hyun]

Thanks!

[glapark]

Hello,

This commit seems to fail four existing test cases in org.apache.hadoop.hive.ql.io.orc.TestOrcFile:

[ERROR] Failures:
[ERROR] TestOrcFile.testMemoryManagementV11:1992 stripe 1 is too long at 18454
[ERROR] TestOrcFile.testMemoryManagementV11:1992 stripe 1 is too long at 18454
[ERROR] TestOrcFile.testMemoryManagementV12:2029 stripe 1 is too long at 9626
[ERROR] TestOrcFile.testMemoryManagementV12:2029 stripe 1 is too long at 9626
[INFO]
[ERROR] Tests run: 44, Failures: 4, Errors: 0, Skipped: 0

In my own manual testing, the four tests succeed with HIVE-24331 (the last commit just before HIVE-24316 in branch-3.1), so there is a good chance that upgrading ORC to 1.5.8 introduces these errors.

The error is generated from the following assertTrue(), so replacing the constant 5000 with 2000 would fix the errors.

for(StripeInformation stripe: reader.getStripes()) {
i += 1;
assertTrue("stripe " + i + " is too long at " + stripe.getDataLength(),
stripe.getDataLength() < 5000);
}

Could someone test the latest commit in branch-3.1 to see if the same errors can be reproduced?

--- Sungwoo

[pgaref]

Hey @glapark

This is most probably related to ORC-361 that changed the ORC MemoryManager implementation (to support multi-threaded writers).

The error is generated from the following assertTrue(), so replacing the constant 5000 with 2000 would fix the errors.

I assume that you replaced it with 20_000 as the stripe sizes are around that number -- we should probably have the assert check the StripeSize which is about 50_000.
More details here: https://github.com/apache/orc/pull/433/files#diff-eae6097544d7965e078d582758f745fe52a6772bc71cbafe6068317a68595583R2493

[dongjoon-hyun]

Thank you guys. I'll take a look at them, too.

[dongjoon-hyun]

Got it. I found what I missed. I'll make a follow-up.