Skip to content

[SPARK-32437][CORE] Improve MapStatus deserialization speed with RoaringBitmap 0.9.0 - #29233

Closed
dongjoon-hyun wants to merge 1 commit into
apache:masterfrom
dongjoon-hyun:SPARK-ROAR
Closed

[SPARK-32437][CORE] Improve MapStatus deserialization speed with RoaringBitmap 0.9.0#29233
dongjoon-hyun wants to merge 1 commit into
apache:masterfrom
dongjoon-hyun:SPARK-ROAR

Conversation

@dongjoon-hyun

@dongjoon-hyundongjoon-hyun commented Jul 25, 2020•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR aims to speed up MapStatus deserialization by 5~18% with the latest RoaringBitmap 0.9.0 and new APIs. Note that we focus on deserialization time because serialization occurs once while deserialization occurs many times.

Why are the changes needed?

The current version is too old. We had better upgrade it to get the performance improvement and bug fixes.
Although MapStatusesSerDeserBenchmark is synthetic, the benchmark result is updated with this patch.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins or GitHub Action.

dongjoon-hyun
dongjoon-hyun commented Jul 25, 2020
View reviewed changes
Comment threadcore/benchmarks/MapStatusesSerDeserBenchmark-jdk11-results.txt
Deserialization 530 535 9 0.4 2651.1 0.3X
-------------------------------------------------------------------------------------------------------------------------
Serialization 175 18312 1.1874.1 1.0X
Deserialization 458 462 60.4 2288.6 0.4X

@dongjoon-hyundongjoon-hyunJul 25, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

14% reduced.

dongjoon-hyun
dongjoon-hyun commented Jul 25, 2020
View reviewed changes
Comment threadcore/benchmarks/MapStatusesSerDeserBenchmark-jdk11-results.txt
Deserialization 495 588 79 0.4 2476.7 0.3X
--------------------------------------------------------------------------------------------------------------------------
Serialization 160 171 8 1.2801.1 1.0X
Deserialization 453 484 380.4 2263.4 0.4X

@dongjoon-hyundongjoon-hyunJul 25, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

18% reduced.

dongjoon-hyun
dongjoon-hyun commented Jul 25, 2020
View reviewed changes
Comment threadcore/benchmarks/MapStatusesSerDeserBenchmark-jdk11-results.txt
Deserialization 94697733 0.2 4730.2 1.8X
---------------------------------------------------------------------------------------------------------------------------
Serialization 164118192520.1 8204.1 1.0X
Deserialization 844882370.2 4219.7 1.9X

@dongjoon-hyundongjoon-hyunJul 25, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10% reduction.

dongjoon-hyun
dongjoon-hyun commented Jul 25, 2020
View reviewed changes
Comment threadcore/benchmarks/MapStatusesSerDeserBenchmark-jdk11-results.txt
Deserialization 92994119 0.2 4645.5 1.5X
----------------------------------------------------------------------------------------------------------------------------
Serialization 13601412730.1 6799.3 1.0X
Deserialization 850859130.2 4249.9 1.6X

@dongjoon-hyundongjoon-hyunJul 25, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9% reduction.

dongjoon-hyun
dongjoon-hyun commented Jul 25, 2020
View reviewed changes
Comment threadcore/benchmarks/MapStatusesSerDeserBenchmark-results.txt
Deserialization 94397032 0.2 4715.81.8X
---------------------------------------------------------------------------------------------------------------------------
Serialization 174019032310.1 8700.0 1.0X
Deserialization 872888240.2 4360.92.0X

@dongjoon-hyundongjoon-hyunJul 25, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8% reduction.

dongjoon-hyun
dongjoon-hyun commented Jul 25, 2020
View reviewed changes
Comment threadcore/benchmarks/MapStatusesSerDeserBenchmark-results.txt
Deserialization 94097037 0.2 4699.1 1.5X
----------------------------------------------------------------------------------------------------------------------------
Serialization 14611469110.1 7306.1 1.0X
Deserialization 871889220.2 4353.9 1.7X

@dongjoon-hyundongjoon-hyunJul 25, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8% reduction.

@dongjoon-hyun

dongjoon-hyun commented Jul 25, 2020•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

cc FYI, @HyukjinKwon since GitHub Action is not triggered here while the very next PR gets GitHub Action.

@SparkQA

SparkQA commented Jul 25, 2020

Copy link
Copy Markdown

Test build #126538 has finished for PR 29233 at commit f784f2c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

srowen
srowen reviewed Jul 25, 2020
View reviewed changes

@srowensrowen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good if it's a simple update and perf win.

@dongjoon-hyun

dongjoon-hyun commented Jul 25, 2020

Copy link
Copy Markdown
MemberAuthor

Thank you, @srowen ! Merged to master.

@dongjoon-hyun
dongjoon-hyun deleted the SPARK-ROAR branch July 25, 2020 15:19
@dongjoon-hyun

dongjoon-hyun commented Jul 25, 2020

Copy link
Copy Markdown
MemberAuthor

Oops.. I missed the dependency update and Jenkins passed without a dependency test. :(
I'll make a hotfix commit.

@HyukjinKwon

HyukjinKwon commented Jul 26, 2020

Copy link
Copy Markdown
Member

LGTM except ^.

holdenk pushed a commit to holdenk/spark that referenced this pull request Oct 27, 2020
@dongjoon-hyun
[SPARK-32437][CORE] Improve MapStatus deserialization speed with Roar…
c99973e
…ingBitmap 0.9.0
### What changes were proposed in this pull request?
This PR aims to speed up `MapStatus` deserialization by 5~18% with the latest RoaringBitmap `0.9.0` and new APIs. Note that we focus on `deserialization` time because `serialization` occurs once while `deserialization` occurs many times.
### Why are the changes needed?
The current version is too old. We had better upgrade it to get the performance improvement and bug fixes.
Although `MapStatusesSerDeserBenchmark` is synthetic, the benchmark result is updated with this patch.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the Jenkins or GitHub Action.
Closesapache#29233 from dongjoon-hyun/SPARK-ROAR.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit f642234)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@LuciferYangLuciferYang mentioned this pull request May 30, 2022
Closed
otterc pushed a commit to linkedin/spark that referenced this pull request Mar 22, 2023
@dongjoon-hyun@Victsm
[SPARK-32437][CORE] Improve MapStatus deserialization speed with Roar…
92640a4
…ingBitmap 0.9.0
This PR aims to speed up `MapStatus` deserialization by 5~18% with the latest RoaringBitmap `0.9.0` and new APIs. Note that we focus on `deserialization` time because `serialization` occurs once while `deserialization` occurs many times.
The current version is too old. We had better upgrade it to get the performance improvement and bug fixes.
Although `MapStatusesSerDeserBenchmark` is synthetic, the benchmark result is updated with this patch.
No.
Pass the Jenkins or GitHub Action.
Closesapache#29233 from dongjoon-hyun/SPARK-ROAR.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Ref: LIHADOOP-56788
RB=2401504
BUG=LIHADOOP-56788
G=spark-reviewers
R=vsowrira,minyang,chsingh,yezhou,mmuralid
A=chsingh
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowensrowensrowen left review comments

Assignees

No one assigned

Labels

BUILDCORE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dongjoon-hyun@SparkQA@HyukjinKwon@srowen