[SPARK-27216][CORE] Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[LantaoJin]

What changes were proposed in this pull request?

HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But RoaringBitmap couldn't be ser/deser with unsafe KryoSerializer.

It's a bug of RoaringBitmap-0.5.11 and fixed in latest version.

This is an update of #24157

How was this patch tested?

Add a UT

[LantaoJin]

@srowen@squito Could you help to review this PR? I can not reopen #24157 since a force pushing. Thanks for your time.

[srowen]

Nit: I think the style checker will flag the space before the colon.

[srowen]

Tiny nit: () after the constructor

[SparkQA]

Test build #4675 has finished for PR 24264 at commit 076411b.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[LantaoJin]

Rename for a better searching or tracing.

[attilapiros]

Please move your test into KryoSerializerSuite where conf.set(KRYO_USE_UNSAFE, false) is given.
And as UnsafeKryoSerializerSuite is inherited from KryoSerializerSuite beside using conf.set(KRYO_USE_UNSAFE, true) in beforeAll() it will inherit your test method so both the safe/unsafe case will be tested this way: once executing KryoSerializerSuite and once UnsafeKryoSerializerSuite.

Summary: you do not need to test both cases in one test method.

[LantaoJin]

Oh, thanks for this reminding.

[srowen]

Run ./dev/test-dependencies.sh --replace-manifest

[squito]

I dont' think this is what @attilapiros meant. I think he meant to (a) leave UnsafeKryoSerializerSuite as it was before and (b) add a new test which just tested serialization of RoaringBitmap with kryo in this class, but didn't tweak confs at all. That way you'd get a test for serializing with kryo.unsafe=false (here in KryoSerializerSuite) and you'd also automatically get a test with kryo.unsafe=true (inherited in UnsafeKryoSerializerSuite).

[attilapiros]

Exactly. I was about to write a similar comment.

[LantaoJin]

Uh... @attilapiros Is it proper now?

[attilapiros]

@LantaoJin No.

1. revert the deletion of the original UnsafeKryoSerializerSuite.scala
2. remove these lines:
   

   spark/core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala

   Lines 531 to 543 in f0f62af

   	classUnsafeKryoSerializerSuiteextendsSparkFunSuitewithSharedSparkContext {
   	conf.set(SERIALIZER, classOf[KryoSerializer].getName)
   	conf.set(KRYO_USER_REGISTRATORS, classOf[MyRegistrator].getName)
   	conf.set(KRYO_USE_UNSAFE, true)
   	test("test RoaringBitmap ser/dser with Kryo unsafe enabled") {
   	valexpected=newRoaringBitmap()
   	expected.add(1787)
   	valunsafeSer=newKryoSerializer(conf).newInstance()
   	valactual:RoaringBitmap= unsafeSer.deserialize(unsafeSer.serialize(expected))
   	assert(actual === expected)
   	}
   	}

[attilapiros]

Finally via sbt test both UnsafeKryoSerializerSuite and KryoSerializerSuite and you should see your RoaringBitmap test executed twice, you can do that like this:

$ ./build/sbt
> project core
> testOnly *.UnsafeKryoSerializerSuite
> testOnly *.KryoSerializerSuite

[attilapiros]

You can revert the deletion by:

git checkout f799e34962c24bb55662e7e570195d8f5aa5799a core/src/test/scala/org/apache/spark/serializer/UnsafeKryoSerializerSuite.scala

But please double check the commit ID (my laptop used for developing Spark runs an OS update and this is my other machine...).

[LantaoJin]

Finally via sbt test both UnsafeKryoSerializerSuite and KryoSerializerSuite and you should see your RoaringBitmap test executed twice, you can do that like this:

$ ./build/sbt
> project core
> testOnly *.UnsafeKryoSerializerSuite
> testOnly *.KryoSerializerSuite

Everything looks fine from my side.
> testOnly *.UnsafeKryoSerializerSuite
[info] UnsafeKryoSerializerSuite:
[info] - test RoaringBitmap ser/dser with Kryo unsafe enabled (264 milliseconds)
[info] ScalaTest
[info] Run completed in 4 seconds, 430 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 97 s, completed Apr 2, 2019 5:32:05 PM
> testOnly *.KryoSerializerSuite
[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list
[info] Packaging /Users/lajin/git/my/spark/core/target/scala-2.12/spark-core_2.12-3.0.0-SNAPSHOT.jar ...
[info] Done packaging.
[info] KryoSerializerSuite:
[info] - SPARK-7392 configuration limits (22 milliseconds)
....
[info] - test RoaringBitmap ser/dser with Kryo unsafe disabled (6 milliseconds)
[info] ScalaTest
[info] Run completed in 5 seconds, 503 milliseconds.
[info] Total number of tests run: 34
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 34, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 34, Failed 0, Errors 0, Passed 34

[LantaoJin]

@attilapiros Oh, I read your comment again. I guess I have understood it. I will refactor it.

[LantaoJin]

@attilapiros What about it now?

[attilapiros]

@LantaoJinNow your change is almost perfect.

But as there was no need to import RoaringBitmap for your test it made me think what kind of tests were here before your PR regarding this class.

I have found this:

spark/core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala

Lines 373 to 396 in 8cc3eb4

	test("SPARK-12222: deserialize RoaringBitmap throw Buffer underflow exception") {
	withTempDir { dir =>
	valtmpfile= dir.toString +"/RoaringBitmap"
	valoutStream=newFileOutputStream(tmpfile)
	valoutput=newKryoOutput(outStream)
	valbitmap=newRoaringBitmap
	bitmap.add(1)
	bitmap.add(3)
	bitmap.add(5)
	// Ignore Kryo because it doesn't use writeObject
	bitmap.serialize(newKryoOutputObjectOutputBridge(null, output))
	output.flush()
	output.close()
	valinStream=newFileInputStream(tmpfile)
	valinput=newKryoInput(inStream)
	valret=newRoaringBitmap
	// Ignore Kryo because it doesn't use readObject
	ret.deserialize(newKryoInputObjectInputBridge(null, input))
	input.close()
	assert(ret == bitmap)
	}
	}

I think it is out-dated and this test can be removed as the statement deserialize RoaringBitmap throw Buffer underflow exception is false, what do you think?

[LantaoJin]

I think it's out-dated. But it needs double check if we plan to merge this PR to multiple branches.

[attilapiros]

As this change contradicts with that statement it is safe to remove that test.

[LantaoJin]

I've checked branch 2.1 and above. The kryo versions in them are all 3.0 or above. According to the description in SPARK-12222, it should be safe. I will remove it.

[attilapiros]

Jenkins retest this please.

[squito]

lgtm assuming tests pass. Thanks for the work on this @LantaoJin & @attilapiros

[SparkQA]

Test build #4679 has finished for PR 24264 at commit 3397957.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[lemire]

You might want to consider upgrading directly to RoaringBitmap 0.8.0. We added serialize(ByteBuffer) and deserialize(ByteBuffer) methods that are quite interesting (and fast).

cc @richardstartin@blacelle

[squito]

@lemire I think that's a good idea for something to do eventually, but I think that can be handled separately. IIUC, to take advantage of those performance improvements, we need to change how we're serializing the RoaringBitmaps inside spark. This is a bug fix we can get in now.

I filed https://issues.apache.org/jira/browse/SPARK-27367 for this if anybody wants to follow up with that

[squito]

I merged this to master, but there were merge conflicts going back farther. @LantaoJin would you like to open another pr, against branch 2.4 at least?