[SPARK-30772][ML][SQL] avoid tuple assignment because it will circumvent the transient tag

[zhengruifeng]

What changes were proposed in this pull request?

it is said in LeastSquaresAggregator that :

// do not use tuple assignment above because it will circumvent the transient tag

I then check this issue with Scala 2.13.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_241)

Why are the changes needed?

avoid tuple assignment because it will circumvent the transient tag

Does this PR introduce any user-facing change?

No

How was this patch tested?

existing testsuites

[zhengruifeng]

I use following code to check this issue:
env: Scala 2.13.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_241)

importjava.io.{ByteArrayInputStream, ByteArrayOutputStream, ObjectInputStream, ObjectOutputStream}
defserialise(value: Any):Array[Byte] = {
valstream:ByteArrayOutputStream=newByteArrayOutputStream()
valoos=newObjectOutputStream(stream)
oos.writeObject(value)
oos.close()
stream.toByteArray
}
defdeserialise(bytes: Array[Byte]):Any= {
valois=newObjectInputStream(newByteArrayInputStream(bytes))
valvalue= ois.readObject
ois.close()
value
}
classAextendsSerializable { @transient lazyvala= {println("get a"); System.currentTimeMillis} }
vala=newA
a.a
vala2= deserialise(serialise(a)).asInstanceOf[A]
a2.a
a.a == a2.a
classBextendsSerializable { @transient lazyval (a,b) = {println("get a & b"); valt=System.currentTimeMillis; (t, -t)} }
valb=newB
b.a
valb2= deserialise(serialise(b)).asInstanceOf[B]
b2.a
b.a == b2.a
b.b == b2.b
classCextendsSerializable { @transient lazyvalt= {println("get a & b"); valt=System.currentTimeMillis; (t, -t)}; @transient lazyvala= t._1; @transient lazyvalb= t._2 }
valc=newC
c.a
valc2= deserialise(serialise(c)).asInstanceOf[C]
c2.a
c.a == c2.a
c.b == c2.b

Result:

scala>classAextendsSerializable { @transient lazyvala= {println("get a"); System.currentTimeMillis} }
defined classA
scala>vala=newA
a:A=A@68ef01a5
scala> a.a
get a
res0:Long=1581333143300
scala>vala2= deserialise(serialise(a)).asInstanceOf[A]
a2:A=A@f017dd0
scala> a2.a
get a
res1:Long=1581333143523
scala> a.a == a2.a
res2:Boolean=false
scala> scala>classBextendsSerializable { @transient lazyval (a,b) = {println("get a & b"); valt=System.currentTimeMillis; (t, -t)} }
defined classB
scala>valb=newB
b:B=B@1d008e61
scala> b.a
get a & b
res3:Long=1581333144022
scala>valb2= deserialise(serialise(b)).asInstanceOf[B]
b2:B=B@6ab826bb
scala> b2.a
res4:Long=1581333144022
scala> b.a == b2.a
res5:Boolean=true
scala> b.b == b2.b
res6:Boolean=true
scala> scala>classCextendsSerializable { @transient lazyvalt= {println("get a & b"); valt=System.currentTimeMillis; (t, -t)}; @transient lazyvala= t._1; @transient lazyvalb= t._2 }
defined classC
scala>valc=newC
c:C=C@7ec01440
scala> c.a
get a & b
res7:Long=1581333144575
scala>valc2= deserialise(serialise(c)).asInstanceOf[C]
c2:C=C@42a698bd
scala> c2.a
get a & b
res8:Long=1581333144713
scala> c.a == c2.a
res9:Boolean=false
scala> c.b == c2.b
res10:Boolean=false

We can see that b2.a does not trigger the println, and the fields .a and .b contains the same values after serialization;
while a2.a and c2.a will trigger the println, and the fields change after serialization;

[zhengruifeng]

friendly ping @srowen , I can not find in the scala community any description or disscussin on this issue.

[SparkQA]

Test build #118156 has finished for PR 27523 at commit 477f99e.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Weird. @sethah do you recall this? you added the comment in 1db1c65#diff-668c79317c51f40df870d3404d8a731fR921 several years ago.

So, if this isn't actually avoiding serialization, but still works, another option is to decide that's fine and just let them serialize. I don't know which of them are actually really big, but, that would keep existing behavior right?

[srowen]

Hm, this pattern is a little ugly / adds extra overhead, but might be necessary vs just referencing "elementIndexVar._1" later in the code or something similar.

[sethah]

What's the question 😛 ? See my comments here: #14109

Also the databricks style guide mentions it: https://github.com/databricks/scala-style-guide#destructuring-binds

[srowen]

Oh ha thanks @sethah ! never seen that. Yeah I was just asking if you knew anything more about why or what is happening here. That's pretty good.

[sethah]

Heh np, I remember it being really subtle! But that's about all I can give you.

[zhengruifeng]

I don't know which of them are actually really big, but, that would keep existing behavior right?

acturally, I do know how big those variables in SQL are. If they are supposed to be transient, I guess that should keep existing behavior.

[zhengruifeng]

retest this please

[SparkQA]

Test build #118202 has finished for PR 27523 at commit 477f99e.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I'm OK with it, I guess I was just saying that the current behavior is not as intended, but still 'works' it seems. Maybe in some cases they aren't that big. But if you're pretty confident they are then this is a 'win'.

[srowen]

Jenkins retest this please

[SparkQA]

Test build #118371 has finished for PR 27523 at commit 477f99e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

@srowen acturally, I do not know how big the objects in SQL are. So I agree to let them alone, and only change the ML side.

[SparkQA]

Test build #118407 has finished for PR 27523 at commit 968ffb3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Oh, I don't think it's 'wrong' to fix. Or if we know the intended behavior isn't working, just undo the intention (i.e. remove transient lazy). Well, either way.

[srowen]

Merged to master