[SPARK-20341][SQL] Support BigInt's value that does not fit in long value range

[kiszk]

What changes were proposed in this pull request?

This PR avoids an exception in the case where scala.math.BigInt has a value that does not fit into long value range (e.g. Long.MAX_VALUE+1). When we run the following code by using the current Spark, the following exception is thrown.

This PR keeps the value using BigDecimal if we detect such an overflow case by catching ArithmeticException.

Sample program:

case class BigIntWrapper(value:scala.math.BigInt)```
spark.createDataset(BigIntWrapper(scala.math.BigInt("10000000000000000002"))::Nil).show

Exception:

Error while encoding: java.lang.ArithmeticException: BigInteger out of long range
staticinvoke(class org.apache.spark.sql.types.Decimal$, DecimalType(38,0), apply, assertnotnull(assertnotnull(input[0, org.apache.spark.sql.BigIntWrapper, true])).value, true) AS value#0
java.lang.RuntimeException: Error while encoding: java.lang.ArithmeticException: BigInteger out of long range
staticinvoke(class org.apache.spark.sql.types.Decimal$, DecimalType(38,0), apply, assertnotnull(assertnotnull(input[0, org.apache.spark.sql.BigIntWrapper, true])).value, true) AS value#0
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290)
at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:454)
at org.apache.spark.sql.Agg$$anonfun$18.apply$mcV$sp(MySuite.scala:192)
at org.apache.spark.sql.Agg$$anonfun$18.apply(MySuite.scala:192)
at org.apache.spark.sql.Agg$$anonfun$18.apply(MySuite.scala:192)
at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:68)
at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
...
Caused by: java.lang.ArithmeticException: BigInteger out of long range
at java.math.BigInteger.longValueExact(BigInteger.java:4531)
at org.apache.spark.sql.types.Decimal.set(Decimal.scala:140)
at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:434)
at org.apache.spark.sql.types.Decimal.apply(Decimal.scala)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:287)
... 59 more

How was this patch tested?

Add new test suite into DecimalSuite

[SparkQA]

Test build #75941 has finished for PR 17684 at commit 9a15ba2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

What if we simply use the the set function for BigDecimal?

defset(bigintval: BigInteger):Decimal= set(BigDecimal(bigintval))

[gatorsmile]

This line needs an update

[gatorsmile]

This line is not needed.

[kiszk]

To always use BigDecimal can make code simple. However, it always create and keep an additional object (i.e. BigDecimal) even if value can be fit into long value range.
This is a design choice: runtime efficiency with longer code v.s. simpler code with runtime inefficiency.
Is the priority for simple code higher than that for runtime efficiency?

[gatorsmile]

Ok, could you write a comment in the code to explain it?

[SparkQA]

Test build #75980 has finished for PR 17684 at commit 655e115.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[kiszk]

Thanks, done.

[gatorsmile]

How about

If the value is not in the range of long, convert it to BigDecimal and the precision and scale is based on the converted value.

[kiszk]

Thanks, updated.

[gatorsmile]

LGTM except one comment.

cc @cloud-fan

[SparkQA]

Test build #76019 has finished for PR 17684 at commit 6425239.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

LGTM, merging to master/2.2!

[cloud-fan]

it conflicts with branch 2.1, can you send a new PR? thanks!