[SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math

[yjshen]

JIRA:
https://issues.apache.org/jira/browse/SPARK-8280
https://issues.apache.org/jira/browse/SPARK-8281

[yjshen]

Log family semantic comparision:

Expression	Hive	Spark SQL
ln(0.0)	NULL	-Infinity
ln(-1)	NULL	NaN

Since we disable udf7 to enable log base between (0.0, 1], we do not have log related comparsion with Hive. So, what about other semantics of Log?

[yjshen]

Still need + 0.0 here, math.atan2 is calling java.lang.Math.atan2 inside

[yjshen]

cc @rxin

[rxin]

Let's follow Hive for log, and make sure we add a unit test.

[yjshen]

So we behave like Hive except that we do support log base in between (0.0, 1.0], and add a unit test like udf7

[rxin]

Yup.

[rxin]

Jenkins, add to whitelist.

[rxin]

Jenkins, ok to test.

[rxin]

Jenkins, test this please.

[rxin]

missed a space here

[rxin]

Jenkins, retest this please.

[SparkQA]

Test build #37596 has finished for PR 7451 at commit fb73b87.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class UnaryLogExpression(f: Double => Double, name: String)
  • case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
  • case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
  • case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")

[SparkQA]

Test build #37605 has finished for PR 7451 at commit 003ab23.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class RFormula(override val uid: String)
  • abstract class LeafExpression extends Expression
  • abstract class UnaryExpression extends Expression
  • abstract class BinaryExpression extends Expression
  • abstract class UnaryLogExpression(f: Double => Double, name: String)
  • case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
  • case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
  • case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")
  • case class Length(child: Expression) extends UnaryExpression with ExpectsInputTypes
  • case class FormatNumber(x: Expression, d: Expression)
  • abstract class BinaryNode extends LogicalPlan

[yjshen]

rebased the code to resolve merge conflicts.

[rxin]

Jenkins, retest this please.

[SparkQA]

Test build #37646 has finished for PR 7451 at commit a83b47c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class UnsafeProjection extends Projection
  • case class FromUnsafeProjection(fields: Seq[DataType]) extends Projection
  • abstract class BaseProjection extends Projection
  • class SpecificProjection extends $
  • class SpecificProjection extends $
  • abstract class UnaryLogExpression(f: Double => Double, name: String)
  • case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
  • case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
  • case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")

[rxin]

Oops there is another conflict. I tried resolving this locally but it was a big one. Can you resolve it?

[yjshen]

yes, I've resolved it locally, push here again in a minute.

[rxin]

Jenkins, retest this please.

[rxin]

Jenkins, retest this please.

[rxin]

Thanks - I've merged this.