[SPARK-8148] Do not use FloatType in partition column inference.

[rxin]

Use DoubleType instead to be more stable and robust.

[rxin]

cc @liancheng

[rxin]

The other thing we should consider, although I'm less sure about, is whether we should skip IntegerType and go straight to LongType.

[SparkQA]

Test build #34396 has finished for PR 6692 at commit f88b4e2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #34398 has finished for PR 6692 at commit 0ac8c31.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GeneratedExpressionCode(var code: Code, var isNull: Term, var primitive: Term)
  • class CodeGenContext
  • case class Pow(left: Expression, right: Expression)
  • case class Rint(child: Expression) extends UnaryMathExpression(math.rint, "ROUND")
  • case class ToDegrees(child: Expression) extends UnaryMathExpression(math.toDegrees, "DEGREES")
  • case class ToRadians(child: Expression) extends UnaryMathExpression(math.toRadians, "RADIANS")

[liancheng]

I'm worrying about skipping FloatType (and possibly IntegerType) might break existing user code because the partition column data types gets changed. Especially when the inferred schema gets persisted in places like Parquet file metadata and metastore.

[rxin]

If it is persisted in the metastore, then inference is no longer used, isn't it?

And why would partition columns be stored in Parquet file metadata?

[liancheng]

Had offline discussion with @rxin. There can be rare corner cases where compatibility issues may arise. However, stop using FloatType in this case can eliminate these corner cases. So we would like to have this and add a note in release notes.

[SparkQA]

Test build #34419 has finished for PR 6692 at commit 4fd761f.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #34457 has finished for PR 6692 at commit 6742ecc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #891 timed out for PR 6692 at commit 4fd761f after a configured wait of 175m.