Support for nullable schema types

[dtpeacock]

This closes #85 and closes #97 .

This seems to be most sensible behavior - if the schema specifies a nullable type, it should return null on an empty field.

To maintain backwards compatibility, I have left it so that in the case of a StringType, "" is passed back rather than null (i.e. fieldname='' does not have to be changed to fieldname=null).

Its worth confirming what the desired behavior should be here as in #86 - should "" be treated as null or an empty string, as here would be the place to change it.

[codecov-io]

Current coverage is 85.31%

Merging #102 into master will not effect coverage as of 3570e1c

@@ master #102 diff @@
======================================
Files 10 10 Stmts 388 388 Branches 117 117 Methods 0 0 ======================================
Hit 331 331 Partial 0 0 Missed 57 57 

Review entire Coverage Diff

Powered by Codecov

[falaki]

I don't think we need to check for nullability for StringType. Basically I think the following logic would be simple an intuitive:

• if StringType just return the token (don't need to check for nullability)
• if any other time, if token is "", return null, else cast.

I suggest moving this logic (along with some comment that explains it) to TypeCast.castTo (maybe as a simple private method). This way you can add a few simple unit tests for it. Added benefit is we keep CsvRelation simpler.

[falaki]

Thanks @dtpeacock for submitting this. I left one comment, please address that and I will merge it. I am planning to cut a release after this.

[dtpeacock]

Thanks, I've addressed the comment and added a couple more tests!

[falaki]

I think this would be simpler:

if (datum ==""&& nullable &&!castType.isInstanceOf[StringType]) {
null
} else {
castType match {
case_: ByteType=> datum.toByte
case_: ShortType=> datum.toShort
case_: IntegerType=> datum.toInt
case_: LongType=> datum.toLong
case_: FloatType=> datum.toFloat
case_: DoubleType=> datum.toDouble
case_: BooleanType=> datum.toBoolean
case_: DecimalType=>newBigDecimal(datum.replaceAll(",", ""))
// TODO(hossein): would be good to support other common timestamp formatscase_: TimestampType=>Timestamp.valueOf(datum)
// TODO(hossein): would be good to support other common date formatscase_: DateType=>Date.valueOf(datum)
case_: StringType=> datum
case _ =>thrownewRuntimeException(s"Unsupported type: ${castType.typeName}")
}
}

[dtpeacock]

Agreed that logic is a bit simpler