Fix schema/type inference issue #216

[tanwanirahul]

The issue with current implementation is that, it bumps up the NullType to StringType in reduce phase of each partition. When the final reduce across partitions takes place the column type is inferred as StringType even though the column has only one integer value because of incorrect bumping up of NullTypes.

Here, in this pull, we don't bump up the Null to String as a part of reduce operation (within and across partitions). After the reduction, when we create Fields of StructType, we convert NullTypes to StringTypes.

[codecov-io]

Current coverage is 85.74%

Merging #244 into master will decrease coverage by -0.33% as of 8979a5a

@@ master #244 diff @@
======================================
Files 12 12 Stmts 517 519 +2
Branches 149 149 Methods 0 0 ======================================
Hit 445 445 Partial 0 0 - Missed 72 74 +2

Review entire Coverage Diff as of 8979a5a

Powered by Codecov. Updated on successful CI builds.

[falaki]

Thanks a lot for submitting this. The code looks good. Would you add another unit test that verify it works end-to-end. Maybe use the input you provided in the issue:

A,B,C,D
1,,,
,1,,
,,1,
,,,1

[tanwanirahul]

Added an end to end test case to validate the proper schema/type inference. @falaki Take a look and merge if everything is fine.

[HyukjinKwon]

Hm.. Shouldn't this go to CsvSuite and remove with BeforeAndAfterAll ,beforeAll() and afterAll() as this test is a end-to-end test?

[tanwanirahul]

I agree that CsvSuite performs all end to end tests but since there was a dedicated suite for SchemaInference, I did prefer to put the schema tests in there. Do you see any issues in that?

[HyukjinKwon]

No, I think it is okay though. I just said this because I see suites have been added in this way.

For example, #235 and #224

[tanwanirahul]

Reverted the refactoring part. Tests I kept as is. If more people feel CsvSuite would rather be the right place to put these tests in, I will make that change.

[tanwanirahul]

@HyukjinKwon Created #248 for the refactoring part.

[falaki]

Nit: the indent is off:

assert(
InferSchema.mergeRowTypes(Array(NullType),
Array(NullType)).deep ==Array(NullType).deep)

[falaki]

@tanwanirahul left some more comments. Once you address I am going to merge this. After that would you mind submitting a patch to spark. Spark-csv is now inlined in Spark 2.0.

[tanwanirahul]

@HyukjinKwon Moved end to end tests to CSVSuite, @falaki Also recommended the same.

@falaki Take a look at the CSVSuite for test case and InferSchemaSuite for Indent comments you left. Also let me know your thoughts on #248

I will push the patch to spark repo soon.

[HyukjinKwon]

I think this is ignoerable though.. I believe this { might better move up to the previous line.

[tanwanirahul]

@HyukjinKwon@falaki Is there any eclipse formatting profile that I could load and apply it for the code I am writing?

[tanwanirahul]

@falaki Could we re-build the commit? I don't think build failed because of the code issue.

[tanwanirahul]

@falaki@HyukjinKwon Could you please look at what is causing the build to fail? Looking at the build error, I don't think its code issue.

[tanwanirahul]

@falaki Could you please suggest next steps for this? Also, is there any date planned for the next release? We are currently using this change by adding an unmanaged dependency.

[falaki]

Would you please try that combination of spark and openjdk version to investigate what goes wrong?

[HyukjinKwon]

Let me take a look when I have some time.

[HyukjinKwon]

@falaki Hm.. It looks like this was failed due to the cached ones in travis.ci. This works fine both on my local and travis with a new build here.

I will create a new PR for this. Would you please make the author @tanwanirahul when you merge that PR by merge_pr.py?

[HyukjinKwon]

Can we maybe close this?