parsing options and serializing arrays

[mohitjaggi]

several parsing options are added. they are organized in classes because there are many of them. a "text" based API to configure options is provided.

another feature is the ability to serialize a column of arrays. the array is "unnested" and column names to use are supplied by user. this is useful for writing out csv after doing transforms on the data that "expand" the number of columns e.g. one hot encode a category. this can be improved later. e.g. sparse vector from mllib can replace the array.

[falaki]

Why do you need this?

[falaki]

Please update it to rebase against master for tests to run. Also please revert changes to build.sbt.

[codecov-io]

Current coverage is 84.79%

Merging #113 into master will decrease coverage by -0.22% as of 516c8a0

@@ master #113 diff @@
======================================
Files 10 11 +1
Stmts 407 526 +119
Branches 125 150 +25
Methods 0 0 ======================================
+ Hit 346 446 +100
Partial 0 0 - Missed 61 80 +19

Review entire Coverage Diff

Powered by Codecov

[mohitjaggi]

@falaki should be ready to merge now.

[falaki]

@mohitjaggi This is fairly large. I am about to publish a release with schema inference and all the recent improvements, and then I will review this.

[falaki]

@mohitjaggi this is packing too much into a single PR. Would you please split it. Please first submit one for number parsing improvements and another for arrays. On arrays within CSV it would be good to post an issue and gather some feedback from community first.

[mohitjaggi]

will do

On Thu, Aug 13, 2015 at 10:14 AM, Hossein Falaki notifications@github.com
wrote:

@mohitjaggihttps://github.com/mohitjaggi this is packing too much into
a single PR. Would you please split it. Please first submit one for number
parsing improvements and another for arrays. On arrays within CSV it would
be good to post an issue and gather some feedback from community first.

—
Reply to this email directly or view it on GitHub
#113 (comment).

[mohitjaggi]

see #124