Uh oh!
There was an error while loading. Please reload this page.
Add the support for ignoreTrailingWhiteSpace and ignoreLeadingWhiteSpace - #306
Add the support for ignoreTrailingWhiteSpace and ignoreLeadingWhiteSpace#306HyukjinKwon wants to merge 8 commits into
Conversation
codecov-io
commented
Mar 23, 2016
Current coverage is |
dbspace
commented
May 3, 2016
I try to use this, but I am facing a strange issue. I am able to create a dataframe from the csv, but if I try to use the dataframe that is created and convert 1 of column from string to date using withColumn and then try to write it out to parquet I get a number format exception. However if I write the dataframe out as parquet first and read the parquet back and use the withColumn to create a new dataframe and write that out everything works fine. I can create a test case if you need more information |
HyukjinKwon
commented
Jul 20, 2016
I can rebase this if it seems worth adding it. |
mrchristine
commented
Aug 17, 2016
Any reason users would want to keep whitespace on one side? I'd recommend simplifying this to add "ignoreHeaderWhitespace" to ignore trailing and leading whitespace. I don't see a use case where users would want to keep one or the other, and it makes referring to columns more complex. Users might run into more issues if they only add 1 option, but wanted both. |
HyukjinKwon
commented
Sep 5, 2016
Closing this as it is now maintenance mode and it seems it is not a critical bug. |
jarvisrjz
commented
Nov 25, 2016
Any plans to add this? I noticed the two ignore whitespace options on the javadoc but was unable to use them |
dereXeus
commented
Jan 6, 2017
I have forked this repo. Any chances you can raise a pull request to it ? |
https://github.com/databricks/spark-csv/issues/304
This PR adds the support for
ignoreTrailingWhiteSpaceandignoreLeadingWhiteSpaceoptions which were exposed but not implemented.For Apache Common CSV Parser (
common), it supports the functionalities only when bothignoreTrailingWhiteSpaceandignoreLeadingWhiteSpaceare set totrueby usingwithIgnoreSurroundingSpaces()atCSVFormat.For Univocity parser (
univocity), it supports this functionalities forignoreTrailingWhiteSpace, and/orignoreLeadingWhiteSpaceby usingsetIgnoreLeadingWhitespaces()and/orsetIgnoreTrailingWhitespaces()atCsvParserSettings.