Skip to content
This repository was archived by the owner on Jun 19, 2026. It is now read-only.

Add the support for ignoreTrailingWhiteSpace and ignoreLeadingWhiteSpace - #306

Closed
HyukjinKwon wants to merge 8 commits into
databricks:masterfrom
HyukjinKwon:ISSUE-304
Closed

Add the support for ignoreTrailingWhiteSpace and ignoreLeadingWhiteSpace#306
HyukjinKwon wants to merge 8 commits into
databricks:masterfrom
HyukjinKwon:ISSUE-304

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

https://github.com/databricks/spark-csv/issues/304

This PR adds the support for ignoreTrailingWhiteSpace and ignoreLeadingWhiteSpace options which were exposed but not implemented.

For Apache Common CSV Parser (common), it supports the functionalities only when both ignoreTrailingWhiteSpace and ignoreLeadingWhiteSpace are set to true by using withIgnoreSurroundingSpaces() at CSVFormat.

For Univocity parser (univocity), it supports this functionalities for ignoreTrailingWhiteSpace, and/or ignoreLeadingWhiteSpace by using setIgnoreLeadingWhitespaces() and/or setIgnoreTrailingWhitespaces() at CsvParserSettings.

@codecov-io

Copy link
Copy Markdown

Current coverage is 87.43%

Merging #306 into master will increase coverage by +0.77% as of 7eb2c30

@@ master #306 diff @@
======================================
Files 12 12 Stmts 540 541 +1
Branches 160 160 Methods 0 0 ======================================
+ Hit 468 473 +5
Partial 0 0 + Missed 72 68 -4

Review entire Coverage Diff as of 7eb2c30

Powered by Codecov. Updated on successful CI builds.

@dbspace

Copy link
Copy Markdown

I try to use this, but I am facing a strange issue. I am able to create a dataframe from the csv, but if I try to use the dataframe that is created and convert 1 of column from string to date using withColumn and then try to write it out to parquet I get a number format exception. However if I write the dataframe out as parquet first and read the parquet back and use the withColumn to create a new dataframe and write that out everything works fine. I can create a test case if you need more information

@HyukjinKwon

Copy link
Copy Markdown
MemberAuthor

I can rebase this if it seems worth adding it.

@mrchristine

Copy link
Copy Markdown

Any reason users would want to keep whitespace on one side? I'd recommend simplifying this to add "ignoreHeaderWhitespace" to ignore trailing and leading whitespace. I don't see a use case where users would want to keep one or the other, and it makes referring to columns more complex. Users might run into more issues if they only add 1 option, but wanted both.

@HyukjinKwon

Copy link
Copy Markdown
MemberAuthor

Closing this as it is now maintenance mode and it seems it is not a critical bug.

@jarvisrjz

Copy link
Copy Markdown

Any plans to add this? I noticed the two ignore whitespace options on the javadoc but was unable to use them

@dereXeus

Copy link
Copy Markdown

I have forked this repo. Any chances you can raise a pull request to it ?

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@HyukjinKwon@codecov-io@dbspace@mrchristine@jarvisrjz@dereXeus