Skip to content

[SPARK-15148][SQL] Upgrade Univocity library from 2.0.2 to 2.1.0 - #12923

Closed
HyukjinKwon wants to merge 3 commits into
apache:masterfrom
HyukjinKwon:SPARK-15148
Closed

[SPARK-15148][SQL] Upgrade Univocity library from 2.0.2 to 2.1.0#12923
HyukjinKwon wants to merge 3 commits into
apache:masterfrom
HyukjinKwon:SPARK-15148

Conversation

@HyukjinKwon

@HyukjinKwonHyukjinKwon commented May 5, 2016

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/SPARK-15148

Mainly it improves the performance roughtly about 30%-40% according to the release note. For the details of the purpose is described in the JIRA.

This PR upgrades Univocity library from 2.0.2 to 2.1.0.

How was this patch tested?

Existing tests should cover this.

@HyukjinKwon

Copy link
Copy Markdown
MemberAuthor

cc @rxin and @jbax (who is the author of Univocity library and suggested this change).

@rxin

rxin commented May 5, 2016

Copy link
Copy Markdown
Contributor

LGTM pending Jenkins.

@holdenk

Copy link
Copy Markdown
Contributor

So doing a quick grep shows that we are using the newly deprecated method setParseUnescapedQuotes and setParseUnescapedQuotesUntilDelimiter, we should update to setUnescapedQuoteHandling as per the release notes ( https://github.com/uniVocity/univocity-parsers/releases )

If its really urgent to get this upgrade in quickly, lets make a follow up JIRA to move away from these deprecated functions for the next release.

@HyukjinKwon

Copy link
Copy Markdown
MemberAuthor

Thanks, @holdenk! It is not urgent. I can do this in this PR.

@holdenk

Copy link
Copy Markdown
Contributor

Great!

@SparkQA

Copy link
Copy Markdown

Test build #2975 has finished for PR 12923 at commit e71c1ee.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

Copy link
Copy Markdown

Test build #57853 has finished for PR 12923 at commit e71c1ee.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

Copy link
Copy Markdown

Test build #57861 has finished for PR 12923 at commit e1e8063.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen

Copy link
Copy Markdown
Member

LGTM, except that it needs a rebase now

@SparkQA

Copy link
Copy Markdown

Test build #57875 has finished for PR 12923 at commit bf7262f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin

rxin commented May 5, 2016

Copy link
Copy Markdown
Contributor

Merging in master/branch-2.0.

@asfgitasfgit closed this in ac12b35May 5, 2016
asfgit pushed a commit that referenced this pull request May 5, 2016
## What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-15148
Mainly it improves the performance roughtly about 30%-40% according to the [release note](https://github.com/uniVocity/univocity-parsers/releases/tag/v2.1.0). For the details of the purpose is described in the JIRA.
This PR upgrades Univocity library from 2.0.2 to 2.1.0.
## How was this patch tested?
Existing tests should cover this.
Author: hyukjinkwon <gurwls223@gmail.com>
Closes#12923 from HyukjinKwon/SPARK-15148.
(cherry picked from commit ac12b35)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@HyukjinKwon
HyukjinKwon deleted the SPARK-15148 branch January 2, 2018 03:40
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@HyukjinKwon@rxin@holdenk@SparkQA@srowen