Skip to content

[SPARK-5123][SQL] Reconcile Java/Scala API for data types. - #3958

Closed
rxin wants to merge 1 commit into
apache:masterfrom
rxin:SPARK-5123-datatype-2
Closed

[SPARK-5123][SQL] Reconcile Java/Scala API for data types.#3958
rxin wants to merge 1 commit into
apache:masterfrom
rxin:SPARK-5123-datatype-2

Conversation

@rxin

@rxinrxin commented Jan 8, 2015

Copy link
Copy Markdown
Contributor

Having two versions of the data type APIs (one for Java, one for Scala) requires downstream libraries to also have two versions of the APIs if the library wants to support both Java and Scala. I took a look at the Scala version of the data type APIs - it can actually work out pretty well for Java out of the box.

As part of the PR, I created a sql.types package and moved all type definitions there. I then removed the Java specific data type API along with a lot of the conversion code.

This subsumes #3925

@rxinrxin mentioned this pull request Jan 8, 2015
Closed
@SparkQA

SparkQA commented Jan 8, 2015

Copy link
Copy Markdown

Test build #25272 has started for PR 3958 at commit b4f9649.

  • This patch merges cleanly.

@SparkQA

SparkQA commented Jan 9, 2015

Copy link
Copy Markdown

Test build #25272 has finished for PR 3958 at commit b4f9649.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Jan 9, 2015

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25272/
Test FAILed.

@rxinrxin mentioned this pull request Jan 11, 2015
Closed
marmbrus
marmbrus reviewed Jan 13, 2015
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/types/dataTypes.scala

@marmbrusmarmbrusJan 13, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double newlines here?

@marmbrus

marmbrus commented Jan 13, 2015

Copy link
Copy Markdown
Contributor

Okay, the other PR is in, if you fix the merge conflicts this LGTM. We also need to figure out how to publish scala/java docs here.

@rxin
rxinforce-pushed the SPARK-5123-datatype-2 branch from b4f9649 to 66505ccCompareJanuary 13, 2015 22:13
@rxin

rxin commented Jan 13, 2015

Copy link
Copy Markdown
ContributorAuthor

I just rebased. Let's see ...

@SparkQA

SparkQA commented Jan 13, 2015

Copy link
Copy Markdown

Test build #25487 has started for PR 3958 at commit 66505cc.

  • This patch merges cleanly.

@SparkQA

SparkQA commented Jan 13, 2015

Copy link
Copy Markdown

Test build #25487 has finished for PR 3958 at commit 66505cc.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Jan 13, 2015

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25487/
Test FAILed.

@rxin

rxin commented Jan 14, 2015

Copy link
Copy Markdown
ContributorAuthor

Jenkins, retest this please.

@SparkQA

SparkQA commented Jan 14, 2015

Copy link
Copy Markdown

Test build #25492 has started for PR 3958 at commit 66505cc.

  • This patch merges cleanly.

@JoshRosen

JoshRosen commented Jan 14, 2015

Copy link
Copy Markdown
Contributor

You might want to copy the commit message over from the other PR.

@rxin

rxin commented Jan 14, 2015

Copy link
Copy Markdown
ContributorAuthor

done - thanks for reminding, josh.

@rxinrxin changed the title [SPARK-5123] Expose only one version of the data type APIs (i.e. remove the Java-specific API).[SPARK-5123][SQL] Reconcile Java/Scala API for data types.Jan 14, 2015
@rxin

rxin commented Jan 14, 2015

Copy link
Copy Markdown
ContributorAuthor

Ok merging in master. Thanks for looking at this.

@SparkQA

SparkQA commented Jan 14, 2015

Copy link
Copy Markdown

Test build #25492 has finished for PR 3958 at commit 66505cc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Jan 14, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25492/
Test PASSed.

@rxin
rxin deleted the SPARK-5123-datatype-2 branch January 14, 2015 08:07
This was referenced Jan 16, 2015
Closed
Closed
asfgit pushed a commit that referenced this pull request Jan 17, 2015
@rxin
[SPARK-5193][SQL] Remove Spark SQL Java-specific API.
61b427d
After the following patches, the main (Scala) API is now usable for Java users directly.
#4056#4054#4049#4030#3965#3958
Author: Reynold Xin <rxin@databricks.com>
Closes#4065 from rxin/sql-java-api and squashes the following commits:
b1fd860 [Reynold Xin] Fix Mima
6d86578 [Reynold Xin] Ok one more attempt in fixing Python...
e8f1455 [Reynold Xin] Fix Python again...
3e53f91 [Reynold Xin] Fixed Python.
83735da [Reynold Xin] Fix BigDecimal test.
e9f1de3 [Reynold Xin] Use scala BigDecimal.
500d2c4 [Reynold Xin] Fix Decimal.
ba3bfa2 [Reynold Xin] Updated javadoc for RowFactory.
c4ae1c5 [Reynold Xin] [SPARK-5193][SQL] Remove Spark SQL Java-specific API.
andmarios pushed a commit to andmarios/spark-examples that referenced this pull request Mar 20, 2015
@rxin
[SPARK-5193][SQL] Remove Spark SQL Java-specific API.
3364cc8
After the following patches, the main (Scala) API is now usable for Java users directly.
apache/spark#4056apache/spark#4054apache/spark#4049apache/spark#4030apache/spark#3965apache/spark#3958
Author: Reynold Xin <rxin@databricks.com>
Closes #4065 from rxin/sql-java-api and squashes the following commits:
b1fd860 [Reynold Xin] Fix Mima
6d86578 [Reynold Xin] Ok one more attempt in fixing Python...
e8f1455 [Reynold Xin] Fix Python again...
3e53f91 [Reynold Xin] Fixed Python.
83735da [Reynold Xin] Fix BigDecimal test.
e9f1de3 [Reynold Xin] Use scala BigDecimal.
500d2c4 [Reynold Xin] Fix Decimal.
ba3bfa2 [Reynold Xin] Updated javadoc for RowFactory.
c4ae1c5 [Reynold Xin] [SPARK-5193][SQL] Remove Spark SQL Java-specific API.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@rxin@SparkQA@AmplabJenkins@marmbrus@JoshRosen