Skip to content

[SPARK-5758][SQL] Use LongType as the default type for integers in JSON schema inference. - #4544

Closed
yhuai wants to merge 1 commit into
apache:masterfrom
yhuai:jsonUseLongTypeByDefault
Closed

[SPARK-5758][SQL] Use LongType as the default type for integers in JSON schema inference.#4544
yhuai wants to merge 1 commit into
apache:masterfrom
yhuai:jsonUseLongTypeByDefault

Conversation

@yhuai

@yhuaiyhuai commented Feb 12, 2015

Copy link
Copy Markdown
Contributor

No description provided.

@yhuaiyhuai mentioned this pull request Feb 12, 2015
Closed
@SparkQA

SparkQA commented Feb 12, 2015

Copy link
Copy Markdown

Test build #27309 has started for PR 4544 at commit 6e2ffc2.

  • This patch merges cleanly.

@rxin

rxin commented Feb 12, 2015

Copy link
Copy Markdown
Contributor

lgtm - assuming tests pass.

@SparkQA

SparkQA commented Feb 12, 2015

Copy link
Copy Markdown

Test build #27309 has finished for PR 4544 at commit 6e2ffc2.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 12, 2015

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27309/
Test FAILed.

@rxin

rxin commented Feb 12, 2015

Copy link
Copy Markdown
Contributor

cc @mengxr this broke MLlib... thoughts?

@davies

davies commented Feb 12, 2015

Copy link
Copy Markdown
Contributor

I think some Integers are dumped into JSON, then come out as LongType, then break some APIs.

@mengxr

mengxr commented Feb 12, 2015

Copy link
Copy Markdown
Contributor

Yes, we use row.getInt() in MLlib. I think we should use json4s directly in MLlib's export/import.

@mengxr

mengxr commented Feb 12, 2015

Copy link
Copy Markdown
Contributor

@yhuai I created a JIRA to use json4s directly in MLlib: https://issues.apache.org/jira/browse/SPARK-5757. Could you make a JIRA for this PR and link them?

@mengxrmengxr mentioned this pull request Feb 12, 2015
Closed
@yhuaiyhuai changed the title [SQL] Use LongType as the default type for integers in JSON schema inference.[SPARK-5758][SQL] Use LongType as the default type for integers in JSON schema inference.Feb 12, 2015
@yhuai

yhuai commented Feb 12, 2015

Copy link
Copy Markdown
ContributorAuthor

@mengxr Sure. I have created https://issues.apache.org/jira/browse/SPARK-5758.

asfgit pushed a commit that referenced this pull request Feb 12, 2015
@mengxr
[SPARK-5757][MLLIB] replace SQL JSON usage in model import/export by …
e26c149
…json4s
This PR detaches MLlib model import/export code from SQL's JSON support, and hence unblocks #4544 . yhuai
Author: Xiangrui Meng <meng@databricks.com>
Closes#4555 from mengxr/SPARK-5757 and squashes the following commits:
b0415e8 [Xiangrui Meng] replace SQL JSON usage by json4s
(cherry picked from commit 99bd500)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
asfgit pushed a commit that referenced this pull request Feb 12, 2015
@mengxr
[SPARK-5757][MLLIB] replace SQL JSON usage in model import/export by …
99bd500
…json4s
This PR detaches MLlib model import/export code from SQL's JSON support, and hence unblocks #4544 . yhuai
Author: Xiangrui Meng <meng@databricks.com>
Closes#4555 from mengxr/SPARK-5757 and squashes the following commits:
b0415e8 [Xiangrui Meng] replace SQL JSON usage by json4s
@yhuai

yhuai commented Feb 12, 2015

Copy link
Copy Markdown
ContributorAuthor

test this please

@SparkQA

SparkQA commented Feb 12, 2015

Copy link
Copy Markdown

Test build #27365 has started for PR 4544 at commit 6e2ffc2.

  • This patch merges cleanly.

@SparkQA

SparkQA commented Feb 12, 2015

Copy link
Copy Markdown

Test build #27365 has finished for PR 4544 at commit 6e2ffc2.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 12, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27365/
Test PASSed.

asfgit pushed a commit that referenced this pull request Feb 12, 2015
@yhuai@marmbrus
[SPARK-5758][SQL] Use LongType as the default type for integers in JS…
b0c79da
…ON schema inference.
Author: Yin Huai <yhuai@databricks.com>
Closes#4544 from yhuai/jsonUseLongTypeByDefault and squashes the following commits:
6e2ffc2 [Yin Huai] Use LongType as the default type for integers in JSON schema inference.
(cherry picked from commit c352ffb)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@yhuai
yhuai deleted the jsonUseLongTypeByDefault branch February 13, 2015 02:30
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@yhuai@SparkQA@rxin@AmplabJenkins@davies@mengxr