Skip to content

[SPARK-6546][Build] Using the wrong code that will make spark compile failed!! - #5198

Closed
pzzs wants to merge 9 commits into
apache:masterfrom
pzzs:FilesBug
Closed

[SPARK-6546][Build] Using the wrong code that will make spark compile failed!!#5198
pzzs wants to merge 9 commits into
apache:masterfrom
pzzs:FilesBug

Conversation

@pzzs

@pzzspzzs commented Mar 26, 2015

Copy link
Copy Markdown
Contributor

wrong code : val tmpDir = Files.createTempDir()
not Files should Utils

@AmplabJenkins

AmplabJenkins commented Mar 26, 2015

Copy link
Copy Markdown

Can one of the admins verify this patch?

@pzzspzzs changed the title A little spell wrong, but this will make spark compile failed!![SPARK-6300][Build] Using the wrong code that will make spark compile failed!!Mar 26, 2015
@pzzspzzs changed the title [SPARK-6300][Build] Using the wrong code that will make spark compile failed!![SPARK-6546][Build] Using the wrong code that will make spark compile failed!!Mar 26, 2015
@pzzs

pzzs commented Mar 26, 2015

Copy link
Copy Markdown
ContributorAuthor

i think this is pressing @marmbrus

@JoshRosen

JoshRosen commented Mar 26, 2015

Copy link
Copy Markdown
Contributor

Jenkins, this is ok to test.

@JoshRosen

JoshRosen commented Mar 26, 2015

Copy link
Copy Markdown
Contributor

Do you know which PR broke this?

@liancheng

liancheng commented Mar 26, 2015

Copy link
Copy Markdown
Contributor

@JoshRosen It should be #4289.

@SparkQA

SparkQA commented Mar 26, 2015

Copy link
Copy Markdown

Test build #29206 has started for PR 5198 at commit 6e0140d.

  • This patch merges cleanly.

@liancheng

liancheng commented Mar 26, 2015

Copy link
Copy Markdown
Contributor

�This LGTM pending Jenkins. #4289 somehow passed Jenkins, however the build history has been cleaned now.

@liancheng

liancheng commented Mar 26, 2015

Copy link
Copy Markdown
Contributor

@JoshRosen@marmbrus#4289 was using Guava's com.google.common.io.Files according to the first commit of that PR, see here. However, #5029 was merged earlier, and deprecated Guava Files by Utils.Files. These two combined caused this build failure. (There're no conflicts in the eyes of Git, but there do exist semantic conflicts.)

@SparkQA

SparkQA commented Mar 26, 2015

Copy link
Copy Markdown

Test build #29206 has finished for PR 5198 at commit 6e0140d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class UnregisterApplication(appId: String)
    • class RegexTokenizer extends UnaryTransformer[String, Seq[String], RegexTokenizer]
    • case class Data(weight: Double, mu: Vector, sigma: Matrix)
    • implicit class DslSymbol(sym: Symbol) extends ImplicitAttribute
    • case class Alias(child: Expression, name: String)(
    • class Column(protected[sql] val expr: Expression) extends Logging

@AmplabJenkins

AmplabJenkins commented Mar 26, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29206/
Test PASSed.

@pzzs

pzzs commented Mar 26, 2015

Copy link
Copy Markdown
ContributorAuthor

you are right @liancheng

@zzcclpzzcclp mentioned this pull request Mar 26, 2015
Closed
zzcclp referenced this pull request Mar 26, 2015
@jeanlyn@marmbrus
[SPARK-5498][SQL]fix query exception when partition schema does not m…
e6d1406
…atch table schema
In hive,the schema of partition may be difference from the table schema.When we use spark-sql to query the data of partition which schema is difference from the table schema,we will get the exceptions as the description of the [jira](https://issues.apache.org/jira/browse/SPARK-5498) .For example:
* We take a look of the schema for the partition and the table
```sql
DESCRIBE partition_test PARTITION (dt='1');
id int None
name string None
dt string None
# Partition Information
# col_name data_type comment
dt string None
```
```
DESCRIBE partition_test;
OK
id bigint None
name string None
dt string None
# Partition Information
# col_name data_type comment
dt string None
```
* run the sql
```sql
SELECT * FROM partition_test where dt='1';
```
we will get the cast exception `java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableLong cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableInt`
Author: jeanlyn <jeanlyn92@gmail.com>
Closes#4289 from jeanlyn/schema and squashes the following commits:
9c8da74 [jeanlyn] fix style
b41d6b9 [jeanlyn] fix compile errors
07d84b6 [jeanlyn] Merge branch 'master' into schema
535b0b6 [jeanlyn] reduce conflicts
d6c93c5 [jeanlyn] fix bug
1e8b30c [jeanlyn] fix code style
0549759 [jeanlyn] fix code style
c879aa1 [jeanlyn] clean the code
2a91a87 [jeanlyn] add more test case and clean the code
12d800d [jeanlyn] fix code style
63d170a [jeanlyn] fix compile problem
7470901 [jeanlyn] reduce conflicts
afc7da5 [jeanlyn] make getConvertedOI compatible between 0.12.0 and 0.13.1
b1527d5 [jeanlyn] fix type mismatch
10744ca [jeanlyn] Insert a space after the start of the comment
3b27af3 [jeanlyn] SPARK-5498:fix bug when query the data when partition schema does not match table schema
@WangTaoTheTonicWangTaoTheTonic mentioned this pull request Mar 26, 2015
Closed
@liancheng

liancheng commented Mar 26, 2015

Copy link
Copy Markdown
Contributor

@DoingDone9 Would you mind to add your name to JIRA and GitHub, so that we can include your name in the credit list of the next release? Also, please add your name in your local Git configuration:

$ git config --global user.name "Your Name"

@liancheng

liancheng commented Mar 26, 2015

Copy link
Copy Markdown
Contributor

Merged into master, thanks!

@pzzs

pzzs commented Mar 26, 2015

Copy link
Copy Markdown
ContributorAuthor

ok i will change @liancheng

@zzcclpzzcclp mentioned this pull request Mar 26, 2015
Closed
@viiryaviirya mentioned this pull request Mar 26, 2015
Closed
@jeanlynjeanlyn mentioned this pull request Mar 27, 2015
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@pzzs@AmplabJenkins@JoshRosen@liancheng@SparkQA