Skip to content

[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile - #1924

Closed
ghost wants to merge 2 commits into
masterfrom
unknown repository
Closed

[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile#1924
ghost wants to merge 2 commits into
masterfrom
unknown repository

Conversation

@ghost

@ghost ghost commented Aug 13, 2014

Copy link
Copy Markdown

No description provided.

@AmplabJenkins

AmplabJenkins commented Aug 13, 2014

Copy link
Copy Markdown

Can one of the admins verify this patch?

@marmbrus

marmbrus commented Aug 13, 2014

Copy link
Copy Markdown
Contributor

test this please

@SparkQA

SparkQA commented Aug 13, 2014

Copy link
Copy Markdown

QA tests have started for PR 1924. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18449/consoleFull

@SparkQA

SparkQA commented Aug 13, 2014

Copy link
Copy Markdown

QA results for PR 1924:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18449/consoleFull

@marmbrus

marmbrus commented Aug 13, 2014

Copy link
Copy Markdown
Contributor

BTW you can check style on your local machine by running sbt/sbt scalastyle.

@ghost

ghost commented Aug 14, 2014

Copy link
Copy Markdown
Author

code fixed.

@marmbrus

marmbrus commented Aug 14, 2014

Copy link
Copy Markdown
Contributor

test this please

@SparkQA

SparkQA commented Aug 14, 2014

Copy link
Copy Markdown

QA tests have started for PR 1924. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18526/consoleFull

@SparkQA

SparkQA commented Aug 14, 2014

Copy link
Copy Markdown

QA results for PR 1924:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18526/consoleFull

@marmbrus

marmbrus commented Aug 14, 2014

Copy link
Copy Markdown
Contributor

The test failures look unrelated.

I've merge this to master and 1.1. Thanks!

asfgit pushed a commit that referenced this pull request Aug 14, 2014
@josephsu @marmbrus
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
221c84e 
…ontext.parquetFile

Author: Chia-Yung Su <chiayung@appier.com>

Closes #1924 from joesu/bugfix-spark3011 and squashes the following commits:

c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir

(cherry picked from commit 078f3fb)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@asfgit asfgit closed this in 078f3fb Aug 14, 2014
@tgravescs

tgravescs commented Aug 14, 2014

Copy link
Copy Markdown
Contributor

I'm seeing build failures from this change on hadoop 0.23. I haven't had time to dig into it.

sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypes.scala:382: value TEMP_DIR_NAME is not a member of object org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
[ERROR] name == FileOutputCommitter.TEMP_DIR_NAME

@marmbrus

marmbrus commented Aug 14, 2014

Copy link
Copy Markdown
Contributor

I'll revert this change.

@marmbrus marmbrus mentioned this pull request Aug 14, 2014
Closed
asfgit pushed a commit that referenced this pull request Aug 14, 2014
@marmbrus
Revert [SPARK-3011][SQL] _temporary directory should be filtered out …
c39a3f3 
…by sqlContext.parquetFile

Reverts #1924 due to build failures with hadoop 0.23.

Author: Michael Armbrust <michael@databricks.com>

Closes #1949 from marmbrus/revert1924 and squashes the following commits:

6bff940 [Michael Armbrust] Revert "[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile"

(cherry picked from commit a7f8a4f)
Signed-off-by: Michael Armbrust <michael@databricks.com>
asfgit pushed a commit that referenced this pull request Aug 14, 2014
@marmbrus
Revert [SPARK-3011][SQL] _temporary directory should be filtered out …
a7f8a4f 
…by sqlContext.parquetFile

Reverts #1924 due to build failures with hadoop 0.23.

Author: Michael Armbrust <michael@databricks.com>

Closes #1949 from marmbrus/revert1924 and squashes the following commits:

6bff940 [Michael Armbrust] Revert "[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile"
@tgravescs

tgravescs commented Aug 14, 2014

Copy link
Copy Markdown
Contributor

Thanks, I took a quick look to see what the problem is and it appears that was removed in hadoop 0.23 and then put back in in hadoop 2.x to be compatible with hadoop 1.x:

https://issues.apache.org/jira/browse/MAPREDUCE-5229

So unfortunately we would need a way to have it not look at that on hadoop 0.23.

@marmbrus

marmbrus commented Aug 14, 2014

Copy link
Copy Markdown
Contributor

We could just hard code the string _temporary with a note about why.
On Aug 14, 2014 1:07 PM, "Tom Graves" notifications@github.com wrote:

Thanks, I took a quick look to see what the problem is and it appears that
was removed in hadoop 0.23 and then put back in in hadoop 2.x to be
compatible with hadoop 1.x:

https://issues.apache.org/jira/browse/MAPREDUCE-5229

So unfortunately we would need a way to have it not look at that on hadoop
0.23.

—
Reply to this email directly or view it on GitHub
#1924 (comment).

@ghost ghost mentioned this pull request Aug 15, 2014
Closed
@ghost

ghost commented Aug 15, 2014

Copy link
Copy Markdown
Author

Submitted another pull request: #1959

asfgit pushed a commit that referenced this pull request Aug 26, 2014
@josephsu @marmbrus
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
b5dc9b4 
…ontext.parquetFile

fix compile error on hadoop 0.23 for the pull request #1924.

Author: Chia-Yung Su <chiayung@appier.com>

Closes #1959 from joesu/bugfix-spark3011 and squashes the following commits:

be30793 [Chia-Yung Su] remove .* and _* except _metadata
8fe2398 [Chia-Yung Su] add note to explain
40ea9bd [Chia-Yung Su] fix hadoop-0.23 compile error
c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir

(cherry picked from commit 4243bb6)
Signed-off-by: Michael Armbrust <michael@databricks.com>
asfgit pushed a commit that referenced this pull request Aug 26, 2014
@josephsu @marmbrus
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
4243bb6 
…ontext.parquetFile

fix compile error on hadoop 0.23 for the pull request #1924.

Author: Chia-Yung Su <chiayung@appier.com>

Closes #1959 from joesu/bugfix-spark3011 and squashes the following commits:

be30793 [Chia-Yung Su] remove .* and _* except _metadata
8fe2398 [Chia-Yung Su] add note to explain
40ea9bd [Chia-Yung Su] fix hadoop-0.23 compile error
c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@josephsu @conviva-zz
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
a869de8 
…ontext.parquetFile

Author: Chia-Yung Su <chiayung@appier.com>

Closes apache#1924 from joesu/bugfix-spark3011 and squashes the following commits:

c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@marmbrus @conviva-zz
Revert [SPARK-3011][SQL] _temporary directory should be filtered out …
91f59d7 
…by sqlContext.parquetFile

Reverts apache#1924 due to build failures with hadoop 0.23.

Author: Michael Armbrust <michael@databricks.com>

Closes apache#1949 from marmbrus/revert1924 and squashes the following commits:

6bff940 [Michael Armbrust] Revert "[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile"
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@josephsu @conviva-zz
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
fbcdc39 
…ontext.parquetFile

fix compile error on hadoop 0.23 for the pull request apache#1924.

Author: Chia-Yung Su <chiayung@appier.com>

Closes apache#1959 from joesu/bugfix-spark3011 and squashes the following commits:

be30793 [Chia-Yung Su] remove .* and _* except _metadata
8fe2398 [Chia-Yung Su] add note to explain
40ea9bd [Chia-Yung Su] fix hadoop-0.23 compile error
c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@AmplabJenkins @marmbrus @SparkQA @tgravescs @josephsu