Skip to content

[SPARK-1459] Use local path (and not complete URL) when opening local lo... - #375

Closed
vanzin wants to merge 1 commit into
apache:masterfrom
vanzin:event-file
Closed

[SPARK-1459] Use local path (and not complete URL) when opening local lo...#375
vanzin wants to merge 1 commit into
apache:masterfrom
vanzin:event-file

Conversation

@vanzin

@vanzinvanzin commented Apr 9, 2014

Copy link
Copy Markdown
Contributor

...g file.

@AmplabJenkins

AmplabJenkins commented Apr 9, 2014

Copy link
Copy Markdown

Can one of the admins verify this patch?

@pwendell

pwendell commented Apr 10, 2014

Copy link
Copy Markdown
Contributor

/cc @andrewor14

@andrewor14

andrewor14 commented Apr 10, 2014

Copy link
Copy Markdown
Contributor

Good catch. The existing code fails when the logDir includes file://. This LGTM.

@andrewor14

andrewor14 commented Apr 10, 2014

Copy link
Copy Markdown
Contributor

Jenkins, test this please.

1 similar comment
@pwendell

pwendell commented Apr 10, 2014

Copy link
Copy Markdown
Contributor

Jenkins, test this please.

@AmplabJenkins

AmplabJenkins commented Apr 10, 2014

Copy link
Copy Markdown

Merged build triggered.

@AmplabJenkins

AmplabJenkins commented Apr 10, 2014

Copy link
Copy Markdown

Merged build started.

@AmplabJenkins

AmplabJenkins commented Apr 10, 2014

Copy link
Copy Markdown

Merged build finished. All automated tests passed.

@AmplabJenkins

AmplabJenkins commented Apr 10, 2014

Copy link
Copy Markdown

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14009/

@vanzin

vanzin commented Apr 11, 2014

Copy link
Copy Markdown
ContributorAuthor

FYI: just found another issue in this same code. It does not handle the case where the default fs is not "file:", and will try to use FileOutputStream even when it's something like hdfs. I'll fix that and push the new code.

@andrewor14

andrewor14 commented Apr 11, 2014

Copy link
Copy Markdown
Contributor

Not sure if I understand. Are you saying that if we somehow configure the default file scheme to be hdfs://, then it will still match the null in the first case statement, even if the path is really for HDFS? Currently the FileLogger requires you to explicitly specify the hdfs:// scheme as it does not have a notion of default file schemes (unlike in say sc.textFile), though this need not be the case.

@vanzin

vanzin commented Apr 11, 2014

Copy link
Copy Markdown
ContributorAuthor

Ok, maybe it's not a bug per se. But I think it somewhat breaks the user's expectations.

Let's say that given a user's configuration, the default fs is hdfs; I'd expect that if I specify the log directory to be "/user/vanzin/logs", that would be in HDFS, not in the local fs. But with the current FileLogger code, it will assume that is a local path, since there is no scheme defined.

Anyway, the current patch fixes the actual bug; if you think the above should be handled too I have the code to do it working, but it's not strictly necessary.

@vanzin

vanzin commented Apr 11, 2014

Copy link
Copy Markdown
ContributorAuthor

Commit that implements what I'm suggesting, if you're interested:
vanzin@15c4d1e

I haven't sent a pull request for that one.

@andrewor14

andrewor14 commented Apr 18, 2014

Copy link
Copy Markdown
Contributor

@vanzin I recently experienced what you mean on a YARN cluster myself. I've looked at your patch and I think it's a good fix for Spark on YARN. Could you submit a PR for it? It will be more convenient for me to leave my comments there.

@andrewor14

andrewor14 commented Apr 18, 2014

Copy link
Copy Markdown
Contributor

On a separate note, @pwendell this PR is ready for merge.

@vanzinvanzin mentioned this pull request Apr 19, 2014
Closed
@pwendell

pwendell commented Apr 22, 2014

Copy link
Copy Markdown
Contributor

Thanks, I've merged this.

asfgit pushed a commit that referenced this pull request Apr 22, 2014
Marcelo Vanzin@pwendell
[SPARK-1459] Use local path (and not complete URL) when opening local…
0a73103
… lo...
...g file.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes#375 from vanzin/event-file and squashes the following commits:
f673029 [Marcelo Vanzin] [SPARK-1459] Use local path (and not complete URL) when opening local log file.
(cherry picked from commit ac164b7)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
vanzin pushed a commit to vanzin/spark that referenced this pull request Apr 22, 2014
Marcelo Vanzin
Honor default fs name when initializing event logger.
592cdb3
This is related to SPARK-1459 / PR apache#375. Without this fix,
FileLogger.createLogDir() may try to create the log dir on
HDFS, while createWriter() will try to open the log file on
the local file system, leading to interesting errors and
confusion.
asfgit pushed a commit that referenced this pull request Apr 23, 2014
Marcelo Vanzin@pwendell
Honor default fs name when initializing event logger.
dd1b7a6
This is related to SPARK-1459 / PR #375. Without this fix,
FileLogger.createLogDir() may try to create the log dir on
HDFS, while createWriter() will try to open the log file on
the local file system, leading to interesting errors and
confusion.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes#450 from vanzin/event-file-2 and squashes the following commits:
592cdb3 [Marcelo Vanzin] Honor default fs name when initializing event logger.
asfgit pushed a commit that referenced this pull request Apr 23, 2014
Marcelo Vanzin@pwendell
Honor default fs name when initializing event logger.
46b30f9
This is related to SPARK-1459 / PR #375. Without this fix,
FileLogger.createLogDir() may try to create the log dir on
HDFS, while createWriter() will try to open the log file on
the local file system, leading to interesting errors and
confusion.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes#450 from vanzin/event-file-2 and squashes the following commits:
592cdb3 [Marcelo Vanzin] Honor default fs name when initializing event logger.
(cherry picked from commit dd1b7a6)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
@vanzin
vanzin deleted the event-file branch May 23, 2014 22:39
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
Marcelo Vanzin@pdeyhim
[SPARK-1459] Use local path (and not complete URL) when opening local…
348c016
… lo...
...g file.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closesapache#375 from vanzin/event-file and squashes the following commits:
f673029 [Marcelo Vanzin] [SPARK-1459] Use local path (and not complete URL) when opening local log file.
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
Marcelo Vanzin@pdeyhim
Honor default fs name when initializing event logger.
7af35f8
This is related to SPARK-1459 / PR apache#375. Without this fix,
FileLogger.createLogDir() may try to create the log dir on
HDFS, while createWriter() will try to open the log file on
the local file system, leading to interesting errors and
confusion.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closesapache#450 from vanzin/event-file-2 and squashes the following commits:
592cdb3 [Marcelo Vanzin] Honor default fs name when initializing event logger.
tangzhankun pushed a commit to tangzhankun/spark that referenced this pull request Jul 25, 2017
@ash211
Fix import order and scalastyle (apache#375)
db5f5be
Test with ./dev/scalastyle
erikerlandson pushed a commit to erikerlandson/spark that referenced this pull request Jul 28, 2017
@ash211@foxish
Fix import order and scalastyle (apache#375)
42f578f
Test with ./dev/scalastyle
mccheah pushed a commit to mccheah/spark that referenced this pull request Nov 28, 2018
Ruben Fiszel@robert3005
[SPARK-24345][SQL] Improve ParseError stop location when offending sy…
d5e10da
…mbol is a token (apache#375)
In the case where the offending symbol is a CommonToken, this PR increases the accuracy of the start and stop origin by leveraging the start and stop index information from CommonToken.
mccheah pushed a commit to mccheah/spark that referenced this pull request Nov 28, 2018
Ruben FiszelRobert Kruszewski
[SPARK-24345][SQL] Improve ParseError stop location when offending sy…
c3eb923
…mbol is a token (apache#375)
In the case where the offending symbol is a CommonToken, this PR increases the accuracy of the start and stop origin by leveraging the start and stop index information from CommonToken.
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
@liusheng@kiwik
Add terraform-provider-huaweicloud-acceptance-test-fusioncloud job (a…
feb0fc1
…pache#375)
This change add the job of
terraform-provider-huaweicloud-acceptance-test-fusioncloud
Closes-Bug: theopenlab/openlab#130
arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020
@mgorbov@ekrivokonmapr
* Bug 32263 - Seek called on unsubscribed partitions (apache#375)
3dbb6b2
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vanzin@AmplabJenkins@pwendell@andrewor14