Skip to content

[SPARK-8605] Exclude files in StreamingContext. textFileStream - #20683

Closed
ConcurrencyPractitioner wants to merge 2 commits into
apache:masterfrom
ConcurrencyPractitioner:master
Closed

[SPARK-8605] Exclude files in StreamingContext. textFileStream#20683
ConcurrencyPractitioner wants to merge 2 commits into
apache:masterfrom
ConcurrencyPractitioner:master

Conversation

@ConcurrencyPractitioner

@ConcurrencyPractitionerConcurrencyPractitioner commented Feb 27, 2018•
edited
Loading

Copy link
Copy Markdown

What changes were proposed in this pull request?

In this PR, a extra boolean expression was added to test if a regex was present. If returned true, then we exclude the file.

How was this patch tested?

No tests were added.

@ConcurrencyPractitioner

ConcurrencyPractitioner commented Feb 27, 2018

Copy link
Copy Markdown
Author

Jenkins test this please

@AmplabJenkins

AmplabJenkins commented Feb 27, 2018

Copy link
Copy Markdown

Can one of the admins verify this patch?

@ConcurrencyPractitionerConcurrencyPractitioner changed the title [SPARK-8605] Exclude files in StreamingContext. textFileStream(direct…[SPARK-8605] Exclude files in StreamingContext. textFileStreamFeb 27, 2018
@jerryshao

jerryshao commented Feb 28, 2018

Copy link
Copy Markdown
Contributor

a extra boolean expression was added to test if a regex was present.

Can you please explain what's the meaning of "if a regex was present"?

Seems the fix is not so necessary. If you want to filter out some temp files, you can write your own filter instead of using Spark Streaming's default one.

@ConcurrencyPractitioner

ConcurrencyPractitioner commented Mar 11, 2018

Copy link
Copy Markdown
Author

@jerryshao In Spark Streaming, I think .tmp is used as a suffix to indicate that the object was a file, although I do not know if this is universal.

@gaborgsomogyi

gaborgsomogyi commented Mar 16, 2018

Copy link
Copy Markdown
Contributor

Don't really understand the issue itself. Which filesystem used this case? Why is it not possible to use Hadoop-compatible filesystem like HDFS for instance? This supports atomic rename. See here

@srowensrowen mentioned this pull request May 11, 2018
Closed
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@srowen@zifeif2
[BUILD] Close stale PRs
0d6c93f
Closesapache#20458Closesapache#20530Closesapache#20557Closesapache#20966Closesapache#20857Closesapache#19694Closesapache#18227Closesapache#20683Closesapache#20881Closesapache#20347Closesapache#20825Closesapache#20078Closesapache#21281Closesapache#19951Closesapache#20905Closesapache#20635
Author: Sean Owen <srowen@gmail.com>
Closesapache#21303 from srowen/ClosePRs.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ConcurrencyPractitioner@AmplabJenkins@jerryshao@gaborgsomogyi