[SPARK-3586][streaming]Support nested directories in Spark Streaming

[zsxwing]

Closes #2765

[SparkQA]

Test build #33990 has finished for PR 6588 at commit 2d85159.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36271 has finished for PR 6588 at commit d7f42c2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[GordonWang]

@zsxwing Do you have any further plan about this PR ? When is it going to be merged?

Thanks.

[zsxwing]

@zsxwing Do you have any further plan about this PR ? When is it going to be merged?

Thanks.

@tdas will look at this one soon. I think this one should be able to deliver in 1.5.0.

[huitseeker]

This (and the other recursive call on l.203 above) may blow up the call stack in very deep hierarchies (think Maildir++, etc). I realise there's a depth limit set, but users may precisely want to set it extremely high for some of those cases.

How about making the exploration using a @tailrec auxiliary function, with a queue of yet-to-be-explored directories in arguments ?

[andrewor14]

@tdas @zsxwing any updates on this? Is this targeted for 1.6 now?

[gprivitera]

@tdas : any update on this PR? Is there anything that should be done yet?

[gprivitera]

@zsxwing : your solution is not going to work on S3, since org.apache.hadoop.fs.FileStatus.getModificationTime() always returns 0L on directories in S3.

[tdas]

Good point!

[srowen]

@zsxwing is this stalled now? It doesn't look like this will proceed.

[zsxwing]

Let me just close it. Maybe visit later.

[stshruthi]

Is there a plan to support nested directory streaming with latest versions of spark?