Fixed the number of worker thread

[fireflyc]

There are a lot of input Block cause too many Worker threads and will
load all data.So it should be to control the number of Worker threads

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Slightly bigger point: both the 'fixed' and 'cached' executors from Executors have some drawbacks:

• 'fixed' always keeps the given number of threads active even if they're not doing anything
• 'cached' may create an unlimited number of threads

It's perfectly possible to create a ThreadPoolExecutor with core size 0 and a fixed maximum size. I wonder if that isn't the best choice here, and actually, in other usages I see throughout Spark? Because a similar issue comes up in about 10 places.

[fireflyc]

Will always have a task to run, when the system is idle all threads will not participate CPU. When need run task need not new a thread.

fixed is great way.

[aarondav]

The tasks launched on an Executor are controlled by the DAGScheduler, and should not exceed the number of cores that executor is advertising. In what situation have you seen this happening?

[fireflyc]

My application have 1000+ Worker Threads.

[aarondav]

Does your patch fix this problem, or do your Executors just hang after you reach enough cores? This behavior should not be happening, even with an unlimited capacity cached tread pool.

[fireflyc]

My program is spark streaming over Hadoop yarn.It work for user click stream.
I read code,number of worker threads and block?
@aarondav

[srowen]

@fireflyc again just on my tangent -- the drawback is you leave N threads allocated taking up non-trivial stack memory and so on. In most of the cases I see the overhead of starting new threads on demand isn't significant. If what you describe is happening, then fixed is certainly an improvement over cached.

[aarondav]

@fireflyc Spark should not be scheduling more than N concurrent tasks on an Executor. It appears that the tasks may be returning "success" but then don't actually return the thread to the thread pool.

This is itself a bug -- could you run "jstack" on your Executor process to see where the threads are stuck?

Perhaps new tasks are just starting before the old threads finish cleaning up, and thus this solution is the right one, but I'd like to find out exactly why.

[pwendell]

Hey there - as Aaron said, the executors should never have more than N tasks active if there are N cores. I think there might be a bug causing this. So I'd recommend we close this issue and open a JIRA to figure out what is going on.