[SPARK-2675] Increase EVENT_QUEUE_CAPACITY by 20x.

[concretevitamin]

JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2675

@pwendell@andrewor14

[concretevitamin]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 1579. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17131/consoleFull

[shivaram]

Is there an estimate on the size of an event ? How much memory would 200000 events take ?

[SparkQA]

QA results for PR 1579:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17131/consoleFull

[lianhuiwang]

i want to know how much memory that @shivaram said before. @concretevitamin can show the number?thanks.

[concretevitamin]

Unfortunately I may not be able to find time to run an experiment soon. If anyone is interested, I think org.apache.spark.util.SizeEstimator is reasonable to try. Let's also see if SPARK-2316 gets fixed in the near future, which may remove the need to increase the queue capacity.

[andrewor14]

However many others also face this issue (evident from the mailing list). We should aim to fix this issue one way or another, either through this PR and/or the fix for SPARK-2316, by the 1.1 release. SizeEstimator is a good place to start.

[pwendell]

@concretevitamin would you mind closing this? The underlying issue has been fixed in #1679?