[SPARK-2755] More general Storage Interface for Shuffle / Spill etc

[colorant]

Hi, this is for https://issues.apache.org/jira/browse/SPARK-2275

The code here is not intended to be merged as current status, instead, I just try to show what I think this change could be.

so I put up this PR as a quick way to verify the idea and see how much things need to be modified. It definitely need to be improved or even restructured.

And this is for solving the problem 1 in the jira ticket, since problem 2 rely on problem 1, So I want to use this PR to present my general ideas and to find out what do you think about this whole thing. Thanks.

[colorant]

Hi @andrewor14 , it seems to me that you are work on some big change related to BlockManager, could you take a look on this one?

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16110/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16111/

[andrewor14]

@colorant Can you up merge this to master? Is this still relevant now that we have FileShuffleBlockManager and IndexShuffleBlockManager?

[colorant]

@andrewor14, this is not quite impact by FileShuffleBlockManager and IndexShuffleBlockManager. While might be irrelevant due to @rxin 's recent ManangedBuffer PR