SPARK-5529 Backported changes to 1.3

[alexrovner]

Sorry about all the previous pull requests. Still running tests on this branch. Mechanically applied the changes based on #4369 without fully understanding whats actually happening since I am not familiar with the codebase. Feedback would be appreciated.

[AmplabJenkins]

Can one of the admins verify this patch?

[alexrovner]

Locally the tests are taking for ever to complete. Seems like everything is working as intended so far.

[srowen]

#4363 right? That looks pretty close, but it's not from a cherry-pick. For example it misses the docs change, and the indentation is wrong. Usually you back-port by starting from a cherry-pick of the original commit into the branch and massage it from there. Can you have a run at it that way? it will also preserve more of the connection to the original and avoid making the branches diverge more than needed, even if in small ways.

[alexrovner]

I started out with a cherry-pick originally, however it did not apply cleanly for some reason. Seemed to work this time around. Here is the new pull: #5793