[SPARK-1021] Defer the data-driven computation of partition bounds in so...

[erikerlandson]

...rtByKey() until evaluation.

[erikerlandson]

Reboot of #1689

[SparkQA]

Test build #22828 has started for PR 3079 at commit 019ac27.

• This patch merges cleanly.

[erikerlandson]

@marmbrus, @scwf, FWIW, the correlationoptimizer14 test appears to be working for me. I ran it using: env _RUN_SQL_TESTS=true _SQL_TESTS_ONLY=true ./dev/run-tests > ~/run-tests-1021.txt 2>&1

Not sure why, but running sbt -Dspark.hive.whitelist=correlationoptimizer14 hive/test was not causing the test to run in my environment

[marmbrus]

@erikerlandson I think you also need -Phive for the tests to run. It is possible some other things changed (or even that that test case changed with the upgrade to hive 13). Perhaps you can include a dummy change to the Hive code to make sure those tests are run in this PR so we can see what Jenkins thinks?

[SparkQA]

Test build #22828 has finished for PR 3079 at commit 019ac27.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22828/
Test FAILed.

[SparkQA]

Test build #22880 has started for PR 3079 at commit 0fc30fe.

• This patch merges cleanly.

[SparkQA]

Test build #22880 has finished for PR 3079 at commit 0fc30fe.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22880/
Test FAILed.

[SparkQA]

Test build #22892 has started for PR 3079 at commit 2183325.

• This patch merges cleanly.

[SparkQA]

Test build #22892 has finished for PR 3079 at commit 2183325.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22892/
Test FAILed.

[squito]

valRD is a kinda confusing name. I think the convention would be to name it _rangeBounds. Eg.

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/FutureAction.scala#L111

[erikerlandson]

For reference, this other issue has some overlap:
https://issues.apache.org/jira/browse/SPARK-4514

[marmbrus]

hi @erikerlandson, thanks for working on this. It would be great to have a solution to this long running problem. Since it looks like there is still some work to be done, I propose we close this issue for now. We are on a renewed mission to keep the PR queue small and limited to active PRs so that things don't fall through the cracks.

[marmbrus]

Do please reopen though once you having something that is passing tests :)