[BACKPORT][SPARK-23040][CORE] Returns interruptible iterator for shuffle reader

[jiangxb1987]

Backport #20449 and #20920 to branch-2.3

---

What changes were proposed in this pull request?

Before this commit, a non-interruptible iterator is returned if aggregator or ordering is specified.
This commit also ensures that sorter is closed even when task is cancelled(killed) in the middle of sorting.

How was this patch tested?

Add a unit test in JobCancellationSuite

[SparkQA]

Test build #88777 has finished for PR 20954 at commit e9a661f.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

retest this please

[SparkQA]

Test build #88781 has finished for PR 20954 at commit e9a661f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to 2.3!

[dongjoon-hyun]

Hi, @cloud-fan and @jiangxb1987 .

It seems that the newly added test case starts to fail at sbt-hadoop2.7. Could you take a look at the failure?

branch-2.3

• https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/lastCompletedBuild/testReport/org.apache.spark/JobCancellationSuite/interruptible_iterator_of_shuffle_reader/

master

• https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4301/testReport/junit/org.apache.spark/JobCancellationSuite/interruptible_iterator_of_shuffle_reader/

[jiangxb1987]

Thanks for reporting, I'll be looking into the flaky test.