SPARK-17424: Fix unsound substitution bug in ScalaReflection.

[rdblue]

What changes were proposed in this pull request?

This method gets a type's primary constructor and fills in type parameters with concrete types. For example, MapPartitions[T, U] -> MapPartitions[Int, String]. This Substitution fails when the actual type args are empty because they are still unknown. Instead, when there are no resolved types to subsitute, this returns the original args with unresolved type parameters.

How was this patch tested?

This doesn't affect substitutions where the type args are determined. This fixes our case where the actual type args are empty and our job runs successfully.

[SparkQA]

Test build #65269 has finished for PR 15062 at commit 931f156.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

Could you add a regression test?

[rdblue]

@hvanhovell, I'm not sure what's causing this to build the test. I can step through with a debugger to see that the problem is caused by an empty actualTypeArgs list, but I don't know why it is that way. Maybe someone with more knowledge of Scala types can help fill in the details. Until then, I'm not sure how to construct a test case for this.

[jodersky]

I don't quite understand what use-case this patch fixes. Can you provide an example (in the form of a test) that reproduces the issue?

[jodersky]

Oh, I was too quick to comment, I see a complete description is in the JIRA.
It would still be good if you could add a test though.

[rdblue]

@jodersky, as I said above, I'm not really sure how to build a test for this because I'm not too familiar with the Scala internals that are misused here. However, since I can confirm that it works in practice, I think it's reasonable to commit it anyway, before committers that know this code well can have a look to make sure this method is tested properly.

Another way to go is to revert PR #10970. That is a work-around because another test was already hitting this problem and the solution was to avoid the parameter substitution.

[srowen]

Jenkins retest this please

[jodersky]

I'm just unclear of why this is needed . According to the linked issue, some tests were failing and this fixed them. Everything just seems very vague to me. It would be very helpful if you could provide an example (it doesn't have to be a unit-test) that reproduces the error.
It would greatly help me understand what's going on and also distill things down to a regression test.

[rdblue]

I think the best way to reproduce this is to revert #10970. I don't have a case that reproduces it that I can share since this was in a fairly large job.

[SparkQA]

Test build #66190 has finished for PR 15062 at commit 931f156.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

Hi @rdblue can you revert #10970 in this PR? Then other reviewers can test this PR locally and verify if it works.

[rdblue]

I tried to reproduce with #10970 reverted, but I didn't hit the issue in testing. I still think it's fine to move forward on this, even if it is hard to reproduce because we know the code is wrong and this fixes it.

[cloud-fan]

retest this please

[cloud-fan]

I'm fine with this change, cc @yhuai@liancheng

[SparkQA]

Test build #68939 has finished for PR 15062 at commit 931f156.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

any chance that is the same issue as https://issues.apache.org/jira/browse/SPARK-17109?
@rdblue When you were debugging this issue, which version of scala did you use? Scala 2.10 or Scala 2.11? If you were using scala 2.10, is it possible to try scala 2.11? Thanks!

[rdblue]

We were seeing the problem when using the datasets API in our 1.6.1 build, which is based on Scala 2.10. I recently tried to reproduce this on master with 2.11 and #10970 reverted, but I didn't get a case that failed. Either way, I think the fix here makes sense: if there are no types to substitute, don't do it.

[rdblue]

This does look the same as SPARK-17109. Does this fix that issue?

[cloud-fan]

retest this please

[SparkQA]

Test build #76863 has finished for PR 15062 at commit 931f156.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

the change looks reasonable, merging to master/2.2/2.1/2.0!