[SPARK-6202] [SQL] enable variable substitution on test framework

[adrian-wang]

No description provided.

[SparkQA]

Test build #28333 has started for PR 4930 at commit cf13ad3.

• This patch merges cleanly.

[SparkQA]

Test build #28333 has finished for PR 4930 at commit cf13ad3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected[hive] class SubstitutedHiveQLQueryExecution(hql: String)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28333/
Test PASSed.

[chenghao-intel]

LGTM

[yhuai]

Can you explain the reason that you need to do the change when we already have substitution in HiveContext.sql?

[yhuai]

Also, add a comment in the code to explain the reason will be great.

[adrian-wang]

we should substitute variables in hql to pass the text to parseSql() as a parameter. Hive parser need substituted text. HiveContext.sql() does this but return a DataFrame, while we need a logicalPlan so we cannot reuse that.
Added as comment.

[SparkQA]

Test build #28775 has started for PR 4930 at commit acecf6a.

• This patch merges cleanly.

[yhuai]

Can you just change this line and do what we have in sql (new VariableSubstitution().substitute(hiveconf, hql))?

[adrian-wang]

but runSqlHive() also require a substituted text for use.

[SparkQA]

Test build #28775 has finished for PR 4930 at commit acecf6a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected[hive] class SubstitutedHiveQLQueryExecution(hql: String)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28775/
Test PASSed.

[adrian-wang]

@yhuairunSqlHive calls Driver.run directly without do substitution, so we have to substitute for this and QueryExecution both.

[yhuai]

Seems eventually Driver.run will call Driver.compile, which will do the substitution?

[chenghao-intel]

@adrian-wang how about adding the substitution in HiveContext.runSqlHive or HiveContext.runHive? Then we probably not necessary to change anything in TestHive.

[SparkQA]

Test build #28905 has started for PR 4930 at commit d202e13.

• This patch merges cleanly.

[SparkQA]

Test build #28905 has finished for PR 4930 at commit d202e13.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28905/
Test PASSed.

[adrian-wang]

ping @yhuai

[SparkQA]

Test build #28981 has started for PR 4930 at commit b1d68bf.

• This patch merges cleanly.

[SparkQA]

Test build #28982 has started for PR 4930 at commit 2ce590f.

• This patch merges cleanly.

[SparkQA]

Test build #28981 has finished for PR 4930 at commit b1d68bf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28981/
Test PASSed.

[SparkQA]

Test build #28982 has finished for PR 4930 at commit 2ce590f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28982/
Test PASSed.

[yhuai]

LGTM. Is there any Hive query test that we should enable?

[adrian-wang]

There is one of "mapjoin_addjar", and I'll add that after I refactor #4586 . So I'd like this one merge first, thanks a lot!

[marmbrus]

Thanks! Merged to master.