[SPARK-3658][SQL]Take thrift server as a daemon

[WangTaoTheTonic]

https://issues.apache.org/jira/browse/SPARK-3658

And keep the CLASS_NOT_FOUND_EXIT_STATUS and exit message in SparkSubmit.scala.

[SparkQA]

QA tests have started for PR 2509 at commit 598e21e.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2509 at commit 598e21e.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • println(s"Failed to load main class $childMainClass.")
  • logInfo("Interrupting user class to stop.")

[SparkQA]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20710/

[pwendell]

@liancheng - can you take a look at this one?

[liancheng]

I feel complicated to add this logic here... SparkSubmit should not be coupled with Thrift server, but it seems that this is the only legitimate place.

[liancheng]

After some thoughts, I think it's OK to add this checking here. Mainly because it avoids using the exit code checking trick, which may cause potential exit code conflict.

[liancheng]

Generally this is a good idea. But it would be better to make spark-daemon.sh more general, rather than making HiveThriftServer2 a special case.

[SparkQA]

QA tests have started for PR 2509 at commit 8ad9f95.

• This patch merges cleanly.

[WangTaoTheTonic]

@liancheng As you said, I put "spark-submit" as an option to achieve generalization and use source(dot) instead of exec to make SUBMISSION_OPTS and APPLICATION_OPTS still work in spark-daemon.sh.

As for the ClassNotFound error message, we could either do like this or just delete it?

please check.thanks

[SparkQA]

QA tests have finished for PR 2509 at commit 8ad9f95.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • println(s"Failed to load main class $childMainClass.")

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20807/

[liancheng]

This file should be executable, please chmod +x.

[liancheng]

Thanks for working on this! I tested this PR locally and it works fine. But there are still some minor issues pending to be resolved, please refer to the comments for details.

[WangTaoTheTonic]

@liancheng I tried export and it worked. Thanks for the suggestion.
Also modified permission of stop-thriftserver.sh.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20937/

[SparkQA]

QA tests have started for PR 2509 at commit 5dcaab2.

• This patch merges cleanly.

[liancheng]

After updating my local repo, I found that stop-thriftserver.sh is still not executable. Make sure to git add this file after chmod +x. This is the only pending issue from my perspective.

[SparkQA]

Tests timed out after a configured wait of 120m.

[WangTaoTheTonic]

eh...I cloned the repository on another laptop and found it's executable, as shown in top-left corner of https://github.com/WangTaoTheTonic/spark/blob/thriftserver/sbin/stop-thriftserver.sh.

Could you verify this again?

[liancheng]

Ah, sorry, my fault. Then this LGTM, thanks!

[pwendell]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 2509 at commit 5dcaab2.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2509 at commit 5dcaab2.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • println(s"Failed to load main class $childMainClass.")

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20958/

[marmbrus]

Thanks! I've merged this to master.