[Bug] Spark Driver cannot be shutdown at the end of a task

[kangzhilv]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

when i use k8s deploy spark task, the task driver pod is not compact.

Affects Version(s)

1.9.3

Kyuubi Server Log Output

Kyuubi Engine Log Output

25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is initialized.
25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is started.
25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:08 ERROR SparkTBinaryFrontendService: Error fetching results: 
25/04/09 03:00:08 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [09f53cfd-2220-4e1e-8dd7-6b61dc8b2f52]
25/04/09 03:00:08 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [09f53cfd-2220-4e1e-8dd7-6b61dc8b2f52]
25/04/09 03:00:10 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [0b434ecc-111a-44f5-b9d5-9ad29f67ff83]
25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [0b434ecc-111a-44f5-b9d5-9ad29f67ff83]
25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [ab9e6ae9-0fef-4210-b432-286a9bf4885e]
25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [ab9e6ae9-0fef-4210-b432-286a9bf4885e]
25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [84a5120e-4238-4a8b-9b4b-20720df1eabc]
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [84a5120e-4238-4a8b-9b4b-20720df1eabc]
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [773ad9ba-687c-4004-a64a-0b0b7ef5089f]
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [773ad9ba-687c-4004-a64a-0b0b7ef5089f]
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:04:16 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
25/04/09 03:04:16 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
25/04/09 03:04:16 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
25/04/09 03:04:16 INFO SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
25/04/09 03:08:19 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [7041cd34-ca41-4593-bbad-7979c9a071da]
25/04/09 03:08:19 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [7041cd34-ca41-4593-bbad-7979c9a071da]

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @kangzhilv,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[kangzhilv]

net information
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 172.26.13.100:52060 172.26.255.198:9083 ESTABLISHED -
tcp6 0 0 172.26.13.100:34905 172.26.255.175:8216 ESTABLISHED -

[yaooqinn]

Can you provide the jstack info to reveal if there is any daemon thread that prevents JVM from exiting?

[pan3793]

Unlike Spark on YARN, in Spark on K8s cluster mode, daemon threads would block your driver JVM exit, see technical details at apache/spark#46889