Skip to content

Reload credentials file config when app starts with checkpoint file i… - #17937

Closed
jiasheng55 wants to merge 1 commit into
apache:masterfrom
jiasheng55:fix-credential-file
Closed

Reload credentials file config when app starts with checkpoint file i…#17937
jiasheng55 wants to merge 1 commit into
apache:masterfrom
jiasheng55:fix-credential-file

Conversation

@jiasheng55

Copy link
Copy Markdown

What changes were proposed in this pull request?

Currently credentials file configuration is recovered from checkpoint file when Spark Streaming applicatioin is restarted, which will lead to some unwanted behaviors, for example:

  1. Submit Spark Streaming application using keytab file with checkpoint enabled in yarn-cluster mode.

spark-submit --master yarn-cluster --principal xxxx --keytab xxx ...

  1. Stop Spark Streaming application;
  2. Resubmit this application after a period of time (i.e. one day);
  3. Credentials file configuration recover from checkpoint file, so value of "spark.yarn.credentials.file" points to old staging directory (i.e. hdfs://xxxx/.sparkStaging/application_xxxx/credentials-xxxx, application_xxxx is the application id of the previous application which was stopped.)
  4. When launching executor, ExecutorDelegationTokenUpdater will update credentials from credentials file immediately. As credentials file was generated one day ago (maybe older), it has already expired, so after a period of time the executor keeps failing.

Some useful logs are shown below :

2017-04-27,15:08:08,098 INFO org.apache.spark.executor.CoarseGrainedExecutorBackend: Will periodically update credentials from: hdfs://xxxx/application_xxxx/credentials-xxxx
2017-04-27,15:08:12,519 INFO org.apache.spark.deploy.yarn.ExecutorDelegationTokenUpdater: Reading new delegation tokens from hdfs://xxxx/application_1xxxx/credentials-xxxx-xx
2017-04-27,15:08:12,661 INFO org.apache.spark.deploy.yarn.ExecutorDelegationTokenUpdater: Tokens updated from credentials file.
...
2017-04-27,15:08:48,156 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token xxxx for xx) can't be found in cache

How was this patch tested?

manual tests

@AmplabJenkins

Copy link
Copy Markdown

Can one of the admins verify this patch?

@jiasheng55

Copy link
Copy Markdown
Author

Comments on last PR, #17782.

@jerryshao

Copy link
Copy Markdown
Contributor

@Victor-Wong can you please update the PR title like other PRs?

By seeing your description, seems the log is from old Spark version, in the latest Spark there's no ExecutorDelegationTokenUpdater and it has renamed to CredentialUpdater, also CredentialUpdater will not update the credential immediately at start, it is controlled by spark.yarn.credentials.updateTime. Can you please check if your problem still exists in latest master code, also what exception will be met?

Also I would guess some more internal configurations should be excluded from checkpoint, like "spark.yarn.credentials.renewalTime", "spark.yarn.credentials.updateTime".

@jerryshao

Copy link
Copy Markdown
Contributor

Besides I guess this issue only exists in yarn cluster mode, can you also verify it?

@HyukjinKwon

Copy link
Copy Markdown
Member

ping @Victor-Wong, how it is going?

@HyukjinKwonHyukjinKwon mentioned this pull request Jun 25, 2017
@gatorsmile

Copy link
Copy Markdown
Member

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!

@jerryshao

Copy link
Copy Markdown
Contributor

This is already fix in #18230 CC @gatorsmile .

zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
## What changes were proposed in this pull request?
This PR proposes to close stale PRs, mostly the same instances with apache#18017
I believe the author in apache#14807 removed his account.
Closesapache#7075Closesapache#8927Closesapache#9202Closesapache#9366Closesapache#10861Closesapache#11420Closesapache#12356Closesapache#13028Closesapache#13506Closesapache#14191Closesapache#14198Closesapache#14330Closesapache#14807Closesapache#15839Closesapache#16225Closesapache#16685Closesapache#16692Closesapache#16995Closesapache#17181Closesapache#17211Closesapache#17235Closesapache#17237Closesapache#17248Closesapache#17341Closesapache#17708Closesapache#17716Closesapache#17721Closesapache#17937
Added:
Closesapache#14739Closesapache#17139Closesapache#17445Closesapache#18042Closesapache#18359
Added:
Closesapache#16450Closesapache#16525Closesapache#17738
Added:
Closesapache#16458Closesapache#16508Closesapache#17714
Added:
Closesapache#17830Closesapache#14742
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#18417 from HyukjinKwon/close-stale-pr.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jiasheng55@AmplabJenkins@jerryshao@HyukjinKwon@gatorsmile