[SPARK-37060][CORE] Handle driver status response from backup masters

[testsgmr]

What changes were proposed in this pull request?

After an improvement in SPARK-31486, contributor uses 'asyncSendToMasterAndForwardReply' method instead of 'activeMasterEndpoint.askSync' to get the status of driver. Since the driver's status is only available in active master and the 'asyncSendToMasterAndForwardReply' method iterate over all of the masters, we have to handle the response from the backup masters in the client, which the developer did not consider in the SPARK-31486 change. So drivers running in cluster mode and on a cluster with multi masters affected by this bug.

Why are the changes needed?

We need to find if the response received from a backup master client must ignore it.

Does this PR introduce any user-facing change?

No, It's only fixed a bug and brings back the ability to deploy in cluster mode on multi-master clusters.

How was this patch tested?

[AmplabJenkins]

Can one of the admins verify this patch?

[Ngone51]

cc @akshatb1@srowen who involved in the previous PR: #28258

[srowen]

I don't know enough to review this. Can you CC the author of the change before this?

[testsgmr]

Could someone please review these changes? CC: @cloud-fan@Ngone51@HeartSaVioR@jiangxb1987

[Ngone51]

@mohamadrezarostami you may need to rebase your branch to pass GA.

[testsgmr]

@mohamadrezarostami you may need to rebase your branch to pass GA.

Done!

[Ngone51]

Thanks, merged to master.

@mohamadrezarostami Could you create PRs to backport this to branch-3.2/branch-3.1?

[testsgmr]

@Ngone51
Thanks for review.
Yes, I will create PRs for branch-3.2 and branch-3.1, too.