Uh oh!
There was an error while loading. Please reload this page.
Enhance ShutdownWorkerRequest and poll calls with worker_instance_key - #686
Merged
Merged
Conversation
cretz
reviewed
Dec 9, 2025
Uh oh!
There was an error while loading. Please reload this page.
yuandrew
marked this pull request as draft
December 10, 2025 17:39
yuandrew
marked this pull request as ready for review
December 10, 2025 17:46
cretz
reviewed
Dec 10, 2025
Uh oh!
There was an error while loading. Please reload this page.
cretz
approved these changes
Jan 23, 2026
rkannan82
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Do we need to add this to PollNexusTaskQueueRequest as well? Not specifically for the server initiated shutdown.
dnr
approved these changes
Jan 23, 2026
dnr
left a comment
Contributor
There was a problem hiding this comment.
update PR description to remove/rewrite
(This also requires all poll requests to contain the Worker Instance Key, can be added in a future PR)
since it's here now
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
5 tasks
rkannan82 added a commit
to temporalio/temporal
that referenced
this pull request
Feb 6, 2026
) ## What changed? - When workflow or activity poll requests are forwarded from one partition to another, the WorkerInstanceKey field was not being copied to the forwarded request. This caused the worker's instance key to be lost. - WorkerInstanceKey is a new field added to all poll requests. See [PR](temporalio/api#686). ## Why? We will be recording the WorkerInstanceKey in the ActivityInfo as part of task matching. This will be used later to send activity cancellation tasks to that worker. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks
stephanos pushed a commit
that referenced
this pull request
Mar 10, 2026
…#686) **What changed?** - Added a new `worker_instance_key` field to `ShutdownWorkerRequest`, as well as all other poll calls. - Added comment to `ShutdownWorkerRequest.sticky_task_queue` saying it may be blank, now that we've expanded the scope of when `ShutdownWorkerRequest` is called. - Added `task_queue` and `task_queue_kind` to `ShutdownWorkerRequest` **Why?** ShutdownWorker was changed to always be sent by SDK (temporalio/sdk-rust#1082), so sticky queue name is now optional. This plus the new heartbeat info we send on shutdown means Server will now have a more accurate map of which workers are shutting down. Adding task queue and task_queue_kind should also allow us to fix a lost task issue, where there is a race when the SDK cancels an outstanding poll rpc and the server decides to send a task to that poller. Technically some of this info exists in the worker heartbeat part of the message, but it needs to be lifted to its own field due to the scenario where worker heartbeating is disabled. **Breaking changes** N/A I think, just adding new fields **Server PR**
rkannan82 added a commit
that referenced
this pull request
Apr 6, 2026
…711) <!-- Describe what has changed in this PR --> **What changed?** Added worker_control_task_queue to poll requests: - PollActivityTaskQueueRequest - PollWorkflowTaskQueueRequest Note: worker_instance_key was already added to these requests in #686. Added worker_instance_key and worker_control_task_queue to: - RespondWorkflowTaskCompletedRequest: This API is used to eagerly fetch activity. <!-- Tell your future self why have you made these changes --> **Why?** To enable server to send control tasks to worker. Each worker provides a worker_control_task_queue (a dedicated per-worker Nexus task queue) so the server can send control tasks directly to it. Example flow: - User cancels a workflow. - Server sends activity cancellation tasks to all workers that could be processing activities belonging to that workflow. - Worker will receive the cancellation message even when activity heartbeat is not enabled. <!-- Are there any breaking changes on binary or code level? --> Breaking changes: None [Server PR](temporalio/temporal#9231) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
spkane31 pushed a commit
that referenced
this pull request
Apr 9, 2026
…711) <!-- Describe what has changed in this PR --> **What changed?** Added worker_control_task_queue to poll requests: - PollActivityTaskQueueRequest - PollWorkflowTaskQueueRequest Note: worker_instance_key was already added to these requests in #686. Added worker_instance_key and worker_control_task_queue to: - RespondWorkflowTaskCompletedRequest: This API is used to eagerly fetch activity. <!-- Tell your future self why have you made these changes --> **Why?** To enable server to send control tasks to worker. Each worker provides a worker_control_task_queue (a dedicated per-worker Nexus task queue) so the server can send control tasks directly to it. Example flow: - User cancels a workflow. - Server sends activity cancellation tasks to all workers that could be processing activities belonging to that workflow. - Worker will receive the cancellation message even when activity heartbeat is not enabled. <!-- Are there any breaking changes on binary or code level? --> Breaking changes: None [Server PR](temporalio/temporal#9231) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
worker_instance_keyfield toShutdownWorkerRequest, as well as all other poll calls.ShutdownWorkerRequest.sticky_task_queuesaying it may be blank, now that we've expanded the scope of whenShutdownWorkerRequestis called.task_queueandtask_queue_kindtoShutdownWorkerRequestWhy?
ShutdownWorker was changed to always be sent by SDK (temporalio/sdk-rust#1082), so sticky queue name is now optional. This plus the new heartbeat info we send on shutdown means Server will now have a more accurate map of which workers are shutting down.
Adding task queue and task_queue_kind should also allow us to fix a lost task issue, where there is a race when the SDK cancels an outstanding poll rpc and the server decides to send a task to that poller.
Technically some of this info exists in the worker heartbeat part of the message, but it needs to be lifted to its own field due to the scenario where worker heartbeating is disabled.
Breaking changes
N/A I think, just adding new fields
Server PR