DurableTask.AzureStorage: CreateTimer fails to complete if called after a continue as new

[sebastianburckhardt]

On DurableTask.AzureStorage, the following orchestration hangs; stepping through the debugger reveals that the timer does not fire.

 public override async Task<string> RunTask(OrchestrationContext context, int input)
 {
 if (input == 0)
 {
 context.ContinueAsNew(1);

 return "continue as new";
 }
 else if (input == 1)
 {
 await context.CreateTimer(context.CurrentUtcDateTime, 0);

 return "OK";
 }

[sebastianburckhardt]

The root cause seems to be that the execution Id is not updated in the orchestration session after a continue as new - therefore, the timer message appears to target a non-existing instance and is thrown away.

[cgillum]

Yes, I anticipated exactly this. For external events we exclude the execution ID so that it can easily be processed for any execution of the orchestration. For timers, the problem is a bit more complicated because there are legitimate cases for discarding the timer message across executions. For example, if someone creates a new orchestration with the same instance ID as a previously completed instance, you don't necessarily want to process abandoned timer messages that were scheduled by that previous instance.

[sebastianburckhardt]

I understand that it makes sense to discard timer events originating from a different execution.

But what I am seeing here is that the timer event from the same execution is discarded because it is compared against the wrong value. The continueAsNew did not update the execution ID for the in-memory OrchestrationInstance object.

[cgillum]

I think this might be caused by the fast-ContinueAsNew change in DurableTask.Core 2.1.0. I believe it may also be causing issues with calling activity functions immediately after ContinueAsNew (some discussion of this here: Azure/azure-functions-durable-extension#529).

[sebastianburckhardt]

I think I understand the cause for the bug, it is indeed related to the fast ContinueAsNew introduced in #251.

What happens is that the loop inside OnProcessWorkItemAsync goes into the second execution of the instance, which has a different execution id, but the execution id in OrchestrationSession is not updated. When the timer fires, the OrchestrationSessionManager executes AddMessageToPendingOrchestration but because the execution id does not match, the message is put back into the queue (with the expectation that the session will disappear and then a new instance would be created by OrchestrationSessionManager.GetNextSessionAsync to handle the message batch). However, this never happens with extended Sessions and fast continueAsNew enabled, so the message is never actually delivered.

I am working on a fix that can update the execution id in OrchestrationSession and deliver the messages to it.