IndexOutOfRangeException at FixedSizeDictionary`3.AddOrReplace

[phil000]

Getting this exception periodically in Handlebars, Version=2.1.0.0 trying to render content from a compiled delagate (HandlebarsTemplate<object, object>)

Not sure what else I can provide because it doesn't seem related to the content being rendered. Debugging though it always works.

We were previously using 2.0.9 on .NET 5. We upgraded our project to .NET 6.0 and Handlebars 2.1.0 and the problem apparently started.

System.IndexOutOfRangeException:
 at HandlebarsDotNet.Collections.FixedSizeDictionary`3.AddOrReplace (Handlebars, Version=2.1.0.0, Culture=neutral, PublicKeyToken=22225d0bf33cd661)
 at HandlebarsDotNet.BindingContext.Initialize (Handlebars, Version=2.1.0.0, Culture=neutral, PublicKeyToken=22225d0bf33cd661)
 at HandlebarsDotNet.BindingContext+BindingContextPool.CreateContext (Handlebars, Version=2.1.0.0, Culture=neutral, PublicKeyToken=22225d0bf33cd661)
 at HandlebarsDotNet.BindingContext.Create (Handlebars, Version=2.1.0.0, Culture=neutral, PublicKeyToken=22225d0bf33cd661)
 at HandlebarsDotNet.HandlebarsEnvironment+<>c__DisplayClass15_0.<Compile>b__0 (Handlebars, Version=2.1.0.0, Culture=neutral, PublicKeyToken=22225d0bf33cd661)
 at HandlebarsDotNet.HandlebarsEnvironment+<>c__DisplayClass16_0.<Compile>b__0 (Handlebars, Version=2.1.0.0, Culture=neutral, PublicKeyToken=22225d0bf33cd661)
 at OnSend.MessagingService.Core.Templating.HandlebarsMessageBuilder.BuildMessage (OnSend.MessagingService.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\OnSend.MessagingService.Core\Templating\HandlebarsMessageBuilder.cs:68)

[phil000]

FYI,
Just checking our logs, we've never seen this issue before upgrading to .NET 6.0 and 2.1.0 a few days ago.

I've temporarily downgraded back to 2.0.9 to see if the issue is relieved by that. We were previously on 2.0.9.

[oformaniuk]

Hello @phil000
Thanks for the issue report.
There were no changes related to the code path in 2.1.0.
Although, it's unclear how .NET upgrade could have caused it.

[phil000]

Thank you for that.

I've reverted to .NET 6.0 + 2.0.9 and deployed to our test environment already.

I'll be able to report in a few days if we see the issue keep occurring, or if it stops.

Either way that will provide a path forward.

There's a lot of calls to 'AddOrReplace' in Initialize so a debug build and maybe a process dump at the time of the exception may help.

[oformaniuk]

@phil000, additional information will definitely help 👍
Looking forward for more info.

[phil000]

Hi.
For your information, we have been monitoring our test environment on .NET 6.0 + 2.0.9 for several days and have not seen the 'IndexOutOfRangeException' occur again.

We have deployed this configuration into production as of now.

Next step is to see if we can get the 'IndexOutOfRangeException' to occur again in the test environment on 2.1.0 and provide a debug stack trace.

[phil000]

This issue hasn't occurred again. Happy to close it as we cannot reproduce it.

[oformaniuk]

@phil000, thanks for keeping me updated.

[jeffclary]

I am seeing a very similar issue in version 2.0.9 and netstandard2.1. My stack trace is a bit different because I am using Handlebars.Net.Extension.Json version 1.0.0.

The behavior is the same, in that I see only periodic errors that I can't explain. Still investigating on my end.

[ChadNedzlek]

We are seeing this in our service as well, several dozen times a day. Though our callstack isn't quite the same:

System.IndexOutOfRangeException: Index was outside the bounds of the array.
 at HandlebarsDotNet.Collections.FixedSizeDictionary`3.AddOrReplace(TKey& key, TValue& value, EntryIndex`1& index)
 at HandlebarsDotNet.Iterators.ListIterator`2.Iterate(EncodedTextWriter& writer, BindingContext context, ChainSegment[] blockParamsVariables, Object input, TemplateDelegate template, TemplateDelegate ifEmpty)
 at HandlebarsDotNet.Iterators.ListIterator`2.HandlebarsDotNet.Iterators.IIterator.Iterate(EncodedTextWriter& writer, BindingContext context, ChainSegment[] blockParamsVariables, Object input, TemplateDelegate template, TemplateDelegate ifEmpty)
 at HandlebarsDotNet.Compiler.Iterator.Iterate(BindingContext context, EncodedTextWriter writer, ChainSegment[] blockParamsVariables, Object target, TemplateDelegate template, TemplateDelegate ifEmpty)
 at lambda_method(Closure , EncodedTextWriter& , BindingContext )
 at HandlebarsDotNet.HandlebarsEnvironment.<>c__DisplayClass15_0.<Compile>b__0(TextWriter writer, Object context, Object data)
 at HandlebarsDotNet.HandlebarsEnvironment.<>c__DisplayClass16_0.<Compile>b__0(Object context, Object data)

It definitely seems like something is weird about the FixedSizeDictionary. It's also not consistent... we run it ~3000 times a day, and it only fails 30 ish times... But that still means 1% of our users get essentially random failures, which is a bit too high for comfort... We are about to attempt to just wrap all the handlebars code in a giant "try{Render}catch{Render... again}" loop, but that's really unfortunate for our service to have to do.

[ChadNedzlek]

My guess is there is some sort of multithreading issue... and some handlebars data structure we are sharing isn't actually thread safe (that's the only way I'd be able to explain the infrequency, and a failure to look up a thing in an immutable input).

Is the "HandlebarsTemplate" delegate returned from IHandlebars.Compile not reusable? Or perhaps not threadsafe (so we could have a pool of them shared, but perhaps they have to be rented)?

[oformaniuk]

@ChadNedzlek , @jeffclary
It is thread safe (by design). FixedSizeDictionary is part of BindingContext which is pooled and should not be shared between different threads at any given moment in time.
However, I suspect that there's some race-condition involved.
I've already tried to tackle the problem but I still cannot reproduce it in a way I can get some useful data about what is happening.
If you see the error constantly and if you'd be able to provide some dump or try to create a consistent repro - it will help.

[ChadNedzlek]

I'll try my best, unfortunately, its server code, so it just causes 500's in our server code, and taking a dump isn't really feasible (especially not one I could share, since it would contain customer data). If it's not shared between threads, it would be weird for it to be a race condition (since what two things would be racing)?

It's definitely causing us serious problems in our server... I'm not sure what we are going to do here.

[ChadNedzlek]

I'm going to try making a small repo that basically just does millions of simultaneous renders, and see if I can trigger it... I'm not sure what callstack that's coming from though, so it's going to take me a while to produce anything. Like I'm not even sure what it's iterating.

[ChadNedzlek]

Hrm... I think it might be a double return of something from a pool. In general, disposing something is supposed to be safe to do multiple times... but a lot of the Disposes in Handlebars appear to return things to the pool unchecked, including "contexts"... so if we disposed something we shouldn't have twice, then two copies of the same object are in the pool, and will just randomly crash later. (documentation here about 6 paragraphs down)

It would probably be good if the returning disposes were multi-call safe. I'm going to check our code base for any potential double disposes and see if I can reproduce the issue.