Selfhosted upgrade from 8.8.2 to 8.8.4 fails

[ivarreflex]

Hello! We are selfhosting exceptionless and currently run version 8.8.2. The upgrade to version 8.8.4 throws the exception below. What can we do to fix this issue?

Let me know if you need more information.

[06:34:32 ERR] Error running DataSeedService startup action: The JSON value could not be converted to Exceptionless.Core.Models.SavedViewColumnSettings. Path: $.columns.level \| LineNumber: 0 \| BytePositionInLine: 472. <StartupActions>
--
Elastic.Transport.UnexpectedTransportException: The JSON value could not be converted to Exceptionless.Core.Models.SavedViewColumnSettings. Path: $.columns.level \| LineNumber: 0 \| BytePositionInLine: 472.
---> System.Text.Json.JsonException: The JSON value could not be converted to Exceptionless.Core.Models.SavedViewColumnSettings. Path: $.columns.level \| LineNumber: 0 \| BytePositionInLine: 472.
at System.Text.Json.ThrowHelper.ThrowJsonException_DeserializeUnableToConvertValue(Type propertyType)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TDictionary& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Deserialize(Utf8JsonReader& reader, ReadStack& state)
at System.Text.Json.JsonSerializer.Read[TValue](Utf8JsonReader& reader, JsonTypeInfo`1 jsonTypeInfo)
at Elastic.Transport.Extensions.TransportSerializerExtensions.Deserialize[T](Serializer serializer, Utf8JsonReader& reader, MemoryStreamFactory memoryStreamFactory)
at Elastic.Clients.Elasticsearch.Serialization.SourceConverter`1.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/SourceConverter.cs:line 83
at Elastic.Clients.Elasticsearch.Core.Search.Json.HitConverter`1.<>c.<Read>b__20_11(Utf8JsonReader& r, JsonSerializerOptions o) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Types/Core/Search/Hit.Converters.g.cs:line 164
at Elastic.Clients.Elasticsearch.Core.Search.Json.HitConverter`1.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Types/Core/Search/Hit.Converters.g.cs:line 164
at Elastic.Clients.Elasticsearch.Serialization.JsonReaderExtensions.<>c__12`1.<ReadCollectionValue>b__12_0(Utf8JsonReader& r, JsonSerializerOptions o) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/JsonReaderExtensions.cs:line 349
at Elastic.Clients.Elasticsearch.Serialization.JsonReaderExtensions.ReadCollectionValue[T](Utf8JsonReader& reader, JsonSerializerOptions options, JsonReadFunc`1 readElement) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/JsonReaderExtensions.cs:line 355
at Elastic.Clients.Elasticsearch.Core.Search.Json.HitsMetadataConverter`1.<>c.<Read>b__3_0(Utf8JsonReader& r, JsonSerializerOptions o) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Types/Core/Search/HitsMetadata.Converters.g.cs:line 40
at Elastic.Clients.Elasticsearch.Core.Search.Json.HitsMetadataConverter`1.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Types/Core/Search/HitsMetadata.Converters.g.cs:line 40
at Elastic.Clients.Elasticsearch.Serialization.JsonReaderExtensions.<>c__22`1.<TryReadProperty>b__22_0(Utf8JsonReader& r, JsonSerializerOptions o) in /_/src/Elastic.Clients.Elasticsearch/_Shared/Next/JsonReaderExtensions.cs:line 719
at Elastic.Clients.Elasticsearch.Json.SearchResponseConverter`1.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Api/SearchResponse.Converters.g.cs:line 77
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.ContinueDeserialize[TReadBufferState,TStream](TReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, T& value)
at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.DeserializeAsync[TReadBufferState,TStream](TStream utf8Json, TReadBufferState bufferState, CancellationToken cancellationToken)
at Elastic.Transport.SystemTextJsonSerializer.DeserializeAsync[T](Stream stream, CancellationToken cancellationToken)
at Elastic.Transport.Products.Elasticsearch.ElasticsearchResponseBuilder.SetBodyCoreAsync[TResponse](Boolean isAsync, ApiCallDetails details, BoundConfiguration boundConfiguration, Stream responseStream, CancellationToken cancellationToken)
at Elastic.Transport.DefaultResponseFactory.CreateCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, Exception ex, Nullable`1 statusCode, Dictionary`2 headers, Stream responseStream, String contentType, Int64 contentLength, IReadOnlyDictionary`2 threadPoolStats, IReadOnlyDictionary`2 tcpStats, CancellationToken cancellationToken)
at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, CancellationToken cancellationToken)
at Elastic.Transport.RequestPipeline.CallProductEndpointCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, Auditor auditor, CancellationToken cancellationToken)
at Elastic.Transport.DistributedTransport`1.RequestCoreAsync[TResponse](Boolean isAsync, EndpointPath path, PostData data, Action`1 configureActivity, IRequestConfiguration localConfiguration, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elastic.Transport.DistributedTransport`1.RequestCoreAsync[TResponse](Boolean isAsync, EndpointPath path, PostData data, Action`1 configureActivity, IRequestConfiguration localConfiguration, CancellationToken cancellationToken)
at Foundatio.Repositories.Elasticsearch.ElasticReadOnlyRepositoryBase`1.FindAsAsync[TResult](IRepositoryQuery query, ICommandOptions options) in /_/src/Foundatio.Repositories.Elasticsearch/Repositories/ElasticReadOnlyRepositoryBase.cs:line 487
at Exceptionless.Core.Seed.PredefinedSavedViewsDataSeed.<>c__DisplayClass10_0.<<SeedAsync>b__0>d.MoveNext() in /app/src/Exceptionless.Core/Seed/PredefinedSavedViewsDataSeed.cs:line 44
--- End of stack trace from previous location ---
at Foundatio.Lock.LockProviderExtensions.TryUsingAsync(ILockProvider locker, String resource, Func`1 work, Nullable`1 timeUntilExpires, Nullable`1 acquireTimeout) in /_/src/Foundatio/Lock/ILockProvider.cs:line 258
at Foundatio.Lock.LockProviderExtensions.TryUsingAsync(ILockProvider locker, String resource, Func`1 work, Nullable`1 timeUntilExpires, Nullable`1 acquireTimeout) in /_/src/Foundatio/Lock/ILockProvider.cs:line 262
at Foundatio.Lock.LockProviderExtensions.TryUsingAsync(ILockProvider locker, String resource, Func`1 work, Nullable`1 timeUntilExpires, Nullable`1 acquireTimeout) in /_/src/Foundatio/Lock/ILockProvider.cs:line 265
at Exceptionless.Core.Seed.DataSeedService.SeedAsync(CancellationToken cancellationToken) in /app/src/Exceptionless.Core/Seed/DataSeedService.cs:line 43
at Exceptionless.Core.Seed.DataSeedService.RunAsync(CancellationToken shutdownToken) in /app/src/Exceptionless.Core/Seed/DataSeedService.cs:line 33
at Foundatio.Extensions.Hosting.Startup.StartupActionRegistration.RunAsync(IServiceProvider serviceProvider, CancellationToken shutdownToken) in /_/src/Foundatio.Extensions.Hosting/Startup/StartupActionRegistration.cs:line 55
at Foundatio.Extensions.Hosting.Startup.StartupExtensions.<>c__DisplayClass0_1.<<RunStartupActionsAsync>b__3>d.MoveNext() in /_/src/Foundatio.Extensions.Hosting/Startup/StartupExtensions.cs:line 59

[ejsmith]

Thanks for the report. Before changing any data, could you provide:

1. How Exceptionless is deployed: Docker Compose, the all-in-one image, Kubernetes/Helm, or something custom.
2. The exact image names/tags and commands used for the app and job containers.
3. Whether the Migration job ran during the upgrade, along with its logs. Please look for MigrateSavedViewColumns or Migrated ... saved view column configurations.
4. A screenshot of System → Migrations, especially migration version 5, or the equivalent migration-state output.
5. Whether app and job instances were stopped and upgraded together, or whether this was a rolling deployment with mixed versions.

The exception shows a saved-view record still using the legacy boolean column format that migration 5 should convert. Since the migration worked in our environments, we want to determine whether the migration job was missing, failed, or was already marked complete before repairing anything manually.

[ivarreflex]

1. To upgrade I deployed the exceptionless/app:8.8.4 image on aws fargate. Previously it was 8.8.2. When I retried the upgrade to fetch the logs I was unable to deploy 8.8.2 again and I had to revert to 8.7.1 to make the image start. So now we run 8.7.1
2. No specific arguments, but this is the environment. If I'm correct when I'm using the /app image I don't need a separate job container right?
   "environment": [ { "name": "EX_SmtpFrom", "value": "Exceptionless <redacted>" }, { "name": "EX_ConnectionStrings__Elasticsearch", "value": "redacted" }, { "name": "EX_EnableAccountCreation", "value": "false" }, { "name": "Serilog__MinimumLevel__Default", "value": "Debug" }, { "name": "EX_ConnectionStrings__MessageBus", "value": "provider=redis" }, { "name": "EX_ApiUrl", "value": "redacted" }, { "name": "EX_ConnectionStrings__Redis", "value": "redacted" }, { "name": "EX_RunJobsInProcess", "value": "true" }, { "name": "EX_ConnectionStrings__Storage", "value": "provider=folder;path=/app/storage" }, { "name": "EX_BaseURL", "value": "redacted" }, { "name": "EX_AppMode", "value": "Production" }, { "name": "EX_ConnectionStrings__Email", "value": "smtp://redacted" }, { "name": "EX_ConnectionStrings__Cache", "value": "provider=redis" }, { "name": "EX_ConnectionStrings__Queue", "value": "provider=redis" } ],
3. I don't think so. I will upload the full startup log.
4. I'm sorry. I don't know what you are referring to.
5. It was a rolling deployment with mixed versions

exceptionless-app.txt

[ivarreflex]

Found it