Hi There, thanks for your time reading this.
We encounter an error when we use Redis from Elasticache. We create large amount of background job but a small percentage of them fail with the following exception (from our Elastic Search log):
{
“Depth”: 0,
“ClassName”: “”,
“Message”: “Background job creation failed. See inner exception for details.”,
“Source”: “Hangfire.Core”,
“StackTraceString”: " at Hangfire.BackgroundJobClient.Create(Job job, IState state)\n at Hippo.Domain.Common.DomainEventDispatcher.DomainEventHandler1.HandleAsync(DomainEventBase domainEvent) in /app/Hippo.Domain/Common/DomainEventDispatcher.cs:line 47\n at Hippo.Domain.Common.DomainEventDispatcher.DispatchAsync(DomainEventBase domainEvent) in /app/Hippo.Domain/Common/DomainEventDispatcher.cs:line 25\n at Hippo.Infrastructure.Interactions.InteractionRepository.DispatchDomainEventsAsync(DomainObject domainObject) in /app/Hippo.Infrastructure/Interactions/InteractionRepository.cs:line 107\n at Hippo.Infrastructure.Interactions.InteractionRepository.<SaveAsync>d__6.MoveNext() in /app/Hippo.Infrastructure/Interactions/InteractionRepository.cs:line 91\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at Hippo.Domain.Interactions.InteractionService.<ProcessInteractionAsync>d__2.MoveNext() in /app/Hippo.Domain/Interactions/InteractionService.cs:line 19\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at Hippo.Api.Interactions.InteractionsController.<PostSingleInteraction>d__11.MoveNext() in /app/Hippo.Api/Interactions/InteractionsController.cs:line 141", "RemoteStackTraceString": "", "RemoteStackIndex": -1, "HResult": -2146233088, "HelpURL": null }, { "Depth": 1, "ClassName": "", "Message": "Timeout performing EXEC, inst: 1, queue: 1462, qu: 0, qs: 1462, qc: 0, wr: 0, wq: 0, in: 12557, ar: 0, clientName: Hangfire, serverEndpoint: Unspecified/hippo-redis-prod.qdm7yi.ng.0001.apse2.cache.amazonaws.com:6379, keyHashSlot: 13969 (Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts)", "Source": "StackExchange.Redis", "StackTraceString": " at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\code\StackExchange.Redis\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 2120\n at StackExchange.Redis.RedisTransaction.Execute(CommandFlags flags) in c:\code\StackExchange.Redis\StackExchange.Redis\StackExchange\Redis\RedisTransaction.cs:line 51\n at Hangfire.Pro.Redis.RedisConnection.CreateExpiredJob(Job job, IDictionary2 parameters, DateTime createdAt, TimeSpan expireIn)\n at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)\n at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass7_0.<CreateWithFilters>b__0()\n at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func1 continuation)\n at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)\n at Hangfire.BackgroundJobClient.Create(Job job, IState state)",
“RemoteStackTraceString”: “”,
“RemoteStackIndex”: -1,
“HResult”: -2146233083,
“HelpURL”: “http://stackexchange.github.io/StackExchange.Redis/Timeouts”
}
Just want to know, in order to use Redis for HangFire with ElasticSearch, is there any special setting we need to do? Any help is appreciated. Thanks.