Connection to Redis isn't available yet, reconnect is in progress: please try again later

bug?
1

Hi,

I am having an issue after updating Hangfire.Pro.Redis to the latest version (2.7.4.0). i keep getting “Connection to Redis isn’t available yet, reconnect is in progress: please try again later.” and my Hangfire server will not start. The log gave :
> at Hangfire.Pro.Redis.RedisStorage.GetMultiplexer()
> at Hangfire.Pro.Redis.RedisStorage.GetDatabase()
> at Hangfire.Pro.Redis.RedisDistributedLock.<.ctor>b__10_0()
> at Hangfire.Pro.Redis.RedisDistributedLock.RetryUntilTrue(Func1 action, TimeSpan timeout) > at Hangfire.Pro.Redis.RedisDistributedLock..ctor(Func1 databaseFactory, ILockExtender lockExtender, String resource, String lockToken, TimeSpan timeout)
> at Hangfire.Pro.Redis.RedisConnection.DistributedLockWrapper…ctor(RedisConnection connection, String resource, TimeSpan timeout)
> at Hangfire.Pro.Redis.RedisConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
> at Hangfire.RecurringJobExtensions.AcquireDistributedRecurringJobLock(IStorageConnection connection, String recurringJobId, TimeSpan timeout)
> at Hangfire.RecurringJobManager.AddOrUpdate(String recurringJobId, Job job, String cronExpression, RecurringJobOptions options)
> at Hangfire.RecurringJobManagerExtensions.AddOrUpdate(IRecurringJobManager manager, String recurringJobId, Job job, String cronExpression, TimeZoneInfo timeZone, String queue)
> at Hangfire.RecurringJobManagerExtensions.AddOrUpdate(IRecurringJobManager manager, String recurringJobId, Job job, String cronExpression)
> at App.Models.RecuringJobs.SetUpRecurringJobs() in RecuringJobs.cs:line 25
> at App.WebApiApplication.Application_Start() in Global.asax.cs:line 25

Kindly assist urgently.

2

Hi @tobbyioa, thanks for sharing. What version did you use before? Did you try to downgrade it to previous versions?

3

Hi @odinserj,

Thanks for your reply. I was formerly using a fairly old version (2.5.3.0). I decided to upgrade it when I begin to notice a high heartbeat interval on the server. This was resulting to aborted jobs.
I have downgraded to version 2.7.1.0 which I have used in a separate project with no issues at all.
I also upgraded the Hangfire.Core to 1.7.11.0, I previously had version 1.7.6.0.
Hangfire now connects fine with Redis. However, while AddorUpdate my recurring jobs (41 of them) was ongoing, the server encountered the exception below:

2020-07-26 15:57:38,899 [ServerJobCancellationWatcher #1] DEBUG Hangfire.Processing.BackgroundExecution Execution loop ServerJobCancellationWatcher:9e51d4ce caught an exception and will be retried in 00:00:01

StackExchange.Redis.RedisTimeoutException: Timeout performing HGETALL HangfireDB_job:ce16fbb5-4fdf-45a9-8ee2-29a38fe200e9:state, inst: 0, queue: 35, qu: 5, qs: 30, qc: 0, wr: 1, wq: 1, in: 0, ar: 0, clientName: Hangfire@OLUFEMI-PC, serverEndpoint: 10.1.7.174:7000, keyHashSlot: 9678, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=1,Free=8190,Min=8,Max=8191) (Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts)
   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisDatabase.HashGetAll(RedisKey key, CommandFlags flags)
   at StackExchange.Redis.KeyspaceIsolation.DatabaseWrapper.HashGetAll(RedisKey key, CommandFlags flags)
   at Hangfire.Pro.Redis.RedisConnection.GetStateData(String jobId)
   at Hangfire.Server.ServerJobCancellationToken.IsJobStateChanged(IStorageConnection connection)
   at Hangfire.Server.ServerJobCancellationToken.CheckJobStateChanged(IStorageConnection connection)
   at Hangfire.Server.ServerJobCancellationToken.TryCheckJobIsAborted(IStorageConnection connection)
   at Hangfire.Server.ServerJobCancellationToken.CheckAllCancellationTokens(String serverId, IStorageConnection connection, CancellationToken cancellationToken)
   at Hangfire.Server.ServerJobCancellationWatcher.Execute(BackgroundProcessContext context)
   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)
   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)

Then the server stopped provisioning the recurring job.
Kindly assist to look into this.