Hangfire.Pro.Redis does not work with Cluster Mode Enabled Elastic Cache

redis
Tags: #<Tag:0x00007f499dc4b678>

#1

Hi Team,
Any body using Hangfire with Elastic Cache Cluster Mode Enabled. It gets stuck after some time While Non-Cluster Mode is working fine.

Thanks,
Gheri.


#2

What version of Hangfire.Pro.Redis are you using?


#3

Hangfire.Pro.Redis == 2.5.1
Hangfire.Pro == 2.2.0
Hangfire.Core == 1.7.6(latest as of now)


#4

Looks like hash tag is not specified in RedisStorageOptions.Prefix property as written in the docs – https://docs.hangfire.io/en/latest/configuration/using-redis.html#redis-cluster-support. Am I right?


#5

Hi Team,
Thanks for Information. I am clarifying on this so that we are on same page. So I have been using {hangfire-app}: as prefix and my connection string is Cluster ConfigurationEnd Point of Elastic Cache [Cluster Mode Enabled].
I have been Analyzing on this. Some Interesting facts I found

  1. Some of Redis Clients are unsubscribed from Server (in this case hangfire servers) which are subscribed in happy working scenarios.
  2. It Occurs Intermittently(Not Always)
  3. One of way to reproduce is to recycle the app pool which is also not consistent way to reproduce this issue.

Thanks,


#6

No, it’s not a prefix for the connection string itself, it’s a prefix for all the Hangfire-related keys, and you can configure it in the following way as written in the article referenced above:

GlobalConfiguration.Configuration.UseRedisStorage(
    "localhost:6379,localhost:6380,localhost:6381",
    new RedisStorageOptions { Prefix = "{hangfire-1}:" });

Regarding unsubscriptions, could you add more details? Unfortunately seems like I don’t understand what you do mean by subscriptions. But before please ensure you are on the latest version, which is 2.5.2 at the moment.

In the latest version a bug was fixed with workers unable to fetch background jobs if the queue was initially empty when fail-over occurred and not all the cluster nodes were referenced in the connection string.


#7

Thanks for the information. I will definitely look into the latest version.
Regarding the Observations.
I have observed that whenever the job is enqueued. Hangfire publishes to Redis channel {hangfire}:queue:default:events where “default” is queue name.
In Working Scenario I can see Redis Clients are subscribed to these channels and everything is working fine.
In this Scenario, When job is enqueued Hangfire publishes to Redis Channel but there are no subscribers to this channel and hence no processing is happening in hangfire.

In Elastic Cache we have cluster configuration endpoint as well as every node endpoint.
Are you suggesting me to use all node endpoints to hangfire or are we good to use only cluster endpoint as connection string to hangfire.

If you need more information please let me know.

Thanks,


#8

Hi Team,
I am running My app with new Hangfire.Pro.Redis[2.5.2] with cluster mode Elastic Cache.
Previously it gets stuck if AppPool gets recycled but now Its been three days it is working fine.
Will update you if in future it agains gets stuck.
Many Thanks!!! @odinserj

Thanks,
Gheri.