Enable automatic failover of master slave Redis Job store

feature storage
1

I am using Redis job store to persist delayed jobs created from Hangfire.
I have a master redis node and 1 replica nodes. Is there any way I can achieve automatic fail over in case the master node goes down from hangfire configuration?

Please suggest.

2

Take a look at Redis Sentinel for enabling High Availability.

3

@odinserj I need to configure Redis on AWS environment. As mentioned here

The entire failover process, from detection to the resumption of normal
caching behavior, will take several minutes. Your application’s caching
tier should have a strategy (and some code!) to deal with a cache that
is momentarily unavailable.

Does Hangfire provides any strategy to handle the Job store which is momentarily unavailable?

4

Hangfire is just a library, it does not know anything about target project, target infrastructure and target needs. For example, it does not know what do you want to do if a job storage is temporary unavailable – return an error to a user, switch to another storage, put a job to the temporal queue, etc, etc. Your chosen strategy may not work for other projects and vice versa.

However, Hangfire contains extension points so you are able to handle the problem by yourself. For example, you can create your wrappers around Hangfire classes (I’ll show you only the concept, it is very naive):

public class FailoverRedisStorage : JobStorage
{
    public FailoverRedisStorage(RedisStorage primary, RedisStorage failover) { /* ... */ }
    public override GetConnection()
    {
        return IsPrimaryAvailable ? primary : failover;
    }
}

public class FailoverBackgroundJobClient : IBackgroundJobClient
{
    public FailoverBackgroundJobClient(FailoverJobStorage storage, IBackgroundJobClient client)
    {
        /* ... */
    }

    public string Create(Job job, /* ... */)
    {
        try
        {
            client.Create(job, ...);
        }
        catch (RedisException)
        {
             _storage.IsPrimaryAvailable = false;
        }
    }
}
1 Like