Avoid concurrency in background processes with multiple web servers

ogborstad · October 13, 2015, 7:13am

I am running multiple web servers behind a load balancer, and would like to put up a Hangfire background process. The work done by the background process should only be done by one server. Do I need to only start the background process on one of the servers, or add my own synchronization mechanism between the servers?

yngndrw · October 14, 2015, 12:26pm

Just to better understand your constraints and scenario, why should the background process only be on one server ?

Is this to limit the performance impact ? What kind of jobs do you intend to run ?

ogborstad · October 14, 2015, 12:33pm

The job is going to do some database intensive work, I cant have all my servers pound the database

yngndrw · October 14, 2015, 12:53pm

So you can just run background process on a single machine, but if that machine fails then your jobs won’t be processed until you start another background process.

You could run the Hangfire background process on all of the machines with a single worker, then have a custom-made job filter which uses a distributed lock to see if another server is executing jobs and if it is, cancel the job execution. (I.e. Upon state change election to the “processing state”, set the job back to “enqueued state”. See the retry attribute for an example of altering the state election process:

github.com

HangfireIO/Hangfire/blob/ad008233257cf2ff574ffc737b6203bc41b97cc5/src/Hangfire.Core/AutomaticRetryAttribute.cs#L145


/// <summary>
/// Gets or sets whether to produce log messages on retry attempts.
/// </summary>
public bool LogEvents
{
    get { lock (_lockObject) { return _logEvents; } }
    set { lock (_lockObject) { _logEvents = value; } }
}


/// <inheritdoc />
public void OnStateElection(ElectStateContext context)
{
    var failedState = context.CandidateState as FailedState;
    if (failedState == null)
    {
        // This filter accepts only failed job state.
        return;
    }


    var retryAttempt = context.GetJobParameter<int>("RetryCount") + 1;

Topic		Replies	Views
Make sure job is excuted from óne specific server question	6	7543	January 4, 2018
Running Hangfire jobs on a specific server question	3	6735	October 2, 2015
Single worker process for entire farm question queues	0	1114	August 1, 2017
One queue for the whole farm, and one queue by server question	7	17326	October 8, 2019
How to ensure that the same job will not run twice on the same time question queues	3	9066	December 21, 2018

Avoid concurrency in background processes with multiple web servers

Related topics