Avoid concurrency in background processes with multiple web servers

ogborstad · October 13, 2015, 7:13am

I am running multiple web servers behind a load balancer, and would like to put up a Hangfire background process. The work done by the background process should only be done by one server. Do I need to only start the background process on one of the servers, or add my own synchronization mechanism between the servers?

yngndrw · October 14, 2015, 12:26pm

Just to better understand your constraints and scenario, why should the background process only be on one server ?

Is this to limit the performance impact ? What kind of jobs do you intend to run ?

ogborstad · October 14, 2015, 12:33pm

The job is going to do some database intensive work, I cant have all my servers pound the database

yngndrw · October 14, 2015, 12:53pm

So you can just run background process on a single machine, but if that machine fails then your jobs won’t be processed until you start another background process.

You could run the Hangfire background process on all of the machines with a single worker, then have a custom-made job filter which uses a distributed lock to see if another server is executing jobs and if it is, cancel the job execution. (I.e. Upon state change election to the “processing state”, set the job back to “enqueued state”. See the retry attribute for an example of altering the state election process:

github.com

HangfireIO/Hangfire/blob/ad008233257cf2ff574ffc737b6203bc41b97cc5/src/Hangfire.Core/AutomaticRetryAttribute.cs#L145


/// <summary>
/// Gets or sets whether to produce log messages on retry attempts.
/// </summary>
public bool LogEvents
{
    get { lock (_lockObject) { return _logEvents; } }
    set { lock (_lockObject) { _logEvents = value; } }
}


/// <inheritdoc />
public void OnStateElection(ElectStateContext context)
{
    var failedState = context.CandidateState as FailedState;
    if (failedState == null)
    {
        // This filter accepts only failed job state.
        return;
    }


    var retryAttempt = context.GetJobParameter<int>("RetryCount") + 1;

Topic		Replies	Views
Running Hangfire jobs on a specific server question	3	6706	October 2, 2015
Single worker process for entire farm question queues	0	1093	August 1, 2017
Best Practice: One Process with Multiple Background Workers, or Multiple Process with One question	1	415	May 27, 2023
Separated reponsabilities with hangfire jobs question queues , aspnetcore	1	718	September 20, 2021
Multitenancy with Hangfire question queues	1	2605	July 5, 2017

Avoid concurrency in background processes with multiple web servers

Related Topics