Avoid concurrency in background processes with multiple web servers

I am running multiple web servers behind a load balancer, and would like to put up a Hangfire background process. The work done by the background process should only be done by one server. Do I need to only start the background process on one of the servers, or add my own synchronization mechanism between the servers?

Just to better understand your constraints and scenario, why should the background process only be on one server ?

Is this to limit the performance impact ? What kind of jobs do you intend to run ?

The job is going to do some database intensive work, I cant have all my servers pound the database

So you can just run background process on a single machine, but if that machine fails then your jobs won’t be processed until you start another background process.

You could run the Hangfire background process on all of the machines with a single worker, then have a custom-made job filter which uses a distributed lock to see if another server is executing jobs and if it is, cancel the job execution. (I.e. Upon state change election to the “processing state”, set the job back to “enqueued state”. See the retry attribute for an example of altering the state election process: