Advice on setting up Hangfire for multi-tenant system

question
1

Hi, I’ve inherited a system using Hangfire and I plan to adjust how it’s working. I’d like some advice on my proposed setup, since I’m new with Hangfire and you’re the experts.

It’s a multi-tenant system with sql server backend where each tenant (ie customer) has their own database. There’s a separate single db for Hangfire jobs with a separate Queue per tenant. There’s a single live webserver at any one time, currently hosting about 50-100 customers. Mostly we use Hangfire for running background jobs immediately, rather than scheduling recurring jobs. All our jobs just trigger a webapi call to the endpoint for the appropriate tenant, rather than running any real .net code within the app hosting the hangfire server. Many of our jobs are long-running, taking 1-15mins to run, and most of that work is within SQL server rather than webserver CPU time. Even with 100 customers and long-running jobs there’s probably never more than 10 jobs running, and mostly fewer than 5.

Currently we do a few things I think are wrong:

  • we don’t set QueuePollInterval, so it defaults to 15s. This means there’s a wait of up to 15s for jobs to be run.
  • we call app.UseHangfireServer() once per tenant, passing in the single queue name for that tenant. This means there’s a ton of worker threads, lots of heartbeats, and lots of polling sqlserver even though interval is 15s.

I’m thinking of changing to:

  • set QueuePollInterval to about 200ms, so jobs will start quicker.
  • have a single call to app.UseHangfireServer(), passing in all the Queue names.
  • set WorkerCount to 20. We’re not CPU bound on the web server.
  • continue to have UsePageLocksOnDequeue = false and not set SlidingInvisibilityTimeout, so we use a DELETE statement with transaction for dequeuing, and row-level locking.

Does this sound sensible? Any other suggestions or thoughts?

I don’t know what DisableGlobalLocks does so I’m hesitant to change it, but a quick look suggests it’s mostly about recurring jobs so not very important for us?

Thanks for any tips.

Rory

2

If it is so few jobs at a time, why not just use a single Hangfire server. You can configure it to handle all the different queues if you want to keep the separate queues for easy overview reasons.