Understanding background/recurring job performance vs within application

aspnetcore
Tags: #<Tag:0x00007f499c06b7a0>
#1

I’m using Hangfire recurring background jobs overnight to update a variety of stuff in our databases. This is an ASP MVC Core 2.0 application running in a 4-core standard azure app service. These run just after midnight and at that point there are no users on the system.

I’ve noticed that the jobs runs considerably slower when triggered by hangfire as opposed to when they are run from within the application. On my local dev machine calling the function from a controller (outside of hangfire) takes about 30 seconds. The same function when triggered from within hangfire takes about 120 seconds, so about 1/4 the speed.

In Azure the difference is more drastic with the same function on the same DB taking up to 3-4 minutes when triggered from hangfire.

I’ve seen that there are options for settings server settings and parallel options for background jobs but I’m curious as to how to set it up so that the functions triggered by hangfire sees similar performance to the application.

The particular job I’m talking about has a lot of DB access with not a lot of processing. I’ve played around with breaking some of it into parallel tasks but I’d like to sort out how I can get similar performance to when I call the function from within the application.

Thanks,
Brian

#2

Need some more details. Is your job sync or async? Is the database the job is working with the same database Hangfire uses for storage? Do you have any extensions (e.g. Hangfire.Console) which may affect job execution time if used extensively? Posting some code might be helpful.

#3

Thanks for the response. Hangfire is using a separate database, it’s a database per tenant saas application. I am not using an extensions. The example I sited and the tests I am running are just running the job against one of the tenants. The code that is called is sync, I’ll see if I can make a simple example with the same performance.

Thanks
Brian

#4

So you’re running a separate Hangfire server for each tenant? How many tenants do you have? Each instance of Hangfire server will run twenty-something threads (by default), and even though they’re sleeping most of the time, they definitely not increase overall performance. Also if all those databases are hosted on the same server, you might want to check the max number of connections configured for your server, as your queries may hit that limit.

Just to be sure: do you have all those Hangfire servers running when testing your job outside Hangfire, or are they stopped at the time?

#5

There is one hangfire server, database per tenant but only running one instance of the application and hangfire. After digging in some more it likely has to do with all the stuff going on at once and limiting performance. If I isolate smaller portions of the jobs and run them I get similar execution times running it from hangfire as I do from a controller or from a console application. Haven’t solved it just yet but it if I find anything interesting I’ll report back.

Thanks

#6

Hey, did you come to any conclusion about this? I have the exact same problem on Azure i.e. code is performing miraculously slower in the Hangfire job than when being executed in a controller.

Regards

#7

I managed to increase performance quite significantly by reducing the number of calls to
IJobCancellationToken.ThrowIfCancellationRequested();