I have 79 “queues” which contain 285 “workers”.
When I check how many hangfire threads are running with the following command I get 996.
dotnet stack report -p 1297860 | grep 'Hangfire.Processing.BackgroundDispatcher.DispatchLoop' | wc -l
Does anyone have any idea why I have 996 active threads which is nowhere close to 285?
That’s really strange, could you post here the full report of the dotnet stack command?
And this is my setup.
public IServiceProvider ConfigureServices(IServiceCollection services)
{
services.AddHangfire(configuration =>
{
configuration.UseRedisStorage(RedisHangfire, new RedisStorageOptions
{
InvisibilityTimeout = TimeSpan.FromHours(6),
FetchTimeout = TimeSpan.FromMinutes(5),
Prefix = "{h}"
});
});
SetupHangfireQueues(services);
}
private void SetupHangfireQueues(IServiceCollection services)
{
var longStopTimeout = TimeSpan.FromHours(1);
if (Settings.IsDevelopment)
{
longStopTimeout = TimeSpan.FromSeconds(7);
}
services.AddServer(50, TaskQueue.Default);
services.AddServer(1, TaskQueue.Import);
services.AddServer(4, TaskQueue.RetroactivePurchase, longStopTimeout);
services.AddServer(3, TaskQueue.RetroactiveAccounting, longStopTimeout);
services.AddServer(1, TaskQueue.RetroactiveAccountingLong, longStopTimeout);
services.AddServer(1, TaskQueue.GenericSingleLine);
services.AddServer(3, TaskQueue.DeviceOrder);
services.AddServer(5, TaskQueue.GenericBatch);
services.AddServer(8, TaskQueue.SyncDevices);
services.AddServer(1, TaskQueue.WriteAccounting, longStopTimeout);
services.AddServer(1, TaskQueue.Notifications);
services.AddServer(5, TaskQueue.NotificationProcessor);
services.AddServer(1, TaskQueue.LimitLog);
for (int i = 0; i < 30; i++)
{
services.AddServer(1, TaskQueue.Invoice + (i + 1), longStopTimeout);
services.AddServer(1, TaskQueue.WriteAccounting + (i + 1), longStopTimeout);
}
services.AddServer(20, TaskQueue.Petrol + (int)PetrolType.Opet);
services.AddServer(20, TaskQueue.Petrol + (int)PetrolType.Bp);
services.AddServer(30, TaskQueue.Petrol + (int)PetrolType.Shell);
services.AddServer(60, TaskQueue.Petrol + (int)PetrolType.PetrolOfisi);
services.AddServer(1, TaskQueue.Petrol + (int)PetrolType.Demo);
services.AddServer(10, TaskQueue.Petrol); // Compound Fleets
}
public static void AddServer(this IServiceCollection app, int workerCount, string queueName,
TimeSpan? stopTimeout = null)
{
var stdStopTimeout = TimeSpan.FromMinutes(15);
if (Settings.IsDevelopment)
{
stdStopTimeout = TimeSpan.FromSeconds(7);
}
app.AddHangfireServer(options =>
{
options.WorkerCount = workerCount;
options.StopTimeout = stopTimeout ?? stdStopTimeout;
options.ShutdownTimeout = stopTimeout ?? stdStopTimeout;
options.Queues = new[] { queueName };
});
}
Background job server is responsible for many tasks – execute jobs, schedule recurring and delayed jobs, perform storage maintenance, report heartbeats. For each such task dedicated server component is created that’s based on a dedicated thread.
It’s only necessary to have single full-fledged server for the whole storage instance, since only workers will and can act in parallel (for other components distributed lock is usually used). It is technically possible to use BackgroundProcessingServer instead of BackgroundJobServer (used by AddHangfireServer), but you will need to deconstruct the AddHangfireServer method. In this case no additional DelayedJobScheduler and RecurringJobScheduler components will be created, but still there will be additional storage-related components.
Or you can create only a single background job server and pass more workers as additional processes. In this case they will not be reflected on the Servers tab of the Dashboard UI, though. And also you’ll not be able to use AddHangfireServer, because in this case you’ll not be able to use the additionalProcess parameter.
But it’s possible to use the underlying BackgroundJobServerHostedService class:
var additionalProcesses = new List< IBackgroundProcess>();
additionalProcesses.Add(new Worker(new [] { "queue-1", "queue-2" }));
// add other workers
services.AddTransient<IHostedService>(provider => new BackgroundJobServerHostedService(
provider.GetService<JobStorage>(),
new BackgroundJobServerOptions { WorkerCount = 4, Queues = new [] { "default" } }), // some default queues
additionalProcesses);
In Hangfire 1.8 possibly there will be “lightweight” servers with only workers defined for cases like this and ability to show them in the Dashboard UI.
So it looks like I have been using Hanfgfire in a very bad way.
So you are saying I can build the same behaviour using workers as parameter, instead of creating 80 job servers 
That sounds great! It looks like I will get rid of 80 distributed locks and many threads.
Thanks. I will keep posted.
> dotnet stack report -p 2329372 | grep 'Hangfire.Processing.BackgroundDispatcher.DispatchLoop' | wc -l
> 294
Great, thanks! By the way, since I am using dependency injection, creating workers like you specified did not work for me, I have implemented a method like the following, may I ask can you see any bad practise here?
Note: I am handling graceful shutdown myself.
public static IApplicationBuilder AddHangfireServer(
this IApplicationBuilder app,
BackgroundJobServerOptions options,
IEnumerable<string> additionalQueues)
{
var applicationServices = app.ApplicationServices;
var storage = applicationServices.GetRequiredService<JobStorage>();
options.Activator ??= applicationServices.GetService<JobActivator>();
options.FilterProvider ??= applicationServices.GetService<IJobFilterProvider>();
options.TimeZoneResolver ??= applicationServices.GetService<ITimeZoneResolver>();
var performer = new BackgroundJobPerformer(options.FilterProvider, options.Activator, TaskScheduler.Default);
var stateChanger = new BackgroundJobStateChanger(options.FilterProvider);
var workers = additionalQueues.Select(c => new Worker(new[] { c }, performer, stateChanger));
var server = new BackgroundJobServer(options, storage, workers);
Servers.Add(server);
return app;
}Yes, I see that the only thing is missing is graceful shutdown, so for other folks who wants to implement something like this I’d also recommend to add support for application lifetime events as shown here – https://github.com/HangfireIO/Hangfire/blob/6469784d55a6f20ee40a1040633f5125483dc903/src/Hangfire.AspNetCore/HangfireApplicationBuilderExtensions.cs#L92-L93.