After deployment of hangfire server does not start processing jobs. Please help to fix this.
We are using hangfire 1.7.30 (under IIS and SQL server) . We have SOA Service application suite. Where we have about 15 hangfire Service component hosted on 2 node. Some service components are on .net framework 4.8 and some on .net core. Our majority of hangfire component are independent component which mean they not with application service hosting.
At the time of CD deployment, we trigger restart app pool for all hangfire component, but Hangfire does not start itself it does not process any recurring job. On further investigation we found that Hangfire.Server table records are not created after deployment. (deployment has app pool restart at the end for all service)
As workaround to start hangfire we have to load dashboard (which is difficult due to multiple node behind LB), or manually restart app pool one by one on both node.
After this workaround hangfire works fine.
We guess starting app pool for all hangfire service together may causing issue. Is it a problem?
Here I have log and code -
For some application below exception
FATAL Execution loop ExpirationManager:8d3594ec was unable to wait for ‘00:00:16’ delay due to an exception. Execution will be stopped. System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)
at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)
For some just this and after
… caught stopping signal…
…All dispatchers stopped
We have generally these hangfire server options settings
var sqlOptions = new SqlServerStorageOptions
{
QueuePollInterval = TimeSpan.FromSeconds(1),
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
UseRecommendedIsolationLevel = true,
DisableGlobalLocks = false,
};
app.UseHangfireServer(options);
Some uses app.UseHangfireAspNet to register backgroundJob.
Please help on this.