Dispatcher is stopped due to an exception. Please report it to Hangfire developers

question
1

We get these error messages about 2 or 3 times a day at random times, even at night when there is no activity happening. I think these two error messages don’t always come together, sometimes they come at the same time, sometimes they are independent.

  • Execution Worker is in the Failed state now due to an exception, execution will be retried no more than in 00:00:04

  • Dispatcher is stopped due to an exception, you need to restart the server manually. Please report it to Hangfire developers.

We have Hangfire running in a website hosted in the Azure Cloud, and we have the “Always On” setting turned on.

It is an ASP.NET 4.61 MVC website.

Here is our configuration in Global.asax:

HangfireAspNet.Use(GetHangfireServers);
.
.
.
private IEnumerable GetHangfireServers()
{
Hangfire.GlobalConfiguration.Configuration
.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
.UseSimpleAssemblyNameTypeSerializer()
.UseRecommendedSerializerSettings()
.UseNinjectActivator(_kernel)
.UseNLogLogProvider()
.UseSqlServerStorage(“SQL connection string”, new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
QueuePollInterval = TimeSpan.Zero,
UseRecommendedIsolationLevel = true,
DisableGlobalLocks = true
});

yield return new BackgroundJobServer();
}

We have this in the Owin Startup.cs file:

app.UseHangfireDashboard("/hangfire", new DashboardOptions
{
Authorization = Enumerable.Empty()
});

We have these packages installed:

  • Hangfire.AspNet version=“0.2.0”
  • Hangfire.Core version=“1.7.28”
  • Hangfire.Ninject version=“1.2.0”
  • Hangfire.SqlServer version=“1.7.28”

Our website (hosted in the Azure Cloud) has 5 webservers that are load-balanced, so we normally have 5 Hangfire servers running.

If we just get this message, I think it means we did not lose a server:

  • Execution Worker is in the Failed state now due to an exception, execution will be retried no more than in 00:00:04

But if we get this message, I think it means we lost a server:

  • Dispatcher is stopped due to an exception, you need to restart the server manually. Please report it to Hangfire developers.

Our Hangfire servers are idle most of the time when we get these errors.

Does anyone have ideas on how we can solve this? Or any ideas on how we can troubleshoot this?

Thank You