Recurring jobs randomly being removed

question
,
1

We have been using Hangfire for approximately 3 years+ with no significant problem but the latest release seems to have caused us a significant issue.

We have 2 servers running in an ASP.NET IIS process with approximately 113 recurring jobs.

On startup we setup recurring jobs which in previous releases and including this one setup all the recurring jobs, this all still works.

Occasionally though we find all the recurring jobs are being removed with no idea why this would occur. I have performed a SQL trace to find out when it happens and conclude it happens with no real schedule.

Last evening it removed them all at 23:54, 00:02 and then 06:47!

The trace shows the SQL being executed as the following, one for each recurring job:

exec sp_executesql N’delete from [HangFire].Hash where [Key] = @key’,N’@key nvarchar(100)’,@key=N’recurring-job:jobid’

All other scheduled jobs remain intact and Hangfire continues to run.

We have code that will schedule and unschedule new recurring jobs using the API but these are only single jobs and we never actively remove all recurring jobs via the API so are a little stumped.

Has anybody experienced this behaviour?

I am going to do more investigation to find out the root cause and will update this once I have more information.

2

I have identified where the jobs are deleted but not the reason why they would not get recreated:

When our app starts we remove all the existing jobs and then add back the jobs we need from our system using Autofac.

Even though we reschedule the jobs after we have removed them there seems to be an opportunity for them to not get recreated, below is a stub of how our Hangfire is started:

jobStorage = GlobalConfiguration.Configuration.UseSqlServerStorage(ConfigurationManager.ConnectionStrings["Default"].ConnectionString + ";Application Name=Hangfire;", storageOptions).Entry;

using (var connection = jobStorage.GetConnection())
{
  foreach (var recurringJob in connection.GetRecurringJobs())
  {
      RecurringJob.RemoveIfExists(recurringJob.Id);
  }
}

if (_enabled)
{
   var options = new BackgroundJobServerOptions();
   options.WorkerCount = 2;
   backgroundJobServer = new BackgroundJobServer(options);

   // This is where new recurring jobs are created
   ConfigureJobs(container);
}

One minor change that was made was moving the removal of recurring jobs from the ConfigureJobs method to just after the jobStorage was created, we will be moving it back as it shouldn’t have been moved in the first place and this may be the cause although I do not understand what difference it would make.

3

Further to my previous post the code has been reverted but recurring jobs still randomly disappear. We have two servers running the ASP.NET server application and it seems that on app pool recycling is when the jobs are removed and subsequently not rescheduled.

We had upgraded Hangfire from version 1.6.17 to version 1.7.6 but not sure if this would have caused this issue so will need to investigate further.