How to prevent App Restarts from throwing Distributed Lock Timeout Exceptions

question
1

The app I’m using caches most of the data it needs and pulls in all the data on app startup. The cache is refreshed on various intervals depending on the data being refreshed. I’m using hangfire to call all the cache refresh methods in a fire and forget manner on app startup and then, after a five minute period, setting up scheduled tasks to refresh each cache item on its own interval.

I’m running into a lot of DistributedLockTimeoutExceptions when I publish or restart the app via IIS because on app startup hangfire is executing the jobs that didn’t finish in the previous session while I am calling the same jobs in my app startup sequence.

Is there a way to tell hangfire to ignore unfinished jobs on app startup and let me requeue everything each app startup. Or is there a way that hangfire to start all my jobs on app startup instead of just the ones executing when the app restarted?

I have tried to delete all the jobs from the previous session but it does not prevent jobs that didn’t finish executing from automatically starting. See method snippet below.

Thanks.

-Kevin

public static void WipeJobs()
    {
        try
        {
            logger.Info("Clear Pending Hangfire Jobs start");
            //On new app startup remove any previous recurring jobs since they will be setup again
            using (IStorageConnection hfCon = JobStorage.Current.GetConnection())
            {
                List<RecurringJobDto> recurringJobs = StorageConnectionExtensions.GetRecurringJobs(hfCon);
                {
                    foreach (RecurringJobDto job in recurringJobs)
                    {
                        RecurringJob.RemoveIfExists(job.Id);
                    }
                }
            }
            //Remove any other enqueued jobs
            var monitor = JobStorage.Current.GetMonitoringApi();
            var toDelete = new List<string>();
            foreach (QueueWithTopEnqueuedJobsDto queue in monitor.Queues())
            {
                for (var i = 0; i < Math.Ceiling(queue.Length / 1000d); i++)
                {
                    monitor.EnqueuedJobs(queue.Name, 1000 * i, 1000)
                        .ForEach(x => toDelete.Add(x.Key));
                    monitor.ScheduledJobs(1000 * i, 1000).ForEach(x => toDelete.Add(x.Key));
                }
            }
            foreach (var job in toDelete)
            {
                BackgroundJob.Delete(job);
            }
            logger.Info("Clear Pending Hangfire Jobs end");
        }
        catch (Exception e)
        {
            logger.Fatal(string.Format("Removal of Previous Sessions Jobs failed, Error {0}", e.Message), e);
            throw;
        }
    }