Handling long running tasks (+ long invisibility timeout) + server restarts

question
13

I know this is a pretty old topic at this point, but I’ve been running into a similar issue. I wanted to throw my contribution to working around jobs which have been aborted due to server shutdown. If automatic retries are disabled (Attempts = 0), or the jobs fails due to server shutdown and is beyond the maximum number of attempts, you can run into this issue. Unfortunately for us, this was causing new jobs to not start processing until the aborted jobs were either manually deleted or re-queued.

Basically, I took the following approach to automatically handle aborted jobs: during startup and after initializing the BackgroundJobServer, I use the MonitoringApi to get all of the currently processing jobs. If there are any, I loop through each and call BackgroundJob.Requeue(jobId). Here’s the code, for reference:

var monitor = Hangfire.JobStorage.Current.GetMonitoringApi();                
if (monitor.ProcessingCount() > 0)
{                    
    foreach (var job in monitor.ProcessingJobs(0, (int)monitor.ProcessingCount()))
    {
        BackgroundJob.Requeue(job.Key);                        
    }
}