Scheduled job starts processing twice

#1

Something is causing a scheduled job to start twice. below is the state table that shows job 19 starts processing twice.

double - start.png1153x409 13 KB

I use additional logging at the start and end of a class method that is executed when the job runs and it shows the method executed twice - once from 5:00:00 to 5:01:24 and again from 5:08:51.310 to 5:08:51.360. ( The short run time on the second execution is not relevant because the job exited when it could not locate a file that during the first run )

There was nothing in the application log. Why is the process be starting twice or what is the next step to debug?

We are running hangfire 1.6.8 on .net 4.6.1 with SQL Server 2016

— more relevant details —

We are running the dashboard on two servers and it looks like the job was actually run one on each server.

Servers tab - what are the servers

servers tab.PNG1209x571 11.2 KB

Jobs tab - shows multiple processing - one by each server

multiple processing.PNG876x837 25.4 KB

#2

The possible reason is an automatic application pool recycle. In this case background job is requeued, and gets processed by a new worker. By the way, there are other circumstances under which your background jobs may be executed twice or more, all the background job code should be idempotent and don’t lead to errors in such cases.

#3

I appreciate that a background job needs to be re-runnable in the event of failure, but looking at the state table the job would have been in a “Processing” state when it was picked up the second time. What are the conditions that allow a “Processing” job to be started again?

#4

Wanted to say my issue with multiple executions of a job was due to an app pool recycling after 30 minutes. I tried everything, invisible timeout, data context timeout, AutomaticRetry(Attempts = 0), SkipConcurrentExecutionAttribute()…I was going bonkers, then I found this thread.

During very early development stages of this WCF DataServices oData API we had set the app pool to recycle after 30 minutes on our test server to avoid locking up. The issues were long since corrected, but we forgot the recycle in IIS.

Thanks,
Mike

#5

I realize this is a bit of a zombie thread, but what exactly are the “circumstances under which your background jobs may be executed twice or more”? This is occurring for us and it’s something that we need to run exactly once for a given time period. If it’s something we can’t work around we’d like to know that so we can look at other ways to achieve our goal.

#6

@siprager I am facing the same issue have you found any workround for it?