Hangfire queueing recurring jobs multiple times?

Tags: #<Tag:0x00007f185da752c0> #<Tag:0x00007f185da751f8> #<Tag:0x00007f185da75130>

I’ve been noticing a lot of duplicated jobs being processed in my project thats running hangfire.

I’m running 1.6.17, using MySQL storage, with a single server pointing to a single database. I have 168 recurring jobs, of which I’ve confirmed none are duplicate, and yet somehow a few times a day I run into some jobs being queued up and run at exactly the same time. Lets say job “A” sends a message to a user, I will end up seeing 2 of the same message being sent to the user. When I check the list of succeeded jobs, I see two of the same job having run at exactly the same time, in the same server, queue, and worker, but both jobs have a different job ID (always 1 job ID higher than the other since they’re apparently started at the exact same time).

I’m really puzzled why this is happening, I can’t see any issues in my code as to why they would run twice, and it seems to happen very randomly. For the same recurring job that runs every ~15 minutes, it may happen 4 or 5 times sporadically throughout a day and then not happen the rest of the time.

I’ve seen some other threads on here and on github where people have seen 1 job ID show “processing” multiple times, but this is happening on 2 separate jobs. Some of the troubleshooting mentioned making sure processes aren’t failing (which they aren’t), workers aren’t recycling in IIS (they aren’t), and that I’m not running two instances pointing at the same DB (I’m not).

I am not sure why this happens. It seems when jobs failed for whatever reason, it attempts to run again even if the next scheduled job is running. I decided to implement semaphore slim so that they are not all running the same task.

Not the same. These jobs aren’t failing or being retried. Retries are a normal feature of Hangfire.