Preventing dequeue during check to shutdown


#1

Let me start off by saying we use Hangfire very oddly. I’m aware of this, I am trying to take steps to solve it, but in the mean time I have somethings I want to figure out just how feasible they are.

We have a check to see if there’s any more work left to do on our Hangfire server instance. If we cannot find jobs assigned to that server, the server shuts down. My concern is that between the check for more work and the time we shut down, it’s possible that another job may get queued and picked up by the server, creating a race condition. I figured we could use a mutex, hold the mutex during the check (and potential shutdown), and have that mutex respected during the dequeue operation as well so no new work could get dequeued.

I saw that SqlServerStorage has a public QueueProviders collection, and thought maybe I could implement IPersistentJobQueueProvider, add my provider to QueueProviders, and have the provider respect the mutex before returning the job queue or something similar. So, I can add to the QueueProviders collection, but there’s no way to fall through to the underlying implementation of JobQueue / JobQueueProvider as all the classes there are internal and there’s nothing that can signal to the QueueProviderCollection to use the established default queue provider.

Given the gap between when I check for remaining work and when I shut down my server, how do I prevent my server from grabbing new work that would lead to cancelled jobs on shutdown?