Setup: Hangfire 1.6.17
We are using SQL Server and MSMQ.
Our Hangfire environment has gotten into a bad state. on the dashboard, we see this:
Note the “15/6” next to the “Enqueued” label, but then there are 659 items in the MEDIUMPRIORITY queue. When you drill into this queue, you see that all of them (apart from 15), are “Job expired”:
If you look in the DB, there is no record in the Job table, for all of the expired ones.
When we enqueue new items on the MEDIUMPRIORITY queue, they go to the back and are being blocked. The “expired” (non-existent) job items are very slowly being cleared out of the queue, but essentially are holding up the new job items for hours, until the expired ones are fully cleared out.
So my questions are:
-How could this have happened?
-Where is Hangfire sourcing these 659 job items from, if they are not in the DB?
-How can we clear these “expired” items out of whatever storage they are in, so that newly enqueued jobs aren’t blocked and can be processed immediately?