Hangifre stops executing jobs after I delete a job

I’m using SQL Server and Hangfire to process background jobs, and it works great.

However, Hangfire has started to hang and not execute any jobs. The hangfire dashboard show new jobs as enqueued and nothing processing.

I can reproduce this issue, and it happens whenever I cancel/delete a job. We use the method BackgroundJob.Delete() to cancel jobs.

Restarting Hangfire fixes the problem.

Has anyone seen this issue? Could this be down to us not handling cancellation tokens properly?

When background processing is stuck, the simplest way is to take a minidump and see (or post them here) all the stack traces. To make it even more simpler, I recently wrote a small utility that lists all the stack traces of a running process to the console without pausing it: https://github.com/odinserj/stdump. Can you run the following command and post here the output?

stdump-x64 process_id > output.txt