Hangfire stopped processing jobs after server backup

question
1

Firstly let me say I’m not sure is specifically a problem with Hangfire :smile:

Last night was the first time I left my application running overnight in our staging environment. After our server backup process ran overnight, Hangfire stopped processing any jobs, and they just sat in the processing queue for several hours, not actually doing anything (no cpu activity).

I tried restarting IIS, which had no effect. Then I restarted the server, which had no effect. Then on the SQL Server (a separate server) I dropped the the whole Hangfire schema and reinstalled it. My application then started processing jobs again.

I have the Hangfire schema in a separate database to the main application database, and one of the server backup tasks is to kick off sql server maintenance (reindexing etc) of the main application database.

So my jobs in Hangfire will want to write to the main application database with results of what they process, so when the maintenance is in progress it would be unavailable, which is what I think has caused the problem.

What I don’t understand is why dropping the Hangfire schema and reinstalling it allowed the jobs to start processing again.

Any advice on how to investigate and prevent this from happening in the future would be appreciated, especially as I’m not sure if it was Hangfire’s fault or my own.

2

Ran all weekend and not encountered the same problem.

3

What I don’t understand is why dropping the Hangfire schema and reinstalling it allowed the jobs to start processing again.

This is very strange, but the cause can be outside of Hangfire. We are continously running it over a couple of months (slightly less than one year). However, this issue should be investigated. Do you have any logs? Application error can explain this, but we should catch it first.

4

Actually think that this particular problem is a bit of a red herring, because I think it coincided with some network issues we were having.

I don’t think this needs pursuing any further, unless it happens again, and even then, I would be more inclined to look for a problem anywhere else than with Hangfire :smile: