We have a Windows Service that is running Hangfire and follows the pattern shown here: https://docs.hangfire.io/en/latest/background-processing/processing-jobs-in-windows-service.html
That service runs fine and it’s been in production for a couple years.
The problem we have is when patching is done in our lower environment and things are being restarted. If the server that is running our Hangfire service comes back up while the database is unavailable, then the service fails to start. Thankfully, this only happens in the lower lane where we don’t have clustering for our database. We’d like to make our service a bit more resilient anyways.
The specific line that fails is this one in the constructor:
GlobalConfiguration.Configuration.UseSqlServerStorage(“connection_string”);
We can of course configure Windows to keep restarting the service on failure (though admittedly, I don’t know if this covers startup failures).
Alternatively, we’d like to let the service startup, and have our service internally retry to setup the Hangfire bits. I imagine what we’d do is we would catch the exception and start a periodic timer that would keep retrying the UseSqlServerStorage until it succeeds, and only once it succeeds, would we then construct the background job server:
_server = new BackgroundJobServer();
Thoughts?