Best way to ensure that Hangfire is always running?

I’m using the latest stable version of hangfire.io in an MVC 5 application and need to ensure that my recurring background tasks get kicked off 24x7 (all day every day).

I’m configuring the site on IIS 7.5 to be always running so that when the app pool is shutdown or the server is rebooted, then the application pool will be guaranteed by IIS to be started up as soon as possible.

The problem with the above is that OWIN, and therefore hangfire itself, won’t be kicked off until the first user request is received by the site. My site is largely an admin tool that must do work throughout the day, but people aren’t really visiting unless there’s a problem.

So, the app pool gets restarted, which is great, but none of my hangfire worker tasks are getting run until the first interactive user shows up.

I’ve solved this in the past by writing a simple pinging service which keeps the site up and running. While this is simple, I’m trying get rid of these extra moving parts and just use the configuration options that IIS gives me.

I’m considering implementing some custom logic in a Preload handler (serviceAutoStartProvider) to bootstrap hangfire if the site is getting started back up through the autoStart path.

Am I heading down the right path here, or did I stray? Is there a different approach that others have used successfully for this?

Thanks!

1 Like

I don’t have the answer to this question, but I’d really like to know what it is. I am in a similar situation, my site is an admin tool that should run 24/7.

It does have a user facing signalr powered dashboard (looks great on the big status monitor in the ops room) so that does slightly negate the issue for me.

Because my site is auto-deployed by Jenkins, I have also considered having a post-deployment task which requests the main page.

This will solve your problem. There is a lot of burden in IIS options related to long-running ASP.NET web applications. Unfortunately they aren’t well documented, and description is spread across different blogs. It is very annoying thing to compile articles together – very often the same terms telling about different things. And very often multiple options messed together and it is hard to understand the responsibility of each other.

So let’s discuss these options. As a disclaimer, I should warn you that the description below is based on my opinions and investigations. I’ll try to do my best, by I’m not working in ASP.NET team and sometimes don’t have access to the sources for deeper details (but thanks to MS for http://referencesource.microsoft.com, it helped a lot). I’ll move from bottom (but will skip the automatic startup of W3SVC and WAS services) to the top (your application).

Enable Automatic Startup

Scope: IIS / Application Pool, Default: Enabled.

This option is turned on by default, I’ve placed it here for the completeness.

… indicates to the World Wide Web Publishing Service (W3SVC) that the application pool should be automatically started when it is created or when IIS is started.
IIS configuration reference

How to apply: http://technet.microsoft.com/en-us/library/cc772112(v=ws.10).aspx

Enable Always Running Mode

Scope: WAS / Application Pool, Default: Disabled, IIS Version: ≥ 7.5.

Observed behavior: tells WAS to start all worker processes owned by application pool immediately – there are W3WP.exe processes in Windows Task Manager. It is interesting that worker processes are being started even when W3SVC service is stopped.

By default worker processes are being started only after first request is being made to the corresponding Web Application.

Specifies that the Windows Process Activation Service (WAS) will always start the application pool. This behavior allows an application to load the operating environment before any serving any HTTP requests, which reduces the start-up processing for initial HTTP requests for the application.
IIS configuration reference

How to apply: usually described along with Autostart or WarmUp features (see below).

Enable Automatic Startup of Your Application

If you turned on the features described above, you have only Application Pool and a set of Worker Processes running. Your application is ready to receive requests, but this is not enough for you – application initialization logic is not being called automatically.

There are two different approaches to start your application automatically. Despite of they serve for the same purposes, their implementation is different to each other.

Service Autostart Providers (Autostart)

Scope: WAS / Worker Process, Default: Disabled, IIS Version: ≥ 7.5

… allows developers to specify assemblies that perform initialization tasks before any HTTP requests are serviced.
IIS Configuration Reference

Service Autostart Providers allow Windows Activation Service to call a method in your code automatically each time your Application Pool is being started or recycled. This is the recommended method to keep your application running all the time.

How to apply: http://weblogs.asp.net/scottgu/auto-start-asp-net-applications-vs-2010-and-net-4-0-series

Unfortunately, you are not able to use Hangfire’s OWIN bootstrapped methods (and OWIN Startup class itself), because the ASP.NET application is not being initialized during the Service Autostart Provider initialization – neither Application_Start method nor OWIN Startup’s Configure method is being called. Please initialize the storage and server classes manually:

public class PreWarmCache : System.Web.Hosting.IProcessHostPreloadClient 
{
    private static readonly object LockObject = new object();
    private static bool _started;

    // Prevent server from being collected by GC
    private static BackgroundJobServer Server;

    public void Preload(string[] parameters) 
    {
        lock (LockObject)
        {
            // Who knows the implementation? It is better
            // to ensure that initialization method is being 
            // called only once programmatically. 
            if (_started) return;
            _started = true;

            JobStorage.Current = new SqlServerStorage("connection_string");
            Server = new BackgroundJobServer();
            Server.Start();
        }
    }
}

Application Initialization Module (WarmUp)

Scope: ??? / Worker Process, Default: Disabled, IIS Version: ≥ 7.5

IIS Application Initialization for IIS 7.5 enables website administrators to improve the responsiveness of their Web sites by loading the Web applications before the first request arrives. By proactively loading and initializing all the dependencies such as database connections, compilation of ASP.NET code, and loading of modules, IT Professionals can ensure their Web sites are responsive at all times even if their Web sites use a custom request pipeline or if the Application Pool is recycled. While an application is being initialized, IIS can also be configured to return an alternate response such as static content as a placeholder or “splash page” until an application has completed its initialization tasks.
www.iis.net

So, you simply need to install the module, enable it by clicking a button in IIS 8 or add a new config option in applicationHost.config in IIS 7.5 and leave your initialization logic in global.asax.cs file in Application_Start method. So, you can use Hangfire’s OWIN bootstrapper methods.

Sounds cool, but I had different problems approximately a year ago when I tried to use it in my web application. There are a lot of articles in the Internet that describe how to install and use this module, but none of them describe how it actually works. So I’ll try to remember the details of the problem without references. The details can be sometimes wrong, so feel free to play with them and send me feedback.

The main problem – Application Initialization Module is unsuitable for running web application continuously. As far as I remember, the automatic request is being triggered only on Worker Process start-up (i.e. only on WAS service startup with startMode="AlwaysRunning" application pool). Recycling events, such as application deployments (including WebDeploy), configuration changes, do not trigger the initialization request.

As far as I remember there was an article that described the way how this module works. And this problem was caused that this module is just an HTTP Module for Integrated Pipeline. IIS 7.5 (or 7.0) added a new event that is being triggered when Worker Processes are being started. And since recycling process does not restart worker processes, the initialization request is not being issued, and your application initialization logic is not being called.

How to apply: http://blogs.msdn.com/b/benjaminperkins/archive/2014/01/07/configure-the-iis-application-initialization-module.aspx

Mixing Autostart & WarmUp together

Do it very carefully, enable logging and look for duplicate Hangfire Server started rows without Hangfire Server stopped between them. If this happen, please, add a comment in this post and turn off the WarmUp feature – I can’t remember the details, but had some issues with this setup.

There are some articles in the Internet that do not make a difference between these features, usually they mess up preloadEnabled="true" and serviceAutostartEnabled="true" together. Read them carefully!

2 Likes

Thanks for the complete answer…but I noticed that in the.net core you don’t need to use IProcessHostPreloadClient anymore and these steps are enough:
1- install the Application Initialization module
Making ASP.NET application always running — Hangfire Documentation
2- In IIs set application pool under which the application runs to:

a. NET CLR version: .NET CLR Version v4.0.30319
Normally, for a .NET Core app, you’d use No managed code, but if you do that, the application preload option won’t work.
b. Managed pipeline mode: Integrated

3- Right-click on the same application pool and select “Advanced Settings”. Update the following values:

Set start mode to “Always Running”.

Setting the start mode to “Always Running” tells IIS to start a worker process for your application right away, without waiting for the initial request.

Set Idle Time-Out (minutes) to 0.
4-In IIs open Advanced Settings on your application and Set Preload Enabled = True:
5- In IIS go to the Configuration Editor on your app, and navigate to system.webServer/applicationInitialization. Set the following settings:

a. doAppInitAfterRestart: True
b. ( in Config Editor system.webServer/applicationInitialization)Open up the Collection. On the next window, click Add and enter the following:

a. hostName: {URL host for your Hangfire application}
b. initializationPage: {path for your Hangfire dashboard, like /hangfire}