Best way to ensure that Hangfire is always running?

question
3

This will solve your problem. There is a lot of burden in IIS options related to long-running ASP.NET web applications. Unfortunately they aren’t well documented, and description is spread across different blogs. It is very annoying thing to compile articles together – very often the same terms telling about different things. And very often multiple options messed together and it is hard to understand the responsibility of each other.

So let’s discuss these options. As a disclaimer, I should warn you that the description below is based on my opinions and investigations. I’ll try to do my best, by I’m not working in ASP.NET team and sometimes don’t have access to the sources for deeper details (but thanks to MS for http://referencesource.microsoft.com, it helped a lot). I’ll move from bottom (but will skip the automatic startup of W3SVC and WAS services) to the top (your application).

Enable Automatic Startup

Scope: IIS / Application Pool, Default: Enabled.

This option is turned on by default, I’ve placed it here for the completeness.

… indicates to the World Wide Web Publishing Service (W3SVC) that the application pool should be automatically started when it is created or when IIS is started.
– IIS configuration reference

How to apply: Configure Automatic Startup for an Application Pool (IIS 7) | Microsoft Learn

Enable Always Running Mode

Scope: WAS / Application Pool, Default: Disabled, IIS Version: ≥ 7.5.

Observed behavior: tells WAS to start all worker processes owned by application pool immediately – there are W3WP.exe processes in Windows Task Manager. It is interesting that worker processes are being started even when W3SVC service is stopped.

By default worker processes are being started only after first request is being made to the corresponding Web Application.

Specifies that the Windows Process Activation Service (WAS) will always start the application pool. This behavior allows an application to load the operating environment before any serving any HTTP requests, which reduces the start-up processing for initial HTTP requests for the application.
– IIS configuration reference

How to apply: usually described along with Autostart or WarmUp features (see below).

Enable Automatic Startup of Your Application

If you turned on the features described above, you have only Application Pool and a set of Worker Processes running. Your application is ready to receive requests, but this is not enough for you – application initialization logic is not being called automatically.

There are two different approaches to start your application automatically. Despite of they serve for the same purposes, their implementation is different to each other.

Service Autostart Providers (Autostart)

Scope: WAS / Worker Process, Default: Disabled, IIS Version: ≥ 7.5

… allows developers to specify assemblies that perform initialization tasks before any HTTP requests are serviced.
– IIS Configuration Reference

Service Autostart Providers allow Windows Activation Service to call a method in your code automatically each time your Application Pool is being started or recycled. This is the recommended method to keep your application running all the time.

How to apply: ScottGu's Blog - Auto-Start ASP.NET Applications (VS 2010 and .NET 4.0 Series)

Unfortunately, you are not able to use Hangfire’s OWIN bootstrapped methods (and OWIN Startup class itself), because the ASP.NET application is not being initialized during the Service Autostart Provider initialization – neither Application_Start method nor OWIN Startup’s Configure method is being called. Please initialize the storage and server classes manually:

public class PreWarmCache : System.Web.Hosting.IProcessHostPreloadClient 
{
    private static readonly object LockObject = new object();
    private static bool _started;

    // Prevent server from being collected by GC
    private static BackgroundJobServer Server;

    public void Preload(string[] parameters) 
    {
        lock (LockObject)
        {
            // Who knows the implementation? It is better
            // to ensure that initialization method is being 
            // called only once programmatically. 
            if (_started) return;
            _started = true;

            JobStorage.Current = new SqlServerStorage("connection_string");
            Server = new BackgroundJobServer();
            Server.Start();
        }
    }
}

Application Initialization Module (WarmUp)

Scope: ??? / Worker Process, Default: Disabled, IIS Version: ≥ 7.5

IIS Application Initialization for IIS 7.5 enables website administrators to improve the responsiveness of their Web sites by loading the Web applications before the first request arrives. By proactively loading and initializing all the dependencies such as database connections, compilation of ASP.NET code, and loading of modules, IT Professionals can ensure their Web sites are responsive at all times even if their Web sites use a custom request pipeline or if the Application Pool is recycled. While an application is being initialized, IIS can also be configured to return an alternate response such as static content as a placeholder or “splash page” until an application has completed its initialization tasks.
– www.iis.net

So, you simply need to install the module, enable it by clicking a button in IIS 8 or add a new config option in applicationHost.config in IIS 7.5 and leave your initialization logic in global.asax.cs file in Application_Start method. So, you can use Hangfire’s OWIN bootstrapper methods.

Sounds cool, but I had different problems approximately a year ago when I tried to use it in my web application. There are a lot of articles in the Internet that describe how to install and use this module, but none of them describe how it actually works. So I’ll try to remember the details of the problem without references. The details can be sometimes wrong, so feel free to play with them and send me feedback.

The main problem – Application Initialization Module is unsuitable for running web application continuously. As far as I remember, the automatic request is being triggered only on Worker Process start-up (i.e. only on WAS service startup with startMode="AlwaysRunning" application pool). Recycling events, such as application deployments (including WebDeploy), configuration changes, do not trigger the initialization request.

As far as I remember there was an article that described the way how this module works. And this problem was caused that this module is just an HTTP Module for Integrated Pipeline. IIS 7.5 (or 7.0) added a new event that is being triggered when Worker Processes are being started. And since recycling process does not restart worker processes, the initialization request is not being issued, and your application initialization logic is not being called.

How to apply: The Best C# Programmer In The World - Benjamin Perkins | articles about C# and numerous other technologies

Mixing Autostart & WarmUp together

Do it very carefully, enable logging and look for duplicate Hangfire Server started rows without Hangfire Server stopped between them. If this happen, please, add a comment in this post and turn off the WarmUp feature – I can’t remember the details, but had some issues with this setup.

There are some articles in the Internet that do not make a difference between these features, usually they mess up preloadEnabled="true" and serviceAutostartEnabled="true" together. Read them carefully!

2 Likes