Some of the servers don't have heartbeat reported within the last minute hangfire

bug?
1

Server automatically stopping when job is running and job is being aborted. It is creating multiple servers.

if (_backgroundJobServer == null)
{
HostingEnvironment.RegisterObject(this);
_backgroundJobServer = new BackgroundJobServer();
}

2

You might have to show more code than that. It does however sound more like your code is being run in parallel or restarted often.
So tell how you are even running this code. Since if you are using an IIS, it is very likely to be a configuration issue with that.

You can check these nicely confusing guides: https://docs.hangfire.io/en/latest/background-processing/index.html

Personally I would suggest just running Hangfire servers in a Windows Service, it is much easier to control and setup. I simply don’t trust the IIS to handle a task managing server. The IIS is designed to handle webpages and webservices.

3

@odinserj @Deantwo Basically my issue is hangfire server is stopping once the deployment is completed. It is aborting the current long running jobs. I just trying to make sure server is always running.
FYI it’s a .net web application.

It would be great if can get the sample solution for this

public class HangfireBootstrapper : IRegisteredObject
{
public static readonly HangfireBootstrapper Instance = new HangfireBootstrapper();

    private readonly object _lockObject = new object();

    private  BackgroundJobServer _backgroundJobServer;

    private HangfireBootstrapper()
    {
    }

    public void Start()
    {
        lock (_lockObject)
        {
            if (_backgroundJobServer == null)
            {
                HostingEnvironment.RegisterObject(this);
                Schedule.GetHangfireConfiguration();
                _backgroundJobServer = new BackgroundJobServer(new BackgroundJobServerOptions
                {
                    CancellationCheckInterval = TimeSpan.FromSeconds(5) // Default value
                });
            }
        }
    }

    public void Stop()
    {
        lock (_lockObject)
        {
            if (_backgroundJobServer != null)
            {
                _backgroundJobServer.Dispose();
            }

            HostingEnvironment.UnregisterObject(this);
            Log.CloseAndFlush();
        }
    }

    void IRegisteredObject.Stop(bool immediate)
    {
        Stop();
    }
}
4

It sounds more like you have designed your background jobs badly then. Having one long running job is not normally a good idea.

You have to remember that your hangfire job can be canceled at any time really. So you need to design them in such a way that they don’t run too long and can continue where they left off if they are restarted halfway through.

For example if your background job is contacting 100 databases with the exact same commands, it is better to make one job create 100 separate jobs. That way each of those separate jobs can fail and be requeued without the 99 other jobs being affected.

If it is instead a background job that has to download a bunch of files and do something with them, you might have to save in a database which step of the process you are in. That way if the job is cancelled or fails halfway through and is restarted, it can skip all the work it has already done.
https://docs.hangfire.io/en/latest/best-practices.html#make-your-background-methods-reentrant

5

@Deantwo you mean we need to requeue the jobs when server stop and start again . Currently we running almost 10 long running jobs which approximately takes 1 hour to run. What if use Windows service do we need to stop and start on every deployment completion

6

No. Hangfire does that itself. What you need to do is make sure that your background jobs can handle being stopped halfway through and then started again.

Those are some long jobs.

You probaby can’t deploy without stopping what is running. But Hangfire should automatically re-enqueue the jobs that it didn’t finish once the server starts up again.
A Windows Service server likely wouldn’t be any different in that regard.