Multiple workers on one job

bug?
1

Hi, I’m using Hangfire 1.5.3.
Two questions:

As you can see in the screenshot it looks as though a job is being processed by multiple workers.
How is that possible?

Also, I see two numbers next to Enqueued.
The queue seems to be empty so what does the first number mean?

hangfire.png1174×759 64.2 KB

2

What storage are you using?

3

SQL Server storage:
var options = new SqlServerStorageOptions { }; Hangfire.GlobalConfiguration.Configuration.UseSqlServerStorage("EasyAdsContext", options);

4

At a glance it looks like your job is taking longer than the schedule. e.g. you setup an every 1 minute job, the job takes 2 minutes and now you’re seeing a backup of jobs. Since you have available workers it just happily picks up the job since hangfire just sees “hey i need to do this”

I actually encountered this in production the other day. I’m only using a single server so i was able to work around this trivially by updating my job to be similar to:

private static volatile bool _isRunning;

public void MyJobAction()
{
    if (_isRunning) return;

    try
    {
        _isRunning = true;

        //i do stuff
    }
    finally
    {
        _isRunning = false;
    }
}

If this is the case for you, either you need to synchronize your work with something distributed not just a volatile bool or just turn down the scheduling of the work so jobs finish before the next one starts. Or just not care, if not caring is a valid option.

1 Like
5

Hi Chris, thanks for your answer but I don’t really understand your explanation.
If a schedule would start the same job before a previous one is finished won’t it just start a new job in hangfire?
I think I have sometime seen the same job with the same parameters in the processing view.

The problem I’m seeing is that the number of available workers decreases over time. I have 20 workers and after a couple of hours I’ll have 16 or something. And a day later I’ll have around 3. Now I’ll just have to restart the web app to get back to 20 workers. It looks like workers stay busy in the background. I’ve found nothing in the logging.

6

@marcselman Did you ever get to the bottom of this ?

Getting the same behavior in 1.6.4

Pasted image877×662 23.7 KB

7

@groveslu, what storage are you using, and what version?

8

Hello, i’m using sqlStorage with ServiceBusQueues

Hangfire.Core - 1.6.4
Hangfire.SqlServer - 1.6.4
Hangfire.Azure.ServiceBusQueue - 2.1.1

9

Good, looks like the problem is caused by Hangfire.Azure.ServiceBusQueue. What options are you using? Could you post here the configuration logic?

10 
            var sqlStorage = new SqlServerStorage(CONNECTION_STR);

            
            Action<QueueDescription> configureAction = qd =>
            {
                qd.MaxSizeInMegabytes = 5120;
                qd.DefaultMessageTimeToLive = new TimeSpan(0, 1, 0);
            };

            
            sqlStorage.UseServiceBusQueues(new ServiceBusQueueOptions
            {
                ConnectionString = QUEUE_CONNECTION_STR,
                Configure = configureAction,
                Queues = QueueNames()
            });

// QueueNames() is just a array of strings

            GlobalConfiguration.Configuration.UseStorage(sqlStorage);
11

I have detected this situation using:

  • Hangfire.core 1.5.9
  • Hangfire.MySqlStorage 1.0.3

I understand that the expected behaviour is: a Job is supposed to be executed by ONE thread (except in the retry case that could be executed by a different thread), is this correct?

12

Thanks you so much.

I’ve been searching for this for days