Many workers execute my job

question
1

Hi all,
I setup my Hangfire server with 3 workers, and class BackgroundJob.Enqueue(methodCall); But now it’s create a job and execute by all workers. Can who tell me is I wrong in where, thanks!

1 Like
2

Hello, what job storage are you using?

3

Thank for your reply, I impl by my custom storage, it’s clone merge of sql server and mongodb storage.

4

Background job fetching is the most interesting part in a job storage implementation. You should know your storage intimately to allow only one worker to fetch a job. In SQL Server implementation is is achieved via a special column, FetchedAt and an atomic query:

update top (1) HangFire.JobQueue set FetchedAt = GETUTCDATE()
output INSERTED.Id, INSERTED.JobId, INSERTED.Queue
where FetchedAt is null

FetchedAt value is null, when a background job was not fetched by a worker yet, otherwise it is non-null. The query above finds a non-fetched job id, updates it and outputs the identifier atomically, that is why only one worker obtains a job.

What storage are you using? Perhaps there is a repository somewhere?

5

Thank you for your help, this is my impl for Dequeue method, please help me review this code, maybe it’s wrong at here.
http://www.mediafire.com/view/cbl8ba7hl75c7l1/DequeueMethod.txt

Thank you very much!

6

Here are the problem lines of code:

jobQueueDto = repository.Table.Where(queryArray[index].And(x => queues.Contains(x.Queue))).FirstOrDefault();

// Consider an year passes between these two lines. Will another
// worker fetch the same job?

if (jobQueueDto != null)
{
    jobQueueDto.FetchedAt = DateTime.UtcNow;
    repository.Update(jobQueueDto);
}

You need to wrap these lines in a repeatable read transaction, and no other worker will be allowed to change the FetchedAt value in-between.

7

Can I apply lock that is fine?

// Declare at class scope
private static readonly object syncLock = new object();

lock (syncLock)
{
jobQueueDto = repository.Table.Where(fetchConditions[index].And(x => queues.Contains(x.Queue))).FirstOrDefault();
if (jobQueueDto != null)
{
jobQueueDto.FetchedAt = DateTime.UtcNow;
repository.Update(jobQueueDto);
}
}

8

Nope, there may be different Hangfire instances running on different processes or even machines. Only distributed lock, or a repeatable read transaction will help us.

9

I setup 1 server only, then try test some see working fine, thanks you very much. I will think of your thing if have more instance or servers. Have a good day!

10

I got the same problem. 2 workers executed same job at same time. I’m using SQL Server for job storage. This is system logs:

2015-03-16 13:42:22,436 [Worker #12]: Background process: Xu ly du lieu … khach hang duoc chon
2015-03-16 13:42:22,436 [Worker #12]: Da luu du lieu cho khach hang co ID la 301378
2015-03-16 13:42:22,436 [Worker #12]: Da luu du lieu cho khach hang co ID la 301374
2015-03-16 13:42:22,436 [Worker #12]: Da luu du lieu cho khach hang co ID la 301390
2015-03-16 13:42:22,451 [Worker #12]: Da luu du lieu cho khach hang co ID la 301383
2015-03-16 13:42:22,451 [Worker #12]: Da luu du lieu cho khach hang co ID la 301401
2015-03-16 13:42:22,451 [Worker #12]: Da luu du lieu cho khach hang co ID la 290166
2015-03-16 13:42:22,451 [Worker #12]: Da luu du lieu cho khach hang co ID la 301354
2015-03-16 13:42:22,451 [Worker #12]: Da luu du lieu cho khach hang co ID la 301399
2015-03-16 13:42:22,467 [Worker #12]: Da luu du lieu cho khach hang co ID la 301362
2015-03-16 13:42:22,467 [Worker #12]: Da luu xong
2015-03-16 13:42:52,436 [Worker #4]: Background process: Xu ly du lieu … khach hang duoc chon
2015-03-16 13:42:52,451 [Worker #4]: Da luu du lieu cho khach hang co ID la 301378
2015-03-16 13:42:52,451 [Worker #4]: Da luu du lieu cho khach hang co ID la 301374
2015-03-16 13:42:52,451 [Worker #4]: Da luu du lieu cho khach hang co ID la 301390
2015-03-16 13:42:52,451 [Worker #4]: Da luu du lieu cho khach hang co ID la 301383
2015-03-16 13:42:52,451 [Worker #4]: Da luu du lieu cho khach hang co ID la 301401
2015-03-16 13:42:52,467 [Worker #4]: Da luu du lieu cho khach hang co ID la 290166
2015-03-16 13:42:52,467 [Worker #4]: Da luu du lieu cho khach hang co ID la 301354
2015-03-16 13:42:52,467 [Worker #4]: Da luu du lieu cho khach hang co ID la 301399
2015-03-16 13:42:52,467 [Worker #4]: Da luu du lieu cho khach hang co ID la 301362
2015-03-16 13:42:52,467 [Worker #4]: Da luu du lieu cho khach hang co ID la 301356
2015-03-16 13:42:52,467 [Worker #4]: Da luu xong