Recurring job not enqueued

bug?
, , ,
1

We are using SQL Server storage and just scheduled ~300 recurring jobs to run over a period of 30 minutes, each with a specific date and time (so they are executed once per year) - the cron looks like 12 15 30 12 *. Strangely, we noticed that roughly half of them are not moved to the enqueued state. For example for December 30th, 15:12 we had 14 jobs and 7 of them were enqueued and processed while the rest were not and their NextExecution was recalculated for 2015-12-30. Note that those are fairly lightweight jobs - PerformanceDuration for the successful ones was ~220, so I don’t think there was an issue with the server being overloaded. Has anyone had similar experience and what have you done to fix it?

Optional reading: To put our setup into context, we basically want to schedule a background job that we can cancel or modify and since scheduled jobs don’t have ID-s, we decided to use recurring jobs that will be executed once - they are deleted after execution. If there’s a more elegant way to do it, we’re open to ideas, but we’d still want to know why our current setup is misbehaving.

1 Like
2

@nirinchev, this looks very strange. I’ll look at it after holidays (Jan, 12). Recurring jobs are meant to be used as static records and are not meant to be created dynamically.

All background jobs have unique identifiers you can use to delete or re-queue them. You can obtain it in the following way:

var id = BackgroundJob.Enqueue(/* ... */);
3

Hi @odinserj, thanks. It seems that my issue is similar to the original issue described in https://github.com/HangfireIO/Hangfire/issues/247. I decided to go with dynamic recurring jobs because they have out of the box support for custom Id’s and I didn’t want to persist the Id returned from BackgroundJob.Enqueue. After some more digging, I noticed you proposed a solution in https://github.com/HangfireIO/Hangfire/issues/181, so I will see if I’ll be able to apply it easily.

4

Ok, Hangfire needs a bunch of small how-tos :smile: But recurring jobs need further investigation, thanks for pointing it out!

5

I am seeing records never leaving the Enqueued state in the Job table.

I am using MSMQ. I have a log entry that is being written write before the method exits, but I the state is never updated.

I turned on the journal in the queue. I see that it get pulled off the queue.

Here are all the SQL statements processed for this job in the order they were executed.

exec sp_executesql N'

insert into HangFire.Job (InvocationData, Arguments, CreatedAt, ExpireAt)
values (@invocationData, @arguments, @createdAt, @expireAt);
SELECT CAST(SCOPE_IDENTITY() as int)',N'@invocationData nvarchar(4000),@arguments nvarchar(4000),@createdAt datetime,@expireAt datetime',@invocationData=N'{"Type":"NLS.Api.Assets.Analyzer.IPowerDeviceDerivationService, nls.api.asset, Version=3.2.1.0, Culture=neutral, PublicKeyToken=null","Method":"ProcessByPowerAssetSafely","ParameterTypes":"[\"System.Guid, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\"]","Arguments":"[\"\\\"767363fe-9aa1-e411-8457-00090ffe0001\\\"\"]"}',@arguments=N'["\"767363fe-9aa1-e411-8457-00090ffe0001\""]',@createdAt='2015-01-22 06:08:28.143',@expireAt='2015-01-22 07:08:28.143'

exec sp_executesql N'
insert into HangFire.JobParameter (JobId, Name, Value)
values (@jobId, @name, @value)',N'@jobId nvarchar(4000),@name nvarchar(4000),@value nvarchar(4000)',@jobId=N'12',@name=N'CurrentCulture',@value=N'"en-US"'

exec sp_executesql N'
insert into HangFire.JobParameter (JobId, Name, Value)
values (@jobId, @name, @value)',N'@jobId nvarchar(4000),@name nvarchar(4000),@value nvarchar(4000)',@jobId=N'12',@name=N'CurrentUICulture',@value=N'"en-US"'

exec sp_executesql N'
insert into HangFire.State (JobId, Name, Reason, CreatedAt, Data)
values (@jobId, @name, @reason, @createdAt, @data);
update HangFire.Job set StateId = SCOPE_IDENTITY(), StateName = @name where Id = @id;',N'@jobId nvarchar(4000),@name nvarchar(4000),@reason nvarchar(4000),@createdAt datetime,@data nvarchar(4000),@id nvarchar(4000)',@jobId=N'12',@name=N'Enqueued',@reason=NULL,@createdAt='2015-01-22 06:08:28.147',@data=N'{"EnqueuedAt":"2015-01-22T06:08:28.1435488Z","Queue":"default"}',@id=N'12'

exec sp_executesql N'update HangFire.Job set ExpireAt = NULL where Id = @id',N'@id nvarchar(4000)',@id=N'12'

exec sp_executesql N'select InvocationData, StateName, Arguments, CreatedAt from HangFire.Job where Id = @id',N'@id nvarchar(4000)',@id=N'12'
6

@nirinchev, your bug was resolved in 1.3.2, please see the release notes.

@youngcm2, looks like your issue related to MSMQ. Do you use transactional queues?

7

@odinserj, yes, already updated :slight_smile: Awesome work :+1:

8

@youngcm2, I’ve just released Hangfire 1.3.3 with sucked jobs in enqueued state with MSMQ.

9

Hi @odinserj,

We have started investigating the switch to use the Pro/Redis. I don’t have the capacity to test this out for you.

Best regards,
Chris Young

10

Hello, @youngcm2, I know you are using Redis. Just informing you about the state of that MSMQ bug and nothing else :smile: