Jobs being auto canceled

kwinterton · December 18, 2017, 5:44pm

Hi,

I was just wondering what might trigger an automatic cancelation of a recurring job. Over the weekend a recurring job I was running seems to have gotten stuck in a state where it was being canceled but not through user intervention. It was a minutely job and the Last Execution time would always just show “Canceled a few seconds ago”, but it was still getting queued up and then canceled again.

I’ve fixed the issue by simply deleting the recurring job and rescheduling it, but I was wondering if there was something I should look for that might cause this to happen in the future. Thanks

jyendapally · July 30, 2019, 2:35pm

Hi: Were you able to figure this out? We are having the same issue.

Thanks,
Jyotsna

Deantwo · May 20, 2020, 2:11pm

I am having the same problem and have not been able to find a cause either.
I have found the following log entries in my Hangfire log:

"2020-05-20 08:30:02.4200";"INFO";"Job that is based on method `MyRecurringJobMethod` has been created with id ``  with parameters  RecurringJobId: MyRecurringJobId,Time: 1589956202,CurrentCulture: da-DK,CurrentUICulture: en-US";"LogEverythingAttribute.OnCreated => Logger.Info => Logger.Log";""
"2020-05-20 08:30:02.4200";"DEBUG";"Recurring job 'MyRecurringJobId' execution at '20-05-2020 06:30:00' has been canceled.";"RecurringJobScheduler.ScheduleRecurringJob => LogExtensions.Debug => NLogLogger.Log";""

The second log entry is written here:

github.com

HangfireIO/Hangfire/blob/ee64dd1ca65883875e308832736076b95f094a4e/src/Hangfire.Core/Server/RecurringJobScheduler.cs#L292


IReadOnlyDictionary<string, string> changedFields;


if (recurringJob.TrySchedule(out var nextExecution, out var error))
{
    if (nextExecution.HasValue && nextExecution <= now)
    {
        backgroundJob = _factory.TriggerRecurringJob(context.Storage, connection, _profiler, recurringJob, now);


        if (String.IsNullOrEmpty(backgroundJob?.Id))
        {
            _logger.Debug($"Recurring job '{recurringJobId}' execution at '{nextExecution}' has been canceled.");
        }
    }


    recurringJob.IsChanged(out changedFields, out nextExecution);
}
else
{
    throw new InvalidOperationException("Recurring job can't be scheduled, see inner exception for details.", error);
}

Which suggest that the job is not created at all because _factory.TriggerRecurringJob(...) returns null.

Deantwo · July 15, 2020, 7:58am

Looking through the code a little more. I think the error is happening here:

github.com

HangfireIO/Hangfire/blob/ee64dd1ca65883875e308832736076b95f094a4e/src/Hangfire.Core/Client/BackgroundJobFactory.cs#L86


{
    if (context == null) throw new ArgumentNullException(nameof(context));


    var filterInfo = GetFilters(context.Job);


    try
    {
        var createdContext = CreateWithFilters(context, filterInfo.ClientFilters);
        return createdContext.BackgroundJob;
    }
    catch (Exception ex)
    {
        var exceptionContext = new ClientExceptionContext(context, ex);


        InvokeExceptionFilters(exceptionContext, filterInfo.ClientExceptionFilters);
        if (!exceptionContext.ExceptionHandled)
        {
            throw;
        }


        return null;

But I can see that I don’t know enough about Hangfire Filters to make it actually log the exception in that catch block.

I would say that it is completely stupid that it simply returns null instead of re-throwing the exception so the exception message could be shown in the dashboard.

Topic		Replies	Views
Not all recurring jobs trigger when they should bug? recurring	1	1157	April 3, 2017
Recurring jobs randomly being removed question recurring , sql-server	2	2286	October 12, 2019
Recurring Jobs spontaneously not working properly from DST question recurring , queues	4	1922	March 15, 2017
Recurring job in hangfire is not fired question	0	1412	May 18, 2018
Queuing one job cancels another? bug? recurring	8	1463	March 7, 2018

Jobs being auto canceled

Related topics