Reschedule failing job on different queue

question
1

Hi, I have an HttpNotifier job that is supposed to send notifications over http.
I want to have one queue where new jobs go, let’s call it “webhooks”, and then another queue “retries”, which is processed on a different server for retries of failing jobs.

How can I make it so that when the job fails, it will be retried on the “retries” queue?

Thanks

2

So far I have tried:

Intercepting “EnqueuedState” and changing the queue → does not work because the AutomaticRetryAttribute creates “ScheduledState”

Intercepting the “FailedState” and creating a new “EnqueuedState” with the “retry” queue → works for a single retry, then the job goes back to original queue (“webhooks”) - seems like a bug with SQL storage? IFetchedJob contains correct queue, but InvocationData is never updated with the new queue name, so the actual handlers receive the original queue.

Any other options?

3

Just for anyone also looking for a solution to this problem, this is how I’m solving it now.

I created a filter that intercepts the “FailedState”, check if the job is not on the retry queue, if it is not, create a duplicate of the job and enqueue it on the retry queue. Move the original job to DeletedState. This plays well with the AutomaticRetryAttribute shipped with Hangfire (Order of our filter should be < 20).

internal class ScheduleFailingOnRetryQueue : JobFilterAttribute, IElectStateFilter
{
    public ScheduleFailingOnRetryQueue()
    {
        Order = 0;
    }
    public void OnStateElection(ElectStateContext context)
	{
		if (context.CandidateState is FailedState && context.BackgroundJob.Job.Queue != "retry")
		{
			var job = new Job(context.BackgroundJob.Job.Type, context.BackgroundJob.Job.Method, context.BackgroundJob.Job.Args, "retry");
			BackgroundJobFactory backgroundJobFactory = new BackgroundJobFactory();
			backgroundJobFactory.Create(new CreateContext(context.Storage, context.Connection, job, new EnqueuedState("retry")));
			context.CandidateState = new DeletedState(){ Reason = $"Rescheduled to retry queue." };
		}
	}
}

There’s a small possibility of duplicate retry job creation due to the lack of transaction. For me this is not a problem, but I did not find an out-of-the-box solution to hook both operations to Hangfire’s own transaction.