Cancelling a batch that contains job continuations

We have some logic that requires certain jobs in a specific, for example, attempting to purchase a set of items in the order they were created, but with isolation, so if item A fails, item B is still attempted.

To do this we use job continuations, so purchase job A has a continuation to purchase job B, marked as continuing on any finished state.

Additionally this is done as a large batch of purchases for a subset of users, we create an empty batch and attach the jobs into the batch so they can begin executing immediately, and we can attach a continuation onto the batch to generate a final overall report once they are complete.

We have noticed if the batch is cancelled, jobs without continuations move to the deleted state, but the continuation jobs are stuck in the pending tab of the batch dashboard, all in the awaiting state.

This is on Hangfire 1.7.9 with Hangfire.Pro 2.2.1, using SQL Server as the storage backend.

Also worth noting we can manually enqueue the jobs with the awaiting state and they immediately go into deleted state.

Thanks for reporting this problem, I’ve fixed filter execution order and just released Hangfire.Pro 2.2.3. The problem was caused by wrong execution order of filters, and filter responsible for running or canceling continuations was running before realizing that background job should be canceled.

Just got some time to test this new version and it works, no more jobs stuck awaiting on a cancelled batch, thank you, great support :smiley:

1 Like

The fix for this seems to have regressed,

We are using Hangfire v1.7.24 & Hangfire.Pro v2.3.0 and our continuation jobs are not getting deleted with the batch, any chance this could be looked at again @odinserj ?

Many Thanks
Joe.

Hi Joe, can you please send some Dashboard UI screenshots to understand what’s happened with your continuations – whether they stuck or not properly deleted after their invocation?

Sure thing,

Please see screen shots below

Batch has been deleted / cancelled

Jobs are still pending and have not been added to finished

The idea here is that we have a batch

The first job in the batch is a BackgroundJob.Schedule

All subsequent jobs in batch are BackgroundJob.ContinueJobWith

Thanks
Joe.

And this post because i can only have 2 attachments per post

Jobs are also still scheduled under the Jobs section

And Continuations are still pending

Thanks for sharing the screenshots! I’m afraid this is by design, because batch cancellation is fully asynchronous and background jobs acknowledge batch cancellation only when worker fetch them for processing – for example that scheduled job will be canceled only after delayed job scheduler schedules it to be queued, when worker fetched it from the queue.

Batches can have millions of background jobs, and it is required to traverse every background job during a synchronous cancellation.

Oh right,

I understand now, so the actual jobs will not be deleted until they are scheduled. That’s fair enough and makes a lot of sense from an architectural point of view.

Thanks for clarifying.

Joe.