There is an opportunity to stop background processing – blocking
BackgroundJobServer.Dispose method, and non-blocking
BackgroundJobServer.SendStop method (appeared in 1.6.0). When they are called, background processing is gracefully stopped, and no new background jobs are processed. However, this requires manual
BackgroundJobServer class lifecycle management.
The main problem is to deliver the Stop command to all of your instances, you can use any distributed pub/sub implementations available, including Redis ones. Not a rocket science
If you have long-running jobs, ensure you are using
IJobCancellationToken, or at least
CancellationToken parameter in your background job method, and checking it from time to time. This will enable your method to finish gracefully, within the points you’ve defined. Nevertheless, with or without graceful shutdown, your backround job identifier will be returned to the queue, and background job will be resumed after the startup. So you need to ensure that your background jobs are idempotent, or at least can restore their execution from some moment.
Idempotence is a required property for any background job in any framework, that have at least once processing semantics (at most once leads to background job loss, and exactly once don’t exist at all due to async nature).