So we have a Hangfire implementation which is running multiple servers to do long running jobs, some of which can be running for 20-60 minutes.
It there a mechanism to put 1 of the servers in “maintenance” or offline mode where it will STOP picking up new jobs from the scheduler, while still being able to complete the jobs in progress on that instance ? Essentially DO what your doing, but don’t pick up any new work for now.
We really don’t want to lose the job which is currently in progress, but as the system has grown to higher usage, it’s become challenging to find windows where the both the servers (looking to add even more job processors) are not busy processing jobs, so we really need to be able to get a server to stop pulling jobs or scheduling work for themselves, while in a Paused/Maintenance state so we can do graceful code updates/maintenance.
Any suggestions would be greatly welcome.