We are working through a design where we intend on running CPU-intensive jobs in a cluster of separate machines. The server process itself in each machine will be an IIS hosted application that is set to be always available. So if a given server has jobs currently running, and the server or server process crashes or is restarted, what happens then? I realize there are really potentially two questions here, one where the server is gracefully shutdown and one where the server crashes or is locked up (or a network error causing the machine to be isolated from the other processing servers). How does hangfire in the these cases determine that the job server is gone (and all the in-flight jobs will never complete)?
Thanks,
–john