I think I’ve hit a bug that I can reproduce here.
I recently had to migrate a bunch of servers from the v1 to v2 cloud on Rackspace. I have 4 servers in a cluster, all on the same time/timezone running Windows Time service.
After migration, the new boxes would come up without the windows time service running and with the wrong clock setting (3 are complete, and all 3 triggered the behaviour). Servers had the correct timezone, etc, just the wrong system time, and since the windows time service wasn’t running, it didn’t auto-sync back up. IIS was set to start automatically, and when it came up, what I saw was the following:
-the dashboard was showing a “last executed” time on all scheduled jobs of X hours in the future where X was offset exactly by the amount of time the clock was off
-all the other servers were no longer registered (only the “new” box with the clock set (incorrectly) forward
-none of the scheduled jobs ran
-I could still manually trigger jobs, which ran perfectly
-no errors in the logs
I guessed that by waiting X hours, the issue would resolve itself once X had passed. Since I hadn’t (yet) put any mission critical jobs in place, this was thankfully a valid solution, however that wouldn’t work normally.
Other things I tried after realizing the issue that did not work:
-fixing the time and restarting IIS
-deleting stuff from the “State” table that appeared to have occurred in the future
-deleting stuff from the “Job” table that appeared to have occurred in the future
No dice on any of it.
TL;DR: if the system clock on one machine in a cluster is set into the future, it borks the rest of the servers in the cluster and also prevents any scheduled jobs from executing until that future time offset is hit.