Hi,
How can have a RecurringJob firing in all servers at the same time. By default the job just run in one of the servers.
Thanks
Hello,
This may be possible if you use the following approach:
- Launch Hangfire for Server1
- Create RecurringJob to fire at 2:30pm.
- Dispose of the Hangfire instance for Server1
- Repeat steps 1-3 for the rest of your servers
I haven’t tested this, but - theoretically - this should work no matter which approach you use for Launching and Disposing Hangfire. Also, set the job to run at a reasonable time; give Hangfire a chance to create the job on all servers, so the job can run at the same time on all servers. Lastly, if this doesn’t work, then try creating the RecurringJob with a Unique ID; meaning, you will have multiple RecurringJobs in Hangfire. I ran into this issue when I tried to create mutiple RecurringJobs for an in-house Web Scheduler, but only one job was being created in Hangfire.
Not sure what you’re asking is possible, but I would personally create a clone of the job on each server, each with its own id, so you’ll also have more control over them. If you have to have the jobs linked you can create that link in your own code.
Hi Francois,
The same service is running in 3 instances, when the service starts each instance it’s starting the same Hangfire recurrent job. The problem is when the job fires, it just fires in one instance (the one that is available at the time - balance) and I want that the job fires in all instance at the same time.
So if I understand your situation correctly it’s essentially as you mentioned a load balancer scenario?
If it’s the same instance on 3 services they probably share the same database, in which case my guess would be that the first one to trigger it updates the job, so even if the others are a fraction later Hangfire doesn’t find any scheduled jobs so doesn’t run it.
Not sure how HF handles load balancing, but you will need to look at the code where HF queries it’s data store for jobs and updates them once picked up, and perhaps add in your own logic to only update the job as started after 3 attempts or somesuch - it might give you a clue as to where to put in your changes.
Not sure the job filters will work, but you can have a look at them as well, however this sounds to me to be even before the filters are checked as you need the job first.
Something else you can try is to do something like an update trigger in sql on the jobs table, then perhaps manipulate the job that way. Not looked at this but might be worth investigating?
Here is how I would do it, which is not the only way, possibly not even the best way (depends on your scenarios).
Use a machine specific queue. When I say machine specific queue I mean a queue that is only serviced by 1 server. For this purpose I use the hostname (minus special characters as Hangfire Queues cannot have special characters in the name) on my system. This is just a matter of setting up the queue as part of BackgroundServerOptions when you start the Hangfire backgroundserver instance. Something like :
var q = string.Concat(Environment.MachineName.ToLowerInvariant().Where(char.IsLetterOrDigit));
var options = new BackgroundJobServerOptions
{
Queues = new[] {q}
};
var server = new BackgroundServer(options);
At this point you can either create a schedule for each server manually or if you want to get fancy, use the monitoring api to get the list of servers so you can use that to create 1 load balanced schedule and have that scheduled job use the dynamic server list to create fire and forget jobs for all the individual serverqueues that do the actual work.
Assuming you’re indeed referring to a load balancer scenario, this is actually a very simple variation of the Service Discovery pattern.
Every instance of you Hangfire server that starts up, registers an ID unique to that server instance (e.g machine name) in a Db table / Redis dictionary.
When you add a job to the queue that you want to run on all registered servers, you first query whatever store you used for your unique server IDs, get all registered servers and add the same job for each server with a unique ID for each job. That should give you the desired effect. They won’t fire at exactly the same time but that’s the nature of how queues work.
Then when scaling down your servers, either automatically or manually, you simply make sure to deregister the server ID from wherever you stored it as part of the shutdown task.
Concretely, how do we do this? In my case I want to run a fire and forget job on each server.
How do you solve this problem？？？
every machine have unique queue ，but execute one job have an error （The queue is empty）