Job Queues and priority problem

question
1

Hi

I have a problem with hangfire queues. I use HF to trigger some background sync processes in the Net Core application using SqlServer

I have 2 queues. Lets call them “1ststep” and “2ndstep”.
For each queue I have defined a several jobs. Jobs inside the “1ststep” queue are triggered every 5th minute, while jobs inside the “2ndstep” queue are triggered every 10th minute. Which means there will be an overlap at some point.

When it comes to that moment of overlap, I want the jobs to be done in the order defined by the queue, which means that I want all the jobs from the queue named “1ststep” to be executed first, followed by all the jobs from the queue named “2ndstep”.

I did the whole hangfire setup inside the Startup.cs file. I defined all possible queues:

services.AddHangfireServer(options => 
            {
                options.Queues = new[] { "1ststep", "2ndstep", "default" };
            });

and registered all the necessary jobs to which I assigned to selected queue:

        recurringJobs.AddOrUpdate("Job1", Job.FromExpression<Class1>(x => x.Method1()), "*/5 * * * *", tzi, "1ststep");
        recurringJobs.AddOrUpdate("Job2", Job.FromExpression<Class1>(x => x.Method2()), "*/5 * * * *", tzi, "1ststep");
        recurringJobs.AddOrUpdate("Job3", Job.FromExpression<Class1>(x => x.Method3()), "*/5 * * * *", tzi, "1ststep");

        recurringJobs.AddOrUpdate("Job5", Job.FromExpression<Class2>(x => x.Method1()), "*/10 * * * *", tzi, "2ndstep");
        recurringJobs.AddOrUpdate("Job6", Job.FromExpression<Class2>(x => x.Method2()), "*/10 * * * *", tzi, "2ndstep");

When I start the application and when in a given time interval there is an overlap of jobs from 2 different queues, they are all executed at the same time. There is no priority between 2 different queues where job from “1ststep” queue will be executed first and than jobs from queue “2ndstep”

Is it even possible to have job execution priority based on queue value or am I doing something wrong ?
I even tried using QueueAttribute on method, same result.

I can limit Worker count with: options.WorkerCount = 1 but then each job will be executed one by one and not in batch (1 batch for each queue) and that takes time.

2

So a queue is an ordered list by definition, why are you creating two ordered lists and expecting them to behave like they’re one ordered list?

Is there an explicit relationship between the Jobs in 1stStep vs Jobs in 2ndStep where they need to occur in that order? i.e. Job1 gets queued to 1stStep queue and must execute before Job2 in 2ndStep queue?

Or are you just trying to prioritize operations and Job2 has no dependency on Job1?

Workers are just going to dequeue work in “roughly” the order that the Jobs were enqueued. If there’s no dependency between Jobs and you’re just prioritizing then just use one queue and enqueue the Jobs in the order you want them prioritized.

If you need Job1 to run before Job2, enqueue Job2 when Job1 has completely successfully, or buy the Pro license and use Hangfire Batches (as it is a more reliable pattern).

3

Execution order of jobs inside queue 1ststep is not important.
All I want is first to finish all jobs inside queue 1ststep so: Job1, Job2 and Job3 (order is not important) and after they are successfully or unsuccessfully executed then I want start processing Job5 and Job6.
Job5 and Job6 have dependencies on Job1-3 thats why I need need to wait for them to finish.

4

I think you need to spend some time to learn the basic concepts involved. I know it can be a bit confusing to understand the relationship between servers, queues, workers and jobs, so here’s a metaphor:

Your design has two queues, think of them as side-by-side line ups at the grocery store. Each person in the line up is some data that needs some work done (aka a Job). You have two cashiers (Workers) and they are serving people from each of the line ups. The cashier is going to serve people at the front of the line up from one queue or the other. Every 5 minutes some people stand in one line up and every 10 minutes people stand in the other line up. That doesn’t affect how the cashiers are serving people at the front of the line up.

How do you get from this design to the one you want?

You have Jobs that need to be finished before other Jobs. So instead of having two lines with two cashiers serving both, you could have a line up with one cashier, and after that person gets served they go stand in line with the other cashier working on them. You can extend this pattern out as far as you need.

I can’t give you much more detail than that because there’s still some missing details in your description as far as granularity, quantity and dependencies between Jobs. Like if you’re literally queuing up 3 long running jobs every 5 minutes (aka - Job 1, 2 & 3) or many instances of 3 types of jobs. Are all the Jobs working on the same set of data (like a big set of data) or are you doing individual record processing? How are your recurring Jobs deciding what data to work on?