Implementing round robin approach for multiple queues

question
,
1

Hello.
I am having trouble implementing round robin approach in Hangfire. My idea is to have a multiple queues (one per user) that are never being processed. Then I would have a single recurrent RoundRobin job every 10 seconds that is going to monitor all the queues from different users. Once the recurrent job finds a job in the User1’s queue and there are no other jobs in Default queue from that User1 are being processed then the recurrent jobs deques from User1 queue and enqueues to Default queue. This way each user would have maximum only one job in the processing state at a given time.
Is there another way of achieving this?

The following code doesn’t work with SqlStorage. The jobs are never dequeued from the original queue according to dashboard though their state are being changed to Succeded.

 public void RoundRobin(PerformContext jobContext)
        {
            var monitoringApi = jobContext.Storage.GetMonitoringApi();

            foreach (var queue in monitoringApi.Queues())
            {
                if (queue.Name.StartsWith("c_") && monitoringApi.EnqueuedCount(queue.Name) > 0)
                {
                    var jobs = monitoringApi.EnqueuedJobs(queue.Name, 0, int.MaxValue).FirstOrDefault();
                    var jobParams = queue.Name.Split('_', 3).ToList();
                    var userId = jobParams[1];
                    
                    // CHECK THAT NO JOBS FROM CUSTOMER ARE IN DEFAULT QUEUE (ENQUEUD OR PROCESSING)
                    bool customerJobIsAlreadyBeingProcessed = false;
                    foreach (var processingJob in monitoringApi.ProcessingJobs(0, int.MaxValue))
                    {
                        if (processingJob.Value.Job.Args[1].ToString() == userId)
                        {
                            customerJobIsAlreadyBeingProcessed = true;
                            break;
                        }
                    }
                    if (!customerJobIsAlreadyBeingProcessed)
                        foreach (var processingJob in monitoringApi.EnqueuedJobs("default", 0, int.MaxValue))
                        {
                            if (processingJob.Value.Job.Args[1].ToString() == userId)
                            {
                                customerJobIsAlreadyBeingProcessed = true;
                                break;
                            }
                        }

                    if (customerJobIsAlreadyBeingProcessed)
                        continue;
                    // ENDCHECK
                

                    var client = new BackgroundJobClient(jobContext.Storage);
                    client.RetryAttempts = 0;
                    client.ChangeState(job.Key, new EnqueuedState("default"));
                }
            }
        }
2

You could track the the last job Id for each user. Then queue the job via ContinueWith passing the job Id. This ensures the previous job is completed before the next one starts. Completed jobs don’t stay in the database for long so I can’t say what happens if the job Id no longer exists in the database.

3

What if any of the jobs fails? All jobs awaiting the one that failed will never run.

4

You can specify to delete the job on failure when queueing the job.