Using Hangfire on Azure K8S

question
1

Hi all, i’m using from several time Hangfire and i don’t have any problem.
Classic use cases that i will have are:

  1. From a web app, user enqueue an operation and can check all pending operations. Where operation ends he has a notification via SignalR and can check output of a job. An example is the same “Azure Portal”, where every operation is enqueued and status is visible in the bell on top right.
  2. Recurring job that runs every few seconds to perform scheduled operations indipendent from user’s interaction. Jobs must have a lock concept, so until one is running, nothing else of the same type should start for that user.

Recently i’m moving to Azure, using k8s for some microservices. I think that i will use a microservice with Hangfire, do you think that Hangfire and microservices can coexist? If i don’t use Hangfire i have to replicate something similar using Azure Function (high cost), manage a storage for job’s state, so will worth it using something different? How do you manage these scenario in microservice architecture?

Bye

2

We use Hangfire in Azure k8s no problem.

3

Can i ask you how you manage different queues? Do you have a deployment for each queue?
Thanks

4

We’re still kind of figuring it out.

After running into bottlenecks when one piece is down and its blocking other jobs causing them not to run in a timely manner, we started giving each job its own queue. Then when once piece is not processing correctly the rest of the system continues chugging along.

We basically create a Q and a Priority-Q for each qName passed in. As the pod heats up and the HPA starts scaling out, each queue scales out with it.

Our current implementation looks something like this:

        public static void ConfigureHangfire( this List<BackgroundJobServer> servers, int cpuMultiplier = 5, bool addDefault = true, params string[] queueNames )
        {
            servers ??= new List<BackgroundJobServer>();
            var allQueueNames = new HashSet<string> { QueueName.Default };
            if ( !addDefault ) allQueueNames.Clear();

            // Add priority queues
            var priorityQueues = queueNames.Select(q => $"{QueueName.Priority}-{q}").ToArray();
            allQueueNames.AddRange(priorityQueues);

            // Add queues
            allQueueNames.AddRange(queueNames);

            // Create a BackgroundJobServer for each queue name
            foreach ( var queue in allQueueNames.Select(( queueName, idx ) => new { queueName, idx }) )
                servers.Add(new BackgroundJobServer(new BackgroundJobServerOptions
                {
                    Queues              = new[] { queue.queueName },
                    IsLightweightServer = queue.idx > 0,
                    WorkerCount         = Environment.ProcessorCount * cpuMultiplier
                }));
        }