Running multiple instances of the same never-ending task

question
1

So, we have workers that read from queues. This is a push process, so the task is never ends. We also would like to be able to scale the worker processes to run multiple instances of the same worker (potentially connected to the same persistence). Would hangfire fit to solve something like this? Or shall we look for another approach?

2

If you could en queue multiple jobs rather than having one never-ending it will scale much better. Isn’t it possible?
Or, one never ending that en queue other jobs? - working like an “orchestrator”, spawning the “processing” of in other job.
So, those “processing” jobs will scale well, but, you will still have a bottleneck on the “orchestrator” one.

3

The queue API is subscribe, e.g. subscribe(handler) and handler gets called when there is anything on the queue for it, and then the job would just be waiting for cancellation. For scaling we are going to run multiple instances of the process using something like k8s, we don’t want to scale on the same process. This should handle the scalebility for us based on queue size or other metrics…

Another thing i have noticed is when i run two instances of the same worker based on hangfire with the same persistence store (mongoDB in my case), that has same tasks it starting to pick up each others task etc…

Hope my answer makes sense…

4

Fedor - I’m not sure why you would want multiple instances all doing subscribe(handler), it would seem more sensible to me to have a single, dedicated thread, that does the queue subscription, and then for every message popped of the queue to then have hangfire process that message. Usually it’s the processing of the messages that needs to scale out, not the retrieval of the messages to process.

-- Run this on a dedicated thread
private void Subscriber()
{
    var someQueue = GetTheQueueFromSomewhere();
    var jobClient = GetTheBackgroundJobClientFromSomewhere();
    someQueue.Subscribe(m => jobClient.Enqueue(() => this.ProcessMessage(m));
}

-- This method does the actual message processing
public void ProcessMessage(QueueMessage m)
{
    -- do stuff here
}

Note the above is pseudocode and hasn’t been anywhere near a compiler, so could be a complete work of fiction!

Hope this helps.

2 Likes
5

Hm… i’m missing something… Just wondering why do you separate retrieving and processing? I mean, I retrieve the message and don’t notify the queue that the message been processed before the processing is actually finished processing (and if it failed processing i want to keep it in the queue). And if we have like 2000 messages coming per second, i think it would be reasonable to scale it out. So say for instance a new user message arrives and we want as a response to this message be creating a bank account details, sending welcome message and auditing new profile. Those three tasks (workers) will all be separate applications and will all be run on separate containers in docker in k8s, each of this tasks will have it’s own queue. Say we are super busy system and we have 1000 users created per minute then we would see that email service is doing alright and don’t need to be scaled, but other two workers cannot handle the load and we would like to create more instances of those.

That kind of scenario i’m talking about, maybe i misunderstood the whole concept of Hangfire, but i would make each of those tasks a hangfire process running inside a container (it has CPU/memory limits), e.g. each of them would have a hangfire server. I’m not sure how much would it help starting multiple same hangfire tasks if the container reached its CPU\memory limits.

Am i missing something?

Thank you

6

It appears you would not need hangfire for this architecture because you will manage your workers (each in its own container) using k8s.