Question about Recuring Jobs Parallelism

question
1

Hello, everyone.

I have a question regarding recurring job processing. Currently, I’m utilizing the recurring feature to execute an action with different parameters, specifically the client ID, to perform processing logic for each client.

foreach (var clientId in await clientsRepository.GetAllAsync(c => c.Id))
{
    RecurringJob.AddOrUpdate<ClientQueueProcessingBackgroundJob>(
        $"ClientQeueuProcessingBJ-{clientId}",
        (job) => job.ExecuteAsync(clientId),
        cronExpression: () => "*/1 * * * *");
}

now, if we have 10 clients, we will have 100 recurring jobs, and if we have 10000 clients we will have 10000 recurring jobs.

what I don’t understand is if we have 10000 or more recurring jobs running at the same moment (because they have the same cron expression) how hangfire will perform the processing? because the Parallelism is limited to the number of Workers I guess. does it select a number of jobs to be processed based on the number of workers and when finished with that batch will start with the remaining ones, but here they are not run on the specified time? I hope someone can explain how this works.

Note:

  • am using ASP Core v7.
  • Hangfire.AspNetCore and Hangfire.SqlServer v1.8.4.
  • Hangfire server is running inside the web application, am using a monolithic.
2

Now sure that will scale. I would suggest one recurring job or one recurring job per job type. Within that job you can queue a job for each client and/or job type.

The recurring job is not a job in the job queue. It is a schedule that, once triggered, will generate a job in the job queue. I can’t speak to how 10k recurring jobs would perform.