Can I limit the rate at which jobs are processed?

question
1

We have jobs which call a a third-party API. This API seems to struggle if too many calls are fired at it too quickly. So I would like to limit the rate at which our jobs are processed, so as not to overwhelm the third-party server.

The problem is that all our jobs are scheduled in one big batch.

Any ideas how I might accomplish this?

I guess I could limit the number of worker threads to 1, but that would then limit the usefulness of HangFire for processing other kinds of jobs.

2

I’m also looking at ways of doing this, but one way I have tried is instead of Enquing all the jobs, I actually Schedule them with a staggered 15 seconds interval between each other. It kinda works, but I wish there was a more “official” way…

3

If you have a single server process, you can create an API manager shared between tasks, which will guard all API calls with a semaphore. So when the number of concurrent API requests reaches the limit, the next calls will wait. That won’t hinder other jobs’ processing, while limiting API access rate.

4

This sounds interesting. You think maybe you could come up with a little code snippet?