[Challenging (impossible?) scenario] Multi tenant queue architecture for rate limiting

Tags: #<Tag:0x00007f23584ed220>

Hello everyone,

We have got a challenging scenario on our hands that we have not been able to find advice for anywhere.

Our system is a multi tenant system and we use hangfire to deal with a large number of background tasks.

Dashboard screenshot:

As you can see we have 5 queues to deal with different job types (so we can have different worker threads per queue) and those queues contain jobs from all customers.

This has worked well so far but now we would like to avoid one customer from starving all others, for example one customer running many layers should not prevent other customers from running their layers.
To achieve this we were looking at the Hangfire.Throttling package but the package only works per job type and the job types are shared across customers.

Every job has the customer code information in it so it knows which customer it belongs to.

Does anybody have any advice on how to limit the number of concurrent execution in this scenario?

Thank you in advance for taking on the challenge.