DisableConcurrentExecution job using all workers

We have a job we only want to run 1 at a time. For this we use DisableConcurrentExecution. Some times we create many of these jobs. the problem is these jobs make take a couple minutes to run and it can takes all the workers available while they wait for each other to finish. Is there a way to stop this from happening? It seem like a worker shouldn’t even pull those in for processing because while those are stuck waiting there are higher priority queues that fill up while the workers are waiting. Is there any way around this?