I don’t have Pro yet, but before I make the jump I need to know if my use case can be handled.
A common scenario would be for a user to dump in a batch of 50000 jobs, then some time later (probably while that’s still processing) another user will add a small batch of 10 jobs. I’d like to throttle both of these batches to process no more than 20 at a time which will leave some workers idle if the system only has one batch in queue. This would allow the system to quickly respond to smaller batches while still making gains on large ones. It also prevent the queue with 500 works from overwhelming any one resource.
I know I could set the batch up as a series of batches, each with 20 jobs, but as I understand it this would require the first 20 jobs to finish before moving to the next 20. What I’m hoping for is that I can tell it to process 20 jobs at a time in this batch until it’s done, then move into whatever that batch continues with (if anything).
Any suggestions and code samples would be appreciated.