When we initially set this up, we didn’t have ACE licenses, so its an attempt to create some level of throttling that would limit outgoing requests to a third party.
Each of the jobs performs a set of operations against a single object in our system. It can be relatively complex depending on the scenario, and we don’t want failures in one object to affect another. We got around failures by having the final attempt of the job actually mark the job as a success, this lets the batch complete and the next continue
We’ve actually got a licenses for ACE now, so we’re planning on using HF Throttling to replace the batching. What this means is that we’re dump all the jobs into a single batch - we need to investigate, but we’re hoping that doing so will improve the performance of the job creation overall. If not, then we’ll need to move job creation into an Init job - that’ll let us take the load off the request itself.