Orchestrating jobs using a batch job

question
1

We’re having a complex import using multiple import and processing jobs. Right now, these jobs are all configured with AddOrUpdate and triggered with a CronJob. As there are multiple dependencies between these job, this either isn’t very stable and leads to timing issues, or takes very long because we have to wait long enough between the different jobs to make sure the dependencies are correctly “resolved”.

We wanted to use Hangfire Pro / Batch processing to orchestrate all the different jobs, while still being able to trigger them manually if the need arises. It seems we misunderstood the documentation there, we’re not exactly sure how to achieve what we tried to do.

What we’re trying to achieve is to create a MasterImportJob, which then orchestrates the other jobs on a daily trigger.
Our naive hope was to do something like this:

BatchJob.StartNew(masterBatch =>
{
    var batch1 = masterBatch.StartNew(batch =>
    {
        // 1. Requeue does not work (Exception: State change is not supported while creating a batch)
        masterBatch.Requeue("ImportJob1.1");
        masterBatch.ContinueJobWith("ImportJob1.1", innerBatch => innerBatch.Requeue("ProcessingJob1.1"));

        masterBatch.Requeue("ImportJob1.2");
        masterBatch.ContinueJobWith("ImportJob1.2", innerBatch => innerBatch.Requeue("ProcessingJob1.2"));

        // 2. Cannot await completion of multiple jobs to ensure batch is only continue when jobs are all finished
    });

    var batch2 = masterBatch.ContinueBatchWith(batch1, batch =>
    {
        masterBatch.Requeue("ImportJob2");
        masterBatch.ContinueJobWith("ImportJob2", innerBatch => innerBatch.Requeue("ProcessingJob2"));
    });
});

So, the idea would be to run ImportJob1.1 and ImportJob1.2 in parallel, and both immediatly trigger ProcessingJob1.1 or ProcessingJob1.2 respectively. When all 1.x jobs are completed, the batch should continue with ImportJob2, which again triggers ProcessingJob2 on completion. And so on

So there’s multiple questions:

  1. (related to 1. in the example code): How can I trigger a Job with a specific JobId in the batch?
  2. (related to 2. in the example code): How can I await the completion of multiple jobs in a batch?
  3. And most important of: Is it even possible what we are trying to achieve? Is the approach we try the correct one? And if not, is there another way?

Takes in advance
Kind regards,
Max