Job ends "successfully" prematurely without error

bug?
1

I have a very strange problem and I am hoping someone here knows what is going on. A brief background of my project:

  • Running on Ubuntu Linux, it consists of two main applications: a .NET 5 website and a .NET 5 console app that works as a background process to run different functions based on command line string parameters that are passed in
  • Both the website and the console app initialize Hangfire so they can queue jobs
  • I have a CommandLineExecutorJobHandler class that allows me to enqueue a job that runs command line executables with parameters, and this class is used to enqueue both scheduled and ad hoc jobs that run the console app using the various CLI string parameters

Everything works fine except in one very specific circumstance. When I enqueue an ad hoc job via my website (a button on a page that calls the function that creates HF jobs and enqueues them) for one specific set of parameters, the job says it processes and succeeds, but the application never gets past the following code (which happens during the ConfigureServices block at the very beginning of the logic):

GlobalConfiguration.Configuration.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSerilogLogProvider()
    .UseDefaultActivator()
    .UseStorage(new MySqlStorage(config.GetConnectionString(HangfireConnStr), new MySqlStorageOptions
    {
        TransactionIsolationLevel = System.Transactions.IsolationLevel.ReadCommitted,
        QueuePollInterval = TimeSpan.FromSeconds(15),
        JobExpirationCheckInterval = TimeSpan.FromHours(1),
        CountersAggregateInterval = TimeSpan.FromMinutes(5),
        PrepareSchemaIfNecessary = true,
        DashboardJobListLimit = 50000,
        TransactionTimeout = TimeSpan.FromMinutes(1),
        TablesPrefix = "Hangfire"
    }));

I’ve put logger entries before and after this block, and I see the log entry before but not after. No error is thrown. I even put in a try/catch block to solely encompass the code above and no exception was caught.

HOWEVER! If I go into the Hangfire dashboard and click “requeue” on the very same job that ended prematurely it runs to full completion exactly as expected! As well, other similar jobs that get enqueued by the website that use the same CommandLineExecutorJobHandler class to run the same console app but use different parameters ALL WORK without problems. But this one fails consistently if enqueued from the website. The weirdest thing is there is no reason that different CLI parameters should make a difference because the ConfigureServices call where things just drop off comes before the code that parses the parameters. Again, just requeuing the job manually by a single click of a button results in it magically working. And finally, this specific scenario makes up less than 1% of the total various similar jobs (enqueue console app with varying parameters). All the others run fine to completion when enqueued from the website, but this one always behaves this way when queued from the website.

Any ideas? I am baffled.