Recurring job stuck in processing after Azure DB transaction exception, blocking other jobs in the queue

bug?
,
1

Hello, we are using Hangfire 1.7.32 and are having an issue with a recurring job running on an Azure instance, it sometimes to intermittently get stuck in ‘processing’ despite the function no longer running. Since we’re set up to run one job at a time, this prevents other enqueued jobs from running until we delete the stuck job and restart the server.

Samples of our code is as follows:

To schedule the job, we call:

RecurringJob.AddOrUpdate(() =Start(), cron, Shared.GeneralUtil.GetLocalTimeZone());

The contents of Start() include:

public static void Start()
{
    int logId = default(int);
    try
    {
        (db intensive work here)
        (EXCEPTION THROWN HERE)
    }
    catch (Exception ex)
    {
        (try and clean up after exception)
    }
}

The exception details are as follows, and notably the Start() function ends immediately upon hitting this exception, so despite being occurring in the try/catch, the catch block doesn’t run:

DevExpress.Xpo.DB.Exceptions.SqlExecutionErrorException
  HResult=0x80131500
  Message=Executing Sql 'update "dbo"."RecurringJobLog" set "JobName"=@p0,"Date"=@p1,"Log"=@p2 where ("Id" = @p3) IF @@ROWCOUNT <1 begin set @r=0 RETURN end  set @r=1' with parameters '{StaffActualHoursCalc},{12/01/2023 1:33:40 PM},{12/01/2023 1:33 PM: Starting log
12/01/2023 1:33 PM: begin fetc...},{2},{Null}' exception 'Microsoft.Data.SqlClient.SqlException (0x80131904): The transaction log for database 'test10' is full due to 'ACTIVE_TRANSACTION'.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\SqlConnection.cs:line 2376
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\SqlInternalConnection.cs:line 773
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\TdsParser.cs:line 1686
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\TdsParser.cs:line 2913
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\SqlCommand.cs:line 5685
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\SqlCommand.cs:line 5511
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\SqlCommand.cs:line 5113
   at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\SqlCommand.cs:line 2005
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery() in H:\tsaagent1\_work\18\s\src\Microsoft.Data.SqlClient\netfx\src\Microsoft\Data\SqlClient\SqlCommand.cs:line 1459
   at DevExpress.Xpo.DB.MSSqlConnectionProvider.<>c__DisplayClass92_0.<Exec>b__0()
   at DevExpress.Xpo.Logger.LogManager.Log(String category, LogHandlerVoid handler, MessageHandler`1 createMessageHandler, ExceptionHandler exceptionHandler)
   at DevExpress.Xpo.DB.MSSqlConnectionProvider.Exec(IDbCommand command, IDictionary parameters)
ClientConnectionId:84ba0653-0aa3-439b-8a25-6276bcfe4305
Error Number:9002,State:4,Class:17'
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

Inner Exception 1:
SqlException: The transaction log for database 'test10' is full due to 'ACTIVE_TRANSACTION'.

If we cannot always prevent these transaction exceptions from occurring, is there a way we can at least automatically clean up the currently processing job after them so we can move on to the next enqueued job?