Unhandled Exception in Hangfire Job Terminates the Service Instead of Marking the Job as Failed

bug?
1

Hi everyone,

I’m encountering an issue where a job executed via Hangfire throws an unhandled exception, and instead of marking the job as failed, the entire Hangfire service crashes. Below are the details:

Context:

  • Environment: .NET Framework 4.8
  • Hangfire Version: 4.16
  • Related Library: Microsoft.ClearScript.V8
  • Deployment: Running as a Windows Service

Error Description:

A job is calling a method that utilizes Microsoft.ClearScript.V8. When this method encounters an error, an unhandled exception is thrown, which terminates the entire Hangfire service. I expected Hangfire to handle this situation by marking the job as failed instead of terminating the process.

Here’s the exception stack trace:

Application: YiQi.Hangfire.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: exception code 80000003, exception address 820FC2EC
Stack:
   at Microsoft.ClearScript.V8.SplitProxy.V8SplitProxyNative+Impl_Windows_X86.V8Isolate_Create(Ptr, Int32, Int32, Double, UInt64, Boolean, Boolean, Boolean, Int32)
   at Microsoft.ClearScript.V8.SplitProxy.V8SplitProxyNative+Impl_Windows_X86.Microsoft.ClearScript.V8.SplitProxy.IV8SplitProxyNative.V8Isolate_Create(System.String, Int32, Int32, Double, UInt64, Boolean, Boolean, Boolean, Int32)
   at Microsoft.ClearScript.V8.SplitProxy.V8IsolateProxyImpl+<>c__DisplayClass3_0.<.ctor>b__1(Microsoft.ClearScript.V8.SplitProxy.IV8SplitProxyNative)
   at Microsoft.ClearScript.V8.SplitProxy.V8SplitProxyNative.Invoke[[Microsoft.ClearScript.V8.SplitProxy.V8Isolate+Handle, ClearScript.V8, Version=7.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](System.Func`2<Microsoft.ClearScript.V8.SplitProxy.IV8SplitProxyNative,Handle>)
   at Microsoft.ClearScript.V8.SplitProxy.V8IsolateProxyImpl+<>c__DisplayClass3_0.<.ctor>b__0()
   at Microsoft.ClearScript.V8.SplitProxy.V8EntityHolder..ctor(System.String, System.Func`1<Handle>)
   at Microsoft.ClearScript.V8.SplitProxy.V8IsolateProxyImpl..ctor(System.String, Microsoft.ClearScript.V8.V8RuntimeConstraints, Microsoft.ClearScript.V8.V8RuntimeFlags, Int32)
   at Microsoft.ClearScript.V8.V8Runtime..ctor(System.String, Microsoft.ClearScript.V8.V8RuntimeConstraints, Microsoft.ClearScript.V8.V8RuntimeFlags, Int32)
   at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(Microsoft.ClearScript.V8.V8Runtime, System.String, Microsoft.ClearScript.V8.V8RuntimeConstraints, Microsoft.ClearScript.V8.V8ScriptEngineFlags, Int32)
   at YiQi.Facade.ScriptRunner.GetV8ScriptEngine(YiQi.Interfaces.IContext, YiQi.Facade.MetaAbstract, System.String)
   at YiQi.Facade.ScriptRunner.RunWith(System.String, YiQi.Core.General.BoTransaction, YiQi.Facade.MetaAbstract, EnumScriptEngine)
   at YiQi.Facade.ScriptRunner.Run(System.String, YiQi.Core.General.BoTransaction)
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at YiQi.Core.Eventos.BoEventoYScript.EjecutarOValidarScript(Int32, System.String, System.String, System.String, YiQi.Core.General.BoTransaction, Int32)
   at YiQi.Core.Eventos.BoEvento.Disparar(YiQi.Model.Event, Int32, System.String, YiQi.Core.General.BoTransaction, System.Nullable`1<Int32>, System.Nullable`1<Int32>)
   at YiQi.Core.General.BoTransaction.FireEvent(EventInstance, Int32, System.String, System.String, System.Nullable`1<Int32>)
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at Hangfire.Server.CoreBackgroundJobPerformer.InvokeSynchronously(System.Object)
   at Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(Hangfire.Server.PerformContext, System.Object, System.Object[])
   at Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext)
   at Hangfire.Server.BackgroundJobPerformer.InvokeNextServerFilter(Enumerator<Hangfire.Server.IServerFilter> ByRef, Hangfire.Server.IBackgroundJobPerformer, Hangfire.Server.PerformContext, Hangfire.Server.PerformingContext)
   at Hangfire.Server.BackgroundJobPerformer.InvokeServerFilter(Enumerator<Hangfire.Server.IServerFilter> ByRef, Hangfire.Server.IBackgroundJobPerformer, Hangfire.Server.PerformContext, Hangfire.Server.PerformingContext)
   at Hangfire.Server.BackgroundJobPerformer.InvokeNextServerFilter(Enumerator<Hangfire.Server.IServerFilter> ByRef, Hangfire.Server.IBackgroundJobPerformer, Hangfire.Server.PerformContext, Hangfire.Server.PerformingContext)
   at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, FilterCollection`1<Hangfire.Server.IServerFilter>)
   at Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext)
   at Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String, Hangfire.BackgroundJob, System.Collections.Generic.IReadOnlyDictionary`2<System.String,System.Object> ByRef)
   at Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object)
   at Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object)
   at Hangfire.Processing.BackgroundDispatcher.DispatchLoop()
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

Questions:

  1. Is there a way to configure Hangfire to better handle unhandled exceptions and prevent service termination?
  2. Are there known issues with Microsoft.ClearScript.V8 in this context that could cause process crashes?
  3. Should Hangfire automatically isolate jobs that may throw exceptions to prevent crashes, or is additional configuration required?

Any insights or recommendations would be greatly appreciated. Thank you for your help!

2

I believe I found the root cause by reviewing Hangfire’s source code. From what I observed, Hangfire has mechanisms to catch ‘manageable’ exceptions, but it seems the exception thrown by ClearScript is not considered manageable and therefore isn’t properly handled by the system.

To resolve this, I created a custom exception. In the block where ClearScript might throw an exception, I caught it and re-threw it as a manageable exception that Hangfire could process correctly. This allowed the job to fail gracefully instead of terminating the entire service.

I hope this solution helps others facing similar issues.