Hangfire as a Service TargetInvocationException error

bug?
1

I am running Hangfire 1.6.7 as a service and utilizing MSMQ. I get a random error in our stage and production environments where the service will throw a TargetInvocationException when trying to run a background job. The job will show in a Processing state, then the invocation of the method causes the service to crash. The job stays in this state until I restart the service. When the service is restarted, the job completes successfully.

What could be causing this to happen? Why would one server not be able to process this method and the next one does?

2017-03-31_11-05-17.png866×892 56.7 KB 

Application: WorkerScheduler.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Reflection.TargetInvocationException
Stack:
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
   at Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(Hangfire.Server.PerformContext, System.Object, System.Object[])
   at Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext)
   at Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass8_0.<PerformJobWithFilters>b__0()
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(Hangfire.Server.IServerFilter, Hangfire.Server.PerformingContext, System.Func`1<Hangfire.Server.PerformedContext>)
   at Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass8_1.<PerformJobWithFilters>b__2()
   at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, System.Collections.Generic.IEnumerable`1<Hangfire.Server.IServerFilter>)
   at Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext)
   at Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String)
   at Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.ServerProcessExtensions.Execute(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.AutomaticRetryProcess.Execute(Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.ServerProcessExtensions.Execute(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.InfiniteLoopProcess.Execute(Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.ServerProcessExtensions.Execute(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.ServerProcessExtensions.RunProcess(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.ServerProcessExtensions+<>c__DisplayClass1_0.<CreateTask>b__0()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.Tasks.Task.ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork(System.Object)
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)
2

It’s hard to diagnose a TargetInvocationException without knowing its InnerException. Try to add more logging to your service.

3

I’m not sure how much more logging I can add. What doesn’t make sense to me is that one instance of the BackgroundWorker throws this exception, while the next one that is created can.

4

I tried to add logging to Hangfire.Core and couldn’t capture the inner exception. I was finally able to get the InnerException

System.Reflection.TargetInvocationException was unhandled
HResult=-2146232828
Message=Exception has been thrown by the target of an invocation.
Source=mscorlib
StackTrace:
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(PerformContext context, Object instance, Object[] arguments)
   at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context)
   at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass8_0.<PerformJobWithFilters>b__0()
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
   at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass8_1.<PerformJobWithFilters>b__2()
   at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters)
   at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context)
   at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)
   at Hangfire.Server.Worker.Execute(BackgroundProcessContext context)
   at Hangfire.Server.ServerProcessExtensions.Execute(IServerProcess process, BackgroundProcessContext context)
   at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context)
   at Hangfire.Server.ServerProcessExtensions.Execute(IServerProcess process, BackgroundProcessContext context)
   at Hangfire.Server.InfiniteLoopProcess.Execute(BackgroundProcessContext context)
   at Hangfire.Server.ServerProcessExtensions.Execute(IServerProcess process, BackgroundProcessContext context)
   at Hangfire.Server.ServerProcessExtensions.RunProcess(IServerProcess process, BackgroundProcessContext context)
   at Hangfire.Server.ServerProcessExtensions.<>c__DisplayClass1_0.<CreateTask>b__0()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork(Object obj)
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)
InnerException: 
     HResult=-2147467261
     Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
     InnerException:
5

I found the issue.

I have StructureMap as the IoC container and it was using HttpContext as the lifecycle which didn’t exist.