I am getting random deadlock errors when running Hangfire 1.6.1 as a windows service against SQL Server (.Net 4.5.1). I was also getting the error with Hangfire 1.5.8.
It looks like it’s running the install script every time a new connection is established to SQL Server, and is randomly deadlocking with itself. Sometimes it will run for days without a deadlock, other times it seems to deadlock every 5 minutes.
I noticed that it looks like this issue occurred before and was resolved:
http://hangfire.discourse.group/t/hangfire-deadlocks-on-server-restart/838
I ran SQL Server profiler and confirmed that the install script is deadlocking with itself. I’ve attached a screenshot showing this.
I think that the issue might be that HangFire is looking to catch error 1205 to attempt a retry, but install.sql already catches error 1205 and returns error 50000 instead (also shown in screenshot).
Deadlock error:
Error: Failed to process the job '22503': an exception occurred.
Exception
=========
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Data.SqlClient.SqlException: Hangfire database migration script failed: Transaction (Process ID 65) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. Changes were rolled back, please fix the problem and re-run the script again.
Installing Hangfire SQL objects...
Database schema [HangFire] already exists
Table [HangFire].[Schema] already exists
Current Hangfire schema version: 5
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader)
at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command)
at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable`1 commandTimeout, Nullable`1 commandType)
at Hangfire.SqlServer.SqlServerObjectsInstaller.Install(DbConnection connection, String schema)
at Hangfire.SqlServer.SqlServerStorage..ctor(String nameOrConnectionString, SqlServerStorageOptions options)
at Hangfire.SqlServerStorageExtensions.UseSqlServerStorage(IGlobalConfiguration configuration, String nameOrConnectionString)
at myapp.Scheduler.Scheduler..ctor() in D:\Dev\myapp\myapp\myapp.Scheduler\Scheduler.cs:line 27
--- End of inner exception stack trace ---
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
at Hangfire.JobActivator.SimpleJobActivatorScope.Resolve(Type type)
at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context)
at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass8_0.<PerformJobWithFilters>b__0()
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters)
at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context)
at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)