We have an hourly recurring job that worked for 48hrs then has piled up on the queue.
Hangfire is configured with standard options on startup of our MVC/WebApi app
The static method invoked has the following attributes:
[AutomaticRetry(OnAttemptsExceeded = AttemptesExceededAction.Fail, LogEvents = true, Attempts = 3)]
The job typically takes 10 mins to run.
Log don’t show failure of invoked method, however, about the time the queue stopped working, the error we got was:
System.Data.SqlClient.SqlException: Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=21005; handshake=0;
---> System.ComponentModel.Win32Exception: The wait operation timed out
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection owningObject, System.UInt32 waitForMultipleObjectsTimeout, System.Boolean allowCreate, System.Boolean onlyOneCheckConnection, System.Data.Common.DbConnectionOptions userOptions, System.Data.ProviderBase.DbConnectionInternal& connection) at offset 286
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection owningObject, System.Threading.Tasks.TaskCompletionSource`1 retry, System.Data.Common.DbConnectionOptions userOptions, System.Data.ProviderBase.DbConnectionInternal& connection) at offset 55
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(System.Data.Common.DbConnection owningConnection, System.Threading.Tasks.TaskCompletionSource`1 retry, System.Data.Common.DbConnectionOptions userOptions, System.Data.ProviderBase.DbConnectionInternal oldConnection, System.Data.ProviderBase.DbConnectionInternal& connection) at offset 493
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(System.Data.Common.DbConnection outerConnection, System.Data.ProviderBase.DbConnectionFactory connectionFactory, System.Threading.Tasks.TaskCompletionSource`1 retry, System.Data.Common.DbConnectionOptions userOptions) at offset 86
at System.Data.SqlClient.SqlConnection.TryOpenInner(System.Threading.Tasks.TaskCompletionSource`1 retry) at offset 69
at System.Data.SqlClient.SqlConnection.TryOpen(System.Threading.Tasks.TaskCompletionSource`1 retry) at offset 122
at System.Data.SqlClient.SqlConnection.Open at offset 93
at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection at offset 36
at Hangfire.SqlServer.SqlServerJobQueue.Dequeue(System.String[] queues, System.Threading.CancellationToken cancellationToken) at offset 87
at Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext context) at offset 47
at Hangfire.Server.AutomaticRetryProcess.Execute(Hangfire.Server.BackgroundProcessContext context) at offset 30
and
System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception: The wait operation timed out
at System.Data.SqlClient.SqlInternalConnection.OnError(System.Data.SqlClient.SqlException exception, System.Boolean breakConnection, System.Action`1 wrapCloseInAction) at offset 39
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(System.Data.SqlClient.TdsParserStateObject stateObj, System.Boolean callerHasConnectionLock, System.Boolean asyncClose) at offset 319
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(System.Data.SqlClient.TdsParserStateObject stateObj, System.UInt32 error) at offset 396
at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync at offset 139
at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket at offset 74
at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer at offset 73
at System.Data.SqlClient.TdsParserStateObject.TryReadByte(System.Byte& value) at offset 25
at System.Data.SqlClient.TdsParser.TryRun(System.Data.SqlClient.RunBehavior runBehavior, System.Data.SqlClient.SqlCommand cmdHandler, System.Data.SqlClient.SqlDataReader dataStream, System.Data.SqlClient.BulkCopySimpleResultSet bulkCopyHandler, System.Data.SqlClient.TdsParserStateObject stateObj, System.Boolean& dataReady) at offset 311
at System.Data.SqlClient.TdsParser.Run(System.Data.SqlClient.RunBehavior runBehavior, System.Data.SqlClient.SqlCommand cmdHandler, System.Data.SqlClient.SqlDataReader dataStream, System.Data.SqlClient.BulkCopySimpleResultSet bulkCopyHandler, System.Data.SqlClient.TdsParserStateObject stateObj) at offset 32
at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(System.Byte[] buffer, System.Data.SqlClient.TransactionManagerRequestType request, System.String transactionName, System.Data.SqlClient.TransactionManagerIsolationLevel isoLevel, System.Int32 timeout, System.Data.SqlClient.SqlInternalTransaction transaction, System.Data.SqlClient.TdsParserStateObject stateObj, System.Boolean isDelegateControlRequest) at offset 572
at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(System.Data.SqlClient.TransactionRequest transactionRequest, System.String transactionName, System.Data.IsolationLevel iso, System.Data.SqlClient.SqlInternalTransaction internalTransaction, System.Boolean isDelegateControlRequest) at offset 434
at System.Data.SqlClient.SqlInternalConnection.BeginSqlTransaction(System.Data.IsolationLevel iso, System.String transactionName, System.Boolean shouldReconnect) at offset 134
at System.Data.SqlClient.SqlConnection.BeginTransaction(System.Data.IsolationLevel iso, System.String transactionName) at offset 67
at Hangfire.SqlServer.SqlServerJobQueue.Dequeue(System.String[] queues, System.Threading.CancellationToken cancellationToken) at offset 99
at Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext context) at offset 47
at Hangfire.Server.AutomaticRetryProcess.Execute(Hangfire.Server.BackgroundProcessContext context) at offset 30
Could this be a bug, am I missing something obvious, or do I need to provide more information?