Consistent error, every 10 minutes

bug?
1

I just pushed hangfire to our various environments (develop, staging, prod). After logging into the dashboard, we see an error, every 10 minutes exactly:

Message: [SqlException: Connection Timeout Expired. The timeout period elapsed during the post-login phase. The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=0; handshake=5837; [Login] initialization=0; authentication=0; [Post-Login] complete=8990; ]
 System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection):286
 System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection):55
 System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection):493
 System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions):86
 System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry):69
 System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry):122
 System.Data.SqlClient.SqlConnection.Open():93
 Hangfire.SqlServer.SqlServerMonitoringApi.UseConnection[T](Func`2 action):42
 System.Lazy`1.CreateValue():198
 System.Lazy`1.LazyInitValue():134
 Hangfire.Dashboard.RazorPage.get_Statistics():21
 Hangfire.Dashboard.DashboardMetrics.<.cctor>b__a(RazorPage page):0
 Hangfire.Dashboard.JsonStats+<Dispatch>d__5.MoveNext():348
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__0.MoveNext():308
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext():133
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__0.MoveNext():464
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext():133
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1+<Invoke>d__0.MoveNext():394
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1+<Invoke>d__0.MoveNext():394
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1+<Invoke>d__0.MoveNext():394
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1+<Invoke>d__0.MoveNext():394
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext():133
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
 Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext+<DoFinalWork>d__2.MoveNext():132
 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
 Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.EndFinalWork(IAsyncResult ar):7
 System.Web.HttpApplication+AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute():215
 System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously):60
Message: [Win32Exception: The wait operation timed out]

hangfireErrors.png1596×726 137 KB

2

Another note, since I woke up and closed those browser sessions from hangfire’s dashboard, the errors seemed to have stopped.

EDIT:
After opening the dashboard again on both sites, the errors are starting again.

3

The timeout period elapsed during the post-login phase. The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.

Are you using SQL Azure? It has a lot of timeouts. Please also ensure your connection pool is large enough for your current worker number. What number of workers do you have, and what number of connections do you have in a pool?

4

No SQL Azure - regular SQL on AWS EC2 instance.

I only have 5 workers currently (the default). I have not specified any connection pooling in my connection string (so I think 100 is the default).