Error occurred during execution of 'Server Heartbeat'-Load Balancer

question
1

hello,
I have the following :

  • 2 web servers running in a load balanced setup.

  • Each of these can enqueue Recurring Jobs with the following code in OwinStartup.cs :

    GlobalConfiguration.Configuration.UseSqlServerStorage(“ConnectionString”)
    .UseFilter(new LogFailureAttribute());
    app.UseHangfireServer();

    RecurringJob.AddOrUpdate(“Job”, () =>FunctionTrigger(), Cron.Daily(5, 0));

The FunctionTrigger is a simple function using Stateless Transaction session to insert records into the Database.

So when am using one Server the jobs running successfully, But in load balance an error happens and no records inserted into the DB .

This is the log error:

Hangfire.Server.ServerHeartbeat - Error occurred during execution of 'Server Heartbeat' component. Execution will be retried (attempt 2 of 2147483647) in 00:00:03 seconds.
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Hangfire.SqlServer.SqlServerStorage.GetConnection()
   at Hangfire.Server.ServerHeartbeat.Execute(CancellationToken cancellationToken)
   at Hangfire.Server.AutomaticRetryServerComponentWrapper.ExecuteWithAutomaticRetry(CancellationToken cancellationToken)

I am using Hangfire 1.4.5

**Is this because the job runs tow times on each server ?! how can i avoid that. ?! **
Thanks.

2

Any one tried this out??!

3

What number of workers do you have on that server? You can see the value on the Servers page. High number of workers may exhaust all the connections from the pool. Depending on your needs, you should increase the Max Pool Size setting in the connection string, or decrease the number of workers.

5

Hello Thank you but,they are 20 Workers in each Server, and the Pool Size is the default i think 100. So where is the porblem