Hangfire Console Server crashes

bug?
1

Hello

I’m right now using Hangfire to host some processes. The architecture is looking like that:

  1. We have 1 hangfire Dashboard
  2. We have a lot of single console applications as backgroundjobservers. 1 application = 1 queue.

My problem is:

When we host a lot of console applications they sometimes are crashing on error:

Unhandled Exception: System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught) —> System.ObjectDisposedException: Cannot access a disposed object.
Object name: ‘System.Net.Sockets.Socket’.
at System.Net.Sockets.Socket.SetSocketOption(SocketOptionLevel optionLevel, SocketOptionName optionName, Int32 optionValue)
at System.Data.SqlClient.SNI.SNITCPHandle…ctor(String serverName, Int32 port, Int64 timerExpire, Object callbackObject, Boolean parallel)
— End of inner exception stack trace —
at System.Data.SqlClient.SqlInternalConnectionTds…ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.PoolCreateRequest(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

How to avoid this problem? After this error applications can not start again automatically.

This is our implementation of single backgroundjobserver

var builder = new ContainerBuilder();

        builder.RegisterType<Executor>().As<IExecutor>().AsImplementedInterfaces().InstancePerBackgroundJob();

        GlobalConfiguration.Configuration.UseAutofacActivator(builder.Build());

        var config = JsonConfigurationProvider.GetConfiguration();

        GlobalConfiguration.Configuration.UseSqlServerStorage(config.GetConnectionString("Hangfire"));

        GlobalConfiguration.Configuration.UseFilter(new PreserveOriginalQueueAttribute());

        var options = new BackgroundJobServerOptions
        {
            Queues = new[] { "executor_" + args[0] + "_" + args[1] },
            WorkerCount = 5
        };

        var cronConfig = new CronConfig();

        config.GetSection("CronConfig").Bind(cronConfig);

        using (var server = new BackgroundJobServer(options))
        {
            RecurringJob.AddOrUpdate<IExecutor>("executor-" + args[0] + "-" + args[1], x => x.Execute(args[0]), args[1] + cronConfig.Setup, queue: "executor" + args[0] + "_" + args[1]);
            Console.WriteLine("Hangfire Server started. Press any key to exit...");
            Console.ReadKey();
        }
2

Hmmm strange… The console application keeps open? So nothing is firing the Console.ReadKey() causing the backgound job server to be disposed, right? Maybe something on the autofac ia dropping the connection?

3

Nothing is firing the Console.ReadKey(). My console applications are hosted with docker on machine with linux. I think Hangfire can’t handle connection error when it can’t connect to its database (sometimes our server is unavailable for few seconds). I found temporary solution. On the BackgroundJobServer Layer I used try catch and when console app crash, in catch block im running again Main method, so it is hosting server again, but I think this is not good solution