System.OutOfMemoryException during sending heartbeat

bug?
1

Hello Hangfire Team,

I am encountering a recurring issue with Hangfire in my production environment, which manifests approximately once every six months. The Hangfire server is running as a Windows Service, and my application is based on the .NET Framework 4.8. The issue involves exceptions that lead to a failure in Hangfire’s processes.

Console Log Entries:

  1. Warning Entry:

    • WARN: Server xxxxxxxx:8080:aab93526 encountered an exception while sending heartbeat
    • EXCEPTION: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at Hangfire.SqlServer.SqlServerStorage.GetConnection() at Hangfire.Server.ServerHeartbeatProcess.Execute(BackgroundProcessContext context)

  2. Error Entry:

    • ERROR: RunTask~Message:Error while generating Snapshot Changes: | FormattedException:Msg: 'Exception of type 'System.OutOfMemoryException' was thrown.', Type: 'OutOfMemoryException'

  3. Fatal Entry:

    • FATAL: Dispatcher is stopped due to an exception, you need to restart the server manually. Please report it to Hangfire developers.
    • EXCEPTION: System.InvalidOperationException: Instance is read-only. at System.Collections.EmptyReadOnlyDictionaryInternal.Add(Object key, Object value) at Hangfire.Processing.BackgroundExecution.HandleException(Guid executionId, Exception exception, TimeSpan& delay) at Hangfire.Processing.BackgroundExecution.Run(Action2 callback, Object state) at Hangfire.Processing.BackgroundDispatcher.DispatchLoop()`

These issues result in significant disruptions, necessitating a manual server restart. The infrequency of the problem makes diagnosis and prediction challenging.

I seek advice or guidance on the following:

  1. Potential causes for these exceptions, especially in the context of a .NET Framework 4.8 application running as a Windows Service.
  2. Best practices or configurations in a .NET Framework 4.8 environment that might help avoid these issues.
  3. Any known issues or patches in Hangfire related to these exceptions under similar environments.

Any insights or recommendations to address or mitigate these issues would be greatly appreciated.

Thank you for your assistance.