Hangfire service is consuming excessive CPU and becomes unresponsive over time

bug?
1
  • High CPU Usage: Hangfire is consuming 50% CPU immediately upon startup.
  • Worker Configuration: The worker count is set to 4.
  • Job Backlog: There are 1.4 million pending jobs in the database.
  • Storage Options:
var options = new SqlServerStorageOptions
{
    UseRecommendedIsolationLevel = true,
    QueuePollInterval = TimeSpan.FromSeconds(30),
    JobExpirationCheckInterval = TimeSpan.FromHours(1),
    CountersAggregateInterval = TimeSpan.FromMinutes(5),
    DashboardJobListLimit = 50000,
    TransactionTimeout = TimeSpan.FromMinutes(1),
    SchemaName = "OHangfire",
    SlidingInvisibilityTimeout = TimeSpan.FromMinutes(15),
};
  • Background Job Server Configuration:
_server = new BackgroundJobServer(new BackgroundJobServerOptions() 
{ 
    WorkerCount = 4, 
    Queues = new[] { "1", "2", "3", "4", "5" }, 
    Activator = JobActivator.Current 
}, JobStorage.Current);
  • Job Execution Details:
    • Jobs send data to another system via HttpClient, which is created as a singleton.
    • Job enqueueing method:
public void SendData(Data data)
{
    if (data != null)
    {
        BackgroundJob.Enqueue("1", () => SendNewData(data.Id, null));
    }
}
  • Job execution method:
[Queue("1")]
[AutomaticRetry(Attempts = 0, DelaysInSeconds = new[] { 600, 1200, 1800 },
    OnAttemptsExceeded = AttemptsExceededAction.Delete)]
public static void SendNewData(long dataId, PerformContext context)
{
    // Loads data from DB and sends it to another system.
}