Scheduled Jobs not being Enqueued

question
,
1

We are on Hangfire 1.8.20, we have 19 million tasks in the scheduled queue, almost all of them are due to be ran now. They are not being enqueued fast enough. We have 19 servers * 5 workers each. Occasionally a burst of tasks will go to the enqueue status, and we then process those tasks very quickly, but then it stalls enqueueing more tasks.

Anyway I can get these tasks enqueued? We’re days behind on some of these tasks due date.

Our settings:

c.UseSqlServerStorage(builder.Config["ConnectionStrings:Hangfire"], new SqlServerStorageOptions
{
    CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
    SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
    QueuePollInterval = TimeSpan.Zero,
    UseRecommendedIsolationLevel = true,
    UsePageLocksOnDequeue = true,                       
    DisableGlobalLocks = true,
    PrepareSchemaIfNecessary = false //Dont try to recreate the tables in production every boot.
});
builder.Services.AddHangfireServer(options =>
{
    options.ServerName = Environment.MachineName;
    options.Queues = new[] { "default", "x0_web", "x1_mail" };
    options.WorkerCount = 5;
    options.MaxDegreeOfParallelismForSchedulers = 2;
});