Performance issue in Hangfire db

question
1

We are using HangFire in production Azure SaaS environment and the DTU is reaching to 100% very often. On further analysis, we are seeing the below statement is causing spike.
(@now datetime,@count int)delete top (@count) T from [HangFire].[Job] T with (forceseek)
where ExpireAt < @now
option (loop join, optimize for (@count = 20000))

  1. How to reduce the batch count from 20000?
  2. Was that configurable? where?
  3. Do we need to do run any optimization scripts like reindex? Any recommendation and scripts for that?
1 Like
2

Hi There

Did you find a solution to this?

We are experiencing the same issue. This query runs on every hour and every half hour. When it does, our DTU hits 100% and we experience SQL timeout issues for other queries at this time.

I was thinking it would be preferable for us to schedule the deletion at a quiet time of day. I assume this is the cleanup job which removes any jobs which are older than (default) 1 day.

I think the properties we need to set are the JobExpirationCheckInterval and DeleteExpiredBatchSize of the SqlServerStorageOptions.

JobExpirationCheckInterval is defaulted to 30 minutes. DeleteExpiredBatchSize is defaulted to -1 on instantiation, so I assume HangFire interprets a -1 value as the default 20,000.

In our case we HangFire is processing a high volume of very small tasks, and once complete, we don’t really care about them. So I think we need to set these jobs to expire very quickly, i.e. 15 minutes rather than 1 day, and then I believe the 30 minute deletion of expired jobs will be more efficient, or we could have the deletion interval set to say every 6 hours, and the batch a higher value such as 50,000.

3

the same, I want specific solutions for development?

Hight CPU 100% when view dashboad and click menu

image
image332×364 8.06 KB

                Hangfire.GlobalConfiguration.Configuration.SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
               .UseSimpleAssemblyNameTypeSerializer()
               .UseSqlServerStorage(ConnectionString, new SqlServerStorageOptions
               {
                   CommandBatchMaxTimeout = TimeSpan.FromMinutes(5), 
                   SlidingInvisibilityTimeout = TimeSpan.FromMinutes(10),
                   QueuePollInterval = TimeSpan.FromSeconds(15),
                   UseRecommendedIsolationLevel = false,được khuyến nghị
                   DisableGlobalLocks = true,
                   JobExpirationCheckInterval = TimeSpan.FromMinutes(10),
                   //UsePageLocksOnDequeue = true, 
               }).WithJobExpirationTimeout(TimeSpan.FromMinutes(30))
                 .UseDashboardMetric(DashboardMetrics.FailedCount);

the ultimate bad solution to make it work
TRUNCATE TABLE [HangFire].[AggregatedCounter]
TRUNCATE TABLE [HangFire].[Counter]
TRUNCATE TABLE [HangFire].[JobParameter]
TRUNCATE TABLE [HangFire].[JobQueue]
TRUNCATE TABLE [HangFire].[List]
TRUNCATE TABLE [HangFire].[State]
DELETE FROM [HangFire].[Job]
DBCC CHECKIDENT (‘[HangFire].[Job]’, reseed, 0)
UPDATE [HangFire].[Hash] SET Value = 1 WHERE Field = ‘LastJobId’