Hi,
I have a MySQL database hosted on AWS RDS as my Hangfire database.
I recently upgraded from .net core 2.2 to .net 6 and whilst everything works, the IOPS from Hangfire has increased 10 fold. I can see that it is hangfire causing it as top SQL b y Load by Waits (AAS) shows:
DELETE FROM DistributedLock WHERE RESOURCE = ?
INSERT INTO DistributedLock ( RESOURCE , CreatedAt ) SELECT ?..
at the top.
I’m continuing to explore the issue and will post here if I find the cause/fix.
I may have foggy memory here of the timelines but at some point I believe there was a change to how aggressively the job scheduling process polls the database. So going from .net 2.2 all the way to .net 6 I wonder how many major/minor versions your Hangfire package upgrade spanned. This led me to thinking you might to compare the old frequency of polling from the package version you were using vs the new version’s default.
Otherwise, iirc it is configurable so you could tweak that to get your IOPS under control in the interim.
Thanks @bhehe. I’m late getting back to this one - unfortunately the problem is still here.
I ran some memory tests using JetBrains dotMemory with .net core 2.1 and .net 6 version of the code and the results are very different. Both have QueuePollInterval set to 15 seconds. The monitoring shows periods of aggressive memory growth on .net 6 that aren’t there under 2.1.
As per the original finding on AWS, it seems like there has been a massive increase in the amount is attempting a huge amount of connections for each poll.
@bhehe when you refer to polling frequency being made more aggressive, I assume you aren’t referring to the setting for QueuePollInterval? I can’t see any other setting that would have an impact.
Of course, it could also be the MySQL hangfire add-on which appears to no longer be maintained.