We have been running HangFire for a year are were facing a few issues.
Yesterday we encountered issue of our DB which HangFire uses hitting to 100% CPU usage.
We stopped the recurring Jobs and also we delete Job and JobQueue table(at that point JobQueue table has got 19000 rows).
However the DB server CPU didn’t went down.
Only when we stop the HangFire IIS app pool then the CPU usage went down. And we turn the app pool back on, it went up again.
Could you please comment on the issue if you have any explanation?
The web server that hosting HangFire hitting 100% CPU usage very frequently.
When that happen there are thousands of Jobs queuing. To resolve this normally we would delete Job and JobQueue tables.
Do you have any explanation on the cause of the issue?
We have some jobs without any retry by setting AutomaticRetry(Attempts = 0). But when they fail we saw them retrying.
Could you advise us how to get around with the issue?
Please let us know if you need more information.
We need to investigate, what causes such a load. Strictly saying, the ideal background processing will always cause 100% CPU usage, if the number of workers is more or equal to the logical processor count – all the CPU cores are busy, and network delays are minimal. But we should ensure that all is ok in your case.
- What queries cause the high CPU usage on your DB server? You can see the answers by using SQL Server Performance Dashboard.
- What methods cause the high CPU usage on your application server? You can use a profiler to get an answer to this question. I prefer to use JetBrains’ dotTrace. Samling method will not cause any significant delays to your application.
- Ensure you are placing the
AutomaticRetryAttribute on the correct method. If you enqueue your background jobs using base classes or interfaces, make sure you are applying the attribute to base classes, instead of derived ones
Thanks for the advice. We will try the tools on SQL DB and continue to check on the cause of the issue.
We were previously placing AutomaticRetry on derived classes and that’s why they didn’t work. They have been changed now and the retry behave as expected.
@Zarni_Aung did that solve your problem?
I am also facing the same issue.But there is a slight difference.The difference is when am adding recurring job in hangfire the cpu usage will go above 90%,no effect on the count of recuring jobs.
Let me know what the issue is?