Here is the premise:
I have a rest service that handle transfer aroudn the world with a thirdparty transfer system.
Thus we get a transfer request and then enqueue a job in hangfire, which process that transfer.
I started doing performance test to determine our service capacity , SLO, etc
Our service is hosted on a VM with 2 processor and 16gb or ram.
Hangfire is configured with 40 worker count.
The test consist in launching 40 jobs on the server and then with 100 users that call an api and measure resquest time, etc etc
Another test consist in launching 100 jobs(60 queued) on the server and then with 100 users that call an api and measure resquest time, etc etc
While doing my test I noticed something really really strange, our service performs better when there is jobs queued versus when there isnt.
I tried on 2 differents machine, used different order of magnitude and the result is always the same… I get better performance when I have at least twice the amount of job queued versus job currently running which does not make sens to me at all because in the end the same amount of jobs is running and I would expect similar result and not an increase in performance.
Thus my question is: Does hangfire act differently (polling, locks, takes more resources,etc ) , when there is queued job vs when there isnt ??!?!