Decrease in the number of jobs per second

question
1

Hello,

I have up a test environment to validate the capacity of hangfire+redis to process a continus flow of short jobs (duration 100ms).

I use last version of hangfire.core and redis pro

  • Hangfire.Pro.Redis 3.0.3
  • Hangfire.AspNetCore 1.8.6

My scenario is basic :

  • I schedule 2000 jobs with a delay of 5s
  • All job use Task.Delay 100ms to simulate a short job

I use a test environment deployed on GCP with Kubernetes managed mode autopilot on, for this test redis is latest version of container redis/redis-stack, no specific customization of configuration used.

Here are the test results this more or less workers (pods)

total time	workers count	rate jobs / s	theorical jobs rates	capacity exploitation
47	20	42,55319149	200	21,27659574
44	20	45,45454545	200	22,72727273
42	20	47,61904762	200	23,80952381
40	20	50	200	25
46	20	43,47826087	200	21,73913043
34	60	58,82352941	600	9,803921569
30	60	66,66666667	600	11,11111111
22	60	90,90909091	600	15,15151515
17	60	117,6470588	600	19,60784314
18	60	111,1111111	600	18,51851852
32	200	62,5	2000	3,125
26	200	76,92307692	2000	3,846153846
24	200	83,33333333	2000	4,166666667
25	200	80	2000	4
30	200	66,66666667	2000	3,333333333
30	400	66,66666667	4000	1,666666667
29	400	68,96551724	4000	1,724137931
31	400	64,51612903	4000	1,612903226
26	400	76,92307692	4000	1,923076923
26	400	76,92307692	4000	1,923076923
  • total time : Is a total time for schedule and process all jobs
  • workers count : Is the worker * number of pods deploed
  • rate jobs / s : is the total number of executions over the number of jobs to be processed (2000)
  • theorical jobs rates : Is the total workers count * 10 (because the duration of 1 job is 100ms)
    capacity exploitation : (“rate jobs/s” / “theorical jobs rates”) * 100

What we are seeing is that on short-term jobs :

  • The theoretical capacity utilisation is low (max 25%)
  • Theoretical capacity decrease with the augmentation of servers and workers count (2% with 20 servers and 400 workers)

Questions :

  • Is there any specific configuration (core / redis) to optimise short-job processing ?
  • Is hangfire a suitable solution for handling a continuous flow of short-jobs ?