Autoscaling support?

question
1

Does Hangfire support autoscaling? I’m looking for something similar to what celery has, to grow or shrink task processors based on load: http://celery.readthedocs.org/en/latest/userguide/workers.html#autoscaling

Thanks!

2

Whooh, autoscaling looks magically. What do you want to achieve with autoscaling (except the autoscaling support itself :smile:)?

3

Thanks for the reply. If the load is high on the system running the tasks I would like to increase the number of workers. Less importantly, I want to kill off workers as the load decreases. If this isn’t currently possible, or isn’t the best approach for some reason, then can you tell me what the load strategy typically is with Hangfire? Is best practice just to start the maximum number of workers (using concurrency control) appropriate for the given machine and let them get used by Hangfire as needed? And then use server autoscaling (e.g. http://aws.amazon.com/autoscaling/) to spin up more machine instances if the single server is not sufficient? Essentially, I want to know the load scaling strategy to use with Hangfire.

4

I have another question about autoscaling. For now I’m using Hangfire with Redis storage on Azure WebSite with autoscale. If the single server not sufficient then starts the second machine instance; on second instance Hangfire uses Redis storage with another prefix. So how can I see information about all jobs (from both of these instances) on one Dashboard?
For now I can see information about jobs that running in one of these instances and I’m not able to open Dashboard of particular instance.

5

@subreptio, why do you use different prefixes for different instances? Hangfire is webfarm-friendly, and there is no issues in running different instances using the same storage at the same time. However, you are able to plugin multiple dashboards:

var filters = new[] { new LocalRequestsOnlyAuthorizationFilter() };

var storage1 = new RedisStorage(
    "localhost:6379", 0, new RedisStorageOptions { Prefix = "hangfire:1" });
app.MapHangfireDashboard("/hangfire/1", filters, storage1);

var storage2 = new RedisStorage(
    "localhost:6379", 0, new RedisStorageOptions { Prefix = "hangfire:2" });
app.MapHangfireDashboard("hangfire/2", filters, storage2);
6

@bitmask, this is a very complex question. Scaling problems are always solved starting from the current bottleneck. Depending on it, you are able to use the following strategies:

  1. Optimize background jobs. Poorly written background job will break all the other strategies :wink:
  2. Increase worker pool size – can increase throughput of a system with high number of I/O intensive tasks. May introduce excessive locking, context switching and other problems related to higher concurrency.
  3. Spread the processing across multiple machines – can increase throughput of a system with high number of CPU intensive jobs.
  4. Place Hangfire Server instance to a different machine – to not to compete with a web application for machine resources (CPU/RAM/etc).
  5. Use other job storage (for example, Redis storage instead of SQL Server storage) – can increase throughput of a system with huge amount of short-running jobs.
  6. Place job storage to a different machine – can increase overall throughput, if previous machine was heavy loaded by other processes.
  7. Use multiple job storages – can overcome limitations of your current storage. For example, 5 Redis storages and a, for example, round-robin background job client that adds each background job to a different storage.
1 Like
7

@odinserj I am truing to run two instances on different servers but with same datastorage and periodically get lock exception

Error occurred during execution of ‘DelayedJobScheduler’ process. Execution will be retried (attempt 18 of 2147483647) in 00:05:00 seconds.

System.TimeoutException Failed to get lock within timeout period TimeoutException System.TimeoutException: Failed to get lock within timeout period
at Hangfire.Redis.StackExchange.RedisLock…ctor(IDatabase Redis, String Key, String LockID, TimeSpan Timeout)
at Hangfire.Redis.StackExchange.FetchedJobsWatcher.ProcessQueue(String queue, RedisConnection connection)
at Hangfire.Redis.StackExchange.FetchedJobsWatcher.Execute(CancellationToken cancellationToken)
at Hangfire.Server.ServerProcessExtensions.Execute(IServerProcess process, BackgroundProcessContext context)
at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context) Void .ctor(StackExchange.Redis.IDatabase, System.String, System.String, System.TimeSpan) at Hangfire.Redis.StackExchange.RedisLock…ctor(IDatabase Redis, String Key, String LockID, TimeSpan Timeout)
at Hangfire.Redis.StackExchange.FetchedJobsWatcher.ProcessQueue(String queue, RedisConnection connection)
at Hangfire.Redis.StackExchange.FetchedJobsWatcher.Execute(CancellationToken cancellationToken)
at Hangfire.Server.ServerProcessExtensions.Execute(IServerProcess process, BackgroundProcessContext context)
at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context)

Ideally I want to keep one server in reserve but don’t know how to do it yet in better way