How to configure concurrency and retry with multiple server instances

question
, ,
1

I’ve created a multi-server setup, with MS SQL storage, where there’s one central server with an hangfire dashboard and the default queue, but no jobs, and all other servers will “plug in” with their own queues and jobs.

The central server with dashboard looks like this:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHangfire(h =>
{
    h.UseSqlServerStorage(builder.Configuration.GetConnectionString("MyConnection"), new SqlServerStorageOptions()
    {
        SchemaName = "hangfire",
    }).UseColouredConsoleLogProvider()
    .UseConsole()
    ;
})
;
builder.Services.AddHangfireServer(x =>
{
    x.Queues = [
        "default"
        ];
    x.WorkerCount = 1;
});
var app = builder.Build();
app.UseHangfireDashboard("", options: new()
{
    DashboardTitle = "Central hangfire",
    Authorization = [new HangfireAuthorizationFilter()]
});

app.Run();

And one of the nodes looks like this:

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHangfire(h =>
{
    h.UseSqlServerStorage(builder.Configuration.GetConnectionString("MyConnection"), new SqlServerStorageOptions()
    {
        SchemaName = "hangfire",
    })
    .UseConsole() ;
})
    .AddHangfireConsoleExtensions();
builder.Services.AddHangfireServer(s =>
{
    s.WorkerCount = 4;
    s.Queues = [nameof(IScheduledJob1).ToLowerInvariant()];
});
builder.Services.AddTransient<IScheduledJob1, ScheduledJob1>();

var app = builder.Build();

app.MapGet("/", () => "Hello scheduled job 1!");
var recurringJob = app.Services.GetRequiredService<IRecurringJobManager>();

recurringJob.AddOrUpdate<IScheduledJob1>("scheduled job 1", nameof(IScheduledJob1).ToLowerInvariant(), s => s.RunAsync(CancellationToken.None), Cron.Never());

app.Run();

The IScheduledJob1 interface exists in a shared library which is referenced by the central dashboard instance as well as the node in which this job implementation exists. This ensures hangfire can create an instance of the job once triggered.

Now I’m trying to configure concurrency and retry on the job using the [AutomaticRetry] and [DisableConcurrentExecution] attributes. But hangfire doesn’t seem to recognise this, so the default settings are applied. I’ve tried this attributes on the shared IScheduledJob1 interface as well as on the actual ScheduledJob1 implementation, but neither seem to work.

    public interface IScheduledJob1
    {
        [AutomaticRetry(Attempts = 3)]
        [DisableConcurrentExecution(60 * 24 * 365)]
        Task RunAsync(CancellationToken ct = default);
    }

    public class ScheduledJob1(ILogger<ScheduledJob1> logger) : IScheduledJob1
    {
        [AutomaticRetry(Attempts = 3)]
        [DisableConcurrentExecution(60 * 24 * 365)]
        public async Task RunAsync(CancellationToken ct = default)
        {
            logger.LogTrace("trace log");
            logger.LogDebug("debug log");
            logger.LogWarning("warning log");
            logger.LogError("error log");
            logger.LogCritical("critical log");
            await Task.Delay(20000);
            //throw new Exception("test");
            logger.LogInformation("Scheduled job 1 executed");
        }
    }

How can I configure this for a specific job within my setup?