Hi,
we use hangfire with batches and Queues and we faced an “issue”.
We have jobs like this
[Hangfire.Queue(“a-altapriorita”)]
public async Task TestJob(string method)
{
[…]
return;
}
differents enqueing method seems to not always use queues
BackgroundJob.Enqueue(x => x.TestJob(“BackgroundJob”));
var batchid = BatchJob.StartNew(x =>{x.Enqueue(x => x.TestJob(“StartNew”));});
BatchJob.Attach(batchid, x =>{x.Enqueue(x => x.TestJob(“Attach”));});
(same for “StartNew”)
Are we missing some configuration?
Really appreciate your work!
Hi, I don’t see the issue, and the Attach job goes to the correct queue.
What Hangfire.Pro version you are using, and is it possible that the Attach method was processed by an older version of the code base without the Queue attribute specified?
we are using redis with this configuration
<TargetFramework>net10.0</TargetFramework>
<PackageReference Include="Hangfire.AspNetCore" Version="1.8.23" />
<PackageReference Include="Hangfire.Pro.Redis" Version="3.3.1" />
<PackageReference Include="Hangfire.Throttling" Version="1.4.3" />
builder.Services.AddHangfire(c => c
.SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
.UseSimpleAssemblyNameTypeSerializer()
.UseRecommendedSerializerSettings()
.UseThrottling()
.UseRedisStorage("XXXXXXXX",
new Hangfire.Pro.Redis.RedisStorageOptions
{
Database = XXXXXX,
})
.UseBatches(TimeSpan.FromDays(1), new ExpirationTimeFilterProvider())
.UseFilter(new ExpirationTimeAttribute()));
builder.Services.AddHangfireServer(options =>
{
options.WorkerCount = Environment.ProcessorCount * XXXXX;
options.Queues = new[]
{
"a-altapriorita",
"b-mediapriorita",
"c-bassapriorita",
"z-default"
"default"
};
});
The methods are enqueued and processed by the same application
we found that this filter was the problem
.UseBatches(TimeSpan.FromDays(1), new ExpirationTimeFilterProvider())
public class ExpirationTimeAttribute : JobFilterAttribute, IApplyStateFilter
{
public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
if (context.NewState is SucceededState)
context.JobExpirationTimeout = TimeSpan.FromHours(8);
}
public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
}
}
public class ExpirationTimeFilterProvider : IJobFilterProvider
{
public ExpirationTimeFilterProvider()
{
}
public IEnumerable<JobFilter> GetFilters(Job job)
{
yield return new JobFilter(new ExpirationTimeAttribute(), JobFilterScope.Global, null);
}
}
It was created to optimize jobs expiration because we noticed that batch jobs clogs our redis server.
I’ve updated it like this
public IEnumerable GetFilters(Job job)
{
foreach (var filter in JobFilterProviders.Providers.GetFilters(job))
{
yield return filter;
}
yield return new JobFilter(new ExpirationTimeAttribute(), JobFilterScope.Global, null);
}
Now the queues works again
Still we don’t understand why in production some jobs enqueued by batches were still using queues and some doesn’t
enqueue with “start new”
enqueued with “attach”
both have the correct attribute
I think that the following line:
.UseBatches(TimeSpan.FromDays(1), new ExpirationTimeFilterProvider())
should be changed to:
.UseBatches(TimeSpan.FromDays(1))
The ExpirationTimeAttribute filter is already registered with the UseFilter method, and batches know about the main filter provider that’s modified when calling the UseFilter. Filter providers are for other use cases, like backed by some IoC container or modules, and it is a very rare case when you’d need a custom filter provider.
I believe the difference was related by the side which created a background job – if it was created by an application, then the queue attribute was applied correctly, but when a batch was responsible for changing state of a job (such as batch continuation), then QueueAttribute filter was completely ignored, since custom filter provider wasn’t aware of it.
