We are working on a user-facing batch job system, using Hangfire as the backend. An important requirement for us is a) support of very large batches (50-100k jobs per batch), and round-robin or otherwise simultaneous execution of batches. Hangfire out of the box doesn’t provide any facility simultaneous batch execution - queues have a priority order, so a batch runs to completion before other batches can be run. In order to work around this, I started modifying the JobStorage back end to provide a different implementation of FetchNextJob in order to provide the scheduling that we needed, and that’s where I immediately ran into many issues.
While it’s true that JobStorage is abstracted out and you can drop in your own replacement, extending the existing implementations is unnecessarily difficult to impossible. Many of the classes you would want to modify are marked as internal, so you can’t (for example) just subclass RedisConnection and override FetchNextJob. Likewise with the SqlServer / MSMQ implentation - the existing implementations are internal, or have many private rather than protected members. I’d like to be able to customize the implementation without needing to entirely fork and / or write an entire storage layer from scratch.
1 Like
It’s always easier to make switch a useful internal thing to public, than error-prone public stuff to internal. Methods and classes also require much more polishing work before releasing them to public, than internal ones. And sub-classing may also produce some weird bugs. And it’s easier to change internal members than public. That’s why there are a lot of internal classes and methods – to make everything easier.
But if there is a reason to make something public, I would be happy to make the change. But before, can you please share what customization you want to perform, to allow me to have better understanding of a problem?
I definitely understand the difficulty that exposing this sort of functionality entails, but without it, the extensibility story of hangfire is essentially limited to “write your own storage implementation”.
The things that I specifically need for my use case:
RedisConnection made public, so I can subclass and override FetchNextJob to implement round-robin queue management.In order to construct the RedisConnection, GetDatabase / GetSubscriber / Options on RedisStorage either need to be public or protected.
We need to use Redis because of the size of our batches, but a prototype implementation on SqlServer / MSMQ had the same problem. MsmqJobQueueProvider and MsmqJobQueueMonitoringApi are both internal, so I can’t re-use or wrap them. Because the source is available for these, of course, forking the existing implementation is possible but I think it’d be a stronger system if we could perform targeted overrides.
The batch filter system in Hangfire.Batches is entirely inaccessible, because all of the interfaces are internal and BatchStateChanger is sealed. Again, I can’t extend anything without doing a near total rewrite. We’d like to add notification alerts on batch completion, which ideally would be handled with IApplyBatchStateFilter.