Prevent Hangfire to make duplicate enqueue or processing jobs

bug?
1

I have a problem in hangfire that I have seen several days ago, I think the HF enqueue duplicate same jobs in Enqueued list, firstly I was thinking this is my logic problem but after doing patch I am facing with this problem again today so I enqueued job very easy like below:

BackgroundJob.Enqueue(() => RoutingCase.Start($"Routing {r.m_PatientName} ({r.m_RoutingFiles.Length + r.m_RoutingTempFiles.Length})=> {r.Server.Description}", r, null));

My changes patch to prevent duplicate jobs by hash of some properties is something like below:

var currentMonitorApi = JobStorage.Current.GetMonitoringApi();
bool detectDuplicatedRouteJob = false;
try
{
    detectDuplicatedRouteJob = currentMonitorApi.Queues().Where(ser => ser.Name.StartsWith("route"))
    .Any(routeQueue =>
    {
        //Check in Routing Processing jobs first
        bool dupl = currentMonitorApi.FetchedJobs(routeQueue.Name, 0, (int)routeQueue.Fetched).Any(pj => (pj.Value.Job.Args[1] as RoutingCase).GetHashCode() == r.GetHashCode())
        || routeQueue.FirstJobs.Any(job => ((job.Value.Job.Args[1]) as RoutingCase).GetHashCode() == r.GetHashCode()); //check from Enqueued Routing Jobs also
        return dupl;
    });
}
catch (Exception exp)
{
    m_Logging.Log(LoggingMode.Warning, $"Failed to Check Duplicaye Route job for, Routing {r.m_PatientName} ({r.m_RoutingFiles.Length + r.m_RoutingTempFiles.Length})=> {r.Server.Description}, EXP:{exp}");
}
if (detectDuplicatedRouteJob)
{
    m_Logging.Log(LoggingMode.Warning, $"Find Duplicate route job for Routing {r.m_PatientName} ({r.m_RoutingFiles.Length + r.m_RoutingTempFiles.Length})=> {r.Server.Description}, " +
        $"Study should be checked in client side to ensure that is exist also...");
}
else
    BackgroundJob.Enqueue(() => RoutingCase.Start($"Routing {r.m_PatientName} ({r.m_RoutingFiles.Length + r.m_RoutingTempFiles.Length})=> {r.Server.Description}", r, null));

duplicate_job
duplicate_job784×157 4.39 KB

as you can see in screenshot there is two duplicate jobs enqueued, I think the problem is in Hangfire that sometime enqueued one job multiple.
is there any solution or patch.
based on questions here and here.

Any help would be truly appreciated.

Thanks in advance.

2

How can you be sure this is a Hangfire problem and not a problem with your application code? Is it possible that the query you’re using to get the data for patient records that you’re queuing up Jobs for is occasionally returning multiple records when you’re only expecting one?

3

I am not sure , I said I think this may be a problem because of those provided links, I am enqueueing the job in one line that is under the control by GetHashCode(), so I am sure about my code because by testing on local and giving duplicate data the detectDuplicatedRouteJob be true, this is the signature of the method with AutomaticRetry attribute:

[DisplayName("{0}")]
[DynamicQueue("router")]
[AutomaticRetry(Attempts = 5, DelaysInSeconds = new int[] { 60, 60 * 3, 60 * 3 * 3, 60 * 3 * 3 * 3, 60 * 3 * 3 * 3 * 3 })]
public static void Route(string caption, RoutingCase rc, PerformContext context)
{ 
}