OnCreating not called

If I go to recurring jobs tab and trigger job myself OnCreating method is never called. If it gets triggered normally everything is fine.

First event that I get is public void OnStateElection(ElectStateContext context) with states going from enqueued -> processing.

Same thing happens if I enqueue a job trough code. Only instance where OnCreating is called when the recurring job gets triggered by itself. OnPerforming is the only one I can rely on calling for every job.

public class DisableMultipleQueuedItemsAttribute : JobFilterAttribute, IClientFilter, IServerFilter, IElectStateFilter
{
    private static readonly TimeSpan LockTimeout = TimeSpan.FromSeconds(10);
    private static readonly List<string> ReleaseStates = new List<string> { DeletedState.StateName, FailedState.StateName };

    public void OnCreating(CreatingContext filterContext)
    {
        Console.WriteLine($"Creating {filterContext.Job.Method.Name}");
    }

    public void OnCreated(CreatedContext filterContext)
    {
        Console.WriteLine($"Created {filterContext.BackgroundJob.Id}");
    }

    public void OnPerforming(PerformingContext filterContext)
    {
        Console.WriteLine($"Performing ({filterContext.BackgroundJob.Id}) {filterContext.BackgroundJob.Job.Method.Name}");

        if (!AddFingerprintIfNotExists(filterContext.Connection, filterContext.BackgroundJob))
        {
            filterContext.Canceled = true;
        }
    }

    public void OnPerformed(PerformedContext filterContext)
    {
        Console.WriteLine($"Performed {filterContext.BackgroundJob.Id}");

        // If job finished with success we remove lock.
        if (filterContext.Exception == null || filterContext.ExceptionHandled)
        {
            RemoveFingerprint(filterContext.Connection, filterContext.BackgroundJob);
        }
    }

    public void OnStateElection(ElectStateContext context)
    {
        Console.WriteLine($"State change ({context.BackgroundJob.Id}) {context.CurrentState} -> {context.CandidateState.Name}");

        // If job is going to release state and it isn't in failed state we release the lock.
        // if job is already in failed state we don't release lock because lock can be from a running job, failed job already had it's lock removed.
        if (ReleaseStates.Contains(context.CandidateState.Name) && context.CurrentState != FailedState.StateName)
        {
            RemoveFingerprint(context.Connection, context.BackgroundJob);
        }
    }

    private static bool AddFingerprintIfNotExists(IStorageConnection connection, BackgroundJob job)
    {
        using (connection.AcquireDistributedLock(GetFingerprintLockKey(job), LockTimeout))
        {
            var fingerprint = connection.GetAllEntriesFromHash(GetFingerprintKey(job));

            if (fingerprint != null && fingerprint.ContainsKey("Timestamp"))
            {
                // Actual fingerprint found, returning.
                return false;
            }

            // Fingerprint does not exist, it is invalid (no `Timestamp` key),
            // or it is not actual (timeout expired).
            connection.SetRangeInHash(
                GetFingerprintKey(job),
                new Dictionary<string, string>
                {
                    { "Timestamp", DateTimeOffset.UtcNow.ToString("o") }
                });

            return true;
        }
    }

    private static string GetFingerprint(BackgroundJob job)
    {
        string parameters = string.Empty;
        if (job.Job.Args != null)
        {
            parameters = string.Join(".", job.Job.Args);
        }

        if (job.Job.Type == null || job.Job.Method == null)
        {
            return string.Empty;
        }

        var fingerprint = $"{job.Job.Type.Name}.{job.Job.Method.Name}.{parameters}";

        return fingerprint;
    }

    private static string GetFingerprintKey(BackgroundJob job)
    {
        return $"Fingerprint:{GetFingerprint(job)}";
    }

    private static string GetFingerprintLockKey(BackgroundJob job)
    {
        return $"{GetFingerprintKey(job)}:lock";
    }

    private static void RemoveFingerprint(IStorageConnection connection, BackgroundJob job)
    {
        using (connection.AcquireDistributedLock(GetFingerprintLockKey(job), LockTimeout))
        {
            using (var transaction = connection.CreateWriteTransaction())
            {
                transaction.RemoveHash(GetFingerprintKey(job));
                transaction.Commit();
            }
        }
    }
}
1 Like

How do you diagnose it is not called?
Keep in mind that filters may be called from different processes (if you have dashboard, client and server running separately).

When fire-and-forget job is triggered, the OnCreate filter will be executed by client process.
When recurring job is being executed on schedule, the OnCreate filter will be executed by server process.
When you trigger the job manually from dashboard, it will be executed by dashboard process.

To reliably work in all cases, your filter must be properly registered for each of the processes.

In your particular case, I believe OnPerforming/OnPerformed (i.e. implementing only IServerFilter) is pretty much enough.

It is, but I’m not sure how it works. Job gets assigned worker and then OnPerforming is called but when OnCreating is called then it’s without worker, as far as I can tell.

I have windows service that is executing jobs and filter is set globally there, but queuing is done from and MVC app that also has dashboard and I don’t have filter set there globally. I see what you are talking about processes and I’ll try that now.

Yes, OnCreating/OnCreated is called before/after the job (either fire-and-forget, or an instance of recurring job) is placed into queue, and OnPerforming/OnPerformed is called before/after it is picked from queue by a worker and executed.

Wait, so OnPerforming is done without worker also? If I have 1000 jobs in queue and 10 workers and I cancel them in OnPerforming method, do they get canceled 10 at the time as the workers get them or will they all be canceled at once before a worker gets assigned to them?

You said yes but then you wrote

OnPerforming/OnPerformed is called before/after it is picked from queue by a worker and executed.

So that means both OnCreating and OnPerforming is called before worker is assigned.

It is like this:

  1. Worker picks next job from the queue.
  2. OnPerforming is called for each registered filter.
    2.1. If one of filters cancels performing, it calls OnPerformed for all previous filters with Canceled flag set to true (so you can properly do cleanup in case you’ve initialized something in corresponding OnPerforming handler, even if the job was canceled) and exits.
  3. Job is being executed.
  4. OnPerformed is called for each registered filter.

So if you cancel jobs in OnPerforming, they’re canceled one after another, not in batches.

I just tried registering same filter globally in my MVC app and triggered job manually from dashboard. OnCreating was still not called, as if the filter is not set. I also tried to put filter on the method but still nothing.

So my last comment was wrong, It’s working now.
Thanks a lot