I have a long-running process. In some cases it seems that Hangfire attempts a retry of the job when the process is not complete yet.
After half an hour, I see a second attempt to run in the Hangfire log. (running on the same engine)
In my own log I see the original task succeeded (after more than half an hour). In my own log I also see the second attempt, which then returns a result, being logged by Hangfire. The result of the original job is not logged by HangFire.
Unfortunately I cannot attach a screenshot, due to forum restrictions.
Any idea why Hangfire attempts the second try? I have implemented job cancellation, but the first run is not cancelled by Hangfire before proceeding to the second try.
To handle unexpected worker terminations, Hangfire has so-called job invisibility timeout. By default it is set to 30 minutes. It is a storage-related option, please see the documentation:
Is there anything I can do to prevent the retry, other setting than the timeout to Timespan.MaxValue?
Side-question, based on the Sql server documentation stating it will set cancellation tokens with the timeout:
I did implement cancellation tokens, but the process does not seem to be cancelled. I queue my tasks on a specific queue, as such:
...
var client = new BackgroundJobClient(JobStorage.Current);
var state = new EnqueuedState("mylowercasequeuename");
client.Create(() => service.Run(JobCancellationToken.Null,...other params), state);
...
Task cancellation seems to work on other occasions as when I now kill a service that runs the job, my code handles the cancellation and logs it, and the service is terminated much faster than before.
The documentation for SQL server, which I’m using, states that the first attempt is to be cancelled after the retry timeout (by default 30 minutes).
My code does implement task cancellation; when a task is cancelled the process exits gracefully, logging the end. This works when I stop the service my Hangfire runner executes in. However, when the automatic retry kicks in, this task does not seem to be cancelled, as I see my code finishing normally in my logs.
The second attempt at running the same task is blocked by my code, returning a success towards Hangfire, internally logging a non-started process.
@odinserj Would you please tell me what happen after invisibility timeout if I use default Invisibility Timeout (30 mins) and set attribute AutomaticRetry = 0?
As my knowledge my job will be terminated and there is no other worker continue to process it.