I am using the “LogEverythingAttribute” class from Using Job Filters — Hangfire Documentation and I have added some code (specifically in the “OnStateElection” method) to log more details about errors.
The following code is incomplete (I removed the details of the way that we log exceptions or send emails, for example), but I hope that it’s helpful; use it or adapt it however you need.
I am using a few snippets of code from places like https://stackoverflow.com/questions/62644191/hangfire-send-emails-after-retry and from some Hangfire forum discussions and some of the comments posted in replies to issues on the Hangfire Github pages (I didn’t save links to any of those).
Without the changes below, the exception that will be logged or reported will only show a stack trace for line “x” in LogEverythingAttribute.cs. The added code retrieves the Message and StackTrace from the code where the original exception occurred.
This code also sends interim messages for retries, as well as fatal errors; and an “OK” message if one of the retry attempts succeeds.
public void OnStateElection(ElectStateContext context)
{
if (context?.CandidateState is FailedState || context?.CandidateState is SucceededState)
{
{
var emailHeader = string.Empty;
var msgText = string.Empty;
int? totalAttemptsAllowed = null;
bool sendLog = false;
bool sendException = false;
var currentRetryCount = context.GetJobParameter<int>("RetryCount");
var retryAttribute = GetCustomAttribute(context.BackgroundJob.Job.Type, typeof(AutomaticRetryAttribute)) as AutomaticRetryAttribute;
if (retryAttribute != null)
{
totalAttemptsAllowed = retryAttribute.Attempts;
}
if (context.CandidateState is FailedState failedState)
{
string exception = "";
var failed = context.CandidateState as FailedState ??
context.TraversedStates.FirstOrDefault(x => x is FailedState) as FailedState;
if (failed != null)
{
exception = $"Error Message: {failed.Exception.Message}" + Environment.NewLine +
$"Stack Trace: {failed.Exception.StackTrace}";
}
//FATAL
if (totalAttemptsAllowed.HasValue && currentRetryCount == totalAttemptsAllowed)
{
msgText = $"FATAL: {GetMessageText(context, currentRetryCount, totalAttemptsAllowed)}" + $" {exception}";
sendException = true;
}
//ERROR
else
{
msgText = $"FAIL: {GetMessageText(context, currentRetryCount, totalAttemptsAllowed)}" + $" {exception}";
// Don't send for every error (note that this assumes a total of 10 retries)
if (currentRetryCount is 1 or 5 or 9) { sendException = true; }
}
}
if (context.CandidateState is SucceededState)
{
if (currentRetryCount > 1) //OK
{
msgText = $"OK: {GetMessageText(context, currentRetryCount, totalAttemptsAllowed)}";
sendLog = true;
}
}
if (sendLog)
{
// Log exception, send email, etc.
Thread.Sleep(500);
}
if (sendException)
{
try
{
throw new InvalidOperationException($"ERROR: {msgText}");
}
catch (Exception ex)
{
// Process exception using your usual exception processing mechanism.
}
}
}
}
}
Please reply if you have any suggestions for improvement to the above code.