Email to Notify Job Has Failed

question
1

Hi,

Is there any way out of the box to Notify a distribution list of people that a Job has failed? Lets say I don’t want Admin guys having to log on to the Web Interface (which is great btw) every morning to check whether any jobs have failed - I just want to send an email from Hangfire to a group of Users saying Job XXXXX has failed - please go to Web Interface to re-queue it?

Thanks

2

Manual checking is annoying, I know, and this is the problem that should be solved with logging. Different logging frameworks provide a feature to send email on some conditions.

HangFire uses Common.Logging library to log different events. The library provides common interfaces and provides adapters for different concrete logging libraries, such as NLog, Log4Net and so on. You can look at the full list of supported adapters at NuGet. To enable logging, you should do the following things:

  1. Choose, install & configure top-level logging library that is supported by the Common.Logging library and satisfies all of your requirements (can send emails, for example).
  2. Install the corresponding adapter for the Common.Logging library.
  3. Configure the Common.Logging to use installed adapter. Here is the documentation for the library.

HangFire uses the following semantic for error levels:

  • Trace – messages related to debugging HangFire itself.
  • Debug – for debugging your jobs, workers and job processing.
  • Info – for informational events, shows that everything was started and stopped, and very few other messages.
  • Warning – events that may lead to errors. For example, messages about failed jobs, that will be retried automatically, fall into this category.
  • Error – there was an error, and you should know about it (storage failure, job failure, etc.). But this error does not need human intervention, because an extra attempt will be made.
  • Fatal – disaster that requires human intervention – background processing does not work.
3

@odinserj

Hi Sergey - Thanks for your response - I am using Log 4 net in my application. Reading your post is it correct then that if I installed the Common.Logging Adapter for Log 4 Net - where would I specify that I want Fatal messages from Hangfire emailed to User - but other Errors I just want to Log to Log.txt file?

4

I confused you with the words “manual intervention”.

You said me that you want to know about job failures. As I said, HangFire emits warnings when they are failing, but will be retried automatically (early problem detection), and emits errors when jobs are failing with no auto-retry (and this is a problem).

I always used practice to send logging messages by email starting from the Warning level to know that some problems are trying to appear before they appear.

5

Ok - let us say for HangFire error levels of:

Trace/Debug/Info - write to log.txt file
Warning/Error/Fatal - send an email

What I am wondering is using Log 4 Net adapter where do I configure that for this first set of warnings for HangFire write to Log File but for second set of warnings for HangFire send an email - note that I would let to make the Fatal/Error/Warning specific for HangFire warnings as for other warnings/Error in the application I want to just write them to a log file - emailing an Admin User will not be a valid scenario for that.

6

I’m unfamiliar with Log4Net, but it seems to me that it allows you to choose different rules for different loggers. All HangFire logger names starting with the HangFire prefix and looks like HangFire.*, so you can try to use this rule.

7

@odinserj - Sergey - Thanks for your response - Yeah I should be able to get rule in place for HangFire.Error etc :smile:

8

HangFire logger names look like:

  • HangFire.Server.WorkerManager
  • HangFire.Server.Worker
  • HangFire.AutomaticRetryAttribute
  • etc

It is better to provide explicit rules to send email for warning and error levels for the whole HangFire.* wildcard.

9

Whilst I appreciate the correct approach may be above - implementing a new logging framework, or writing an adapter for Common.Logging is a considerable amount of work for the task of sending an email on failures.

Hence, I’m simply looking at the [HangFire].[Job] table, for anything with the StateName of “Failed”.

This could obviously break if and when Hangfire changes it’s table structure, or job state naming, but it works for now.

It will also only work for SQL Server backed implementations. There are certainly a few things where looking straight in the database seems easier than using the “correct process”.

It also seems job filters could be used to do the job, although I’ve not tried it myself:

http://docs.hangfire.io/en/latest/extensibility/using-job-filters.html

Thanks again to @odinserj for all the excellent work!

1 Like