Failed job breaking dashboad

bug?
1

I have jobs failing and constantly breaking the /hangfire/jobs/failed page. I can circumvent it by using the query string ?from=x&count=y where the set returned omits the culprit.

I have updated Admin locally and pointed to our live Redis to no avail.

[NullReferenceException: Object reference not set to an instance of an object.]
   Hangfire.Dashboard.Pages.FailedJobsPage.Execute() +2127
   Hangfire.Dashboard.RazorPage.TransformText(String body) +26
   Hangfire.Dashboard.RazorPage.ToString() +7
   Hangfire.Dashboard.RazorPageDispatcher.Dispatch(DashboardContext context) +65
   Hangfire.Dashboard.<>c__DisplayClass1_2.<UseHangfireDashboard>b__1(IDictionary`2 env) +430
   Microsoft.Owin.Mapping.<Invoke>d__0.MoveNext() +274
   System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +92
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58
   Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<RunApp>d__5.MoveNext() +182
   System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +92
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58
   Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<DoFinalWork>d__2.MoveNext() +180
   Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) +69
   Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.EndFinalWork(IAsyncResult ar) +64
   System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +380
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155

Has anyone else had the same issue? If any further information is required let me know.

2

What storage are you using? Are you performing manual clean-ups in your storage bypassing Hangfire methods?

3

Hi Odinserj,

I have exactly same problem. I’am using Redis Storage

4

This problem may be caused by wrong maxmemory-policy setting in Redis. The only correct value for non-cache usage of Redis (like with Hangfire) is noeviction. @gungorugur, what is the value for this setting in your case? Could you paste here the output of the info Redis command?

5

Hi Odin
I pasted output here , thanks for helping.

6

I see a lot of evicted keys. What’s the exact value for the maxmemory-policy setting? It should be set to noeviction as written in the docs (for Hangfire.Pro.Redis, but anyway). You can see or change it in the AWS Dashboard.

After setting this change, you should manually perform cleanup of the following entries, based on your background job id:

del hangfire:job:{jobId} 
del hangfire:job:{jobId}:history
del hangfire:job:{jobId}:state
zrem hangfire:failed {jobId}
7

Hi @odinserj,

I am also facing the same NullReferenceException problem with Redis Storage.

The maxmemory-policy is set to noeviction .

I am using
Hangfire.Core = 1.7.6
Hangfire.Pro = 2.2.1
Hangfire.Pro.Redis = 2.5.2

Below is the stacktrace

[NullReferenceException: Object reference not set to an instance of an object.] Hangfire.Dashboard.Pages.FailedJobsPage.Execute() +2172 Hangfire.Dashboard.RazorPage.TransformText(String body) +30 Hangfire.Dashboard.RazorPageDispatcher.Dispatch(DashboardContext context) +88 Hangfire.Dashboard.<>c__DisplayClass1_1.<UseHangfireDashboard>b__1(IDictionary2 env) +617
Microsoft.Owin.Mapping.d__0.MoveNext() +485
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +32
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +62
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.d__5.MoveNext() +197
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +32
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +62
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.d__2.MoveNext() +184
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +32
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) +118
System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +417
System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +75
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +158`

8

hi @odinserj,

I upgraded the Hangfire.Core to 1.7.8 as in the Release notes I saw the below bug fix
• Fixed – Argument with a null value is skipped when displaying job arguments in Dashboard UI

But even after upgrading it to the latest version I am still facing the NullReferenceError for “hangfire/jobs/failed” on Dashboard.

Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.8.3815.0

Please let me know if you know anything related to this issue.

9

@odinserj,

I found out what the issue was, we’re adding a expiry_date to the failed jobs and when the job is deleted after the expiry the job value is null and there are no null checks in the below code.

It seems I don’t have the access to create a new branch and a pull request. Could you please give me the access to create a pull request or could you please add null checks as you have added for SucceededJobs.