Dashboard appears broken in Hangfire 1.5.2

Tags: #<Tag:0x00007f066bb0dc18>

I just recently upgraded to Hangfire 1.5.2, and the Dashboard appears to be broken when using Hangfire with MSMQ. Clicking on the “Jobs” tab yields the following exception:

    System.ApplicationException: Unable to parse PRIVATE=cfcc46cd-5782-4fa9-930a-01ec1f8bdc0a\00000012
   at MQTools.MessageQueueExtensions.GetCount(MessageQueue messageQueue)
   at Hangfire.SqlServer.Msmq.MsmqJobQueueMonitoringApi.GetEnqueuedAndFetchedCount(String queue)
   at Hangfire.SqlServer.SqlServerMonitoringApi.Queues()
   at Hangfire.Dashboard.Pages.QueuesPage.Execute()
   at Hangfire.Dashboard.RazorPage.TransformText(String body)
   at Hangfire.Dashboard.RazorPage.ToString()
   at Hangfire.Dashboard.RazorPageDispatcher.Dispatch(RequestDispatcherContext context)
   at Hangfire.Dashboard.MiddlewareExtensions.<>c__DisplayClass6.<>c__DisplayClass8.<UseHangfireDashboard>b__4(IDictionary`2 env)
   at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.<DoFinalWork>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.EndFinalWork(IAsyncResult ar)
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

However, it appears that Hangfire is otherwise functioning correctly (jobs are being pulled of the queue and processed).

I’ve tried deleting and re-creating the MSMQ, to no avail.

I’ve confirmed this behavior in two separate environments.

What queue path are you passing to Hangfire?

I can also confirm this with my environment, I just upgraded to 1.5.2 and have the same issue when viewing the jobs queue in the dashboard. The path I’m passing is basically the same as it is in the documentation since I only have one default queue.

GlobalConfiguration.Configuration.UseSqlServerStorage("Hangfire").UseMsmqQueues(".\hangfire-{0}")

And yes it is VB.Net :smile:

I’m passing “.\private$\hangfire-bfwf-{0}” for the queue path. It’s for a Private Queue with the label, “private$\hangfire-bfwf-default”.

Everything was working fine using Hangfire 1.3.4, but stopped working when I upgraded to 1.5.2.

This appears to be an issue introduced in HangFire 1.5.0. I systematically downgraded until the issue is not present in HangFire 1.4.7.

You can downgrade to 1.4.7 using NuGet, but you’ll have to manually change (or delete) the backing database schema to get back up-and-running.

Hmmm… per your comment on GitHub, it appears that the queue path format has changed?

You indicated an example format of:

FormatName:DIRECT=OS:XYZZY\private$\MyQueue

But I’m not sure how to parse that. Is there additional documentation for this new format?

well, in the startup.cs of my mvc website, which is hangfire client of a windows service located in another machine as hangfire server, i’m using:

config.UseSqlServerStorage(HangfireConnectionString).UseMsmqQueues(@“FormatName:Direct=OS:queueServerName\private$\hangfire-{0}”, “queue1”, “queue2”, “queue3”)

As i understood, that should be the way

I have tried with the …UseMsmqQueues(@“FormatName:Direct=OS:queueServerName\private$\hangfire-{0}”, “queue1”, “queue2”, “queue3”) but when I try to open the Jobs tab in Hangfire dashboard then I get this error:

[Win32Exception (0x80004005): Unknown error (0xc00e0069)]
   MQTools.MessageQueueExtensions.GetCount(String computerName, String queuePath) +528
   MQTools.MessageQueueExtensions.GetQueueCount(String computerName, String queueType, String queue) +156
   MQTools.MessageQueueExtensions.GetCount(MessageQueue messageQueue) +418
   Hangfire.SqlServer.Msmq.MsmqJobQueueMonitoringApi.GetEnqueuedAndFetchedCount(String queue) +148...

I have downloaded the 1.5.3 version of Hangfire source code, and run the tests. MsmqJobQueueMonitoringApiFacts.GetEnqueuedAndFetchedCount_ReturnsCorrectCounters fails for the same original parsing problem: System.ApplicationExceptionUnable to parse PRIVATE=a50790d8-7e9c-457b-b07a-fc902c5134c8\00000044

In MQTools.MessageQueueExtensions.GetCount(this MessageQueue messageQueue), I believe the correct member to parse for queue path should be messageQueue.QueueName instead of messageQueue.FormatName.

I.e.:

public static long GetCount(this MessageQueue messageQueue)
{
    var matches = regex.Matches(messageQueue.QueueName);

     if (matches.Count != 1)
         throw new ApplicationException("Unable to parse " + messageQueue.QueueName);
    ...
}

I don’t have the time to work this into a pull request before going on break, but I believe that should fix it.

For me at least, this appears to happen when AD integration is installed with MSMQ. If you don’t need it, uninstalling it worked as a workaround for me

Happens also for me when using MSMQ (public queue)

[ApplicationException: Unable to parse PUBLIC=3c69b92b-726f-4fa9-90f4-d34f944f5ba9] MQTools.MessageQueueExtensions.GetCount(MessageQueue messageQueue) +308 Hangfire.SqlServer.Msmq.MsmqJobQueueMonitoringApi.GetEnqueuedAndFetchedCount(String queue) +98 Hangfire.SqlServer.SqlServerMonitoringApi.Queues() +502 Hangfire.Dashboard.Pages.QueuesPage.Execute() +105 Hangfire.Dashboard.RazorPage.TransformText(String body) +46 Hangfire.Dashboard.RazorPage.ToString() +23 Hangfire.Dashboard.RazorPageDispatcher.Dispatch(RequestDispatcherContext context) +138 Hangfire.Dashboard.<>c__DisplayClass8.<UseHangfireDashboard>b__4(IDictionary2 env) +365`

as mentioned by Manuel, this format works fine in version 1.5.3.0.

I tested with private queues only as of now.

I’m getting the same error. Server functions, but the “Jobs” page returns Unknown Error on MessageQueueExtensions.GetCount().

Using 1.5.3.0.

Reading queue name from an web.config key setting and it looks like:
“FormatName:DIRECT=OS:server.abc.xyz\private$\hangfire-{0}”
I’ve also tried local notation like
““FormatName:DIRECT=OS:.\private$\hangfire-{0}””

Everyone has full control of the queue. Have removed AD integration feature.

If I run the code so that the queue is remote, I get the following stack instead:

[MessageQueueException (0x80004005)]
System.Messaging.MQCacheableInfo.get_ReadHandle() +252
System.Messaging.MessageEnumerator.get_Handle() +72
System.Messaging.MessageEnumerator.MoveNext(TimeSpan timeout) +82
Hangfire.SqlServer.Msmq.MsmqJobQueueMonitoringApi.GetEnqueuedJobIds(String queue, Int32 from, Int32 perPage) +318
Hangfire.SqlServer.SqlServerMonitoringApi.Queues() +664
Hangfire.Dashboard.Pages.QueuesPage.Execute() +224
Hangfire.Dashboard.RazorPage.TransformText(String body) +31
Hangfire.Dashboard.RazorPageDispatcher.Dispatch(RequestDispatcherContext context) +146
Hangfire.Dashboard.<>c__DisplayClass8.b__4(IDictionary`2 env) +467
Microsoft.Owin.Mapping.d__0.MoveNext() +462
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.d__5.MoveNext() +203
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.d__2.MoveNext() +193
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) +96
System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +363
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +137

Guys, try to upgrade to the latest 1.5.7 version – http://hangfire.io/blog/2016/05/30/hangfire-1.5.7.html