Hangfire and Service Fabric

question
1

Hi,
I am trying to deploy a ASP.NET Core application that is using Hangfire and it is hosted inside Azure service Fabric.
If the app is deployed to only 1 node, there aren’t any issues. However, when the app is deployed across multiple nodes inside Service Fabric, I receive the following error on startup on the app.UseHangfireServer().

The exception message is:
Cannot access a disposed object.
Object name: ‘LoggerFactory’.

Here is the stack trace:

at Microsoft.Extensions.Logging.LoggerFactory.CreateLogger(String categoryName)
at Hangfire.AspNetCore.AspNetCoreLogProvider.GetLogger(String name)
at Hangfire.Logging.LogProvider.GetLogger(String name)
at Hangfire.Server.AutomaticRetryProcess…ctor(IServerProcess innerProcess)
at Hangfire.Server.BackgroundProcessingServer…ctor(JobStorage storage, IEnumerable1 processes, IDictionary2 properties, BackgroundProcessingServerOptions options)
at Hangfire.BackgroundJobServer…ctor(BackgroundJobServerOptions options, JobStorage storage, IEnumerable1 additionalProcesses) at Hangfire.HangfireApplicationBuilderExtensions.UseHangfireServer(IApplicationBuilder app, BackgroundJobServerOptions options, IEnumerable1 additionalProcesses, JobStorage storage)
at OurService.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, IServiceProvider serviceProvider)

I am using Hangfire version 1.6.12.0

How can I resolve this issue?

2

@Treach Did you get a solution to this problem? We are considering a similar approach of hosting the hangfire server in azure service fabric.

@odinserj Is is scenario supported? Can Hangfire server work fine when installed in stateless nodes? Any help is greatly appreciated.

Thanks.

3

Hi,

Same here! I would like to know if multi node support is a possiblity.

Thanks.

4

Hi,

Took a little digging as Service Fabric dev environment seems to eat exceptions but Hangfire appears to work fine under service fabric once the service endpoints are configured in a way that doesn’t blow up in a single server development environment.

If your ServiceManifest.xml has a static port defined like:

<Endpoint Name="ServiceEndpoint" Protocol="http" Port="5080" />

Each instance/node of the service is going to try starting on that port. Since you have a single computer in the SF dev environment this means all nodes are on the same server and sharing the same ports. When a second instance of your Hangfire service tries to start up it will blow up due to an error like:

Microsoft.Net.Http.Server.WebListenerException (0x80004005): The prefix 'http://+:5081/' is already registered. 
at Microsoft.Net.Http.Server.UrlGroup.RegisterPrefix(String uriPrefix, Int32 contextId)
at Microsoft.Net.Http.Server.UrlPrefixCollection.RegisterAllPrefixes(UrlGroup urlGroup)

An easy fix for this is to let service fabric automatically determine an available port to access your application by modifying the ServiceManifest like:

<Endpoint Protocol="http" Name="ServiceEndpoint"/>

Also watch out if you are using the ASP Core service fabric integration like:

.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.UseUniqueServiceUrl)

it’ll be very difficult to navigate to the Hangfire dashboard as the URL will need to include the PartitionId/Replica ID. It’s easier if you’ll be accessing the dash to use ServiceFabricIntegrationOptions.None.