I am trying to deploy a ASP.NET Core application that is using Hangfire and it is hosted inside Azure service Fabric.
If the app is deployed to only 1 node, there aren’t any issues. However, when the app is deployed across multiple nodes inside Service Fabric, I receive the following error on startup on the app.UseHangfireServer().
The exception message is: Cannot access a disposed object. Object name: ‘LoggerFactory’.
Here is the stack trace:
at Microsoft.Extensions.Logging.LoggerFactory.CreateLogger(String categoryName)
at Hangfire.AspNetCore.AspNetCoreLogProvider.GetLogger(String name)
at Hangfire.Logging.LogProvider.GetLogger(String name)
at Hangfire.Server.AutomaticRetryProcess…ctor(IServerProcess innerProcess)
at Hangfire.Server.BackgroundProcessingServer…ctor(JobStorage storage, IEnumerable1 processes, IDictionary2 properties, BackgroundProcessingServerOptions options)
at Hangfire.BackgroundJobServer…ctor(BackgroundJobServerOptions options, JobStorage storage, IEnumerable1 additionalProcesses) at Hangfire.HangfireApplicationBuilderExtensions.UseHangfireServer(IApplicationBuilder app, BackgroundJobServerOptions options, IEnumerable1 additionalProcesses, JobStorage storage)
at OurService.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, IServiceProvider serviceProvider)
Took a little digging as Service Fabric dev environment seems to eat exceptions but Hangfire appears to work fine under service fabric once the service endpoints are configured in a way that doesn’t blow up in a single server development environment.
If your ServiceManifest.xml has a static port defined like:
Each instance/node of the service is going to try starting on that port. Since you have a single computer in the SF dev environment this means all nodes are on the same server and sharing the same ports. When a second instance of your Hangfire service tries to start up it will blow up due to an error like:
Microsoft.Net.Http.Server.WebListenerException (0x80004005): The prefix 'http://+:5081/' is already registered.
at Microsoft.Net.Http.Server.UrlGroup.RegisterPrefix(String uriPrefix, Int32 contextId)
at Microsoft.Net.Http.Server.UrlPrefixCollection.RegisterAllPrefixes(UrlGroup urlGroup)
An easy fix for this is to let service fabric automatically determine an available port to access your application by modifying the ServiceManifest like:
it’ll be very difficult to navigate to the Hangfire dashboard as the URL will need to include the PartitionId/Replica ID. It’s easier if you’ll be accessing the dash to use ServiceFabricIntegrationOptions.None.