We have been updating our redis connections to support sentinel and recently upgraded Hangfire.Pro.Redis from 3.0.0 to 3.2.0, and since doing so have been running into performance issues on our performance testing environments.
Most of our API requests that write to the database will also enqueue a hangfire job into redis to dispatch some messages in the background.
Here is a normal percentile graph from our perf suite:
Here is a percentile graph with our redis updates and updating to Hangfire.Pro.Redis 3.2.0.
Today we started running tests on the new version, and then deployed the old version of the code mid test, and we can see the percentiles immediately improve. We then created a new temporary branch that contains all of the logic changes, but the only change we have is downgrading Hangfire.Redis.Pro from 3.2.0 back to 3.0.0.
On this run:
10:54 - 11:12 = newest build with logic changes + updated libraries
11:12 - 11:40 = revert to build before logic changes + old libraries
11:40 onwards = custom build with new logic changes + using old Hangfire.Redis.Pro library.
We also see when we are running the tests the background job creation API now randomly throws back an exception that it is attempting to access a disposed connection.
Method: System.String Create(Hangfire.Common.Job, Hangfire.States.IState, System.Collections.Generic.IDictionary`2[System.String,System.Object]) (163, 17)
Exception 02 inner --------------------------
Type: StackExchange.Redis.RedisConnectionException
Source: StackExchange.Redis.ConnectionMultiplexer, Hangfire.Pro.Redis, Version=3.2.0.0, Culture=neutral, PublicKeyToken=null
Message: ConnectionDisposed on EXEC
Trace: at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 2237
at Hangfire.Pro.Redis.RedisConnection.CreateExpiredJob(Job job, IDictionary`2 parameters, DateTime createdAt, TimeSpan expireIn) in C:\projects\hangfire-pro-redis\src\Hangfire.Pro.Redis\RedisConnection.cs:line 92
at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[TContext,TResult](Int32& attemptsLeft, Func`3 action, TContext context) in C:\projects\hangfire-525\src\Hangfire.Core\Client\CoreBackgroundJobFactory.cs:line 154
--- End of stack trace from previous location ---
at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[TContext,TResult](Int32& attemptsLeft, Func`3 action, TContext context) in C:\projects\hangfire-525\src\Hangfire.Core\Client\CoreBackgroundJobFactory.cs:line 179
at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context) in C:\projects\hangfire-525\src\Hangfire.Core\Client\CoreBackgroundJobFactory.cs:line 73
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(Enumerator& enumerator, IBackgroundJobFactory innerFactory, CreateContext context, CreatingContext preContext) in C:\projects\hangfire-525\src\Hangfire.Core\Client\BackgroundJobFactory.cs:line 141
at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context) in C:\projects\hangfire-525\src\Hangfire.Core\Client\BackgroundJobFactory.cs:line 76
at Hangfire.BackgroundJobClient.Create(Job job, IState state, IDictionary`2 parameters) in C:\projects\hangfire-525\src\Hangfire.Core\BackgroundJobClient.cs:line 156
Location: C:\projects\stackexchange-redis\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs
Method: T ExecuteSyncImpl[T](StackExchange.Redis.Message, StackExchange.Redis.ResultProcessor`1[T], StackExchange.Redis.ServerEndPoint) (2237, 37)
I’m currently asking whoever has the private nuget feed to grab a copy of 3.1.0 to see if it also has issues.