I am using Hangfire v1.7.31 and Hangfire.Storage.SQLite v0.3.2 in a self hosted windows service.
After three days of solid running, I suddenly get this error on all my in-process HangFireServers
(I run 4 in the same process, with different set of workers ).
After this, log is continually reporting the same error, without any recurring jobs being scheduled or processed thereafter.
To me this seems like a bug, although I am not sure if it is in the SQLite adapter or the Hangfire library.
I would be much obliged for any advice how to repair or mitigate this situation.
Cheers,
Daniel
{"Timestamp":"2022-12-23T16:31:03.0188605+01:00","Level":"Debug","MessageTemplate":"Execution RecurringJobScheduler is in the Faulted state now due to an exception, execution will be retried no more than in 00:00:01","Exception":"System.ArgumentException: An item with the same key has already been added. Key: LastJobId\r\n at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)\r\n at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)\r\n at Hangfire.Storage.SQLite.HangfireSQLiteConnection.GetAllEntriesFromHash(String key)\r\n at Hangfire.RecurringJobExtensions.GetRecurringJob(IStorageConnection connection, String recurringJobId, ITimeZoneResolver timeZoneResolver, DateTime now)\r\n at Hangfire.Server.RecurringJobScheduler.TryEnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, DateTime now)\r\n at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)\r\n at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)\r\n at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)\r\n at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\r\n at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)","Properties":{"SourceContext":"Hangfire.Processing.BackgroundExecution","Application":"Scheduling.Service","Instance":"ACC"}}
{"Timestamp":"2022-12-23T16:31:03.0193537+01:00","Level":"Debug","MessageTemplate":"Execution loop RecurringJobScheduler:d8f7f968 caught an exception and will be retried in 00:00:01","Exception":"System.ArgumentException: An item with the same key has already been added. Key: LastJobId\r\n at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)\r\n at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)\r\n at Hangfire.Storage.SQLite.HangfireSQLiteConnection.GetAllEntriesFromHash(String key)\r\n at Hangfire.RecurringJobExtensions.GetRecurringJob(IStorageConnection connection, String recurringJobId, ITimeZoneResolver timeZoneResolver, DateTime now)\r\n at Hangfire.Server.RecurringJobScheduler.TryEnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, DateTime now)\r\n at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)\r\n at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)\r\n at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)\r\n at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\r\n at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)","Properties":{"SourceContext":"Hangfire.Processing.BackgroundExecution","Application":"Scheduling.Service","Instance":"ACC"}}
{"Timestamp":"2022-12-23T16:31:03.0195135+01:00","Level":"Debug","MessageTemplate":"Execution loop RecurringJobScheduler:d8f7f968 will be retried in 00:00:01...","Properties":{"SourceContext":"Hangfire.Processing.BackgroundExecution","Application":"Scheduling.Service","Instance":"ACC"}}
{"Timestamp":"2022-12-23T16:31:03.0408965+01:00","Level":"Debug","MessageTemplate":"Execution RecurringJobScheduler is in the Faulted state now due to an exception, execution will be retried no more than in 00:00:01","Exception":"System.ArgumentException: An item with the same key has already been added. Key: LastJobId\r\n at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)\r\n at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)\r\n at Hangfire.Storage.SQLite.HangfireSQLiteConnection.GetAllEntriesFromHash(String key)\r\n at Hangfire.RecurringJobExtensions.GetRecurringJob(IStorageConnection connection, String recurringJobId, ITimeZoneResolver timeZoneResolver, DateTime now)\r\n at Hangfire.Server.RecurringJobScheduler.TryEnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, DateTime now)\r\n at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)\r\n at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)\r\n at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)\r\n at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\r\n at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)","Properties":{"SourceContext":"Hangfire.Processing.BackgroundExecution","Application":"Scheduling.Service","Instance":"ACC"}}
{"Timestamp":"2022-12-23T16:31:03.0411965+01:00","Level":"Debug","MessageTemplate":"Execution loop RecurringJobScheduler:259e49fc caught an exception and will be retried in 00:00:01","Exception":"System.ArgumentException: An item with the same key has already been added. Key: LastJobId\r\n at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)\r\n at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)\r\n at Hangfire.Storage.SQLite.HangfireSQLiteConnection.GetAllEntriesFromHash(String key)\r\n at Hangfire.RecurringJobExtensions.GetRecurringJob(IStorageConnection connection, String recurringJobId, ITimeZoneResolver timeZoneResolver, DateTime now)\r\n at Hangfire.Server.RecurringJobScheduler.TryEnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, DateTime now)\r\n at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)\r\n at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)\r\n at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)\r\n at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\r\n at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)","Properties":{"SourceContext":"Hangfire.Processing.BackgroundExecution","Application":"Scheduling.Service","Instance":"ACC"}}
{"Timestamp":"2022-12-23T16:31:03.0413447+01:00","Level":"Debug","MessageTemplate":"Execution loop RecurringJobScheduler:259e49fc will be retried in 00:00:01...","Properties":{"SourceContext":"Hangfire.Processing.BackgroundExecution","Application":"Scheduling.Service","Instance":"ACC"}}