Lock timeout problems!

postgres
Tags: #<Tag:0x00007f499e41fe80>

#1

Started seeing a lot of errors on my staging deployment. Could not place a lock on the resource ‘HangFire:locks:schedulepoller’ and ‘HangFire:recurring-jobs:lock’: Lock timeout
Can anyone offer suggestions on how to resolve this?

Also, how can I reduce the number of retry attempts? 2147483647 is a lot.
I am initializing hangfire with the following attribute
Hangfire.GlobalJobFilters.Filters.Add(new AutomaticRetryAttribute { Attempts = 3 });

Error occurred during execution of ‘DelayedJobScheduler’ process. Execution will be retried (attempt 15 of 2147483647) in 00:03:41 seconds. Hangfire.PostgreSql.PostgreSqlDistributedLockException: Could not place a lock on the resource ‘HangFire:locks:schedulepoller’: Lock timeout. at Hangfire.PostgreSql.PostgreSqlDistributedLock.PostgreSqlDistributedLock_Init_Transaction(String resource, TimeSpan timeout, IDbConnection connection, PostgreSqlStorageOptions options) at Hangfire.PostgreSql.PostgreSqlConnection.AcquireDistributedLock(String resource, TimeSpan timeout) at Hangfire.Server.DelayedJobScheduler.EnqueueNextScheduledJob(BackgroundProcessContext context) at Hangfire.Server.DelayedJobScheduler.Execute(BackgroundProcessContext context) at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context)

Error occurred during execution of ‘RecurringJobScheduler’ process. Execution will be retried (attempt 13 of 2147483647) in 00:02:44 seconds. Hangfire.PostgreSql.PostgreSqlDistributedLockException: Could not place a lock on the resource ‘HangFire:recurring-jobs:lock’: Lock timeout. at Hangfire.PostgreSql.PostgreSqlDistributedLock.PostgreSqlDistributedLock_Init_Transaction(String resource, TimeSpan timeout, IDbConnection connection, PostgreSqlStorageOptions options) at Hangfire.PostgreSql.PostgreSqlDistributedLock…ctor(String resource, TimeSpan timeout, IDbConnection connection, PostgreSqlStorageOptions options) at Hangfire.PostgreSql.PostgreSqlConnection.AcquireDistributedLock(String resource, TimeSpan timeout) at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context) at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context)


#2

I am seeing similar in Mongo.Hangfire after my windows service runs for a few days. I have opened a bug in mongo.hangfire: bug link


#3

I am having the same problem in Postgre.Hangfire. I can’t see my scheduller process running in client side, here in desenv side and test side, the proccess run correctly. Someone can help us?


#4

has anyone tried perhaps upgrading to newer versions of npgsql?


#5

I did, but my works have not been executed at the specified time (11:55 PM)
About the erros, I cleaned the table “hangfire.lock” and the erros were gone, but I could not figure out if this is the right way.


#6

When you say “cleaned the table”, you mean you truncated all records from it? just emptied the thing, is that right?


#7

so it do seem to stop these lock timeouts when you clear the lock table, had two locks. deleted both.

i now see one of the lock entries has been reinstated and continue to cause lock timeout problems once again. can anyone enlighten me on how hangfire.lock is used? Could not place a lock on the resource ‘HangFire:locks:schedulepoller’: Lock timeout.

Table has two fields; a resource and an update count
as far as I could see Update count is always 0. Resource entry is a string that looks like HangFire:locks:schedulepoller’


#8

I still get this error 3 years later, are there any fixes for this?

Hangfire.PostgreSql.PostgreSqlDistributedLockException
Could not place a lock on the resource 'HangFire:MigrateJob.RunAsync': Lock timeout.

Hangfire.PostgreSql.PostgreSqlDistributedLockException: Could not place a lock on the resource 'HangFire:MigrateJob.RunAsync': Lock timeout.
   at Hangfire.PostgreSql.PostgreSqlDistributedLock.PostgreSqlDistributedLock_Init_Transaction(String resource, TimeSpan timeout, IDbConnection connection, PostgreSqlStorageOptions options)
   at Hangfire.PostgreSql.PostgreSqlDistributedLock..ctor(String resource, TimeSpan timeout, IDbConnection connection, PostgreSqlStorageOptions options)
   at Hangfire.PostgreSql.PostgreSqlConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
   at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
   at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
   at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstance instance, Func`2 action, String message)
   at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)

#9

That is expected though, isn’t it, when looking at the stacktrace (@Peter_Andersson) you are using DisableConcurrentExecutionAttribute and that places a lock - if the “concurrent job” doesn’t finish before the other, the lock times out and it fails - i.e. it couldn’t execute within the timeout specified in the filter.

That said, I think the signature should include the arguments of the method, not just the action/method name - if you look at the code for the filter as it creates a signature from the action name.