Hi all, we are currently attempting to configure hangfire against a deployed multi master instance of keydb (https://keydb.dev/) in k8s with https://github.com/Enapter/charts/tree/master/keydb.
Hangfire connects and works well, however we are seeing instances of duplicate job executions for a single job execution against multiple servers. I want to confirm if there will be support for this in the road map moving forward? We are looking to keep our redis in a HA configuration.
Background job identifier is fetched from its queue using the RPOPLPUSH that atomically moves background job id from its queue and put it to the pending list. With the behaviour you get it looks like multiple workers were able to pick up that job identifier.
So I’m afraid the question should be asked to KeyDB developers instead why KeyDB allows multiple clients to perform the same RPOPLPUSH operation concurrently, e.g. why does it allow concurrent access to the same key, an entity, that’s considered to be sequential – because any concurrent access in this case does not make any sense.
I believe this behaviur is due to multi-master configuration. So the full question is “How KeyDB serialise access to a particular key when multiple masters involved?” Because the only answer is “Nohow”, otherwise a Holy Grail of distributed processing belong to them and many patents should be protecting it.
High availability is possible, but with primary/replica scenario (as in Redis Cluster), otherwise transactions will not work or give unexpected results and you’ll end up with inconsistent state, where everything is possible, but nothing is predictable.