Hangfire.Pro.Redis lingering connections?

dashboard
Tags: #<Tag:0x00007f499bacce70>

#1

After moving from Hangfire.SQLServer to Hangfire.Pro.Redis, i have noticed that in the Redis Client List command, there are a maximum number of connections being hit.

Can you please explain how the connections are handled in terms of idle / open / closed. Does Hangfire.Pro.Redis open a connection as a new job is created, processed and closes it after connection ?

We are hitting upper limits because after each job, we are seeing the connections staying open for a indefinite period of time.

Thank you.


#2

Hangfire.Pro.Redis creates a single multiplexed connection that spawns two connections – one for commands, and another one for subscriptions. Those connections are used for the whole processing server lifespan. If something happens with that connection, it is disposed and the one is created. So Hangfire on its own side doesn’t create many connections.

Could you show me the output of the CLIENT LIST command?

It’s possible that some connections are killed by firewall or other network issues, causing those connections to remain opened on Redis side, because default value of the tcp-keepalive setting in Redis is zero. But this reason shouldn’t cause huge number of connections to be opened.

Also is it possible that there’s another another feature in the application that uses Redis?


#3

Thank you for getting back to me so quickly. Sure, here is the list. I have just restarted my azure redis, and turned on my web application (the only thing using redis, is Hangfire, i even just changed the password / keys to ensure that and to test).

id=578 addr=1.1.1.1:55519 fd=28 name=Hangfire@pcname age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=579 addr=1.1.1.1:55518 fd=29 name=Hangfire@pcname age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=ping numops=10
id=586 addr=1.1.1.1:55538 fd=34 name=Hangfire@pcname age=8 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=587 addr=1.1.1.1:55537 fd=35 name=Hangfire@pcname age=8 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=ping numops=10
id=582 addr=1.1.1.1:55527 fd=31 name=Hangfire@pcname age=9 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=583 addr=1.1.1.1:55526 fd=21 name=Hangfire@pcname age=9 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=ping numops=10
id=569 addr=1.1.1.1:55507 fd=20 name=Hangfire@pcname age=33 idle=33 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=ping numops=11
id=601 addr=1.1.1.1:55541 fd=40 name=Hangfire@pcname age=6 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=602 addr=1.1.1.1:55542 fd=30 name=Hangfire@pcname age=6 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=ping numops=10
id=576 addr=1.1.1.1:55517 fd=26 name=Hangfire@pcname age=29 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=577 addr=1.1.1.1:55516 fd=18 name=Hangfire@pcname age=29 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=ping numops=10
id=570 addr=1.1.1.1:55508 fd=22 name=Hangfire@pcname age=33 idle=33 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=537 addr=1.1.1.1:55468 fd=12 name=Hangfire@pcname age=84 idle=8 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=subscribe numops=10
id=588 addr=1.1.1.1:55539 fd=36 name=Hangfire@pcname age=6 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=589 addr=1.1.1.1:55540 fd=37 name=Hangfire@pcname age=6 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=ping numops=10
id=584 addr=1.1.1.1:55533 fd=32 name=Hangfire@pcname age=8 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=585 addr=1.1.1.1:55532 fd=33 name=Hangfire@pcname age=8 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=exec numops=27
id=505 addr=127.0.0.1:17034 fd=15 name=PORTAL_CONSOLE age=141 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=client numops=18
id=573 addr=1.1.1.1:55513 fd=24 name=Hangfire@pcname age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=574 addr=1.1.1.1:55512 fd=25 name=Hangfire@pcname age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=ping numops=10
id=536 addr=1.1.1.1:55467 fd=17 name=Hangfire@pcname age=84 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=rpoplpush numops=381
id=571 addr=1.1.1.1:55510 fd=4 name=Hangfire@pcname age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=echo numops=4
id=572 addr=1.1.1.1:55509 fd=23 name=Hangfire@pcname age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=exec numops=27
id=540 addr=1.1.1.1:55476 fd=19 name=Hangfire@pcname age=71 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=rpoplpush numops=363
id=541 addr=1.1.1.1:55477 fd=13 name=Hangfire@pcname age=71 idle=7 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 ow=0 owmem=0 events=r cmd=subscribe numops=9


#4

So the way we create a new job, is when a device connects to our server and creates this job command, that is put into hangfire, when using Hangfire SQLServer it was working ok, but now seeing all these ping, echo, subscribe events, i’m not sure why redis wont work.

Do you think this is something with a firewall, i’ve already checked and unblocked the relevant SSL Ports, 6380 and because can see the dashboard and some incoming jobs proceess and succeed would rule that out.

It seems that when a job fires and a guid id is returned, these type recurring threads are create.

Any thoughts please ?


#5

Found this was a issue due to our service architecture, and only saw this after changing to redis. Therefore, not related to a hangfire issue. So this is resolved.