Hangfire in AWS Beanstalk behind Load Balancer

Tags: #<Tag:0x00007f459065a748> #<Tag:0x00007f459065a540>

We have implemented Hangfire within the AWS world using Elastic Beanstalk for our .Net Core API. We have configured our environments (Production, Staging, QA, and Development) to be load balancing. For Production we have at least 15 instances and at most 20 instances. We scale up by two and scale down by one.

Within the Production environment we started seeing jobs aborted because the server is not found: "The job was aborted - it is processed by server XXX which is not in the active servers list for now …" And this makes sense since we may have servers coming and going depending upon load balancing needs.

Based upon what was found off the internet, we have created our own unique server names. But from what I am gather that is the crux of our problem. The server may be gone when Hangfire says do the work. I now need to figure out how to resolve this dilemma. Any suggestions/guidance will be appreciated.