Many workers execute my job

Background job fetching is the most interesting part in a job storage implementation. You should know your storage intimately to allow only one worker to fetch a job. In SQL Server implementation is is achieved via a special column, FetchedAt and an atomic query:

update top (1) HangFire.JobQueue set FetchedAt = GETUTCDATE()
output INSERTED.Id, INSERTED.JobId, INSERTED.Queue
where FetchedAt is null

FetchedAt value is null, when a background job was not fetched by a worker yet, otherwise it is non-null. The query above finds a non-fetched job id, updates it and outputs the identifier atomically, that is why only one worker obtains a job.

What storage are you using? Perhaps there is a repository somewhere?