I'd rather surprised it worked at all.
SQLite is designed to be lightweight and support concurrent reading, but totally sucks at concurrent writing. When writing, the writer holds exclusive database lock. Hangfire, on the other hand, opens at least one connection per executing job, and they all need to access DB.
If you want performance and concurrency, there's plenty other databases to consider. Why not to use something more suitable for your task?