Allright, so I found the issue after seeing this comment: Hangfire causes application performance issue
I did some file caching within the bin folder of my ASP.NET Web API, which caused the API to wait apparently for the IO to finish. I moved the file caching to somewhere outside my bin folder and it seems to work like a charm now.