w3wp slows down with a lot of HF jobs

We usually enqueue a fair number of jobs onto the HF queue from within an ASP.NET application and then they eventually get processed by the w3wp processes within the ASP.NET application. We experience intermittent slowness in our ASP.NET UI (i.e. browser requests). One theory is that the HF processing is using up all the w3wp processes we have allocated (we set 5 worker processes and about 30 HF workers) and therefore the browser requests aren’t getting catered to effectively.

Does anyone else have this same problem? What possible solutions are there? I guess we could move the processing of the HF jobs onto a windows service to free up the w3wp processes?! Are there any alternatives?