Possible memory leak when running hangfire in linux container

Hi everyone!

We are running Hangfire 1.8.14 with Hangfire.PostgreSql 1.20.9 on .NET Core 8.

It seems that there is a memory leak that only occurs when running it in a linux container deployed to Azure container apps.

When run locally we did not observe the behavior.

Currently we have the following projects:
API - with the endpoints used to enqueue new jobs,
HangfireServer - ASP.NET Core app where Hangfire is hosted and,
Jobs - a seperate project/dll that contiains the classes with methods that represent the jobs.

The API only calls IBackgroundJobClient.Enqueue() and does no further processing.

So for example I have a simple class:

public class KillMemoryJob
{
    public async Task Run()
    {
        long bytesToAllocate = (long)(1.5 * 1024 * 1024 * 1024);
        byte[] memory = new byte[bytesToAllocate];

        for (long i = 0; i < memory.Length; i++)
        {
            memory[i] = 0;
        }

        Console.WriteLine("Successfully allocated 1.5 GB of memory.");
        await Task.Delay(1000 * 60 * 5); // wait five minutes to finish
    }
}

that allocates 1.5GB of memory.

The API endpoint is defined as:

app.MapGet("api/debug/kill-memory", 
    (IBackgroundJobClient backgroundJobClient) =>
    {
        string jobId = backgroundJobClient.Enqueue<KillMemoryJob>(zbj =>
            zbj.Run());

        return Results.Ok(jobId);
    });

The behavior I expect to observe is that the API enqueues the job, the server takes it and starts processing. After the processing finishes and the Run() method exits (after a five minute delay), the allocated memory is released.
That works as expected when I run it locally on my Windows machine, but when deployed to Azure Container Apps the memory never gets released.

I can provide a dump file if that would help figure out the issue.

2 Likes

I have the exact same configuration, and I’m facing an issue with memory release in a Linux environment. Locally, this problem doesn’t occur. After analyzing the memory dump, I found that the leak happens because TimerQueueTimer objects are not being disposed of. If you make progress and find a solution, please let me know :v:

1 Like

Watching this issue too, as we are also experiencing this issue when running hangfire on a linux container in Azure, only difference is we’re using Hangfire.SqlServer.

Hangfire version 1.8.14, Hangfire.SqlServer 1.8.14 on .NET 8.

Bumping this thread up and hoping for a reply or a resolution.

1 Like