Starting a process (System.Diagnostics.Process) inside a reoccuring background job throws thread aborted

bug?
1

Hello,

First or all I am not sure if this is a bug, it might be because I am missing something.

I have a re-occuring hangfire job which starts a System.Diagnostics.Process (to run a python script).
The problem is that some time (up to minute) after the python process is started, it throws an ThreadAbortedException. I am not sure why this happens, but it kind of seems like it happens when the python script takes some time to execute.This special exception has no details about what happened, and it is re-thrown all way up the stack.

If I run the same job outside of hangfire it works fine.
Here is some more details:

We use Owen type startup.

    [assembly: OwinStartup(typeof(Dashboard.Startup))]
    namespace Dashboard
    {
        public class Startup
        {
            public void Configuration(IAppBuilder app)
            {
                app.UseStageMarker(PipelineStage.AcquireState);
                app.UseStageMarker(PipelineStage.Authenticate);
                app.UseStageMarker(PipelineStage.Authorize);
                app.UseStageMarker(PipelineStage.MapHandler);
                app.UseStageMarker(PipelineStage.PostAcquireState);
                app.UseStageMarker(PipelineStage.PostAuthenticate);
                app.UseStageMarker(PipelineStage.PostAuthorize);
                app.UseStageMarker(PipelineStage.PostMapHandler);
                app.UseStageMarker(PipelineStage.PostResolveCache);
                app.UseStageMarker(PipelineStage.PreHandlerExecute);
                app.UseStageMarker(PipelineStage.ResolveCache);

                HangfireBootstrapper.Instance.Start();

                app.UseHangfireDashboard("/hangfire",
                    new DashboardOptions(),
                    new SqlServerStorage(ConfigurationManager.ConnectionStrings["DB"].ConnectionString)
                );
            }
        }
    }

The code to bootstrap Hangfire (in HangfireBootstrapper) is:

    HostingEnvironment.RegisterObject(this);
    JobStorage.Current = new SqlServerStorage(ConfigurationManager.ConnectionStrings["DB"].ConnectionString);
    _backgroundJobServer = new BackgroundJobServer(new BackgroundJobServerOptions(), JobStorage.Current);
    GlobalJobFilters.Filters.Add(new AutomaticRetryAttribute { Attempts = 0 });                
    SetupRecurringHangfireJobs();

The code that creates the new process look like this:

    Process process = null;
    try
    {
        ProcessStartInfo info = new ProcessStartInfo();
        info.UseShellExecute = false;
        info.CreateNoWindow = true;
        info.RedirectStandardOutput = true;
        info.RedirectStandardError = true;              
        info.FileName = "python.exe";
        info.Arguments = "Script.py";
        info.WorkingDirectory = "C:\Temp"; 
        using (process = Process.Start(info))
        {                    
                using (var reader = process.StandardError)
                    standardError.Add(reader.ReadToEnd());
                using (var reader = process.StandardOutput)
                    standardOutput.Add(reader.ReadToEnd());                    
                return
                    process.ExitCode;
       }
   }
   catch (ThreadAbortException e)
   {   /* Thread.ResetAbort();  */  }

It throws the exception as soon as I get to standardError.Add(reader.ReadToEnd());
The python script loads data from a file and dumps it into a database. I have even used a python script where all I am doing is time.sleep(60) (60 seconds) and was able to reproduce the ThreadAbortException.

Please let me know if there is more information you would like.
Thank you

2

Have you tried setting the InvisibilityTimeout option on your storage configuration?

3

Just tried InvisibilityTimeout option set to 2 hours and still getting ThreadAbortedException

app.UseHangfireDashboard("/hangfire",
     new DashboardOptions(),
     new SqlServerStorage(
         ConfigurationManager.ConnectionStrings["DB"].ConnectionString,
         new SqlServerStorageOptions()
         {
             InvisibilityTimeout = TimeSpan.FromHours(2)
         }                
     )
 );
4

I actually tried this same code with Quartz.net and got the same problem. I then created a console app with Hangfire and ran that job and worked fine. I guess for longer jobs that spin new up processes, the asp.net web environment is not the ideal environment.
I am going ahead with the console solution.

5

Ah I must admit I did not consider you were running under ASP.Net.

Personally I avoid it as the plague for a number of reasons although most of them are not really related to ASP.Net itself and more to how it is implemented where I work. But what you report seems to be a common impression/issue judging from other messages in this forum.

6

I use Process like this in one of my jobs, I just use process.WaitForExit() in my job.

7

I actually tried using WaitForExit() and it threw the ThreadAbortException right away. When reading from the process StandardOutput stream (until it closes), it actually looks like it does do some work before that same exception is thrown.

8

I also replied to soon, we are processing jobs in Windows Services. Like Hans_Engelen said, processing the jobs (especially longer running jobs) wasn’t possible in asp.net host.