We are using Hangfire recurring job with MySql Database as backend. Both are running on AZURE.
The job is add/update the data records into MySqlDatabase for every 30 mins. The logic which i have in job is working fine. it is adding and updating into DB.But before finishing the job, we are receiving the below error. it is running till last line of code in the job but not completing and throwing this below error and retrying again.
We are using Hangfire, Hangfire.Inmemory, Hangfire.MySqlStorage packages.
Also We tried adding ConnectionIdleTimeout, ConnectionTimeout in the connection stings. but no luck and not able to find the root cause of this issue.
The issue is not all the time. Also not able to reproduce with local code setup. it is occurred in Azure environments.
Failed An exception occurred during performance of the job. System.IO.IOException Unable to write data to the transport connection: Connection reset by peer. System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException (104): Connection reset by peer at System.Net.Sockets.NetworkStream.Write(Byte buffer, Int32 offset, Int32 count) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Write(Byte buffer, Int32 offset, Int32 count) at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) at System.Net.Security.SslStream.Write(Byte buffer, Int32 offset, Int32 count) at System.IO.Stream.Write(ReadOnlySpan`1 buffer) at MySqlConnector.Protocol.Serialization.StreamByteHandler.WriteBytesAsync(ReadOnlyMemory`1 data, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/StreamByteHandler.cs:line 105 --- End of stack trace from previous location --- at MySqlConnector.Protocol.Serialization.ProtocolUtility.<WritePacketAsync>g__WritePacketAsyncAwaited|8_0(ValueTask task, Byte buffer) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 562 at MySqlConnector.Core.ServerSession.SendReplyAsyncAwaited(ValueTask task) in /_/src/MySqlConnector/Core/ServerSession.cs:line 991 at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList`1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 55 at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 296 at MySqlConnector.MySqlCommand.ExecuteNonQuery() in /_/src/MySqlConnector/MySqlCommand.cs:line 107 at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader) in C:\projects\dapper\Dapper\SqlMapper.cs:line 2836 at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command) in C:\projects\dapper\Dapper\SqlMapper.cs:line 570 at Hangfire.MySql.MySqlDistributedLock.Release() at Hangfire.MySql.MySqlDistributedLock.Dispose() at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple) at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstance instance, Func`2 action, String message) at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message) at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)