Dear Team,
We are using Hangfire recurring job with MySql Database as backend. Both are running on AZURE.
The job is add/update the data records into MySqlDatabase for every 30 mins. The logic which i have in job is working fine. it is adding and updating into DB.But before finishing the job, we are receiving the below error. it is running till last line of code in the job but not completing and throwing this below error and retrying again.
We are using Hangfire, Hangfire.Inmemory, Hangfire.MySqlStorage packages.
Also We tried adding ConnectionIdleTimeout, ConnectionTimeout in the connection stings. but no luck and not able to find the root cause of this issue.
The issue is not all the time. Also not able to reproduce with local code setup. it is occurred in Azure environments.
Failed
An exception occurred during performance of the job.
System.IO.IOException
Unable to write data to the transport connection: Connection reset by peer.
System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer.
---> System.Net.Sockets.SocketException (104): Connection reset by peer
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.Write(ReadOnlySpan`1 buffer)
at MySqlConnector.Protocol.Serialization.StreamByteHandler.WriteBytesAsync(ReadOnlyMemory`1 data, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/StreamByteHandler.cs:line 105
--- End of stack trace from previous location ---
at MySqlConnector.Protocol.Serialization.ProtocolUtility.<WritePacketAsync>g__WritePacketAsyncAwaited|8_0(ValueTask task, Byte[] buffer) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 562
at MySqlConnector.Core.ServerSession.SendReplyAsyncAwaited(ValueTask task) in /_/src/MySqlConnector/Core/ServerSession.cs:line 991
at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList`1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 55
at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 296
at MySqlConnector.MySqlCommand.ExecuteNonQuery() in /_/src/MySqlConnector/MySqlCommand.cs:line 107
at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader) in C:\projects\dapper\Dapper\SqlMapper.cs:line 2836
at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command) in C:\projects\dapper\Dapper\SqlMapper.cs:line 570
at Hangfire.MySql.MySqlDistributedLock.Release()
at Hangfire.MySql.MySqlDistributedLock.Dispose()
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstance instance, Func`2 action, String message)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)