MSDTC error on Hangfire 1.5.3 (and 1.4.6)

question
1

We’re queuing a job that processes data within a transaction scope and on a single realm. Sometimes this job will fail with the error MSDTC on server '[servername]' is unavailable. We intentionally avoid using the MSDTC since our jobs shouldn’t need to be elevated.

I see that this error has been referenced a handful of times in the past and addressed, but we’re still seeing it, so I don’t know what’s particularly unique about our circumstances. The job fails consistently locally and frequently on production, though not always. When we were encountering this in 1.4.6, we tried updating locally to 1.5.3 but are still seeing it.

It appears to be related to creating an expired job, perhaps because the job itself is executed within a transaction? Here’s the stack trace of the inner exception:
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) +388 System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +717 System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) +4515 System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() +61 System.Data.SqlClient.SqlDataReader.get_MetaData() +134 System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest) +1370 System.Data.SqlClient.TdsParser.GetDTCAddress(Int32 timeout, TdsParserStateObject stateObj) +68 System.Data.SqlClient.SqlInternalConnectionTds.GetDTCAddress() +208 System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx) +615 System.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction) +6572836 System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) +1840 System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) +116
System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) +1079 System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) +6610951
System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource1 retry) +233 System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry) +278
System.Data.SqlClient.SqlConnection.Open() +239
Hangfire.SqlServer.SqlServerStorage.UseConnection(Func2 func) +112 Hangfire.SqlServer.SqlServerConnection.CreateExpiredJob(Job job, IDictionary2 parameters, DateTime createdAt, TimeSpan expireIn) +226
Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context) +389
Hangfire.Client.<>c__DisplayClass3.b__0() +38
Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func1 continuation) +323 Hangfire.Client.BackgroundJobFactory.CreateWithFilters(CreateContext context, IEnumerable1 filters) +214
Hangfire.Client.BackgroundJobFactory.Create(CreateContext context) +343
Hangfire.BackgroundJobClient.Create(Job job, IState state) +145

Any thoughts? Happy to provide more information that would be helpful.

2

Update:

We were managing the enqueue/dequeue within a transaction, and the actions that were being run were within a transaction, too. Hence the elevation to MSDTC. By wrapping the queueing in a TransactionScopeOptions.Suppress transaction, we were able to resolve this issue.