Exception when processing background jobs

bug?
1

I have just implemented hangfire in a .net web api project using sql server as the storage. Everything seems to work fine unless I try to process jobs in quick succession (ex. through a testing framework like Fitness or jMeter). The exact same job will work fine one moment, then fail repeatedly with the following exception in the Hangfire.Status table:

Any help getting around this issue would be greatly appreciated!!!

Newtonsoft.Json.JsonSerializationException: Error converting value "CloudConnect.Dto.V2.PartnerMessageDto, CloudConnect.Dto, Version=14.21.0.821, Culture=neutral, PublicKeyToken=53691774cc579a62" to type 'System.Type'. Path '[0]', line 1, position 129. ---> 

System.ArgumentException: Could not cast or convert from System.String to System.Type.
   at Newtonsoft.Json.Utilities.ConvertUtils.EnsureTypeAssignable(Object value, Type initialType, Type targetType)
   at Newtonsoft.Json.Utilities.ConvertUtils.ConvertOrCast(Object initialValue, CultureInfo culture, Type targetType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
   --- End of inner exception stack trace ---
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Hangfire.Storage.InvocationData.Deserialize()
2

Can you show me the line where you create a background job (e.g. BackgroundJob.Enqueue(/* ... */)) and the corresponding method signature?

3

Thank you for the quick response! Here you go…

BackgroundJob.Enqueue(() => _partnerMessageService.ProcessPartnerMessage(value, sdkVersion, User.Identity.Name));

public ApiResponse ProcessPartnerMessage(Dto.V2.PartnerMessageDto value, String sdkVersion, String userName)
{
    UserName = userName;
    
    var meterData = new MeterData
    {
        InitialTimeStamp = DateTime.UtcNow,
        IsSuccessful = true,
        IsAuthorized = true,
        CacheHit = false,
        SdkVersion = sdkVersion
    };

    if (value == null)
    {
        var response = CreateResponseObject(ApiResponseType.Error, AuditEvents.PartnerMessageDtoMissing);

        meterData.IsSuccessful = false;
        ProcessMeterData(meterData, "", null, 0, false);

        _auditService.Log(AuditEvents.PartnerMessageDtoMissing, null, response.ResponseMsg, User.UserId);
        return response;
    }

    return ProcessMessage(meterData, value.MessageType, value.Identifier, value.Payload, value.SiteId, value.ServiceName, null);
}
4

Looks like Newtonsoft.Json unable to convert string value "CloudConnect.Dto.V2.PartnerMessageDto, CloudConnect.Dto, Version=14.21.0.821, Culture=neutral, PublicKeyToken=53691774cc579a62" to its corresponding Type instance. In what assembly you are creating an instance of the BackgroundJobServer class (manually, or through the UseServer extension method)? Does it have a reference to the assembly where PartnerMessageDto class is defined?

5

As mentioned in my first post, this job succeeds if only run once, however, when run multiple times it fails. The assembly references are there.

6

Investigation

Let’s debug the code :smile: We have the following exception thrown from the ConvertUtils.EnsureTypeAssignable method.

System.ArgumentException: Could not cast or convert from System.String to System.Type.
   at Newtonsoft.Json.Utilities.ConvertUtils.EnsureTypeAssignable(Object value, Type initialType, Type targetType)

This method throws exception from the last line:

throw new ArgumentException("Could not cast or convert from {0} to {1}.".FormatWith(
    CultureInfo.InvariantCulture, 
    (initialType != null) ? initialType.ToString() : "{null}", 
    targetType));

From the exception message string we could infer the following things:

initialType == typeof(String)
targetType == typeof(Type)

As these variables never change in the EnsureTypeAssignable method, we could infer that the following line was called from the ConvertUtils.ConvertOrCast method. If we dig into upper method, we can either infer that value argument equals to the following string.

EnsureTypeAssignable(
    "CloudConnect.Dto.V2.PartnerMessageDto, CloudConnect.Dto, Version=14.21.0.821, Culture=neutral, PublicKeyToken=53691774cc579a62",
    typeof(String),
    typeof(Type));

Indeed, the ConvertOrCast method makes this call, but it invokes it only if the call to the TryConvert method returns false. The latter method simply calls the TryConvertInternal method within a try-catch block, so it can return false in following cases:

  1. TryConvertInternal returned false.
  2. TryConvertInternal throws any exception.

Let’s look at TryConvertInternal method and remember that:

initialValue = "CloudConnect.Dto.V2.PartnerMessageDto, CloudConnect.Dt..."
targetType == typeof(Type)

That is why we’ll end up with the following statements (and since Type does not implement the IConvertible interface):

if (typeof(Type).IsAssignableFrom(targetType))
{
    value = Type.GetType((string)initialValue, true);
    return ConvertResult.Success;
}

Notice the second argument in the GetType invocation, it equals to true that means that the method will throw if there was an error locating the type. And it is throwing an exception, because otherwise we’ll end up with another exception.

The problem

The following code invocation sometimes (by your words) throws an exception. Do you know why this may happen?

Type.GetType(
    "CloudConnect.Dto.V2.PartnerMessageDto, CloudConnect.Dto, Version=14.21.0.821, Culture=neutral, PublicKeyToken=53691774cc579a62", 
    true);

Further investigation

This MSDN article lists all exceptions that may be thrown in this case (please see the Exceptions section). What case may be yours?

But before, try to update the Newtonsoft.Json library.

7

Thank you so very much for taking the time to go through this explanation. It seems that I’ve discovered the problem. We had two systems that were queuing jobs. One system (a dev box) didn’t have versioned assemblies, while the other did. So that when the jobs were being processed, they were being processed by the incorrect server.