Lots of jobs "scheduled" for past?

I have tons of jobs that seem to be stuck in my server not processing.
I’m using HF with SQL Server storage on AWS/RDS, and just today it seems like I have thousands of “scheduled” jobs that should have triggered, but haven’t.

When I look at a given job, it shows something like “Created 5 hours ago”, and “Enqueue at: 7 hours ago”… AKA this job should have fired off 7 hours ago. Yet there are no additional statuses.

I have tried:

  • Deleting and re-creating this job, same problem
  • Rebooting MSSQL server, does nothing
  • Rebooting IIS/app, does nothing
  • Rebooting whole Windows server, does nothing
  • Re-publishing my app, does nothing

This is what my Startup.vb looks like:

Public Sub Configuration(app As IAppBuilder)
	app.UseHangfireAspNet(AddressOf GetHangfireServers)
End Sub

Private Iterator Function GetHangfireServers() As IEnumerable(Of IDisposable)
			New SqlServerStorageOptions With {
				.PrepareSchemaIfNecessary = True,
				.CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
				.SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
				.QueuePollInterval = TimeSpan.Zero,
				.UseRecommendedIsolationLevel = True,
				.UsePageLocksOnDequeue = True,
				.DisableGlobalLocks = True

	Dim options = New BackgroundJobServerOptions() With {
		.ServerName = dbName,
		.WorkerCount = 20,
		.Queues = {"myQueue1", "myQueue2", "myQueue3"}

	Yield New BackgroundJobServer(options)
End Function

Actually scratch that last message, I don’t have recurring jobs, these are just one-off scheduled jobs.

I noticed in the “Set” table, all of the “Score” values are negative. If I set some to 0, the job fires. Some keep ending back up at negative though.