Is Hangfire a good fit for what I want to do?

Hello,
I’m looking at possible solutions to help us achieve some pretty straightforward ETL processing goals.
Basically the monitoring and controlling of ETL jobs.

On the face of it Hangfire has some really impressive facilities which I am hoping would allow us to achieve the basic goals and give us loads of scope for expanding the processing in the future.

Before I dive into installing and trialling I was hoping someone could maybe help me understand if our goals are in line with the uses that Hangfire is intended for.

Our most common ETL tasks involve extracting data (from files or oracle database tables), transforming it to JSON and sending it to a REST api endpoint.

Our initial goal is to have a dashboard to monitor the progess of these tasks and allow us to control them (start, stop, schedule)

Some points to note:

  • The files can contain up to 4 million rows, so the task could take a couple of days to run.
  • If resources come under pressure we might want to pause the task, or stop it completely.
  • At other times we might want to schedule the task to start at, say, 10pm on a Friday night and stop again at 10pm on Sunday night.
  • The tasks need to have different command line (or other) params depending on the run.

So, I suppose it boils down to the following questions…

  • Does Hangfire map to these kinds of tasks?
  • Does the Dashboard allow users to stop or start tasks manually?
  • Does the Dashboard have a GUI for setting schedules ?

I see there is an extension for Hangfire called MissionControl… I wonder would that provide us with a manual interface to start and stop jobs ? Being able to stop/interrupt jobs would be a crucial element.

Thanks for your time.

I would personally look at ETL specific frameworks/applications. Hangfire is just a shell for queueing background processes. The issue I would see with what you want to do is Hangfire won’t provide any features for extracting, transforming and loading the data. It will also not provide you any error handling functionality beyond retrying.

The files can contain up to 4 million rows, so the task could take a couple of days to run.

This is fine. Hangfire is meant for long running tasks though I have never written anything that would run for that long.

If resources come under pressure we might want to pause the task, or stop it completely.

This has been answered in another question you have posted. I have not seen a way to pause a job. You could cancel/delete the job and track where you left off outside of the job. Then continue from that point when the job is run again.

At other times we might want to schedule the task to start at, say, 10pm on a Friday night and stop again at 10pm on Sunday night.

Hangfire supports CRON jobs so you can specify that kind of schedule.

The tasks need to have different command line (or other) params depending on the run.

This is vague. I don’t know where the command line would come from. You can pass the command line as a string to the job as a parameter and have the implementation of the job parse it out. Your mentioned “MissionControl” may help with that.

Thanks very much for taking the time to answer my questions.
As you mentioned I have other posts happening in the forum now that are serving to document my learning journey here…

I’m trying to get an asp.net core 6 demo of what I want up and an running … the progress could be followed here
I eventually hope to be able to make a contribution in the form of a completed example.

As for the “different command line” I apologise for the ambiguity… all I meant was that I need to pass params to the jobs. I think I can see how that would be done now.

Also, for the ETL tasks, I hope to code them myself in the jobs and a lot of what we want to do is non-standard so I figured it would be the best approach.
As and aside though, if you can reccomend any ETL specific frameworks I would be interested to take a look… because of the quirky nature of our equipement set-up things like SSIS and BI are not available to us as options … hence the interest in looking at open source options like this.

Thanks again for your input… its greatly appreciated. :grinning: