Installing Apache Airflow On Ubuntu, CentOS Cloud Server

Apache Airflow possibly needs a small introduction. It is basically a usable part for managing servers running Big Data tools. Airflow scheduler executes tasks on an array of workers while following the specified dependencies. There is command line utilities. Similar technology is behind Luigi, Azkaban, Oozie etc. Luigi is simpler in scope than Apache Airflow. Here are the steps for installing Apache Airflow on Ubuntu, CentOS running on cloud server. One may use Apache Airflow to author workflows as directed acyclic graphs of tasks.

Installing Apache Airflow On Ubuntu, CentOS Cloud Server

You can install Apache Airflow anywhere – including your Mac. Because it is just a Python package with submodules. On your Mac if you run :

Then you’ll get this kind of result :

Those things are written here :

As we are starting with a possible blank server running a server OS like Ubuntu, we need to install these things :

For CentOS commands will be like this :

If want a message broker then you will need to install RabbitMQ. We are not showing that steps as it is optional.
If you want to use MySQL as database repo you will need to install some MySQL dependencies :

Now steps are common for any operating system. Run :

We need Python 2.7.X, thats it. Then install pip :

Installing AirFlow or components is easy :

Commonly you’ll need this set :

Possibly you’ll set $HOME of Airflow from some sort of dot file :

Initiate Airflow :

Setup MySQL, create an airflow database, grant all on airflow database. We are not showing the steps.

In that $HOME of Airflow, you need to have dags, logs directories :

At $HOME of Airflow, there is a airflow.cfg file. That like starts like :

That will need to have settings like this :

Again run :

Now the last steps :

You can now navigate to the Airflow UI on Browser :

And Flower web UI :