• Home
  • Archive
  • Tools
  • Contact Us

The Customize Windows

Technology Journal

  • Cloud Computing
  • Computer
  • Digital Photography
  • Windows 7
  • Archive
  • Cloud Computing
  • Virtualization
  • Computer and Internet
  • Digital Photography
  • Android
  • Sysadmin
  • Electronics
  • Big Data
  • Virtualization
  • Downloads
  • Web Development
  • Apple
  • Android
Advertisement
You are here: Home » Installing Apache Airflow On Ubuntu, CentOS Cloud Server

By Abhishek Ghosh August 7, 2017 6:39 pm Updated on August 7, 2017

Installing Apache Airflow On Ubuntu, CentOS Cloud Server

Advertisement

Apache Airflow possibly needs a small introduction. It is basically a usable part for managing servers running Big Data tools. Airflow scheduler executes tasks on an array of workers while following the specified dependencies. There is command line utilities. Similar technology is behind Luigi, Azkaban, Oozie etc. Luigi is simpler in scope than Apache Airflow. Here are the steps for installing Apache Airflow on Ubuntu, CentOS running on cloud server. One may use Apache Airflow to author workflows as directed acyclic graphs of tasks.

Installing Apache Airflow On Ubuntu, CentOS Cloud Server

 

Installing Apache Airflow On Ubuntu, CentOS Cloud Server

 

You can install Apache Airflow anywhere – including your Mac. Because it is just a Python package with submodules. On your Mac if you run :

Vim
1
pip search airflow

Then you’ll get this kind of result :

Advertisement

---

Vim
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
abel-airflow (1.7.1.3.post3)     - Programmatically author, schedule and
                                   monitor data pipelines
airflow-alt-ldap (0.0.1)         - Alternative LDAP auth backend for airflow
                                   to support openLDAP installation without
                                   memberOf overlay
airflow-declarative (1.0)        - Airflow DAGs done declaratively
apache-airflow (1.8.2rc1)        - Programmatically author, schedule and
                                   monitor data pipelines
airflow (1.8.0)                  - Programmatically author, schedule and
                                   monitor data pipelines
airflow-imaging-plugins (2.4.1)  - Airflow plugins to support Neuroimaging
                                   tasks.
airflow_plugin_honeypot (1.0.0)  - Airflow plugin that captures, parses, and
                                   visualizes Hive log files
airflow_utils (0.3)              - collection of helpers to create airflow
                                   dags
AirflowOnTheDumpTruck (0.1.0)    - AirflowOnTheDumptruck bunch of airflow
                                   operators and hooks
fairflow (0.1.2)                 - Functional airflow.
hovertools (0.1.6)               - Tools for airflow-hovercraft
ussd_airflow (0.0.4.7)           - Ussd Airflow Library

Those things are written here :

Vim
1
https://airflow.incubator.apache.org/installation.html

As we are starting with a possible blank server running a server OS like Ubuntu, we need to install these things :

Vim
1
2
3
4
apt update
apt upgrade
apt install unzip build-essential python-dev libsasl2-dev python-pandas
apt install binutils gcc pandas

For CentOS commands will be like this :

Vim
1
2
3
yum update
yum groupinstall "Development tools"
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel python-devel wget cyrus-sasl-devel.x86_64

If want a message broker then you will need to install RabbitMQ. We are not showing that steps as it is optional.
If you want to use MySQL as database repo you will need to install some MySQL dependencies :

Vim
1
2
3
4
apt install python-dev libmysqlclient-dev mariadb-devel
# or
yum install -y mysql-devel python-devel python-setuptools
pip install MySQL-python

Now steps are common for any operating system. Run :

Vim
1
python -V

We need Python 2.7.X, thats it. Then install pip :

Vim
1
2
3
4
5
6
7
wget https://bootstrap.pypa.io/ez_setup.py
python ez_setup.py
unzip setuptools-*.zip
cd setuptools-*
easy_install pip
# AirFlow needs it
pip install -U boto

Installing AirFlow or components is easy :

Vim
1
2
3
4
pip install airflow==1.8.0
pip install airflow[hive]==1.8.0
pip install airflow[celery]==1.8.0
...

Commonly you’ll need this set :

Vim
1
pip install airflow[async,celery,crypto,druid,jdbc,hdfs,hive,kerberos,ldap,mysql,password,rabbitmq,vertica]

Possibly you’ll set $HOME of Airflow from some sort of dot file :

Vim
1
export AIRFLOW_HOME=~/airflow

Initiate Airflow :

Vim
1
airflow initdb

Setup MySQL, create an airflow database, grant all on airflow database. We are not showing the steps.

In that $HOME of Airflow, you need to have dags, logs directories :

Vim
1
2
mkdir dags
mkdir logs

At $HOME of Airflow, there is a airflow.cfg file. That like starts like :

Vim
1
2
3
4
[core]
# The home folder for airflow, default is ~/airflow
airflow_home = /usr/local/airflow
...

That will need to have settings like this :

Vim
1
2
3
4
5
6
7
8
9
10
11
executor = CeleryExecutor
sql_alchemy_conn = mysql://{USERNAME}:{PASSWORD}@{MYSQL_HOST}:3306/airflow
dags_are_paused_at_creation = True
load_examples = False
celery_result_backend = db+mysql://{USERNAME}:{PASSWORD}@{MYSQL_HOST}:3306/airflow
# optional
default_queue = {YOUR_QUEUE_NAME_HERE}
# for RabbitMQ
broker_url = amqp://guest:guest@{RABBITMQ_HOST}:5672/
# for AWS SQS:
broker_url = sqs://{ACCESS_KEY_ID}:{SECRET_KEY}@

Again run :

Vim
1
airflow initdb

Now the last steps :

Vim
1
2
3
4
5
6
7
8
# start server
nohup airflow webserver $* >> ~/airflow/logs/webserver.logs &
# start Celery workers
nohup airflow worker $* >> ~/airflow/logs/worker.logs &
# start Scheduler
nohup airflow scheduler >> ~/airflow/logs/scheduler.logs &
# start Flower
nohup airflow flower >> ~/airflow/logs/flower.logs &

You can now navigate to the Airflow UI on Browser :

Vim
1
http://IP.Address:8080/admin/

And Flower web UI :

Vim
1
http://IP.Address:5555/

Tagged With airflow install OS , install apache-airflow 2 4 3 on centos 7 , install apache air flow in centos 7 , how to install apache air flow , how to configure apache airflow in centos , airflow apache installation , airflow installation centos , airflow installation , download apache airflow as a tar , install apache airflow in centos 7

This Article Has Been Shared 597 Times!

Facebook Twitter Pinterest

Abhishek Ghosh

About Abhishek Ghosh

Abhishek Ghosh is a Businessman, Surgeon, Author and Blogger. You can keep touch with him on Twitter - @AbhishekCTRL.

Here’s what we’ve got for you which might like :

Articles Related to Installing Apache Airflow On Ubuntu, CentOS Cloud Server

  • Install Apache Kafka on Ubuntu 16.04 : Single Cloud Server

    Here Are The Steps On How To Install Apache Kafka on Ubuntu 16.04 Running One Single Cloud Server Instance. Apache Kafka

  • Install Apache Zeppelin On Ubuntu 16.04

    Zeppelin Can Be Pre-Built Package Or Can Be Build From Source. Here Is How To Install Apache Zeppelin On Ubuntu 16.04 Building From Source.

  • List Of Open Source Big Data Visualization Tools

    Here Is List Of Open Source Big Data Visualization Tools Which Will Convert Data In To Graphs, Charts Usable With Tools Like Spark, Hadoop.

  • What is Data Refining in Big Data?

    What is Data Refining in Big Data? In Data Refining we refine disparate data to increase understanding of the data & remove data variability.

  • Install Apache SystemML Machine Learning System on Ubuntu

    Here is How To Install Apache SystemML Machine Learning System on Ubuntu 16.04. SystemML needs a minimum guidance to get started to use it.

Additionally, performing a search on this website can help you. Also, we have YouTube Videos.

Take The Conversation Further ...

We'd love to know your thoughts on this article.
Meet the Author over on Twitter to join the conversation right now!

If you want to Advertise on our Article or want a Sponsored Article, you are invited to Contact us.

Contact Us

Subscribe To Our Free Newsletter

Get new posts by email:

Please Confirm the Subscription When Approval Email Will Arrive in Your Email Inbox as Second Step.

Search this website…

 

Popular Articles

Our Homepage is best place to find popular articles!

Here Are Some Good to Read Articles :

  • Cloud Computing Service Models
  • What is Cloud Computing?
  • Cloud Computing and Social Networks in Mobile Space
  • ARM Processor Architecture
  • What Camera Mode to Choose
  • Indispensable MySQL queries for custom fields in WordPress
  • Windows 7 Speech Recognition Scripting Related Tutorials

Social Networks

  • Pinterest (24.3K Followers)
  • Twitter (5.8k Followers)
  • Facebook (5.7k Followers)
  • LinkedIn (3.7k Followers)
  • YouTube (1.3k Followers)
  • GitHub (Repository)
  • GitHub (Gists)
Looking to publish sponsored article on our website?

Contact us

Recent Posts

  • Cyberpunk Aesthetics: What’s in it Special January 27, 2023
  • How to Do Electrical Layout Plan for Adding Smart Switches January 26, 2023
  • What is a Data Mesh? January 25, 2023
  • What is Vehicular Ad-Hoc Network? January 24, 2023
  • Difference Between Panel Light, COB Light, Track Light January 21, 2023

About This Article

Cite this article as: Abhishek Ghosh, "Installing Apache Airflow On Ubuntu, CentOS Cloud Server," in The Customize Windows, August 7, 2017, January 29, 2023, https://thecustomizewindows.com/2017/08/installing-apache-airflow-ubuntu-centos-cloud-server/.

Source:The Customize Windows, JiMA.in

PC users can consult Corrine Chorney for Security.

Want to know more about us? Read Notability and Mentions & Our Setup.

Copyright © 2023 - The Customize Windows | dESIGNed by The Customize Windows

Copyright  · Privacy Policy  · Advertising Policy  · Terms of Service  · Refund Policy

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT