• Home
  • Archive
  • Tools
  • Contact Us

The Customize Windows

Technology Journal

  • Cloud Computing
  • Computer
  • Digital Photography
  • Windows 7
  • Archive
  • Cloud Computing
  • Virtualization
  • Computer and Internet
  • Digital Photography
  • Android
  • Sysadmin
  • Electronics
  • Big Data
  • Virtualization
  • Downloads
  • Web Development
  • Apple
  • Android
Advertisement
You are here: Home » Integration of Apache Hadoop With OpenStack Swift

By Abhishek Ghosh May 14, 2017 1:27 am Updated on May 14, 2017

Integration of Apache Hadoop With OpenStack Swift

Advertisement

Commonly either peoples are versed with Apache Hadoop or Openstack Swift. The topic integration of Apache Hadoop with Openstack Swift is not exactly new. Good experience with both together may be rare. You can follow our this guide specially for handling OpenStack part without searching here and there. Further you can use this website’s search function to find our old guides apart from the linked articles.

 

Integration of Apache Hadoop With OpenStack Swift : Foreword

 

You must know what you are doing. Hadoop file system widely used with HDFS, but most of them are not built to work out of the box with object storage. It is not odd to get odd behaviour as response. OpenStack installations can differ from vendor to vendor. We have good number of guides on Rackspace and HP Cloud fully separately. Access can be API based or username password based.

Integration of Apache Hadoop With OpenStack Swift

 

Integration of Apache Hadoop With OpenStack Swift

 

We guess that you already have Apache Hadoop up and running, if not please follow our guide for installation and setup of Apache Hadoop on single server instance.

Advertisement

---

Coming to OpenStack Swift, it has client which we described on earlier series of guides like Installation and Setup of OpenStack Python Packages, Uploading to a Swift Container (HP Cloud), Emptying a Swift Container (HP Cloud), mount OpenStack Swift on Ubuntu server (Rackspace) and so an.

After getting used with ordinary files with OpenStack Swift, you can follow Apache Hadoop’s official guide :

Vim
1
https://hadoop.apache.org/docs/current2/hadoop-openstack/index.html

In the same way, OpenStack Swift has official guide :

Vim
1
https://docs.openstack.org/developer/sahara/userdoc/hadoop-swift.html

Regardless of cloud vendor, you’ll need these to configure Apache products including Hadoop, Spark to access :

  • username
  • region for your container
  • authorization URL
  • API key OR password

Please remember that we are not talking about tenant name. Above linked guides have easy integration of those basic things with bash or ZSH. To configure Hadoop for Swift, at location /usr/share/hadoop/etc/hadoop, you’ll find hadoop-env.sh. You need to add a line in this format :

Vim
1
export HADOOP_CLASSPATH=/usr/share/hadoop/share/hadoop/tools/lib/hadoop-openstack-VERSION.jar:/usr/share/hadoop/share/hadoop/tools/lib/httpclient-x.y.z.jar:/usr/share/hadoop/share/hadoop/tools/lib/httpcore-x.y.z.jar:$HADOOP_CLASSPATH

Another file is usr/share/hadoop/etc/hadoop/core-site.xml, swift_test is our example name :

Vim
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<property>
<name>fs.swift.service.swift_test.auth.url</name>
<value>https://identity.vendor.openstack.replace.this.url/v2.0/tokens</value>
<description>VendorName US (multiregion)</description>
</property>
<property>
<name>fs.swift.service.swift_test.username</name>
<value>OS_USER</value>
</property>
<property>
<name>fs.swift.service.swift_test.region</name>
<value>OS_REGION</value>
</property>
<property>
<name>fs.swift.service.swift_test.apikey</name>
<value>OS_APIKEY</value>
</property>

In those official guides you’ll find another variable in case you are running Hadoop outside of OpenStack Swift provider’s datacenter. We already talked about installation of Apache Spark with Hadoop. For, Spark you need to add these lines to /usr/share/spark/conf/spark-env.sh .

Vim
1
2
export SPARK_DIST_CLASSPATH=$(/usr/share/hadoop/bin/hadoop classpath)
export HADOOP_CONF_DIR=/usr/share/hadoop/etc/hadoop

Tagged With paperuri:(65a52f45cf9a751247a26023d6ed10c0)

This Article Has Been Shared 189 Times!

Facebook Twitter Pinterest
Abhishek Ghosh

About Abhishek Ghosh

Abhishek Ghosh is a Businessman, Orthopaedic Surgeon, Author and Blogger. You can keep touch with him on Twitter - @AbhishekCTRL.

Here’s what we’ve got for you which might like :

Articles Related to Integration of Apache Hadoop With OpenStack Swift

  • Installing Apache Airflow On Ubuntu, CentOS Cloud Server

    Airflow Authors, Schedules, Monitors Workflows. Here Are The Steps For Installing Apache Airflow On Ubuntu, CentOS Running On Cloud Server.

  • Big Data Analytics Solutions: On-Premise versus in the Cloud

    Objective of this article Big Data analytics solutions on-premise versus in the cloud is not limited to comparing on-premise & in the cloud.

  • How to Install Apache Gora on Ubuntu Server

    Apache Gora provides an in-memory data model and persistence for big data. Here is How to Install Apache Gora on Ubuntu Server.

  • How to Install Apache Accumulo on Ubuntu (Single Cloud Server Instance)

    Here Are the Steps on How to Install Apache Accumulo on Ubuntu Running on Single Cloud Server Instance. Accumulo provides robust, scalable data storage.

  • How To Use IBM Cloud CDN and Cloud Object Storage

    IBM Cloud Offers Premium Quality Cloud Storage For Different Purposes. Here is Easy Way To Use IBM Cloud CDN and Cloud Object Storage.

Additionally, performing a search on this website can help you. Also, we have YouTube Videos.

Take The Conversation Further ...

We'd love to know your thoughts on this article.
Meet the Author over on Twitter to join the conversation right now!

If you want to Advertise on our Article or want a Sponsored Article, you are invited to Contact us.

Contact Us

Subscribe To Our Free Newsletter

You can subscribe to our Free Once a Day, Regular Newsletter by clicking the subscribe button below.

Click To Subscribe

Please Confirm the Subscription When Approval Email Will Arrive in Your Email Inbox as Second Step.

Search this website…

 

Popular Articles

Our Homepage is best place to find popular articles!

Here Are Some Good to Read Articles :

  • Cloud Computing Service Models
  • What is Cloud Computing?
  • Cloud Computing and Social Networks in Mobile Space
  • ARM Processor Architecture
  • What Camera Mode to Choose
  • Indispensable MySQL queries for custom fields in WordPress
  • Windows 7 Speech Recognition Scripting Related Tutorials

Social Networks

  • Pinterest (21K Followers)
  • Twitter (5.3k Followers)
  • Facebook (5.7k Followers)
  • LinkedIn (3.7k Followers)
  • YouTube (1.3k Followers)
  • GitHub (Repository)
  • GitHub (Gists)
Looking to publish sponsored article on our website?

Contact us

Recent Posts

  • Basics on Python Tornado (web server) March 8, 2021
  • What You Need to Know About Hybrid Mobile App Development March 6, 2021
  • Why Not to Use Your Host for Email Marketing March 5, 2021
  • What You Need to Know About the Microservices March 4, 2021
  • Fix Missing/Bad FileProvider for Freshchat (Android error code 354) March 3, 2021

 

About This Article

Cite this article as: Abhishek Ghosh, "Integration of Apache Hadoop With OpenStack Swift," in The Customize Windows, May 14, 2017, March 8, 2021, https://thecustomizewindows.com/2017/05/integration-apache-hadoop-openstack-swift/.

Source:The Customize Windows, JiMA.in

 

This website uses cookies. If you do not want to allow us to use cookies and/or non-personalized Ads, kindly clear browser cookies after closing this webpage.

Read Cookie Policy.

PC users can consult Corrine Chorney for Security.

Want to know more about us? Read Notability and Mentions & Our Setup.

Copyright © 2021 - The Customize Windows | dESIGNed by The Customize Windows

Copyright  · Privacy Policy  · Advertising Policy  · Terms of Service  · Refund Policy