• Home
  • Archive
  • Tools
  • Contact Us

The Customize Windows

Technology Journal

  • Cloud Computing
  • Computer
  • Digital Photography
  • Windows 7
  • Archive
  • Cloud Computing
  • Virtualization
  • Computer and Internet
  • Digital Photography
  • Android
  • Sysadmin
  • Electronics
  • Big Data
  • Virtualization
  • Downloads
  • Web Development
  • Apple
  • Android
Advertisement
You are here: Home » How to Install Apache Hama (HDFS Installation)

By Abhishek Ghosh July 2, 2018 9:33 pm Updated on July 2, 2018

How to Install Apache Hama (HDFS Installation)

Advertisement

Apache Hama is a Distributed Computing Framework For Massive Scientific Computations. Hama consists of 3 major components – BSPMaster, GroomServers and Zookeeper. It is a framework for Big Data analytics which uses the Bulk Synchronous Parallel (BSP) computing model. It provides BSP programming model, vertex and neuron centric programming models. Hama can be installed in local/pseudo-distributed mode or as HDFS installation for multiple node. We can set up Apache Hama to work with Apache Mesos, Apache Hadoop, Apache Spark. In order to run Hama on Mesos it is required that Mesos already be installed on the cluster. Instructions to set up Mesos may be found at the project website. Here is How to Install Apache Hama as HDFS Installation.

How to Install Apache Hama

 

How to Install Apache Hama

 

We have to assume that you have already set up HDFS (may be CDH on your cluster) and Apache Mesos. CDH, HDP, and Mesosphere does not need any installation. We must point that some versions of HDFS CDH may be odd error and version of Apache Hama older than 0.70 may have different guides. Officially requirements to run Hama is hadoop-1.0.x or higher version (non-secure version), Sun Java JDK 1.7.x or higher version and SSH access to manage BSP deamons.

Download latest Hama :

Advertisement

---

Vim
1
http://www.apache.org/dyn/closer.cgi/hama

First clean previous versions on Hama on HDFS:

Vim
1
hadoop fs -rm /hama.tar.gz

Now build Hama for the particular HDFS and Mesos version, command will be like :

Vim
1
mvn clean install -Phadoop2 -Dhadoop.version=2.3.0-cdh5.1.2 -Dmesos.version=0.20.0 -DskipTests

We have to put it to HDFS (name of tar file same as your configuration file) :

Vim
1
hadoop fs -put dist/target/hama-0.7.1-SNAPSHOT.tar.gz /hama.tar.gz

Move to the distribution directory:

Vim
1
cd dist/target/hama-0.7.0-SNAPSHOT/hama-0.7.1-SNAPSHOT/

Set LD_LIBRARY_PATH or MESOS_NATIVE_LIBRARY environment variables correctly to mesos installation libraries. Default is usr/lib/mesos :

Vim
1
export LD_LIBRARY_PATH=/root/mesos-installation/lib/

Without setting them correctly, bspmaster will not start throwing Expecting an absolute path of the library error on log. Configure Hama with Mesos, you can follow instructions from the Hama wiki. This is an example configuration :

Vim
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
<configuration>
<property>
<name>bsp.master.address</name>
<value>euca-10-2-112-10.eucalyptus.internal:40000</value>
<description>The address of the bsp master server. Either the
literal string "local" or a host[:port] (where host is a name or
IP address) for distributed mode.
</description>
</property>
<property>
<name>bsp.master.port</name>
<value>40000</value>
<description>The port master should bind to.</description>
</property>
<property>
<name>bsp.master.TaskWorkerManager.class</name>
<value>org.apache.hama.bsp.MesosScheduler</value>
<description>Instructs the scheduler to use Mesos to execute tasks of each job
</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://euca-10-2-112-10.eucalyptus.internal:9000</value>
<description>
The name of the default file system. Either the literal string
"local" or a host:port for HDFS.
</description>
</property>
<property>
<name>hama.mesos.executor.uri</name>
<value>hdfs://euca-10-2-112-10.eucalyptus.internal:9000/hama.tar.gz</value>
<description>This is the URI of the Hama distribution
</description>
</property>
<!-- Hama requires one cpu and memory defined by bsp.child.java.opts for each slot.
This means that a cluster with bsp.tasks.maximum.total set to 2 and bsp.child.jova.opts set to -Xmx1024m
will need at least 2 cpus and and 2048m of memory. -->
<property>
<name>bsp.tasks.maximum.total</name>
<value>2</value>
<description>This is an override for the total maximum tasks that may be run.
The default behavior is to determine a value based on the available groom servers.
However, if using Mesos, the groom servers are not yet allocated.
So, a value indicating the maximum number of slots available in the cluster is needed.
</description>
</property>
<property>
<name>hama.mesos.master</name>
<value>zk://euca-10-2-112-10.eucalyptus.internal:2181/mesos</value>
<description>This is the address of the Mesos master instance.
If you're using Zookeeper for master election, use the Zookeeper address here (i.e.,zk://zk.apache.org:2181/hadoop/mesos).
</description>
</property>
<property>
<name>bsp.child.java.opts</name>
<value>-Xmx1024m</value>
<description>Java opts for the groom server child processes.
</description>
</property>
<property>
<name>bsp.system.dir</name>
<value>${hadoop.tmp.dir}/bsp/system</value>
<description>The shared directory where BSP stores control files.
</description>
</property>
<property>
<name>bsp.local.dir</name>
<value>/mnt/bsp/local</value>
<description>local directory for temporal store.</description>
</property>
<property>
<name>hama.tmp.dir</name>
<value>/mnt/hama/tmp/hama-${user.name}</value>
<description>Temporary directory on the local filesystem.</description>
</property>
<property>
<name>bsp.disk.queue.dir</name>
<value>${hama.tmp.dir}/messages/</value>
<description>Temporary directory on the local message buffer on disk.</description>
</property>
<property>
<name>hama.zookeeper.quorum</name>
<value>euca-10-2-112-10.eucalyptus.internal</value>
<description>Comma separated list of servers in the ZooKeeper Quorum.
For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
By default this is set to localhost for local and pseudo-distributed modes
of operation. For a fully-distributed setup, this should be set to a full
list of ZooKeeper quorum servers. If HAMA_MANAGES_ZK is set in hama-env.sh
this is the list of servers which we will start/stop zookeeper on.
</description>
</property>
<property>
<name>hama.zookeeper.property.clientPort</name>
<value>2181</value>
<description>The port to which the zookeeper clients connect
</description>
</property>
</configuration>

Run bspmaster:

Vim
1
./bin/hama-daemon.sh start bspmaster

Check the log files for error :

Vim
1
2
less hama-root-bspmaster-$HOSTNAME.log
less hama-root-bspmaster-$HOSTNAME.out

If everything is fine then you can run some examples, like :

Vim
1
./bin/hama jar hama-examples-0.7.1-SNAPSHOT.jar gen fastgen 100 10 randomgraph 2

You will get output two files as two partitions of a graph, now run :

Vim
1
./bin/hama jar hama-examples-0.7.0-SNAPSHOT.jar pagerank randomgraph pagerankresult 4

List files in HDFS :

Vim
1
hadoop fs -ls /user/root/randomgraph

We guess, now you can use more examples.

Tagged With awaysac , exclaimediqp , installation apache hama , J5BV , risingx8a , silencehut , subject5we

This Article Has Been Shared 392 Times!

Facebook Twitter Pinterest

Abhishek Ghosh

About Abhishek Ghosh

Abhishek Ghosh is a Businessman, Surgeon, Author and Blogger. You can keep touch with him on Twitter - @AbhishekCTRL.

Here’s what we’ve got for you which might like :

Articles Related to How to Install Apache Hama (HDFS Installation)

  • Install Apache SystemML Machine Learning System on Ubuntu

    Here is How To Install Apache SystemML Machine Learning System on Ubuntu 16.04. SystemML needs a minimum guidance to get started to use it.

  • Approaches of Deep Learning : Part 4 (Conclusion)

    This article will end our series on Approaches of Deep Learning which has total four parts – PART 1, PART 2, PART 3 and this current one. After reading so many thousands of words, the reader possibly have some theoretical idea around how to independently learn Deep Learning. This particular article in essence will draw […]

  • How To Process Healthcare Data in Hadoop, Pig (IBM Demo Cloud)

    Here is Basic Commends Showing How To Process Healthcare Data in Hadoop, Pig Using IBM Demo Cloud. Unlike server log, healthcare data lacks universal format.

  • Visualization of SQL Data in Jupyter Notebook & Embedding in WordPress Post

    Here is How To On Visualization of SQL Data in Jupyter Notebook & Embedding in WordPress Post in Easy Language, in All Steps.

  • How to Install Apache Tika on Ubuntu Server

    Apache Tika is a Content Analysis Framework Which Can Be Configure With Web Software Like WordPress For Metadata Extraction of PDF, doc. Here is How to Install Apache Tika on Ubuntu Server.

Additionally, performing a search on this website can help you. Also, we have YouTube Videos.

Take The Conversation Further ...

We'd love to know your thoughts on this article.
Meet the Author over on Twitter to join the conversation right now!

If you want to Advertise on our Article or want a Sponsored Article, you are invited to Contact us.

Contact Us

Subscribe To Our Free Newsletter

Get new posts by email:

Please Confirm the Subscription When Approval Email Will Arrive in Your Email Inbox as Second Step.

Search this website…

 

Popular Articles

Our Homepage is best place to find popular articles!

Here Are Some Good to Read Articles :

  • Cloud Computing Service Models
  • What is Cloud Computing?
  • Cloud Computing and Social Networks in Mobile Space
  • ARM Processor Architecture
  • What Camera Mode to Choose
  • Indispensable MySQL queries for custom fields in WordPress
  • Windows 7 Speech Recognition Scripting Related Tutorials

Social Networks

  • Pinterest (24.3K Followers)
  • Twitter (5.8k Followers)
  • Facebook (5.7k Followers)
  • LinkedIn (3.7k Followers)
  • YouTube (1.3k Followers)
  • GitHub (Repository)
  • GitHub (Gists)
Looking to publish sponsored article on our website?

Contact us

Recent Posts

  • Projector Screen Basics February 6, 2023
  • What is Configuration Management February 5, 2023
  • What is ChatGPT? February 3, 2023
  • Zebronics Pixaplay 16 : Entry Level Movie Projector Review February 2, 2023
  • What is Voice User Interface (VUI) January 31, 2023

About This Article

Cite this article as: Abhishek Ghosh, "How to Install Apache Hama (HDFS Installation)," in The Customize Windows, July 2, 2018, February 6, 2023, https://thecustomizewindows.com/2018/07/how-to-install-apache-hama-hdfs-installation/.

Source:The Customize Windows, JiMA.in

PC users can consult Corrine Chorney for Security.

Want to know more about us? Read Notability and Mentions & Our Setup.

Copyright © 2023 - The Customize Windows | dESIGNed by The Customize Windows

Copyright  · Privacy Policy  · Advertising Policy  · Terms of Service  · Refund Policy

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT