Installing Local Data Lake on Ubuntu Server : Part 1

In previous guides, we have covered some important basic installation and setup guide for the major known Big Data softwares. Here is Part 1 of Installing Local Data Lake on Ubuntu Server With Hadoop, Spark, Thriftserver, Jupyter etc To Build a Prediction System. We suggest to use servers from VPSDime as they cost very low – $7 per month for 6GM RAM. We talked about some limitations of OpenVZ virtualization. VPSDime is great for test setups unless you are breaking their rules. 12GB is minimum need of RAM. Our older guides went towards analysis of data like log files as one path. Prediction software system is another path. We will use Ubuntu server as most user can use.

I can not give warranty about the version number related typo. At worst, currently WordPress has been just bad with hundreds of funky features and configs becomes odd if wrongly switched from Text to Visual.

 

Installing Local Data Lake on Ubuntu Server : What is Data Lake?

 

Data lake is a method of storing data within a system to facilitate the collocation of data in various schemata and structural forms for various tasks like reporting, visualization, analytics and machine learning. Apache Hadoop distributed file system itself is example data lake.

 

Installing Local Data Lake on Ubuntu Server

 

Please follow our previous guides to install the needed components :

1. Install Hadoop (configure exactly in that way)
2. Install Spark

Also you need to create OpenSSL cert :

So, till this step, we already have configured ssh, added user and installed some softwares. Now install python with some packages like textblob, sklearn, jupyter notebook which you can use to test :

If you run this :

You’ll get :

Complete the steps. Next configure :

I have a sample configuration file as gist, you should fork or copy-paste and edit it.

You should use domain name and follow our guide to install Let’s Encrypt SSL certificate. But, at this state, you can go to :

Now install MySQL from Repo :

Now, we will install Apache Hive, latest version of which you’ll get on :

apache-hive-2.1.1 was latest in my case. So these are commands :

Basically I forgot the path of sample MySQL (it was different before), it should be like :

I am roughly saying the steps of configuring Hive. You have to go to that ../../upgrade/mysql/ and run these :

You’ll get detailed step on official website. Now you have to install Scala and Maven :

This is example of configuring :

You need to edit :

to this way :

Edit :

to this way :

Edit :

to :

Edit :

to :

Edit :

to :

Edit :

to :

Perform :

Edit :

to :

Edit :

to :

Install Zeppelin, IntelliJ IDEA :

Add these kind of line :

Run:

Start all the services :

Now you can visit http://ip_address:8080/.