Home built Hadoop analytics cluster: Part 3

Home built Hadoop analytics cluster: Part 3

In my previous post I covered Bill of Materials (BOM), hardware assembly, and installing Linux (Ubuntu).  In this post I will cover how I installed Ambari.

Installing Ambari
Rather than build from source, I opted to use the distribution from Cloudera (formerly HortonWorks). Ambari 2.7.5 requires official support from Cloudera, so I went down to 2.7.3 which doesn't require a support agreement with Cloudera.

Install some pre-requisites
sudo apt install python-dev
sudo apt install gcc

Add Cloudera as a distribution
sudo wget -O /etc/apt/sources.list.d/ambari.list http://public-repo-1.hortonworks.com/ambari/ubuntu18/2.x/updates/2.7.3.0/ambari.list
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
sudo apt-get update

Verify packages show up
apt-cache showpkg ambari-server
apt-cache showpkg ambari-agent
apt-cache showpkg ambari-metrics-assembly

Install and setup the server on primary
sudo apt-get install ambari-server 

Install and setup agent / metrics on all
sudo apt-get install ambari-agent 
sudo apt-get install ambari-metrics-assembly
 

Cloudera also had some instructions that I followed on how to configure the Ambari server once installed.

Next up will be building the remaining boxes of the cluster and installing the agent on those.


Popular posts from this blog

Home built Hadoop analytics cluster: Part 4

Image
Home built Hadoop analytics cluster: Part 4 So yay!  As mentioned for my next goals in my previous post , I finally got the remaining two boxes built out and added into my home network.  I opted to put the Hadoop cluster on it's own subnet with a dedicated unmanaged switch only for the cluster (primary and nodes). I added the agent and metrics to all of the nodes and rebooted the servers. Then I followed the instructions to set up the cluster, naming it "ds730" after the class that I'm currently taking - DS730: Big Data - High Performance Computing. I also made sure I had DNS setup correctly by modifying /etc/systemd/resolved.conf and fixed my name resolution issues. Removed firewall rules. Removed timedatectl by doing: sudo timedatectl set-ntp no Then installed ntp: sudo apt install ntp Now I need to look at installing some database drivers, however I think I'm going to call it a night.  
Read more

Home built blob storage server

Image
In the hopes of creating a "blob" like storage like Amazon S3 , I recently did a Google for open source blob storage.  To my pleasure, I discovered minio .  Minio allows me to expose an S3 compatible service locally on my home network.  I can now work with large datasets in a S3-like fashion locally without having the overhead of dealing with an Internet connection. I can also set up Minio to be a gateway to Amazon S3 or even to my local Hadoop cluster . I also am able to set up the AWS CLI to interact with minio or have the minio client interact with AWS S3. While redundancy can be implemented with minio, I'll save that as a project for later. I picked up a Raspberry Pi 4 8 GB model from Amazon ($150) and a 8 TB external USB drive from Costco ($120).  One can always step down to a lower model / storage space if needed - just couldn't resist the savings on an 8 TB drive from Costco. :) I downloaded the Raspbian Lite image , set up regionalization and my hostname...
Read more

Setbacks

So, for some reason some recent updates to Ubuntu decided to cause the resolution of my monitor to go down. So I tried fixing it doing some updates, and lo and behold, nothing worked. Then on top of that, my virtual machine that I was using for my DS730 class decided to go belly up and didn't come back after 2 reboots. Not having a good day. So, now I'm going to document the installation of my workstation using a blog post so I have some documentation to fall back onto when things go haywire again.  It looks like I'm going to reinstall Ubuntu again. Then I need to hope that rebooting the virtual machine one last time will work.
Read more