Home built Hadoop analytics cluster: Part 5

Home built Hadoop analytics cluster: Part 5

Got the mysql database installed and configured on my secondary node.  Installed the driver on the primary node.  Set up a few users and a database.  Tested the connections.

Now hopefully all goes well with the install!

Home built blob storage server

In the hopes of creating a "blob" like storage like Amazon S3, I recently did a Google for open source blob storage.  To my pleasure, I discovered minio.  Minio allows me to expose an S3 compatible service locally on my home network.  I can now work with large datasets in a S3-like fashion locally without having the overhead of dealing with an Internet connection.

I can also set up Minio to be a gateway to Amazon S3 or even to my local Hadoop cluster.

I also am able to set up the AWS CLI to interact with minio or have the minio client interact with AWS S3.

While redundancy can be implemented with minio, I'll save that as a project for later.

I picked up a Raspberry Pi 4 8 GB model from Amazon ($150) and a 8 TB external USB drive from Costco ($120).  One can always step down to a lower model / storage space if needed - just couldn't resist the savings on an 8 TB drive from Costco. :)

I downloaded the Raspbian Lite image, set up regionalization and my hostname.  Attached the USB drive and created a brand new ext3 partition on the USB drive, wiping out everything else.  Formatted and attached the drive and made sure it came up on reboots.

Then I downloaded minio using the wget process.  

    wget https://dl.minio.io/server/minio/release/linux-arm/minio

    wget https://dl.minio.io/client/mc/release/linux-arm/mc

    sudo ln -s /home/pi/minio /usr/bin/minio

    sudo ln -s /home/pi/mc /usr/bin/mc

Then I made a simple shell script (start.sh) to launch minio.

    #!/bin/bash

 

    export MINIO_ACCESS_KEY=SuperSecretAccessKey

    export MINIO_SECRET_KEY=SuperSecretSecretKey

    export MINIO_DOMAIN=blobstorage

    export MINIO_DISK_USAGE_CRAWL=off

 

    /usr/bin/minio server /mnt/data

Then I made a file 

    /etc/systemd/system/minio.service

Inside the file I put the following:

    [Unit]

    Description=Minio Storage Service

    After=network-online.target mnt-data.mount

    [Service]

    ExecStart=/home/pi/start.sh

    WorkingDirectory=/mnt/data

    StandardOutput=inherit

    StandardError=inherit

    Restart=always

    User=pi

 

    [Install]

    WantedBy=multi-user.target

I then verified the service worked as intended:

    $ sudo systemctl start minio

    $ sudo systemctl status minio

Opened my browser, and once I logged in, I was able to access the minio service via browser and I made an alias for my minio client.