Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

Knowing multiple technologies is not a big thing. You need to know how to integrate these technologies with each other to solve some real-use cases. Here I’m trying to solve one real challenge which we mostly facing in the Big Data world or in the HADOOP setup.

In Hadoop cluster how to give Dynamic storage to the cluster so you can increase and decrease the size of cluster? 

In this Hadoop cluster I will use 1 slave and 1 Name-node and attach 16GB LVM with this Data-node for provide dynamic storage and I will use 10GB from this LVM so we can increase and decrease storage of cluster in future when we want.

So , First create LVM and integrate LVM with Hadoop.

1.Creating LVM :

👉 First , type fdisk -l command

Here we use two hard disk for create 16 GB LVM.

👉 second ,create the Pv(physical volume)

pvcreate <disk name>

👉 Third ,create the vg(Volume group)

vgcreate <Enter the vg name> <pv name> <pv name>

👉 Fourth , Create LV

lvcreate --size <Size in GB OR MB> --name <ENter the lv name> <VG name>

👉 Five , Format the New LV

👉 Now ,last Mount the lv then our 10GB LV is created successfully.

df -h : For see 10Gb /mylv1 directory is created.

2) Attach LVM with Hadoop

192.168.0.104 is IP of name-node:

👉 Configure name-node hdfs-sie.xml file as below.

👉 Configure name-node core-sie.xml file as below.

👉start name node

hadoop-daemon.sh start namenode

👉 Configure data-node hdfs-sie.xml file as below.

👉 Configure data-node core-sie.xml file as below.

👉 start datanode

hadoop-daemon.sh start datanode

👉 See report for how many data node Connected with namenode.

hadoop dfsadmin -report

Before Lvm Static 50GB hd is used but provide dynamic storage we will attach the LVM .👇 👇

👉Attach LVM with data node :

For attach LVM configure data node Hdfs-site file . In this we will ADDLVM directory (We already make above) in <value> Tage.

Restart The data node

Now see report :

So , successfully we attach 10GB LVM with hadoop cluster.

👉 Now Main part is we our data node is full then how we can increase the size of hard disk without affect the data👈

Increase the lvm size from 10Gb to 14GB :

by two step

1)

lvextend --size <Size> <path of LV>

2)

resize2fs <path of LV>

3)

✨ SO , Finally our data — node Size increase from 10GB to 14 GB✨

Conclusion:

So this post Show that the How LVM is use in Hadoop for Dynamic storage.

Thank you