Managing Storage with LVM

LVM or Logical Volume Manager is a key tool for system administrators managing storage on Linux servers. LVM introduces three layers of abstraction b/n the underlying storage devices on a Linux server and the file system which is accessed by the users.

LVM subsystem

This example has 3 hard drive devices – sdc1, sdd1 and sde. We will use LVM to aggregate these 3 hard drives to provide storage, extend 3 formatted file system and mount it under a directory called ‘/lvmdata’. This additional abstraction between the underlying storage devices and the user accessible file system enables us to manage storage more effectively and with greater flexibility.

Let’s discuss each LVM layer in sequence and take a look at some specific command used to configure each layer.

The 1st layer is know as LVM Physical Volumes (PV’s). A storage device must be formatted as a Physical Volume (PV) before it can participate in the LVM infrastructure that we will be building.

pvcreate /dev/sdc1 /dev/sdd1

You can check the newly created PV using the command ‘pvdisplay’. This command shows the 2 physcal volumes (sdc1 and sdd1) that we used right now.

Next we need to aggregate the PV into a single contiguous pool of storage known as Volume Group (VG).

vgcreate exampeVG /dev/sdc1 /dev/sdd1

You can check the newly created VG using ‘vgdisplay’ command. VGdisplay tells you the volume group name, how much size the volume has and so on.

Finally, we need to carve out a section of VG that we can then place our file system on. This carved out piece of storage is known as Logical Volume (LV).

lvcreate -n exampleLV -L 1G exampleVG

You can check the newly created LV using the command ‘lvdisplay’.

Now, create a file system, create a mount point (/lvmdata/) and then finally mount the new LVM file system back to the mount point.

mkfs.ext3 /dev/exampleVG/exampleLV

mkdir /lvmdata

mount /dev/exampleVG/exampleLV /lvmdata/

We can now expand our /lvmdata filesystem in real time while all the data remains online and fully accessible.

You can look the current storage utilization using df command:

df -h

You can see the new LVM (/lvmdata/) partition which is of 1 gig in size.

Now to extend the /lvmdata partition, you can use the command ‘lvextend’ as follows:

lvextend -L +500M /dev/exampleVG/exampleLV

Here we are adding additional 500 megs of space to the existing exampleLV logical volume. Note the output of df command now. You will see the additional 500 megs is not reflected on the /lvmdata although we have already added 500 megs to it. This is because we have extended the underlying logical volume, but not the file system itself. The command ‘resize2fs’ will do the trick.

resize2fs /dev/exampleVG/exampleLV

Now, check the ‘df -h’ command and see the difference. You will see the additional 500 megs added to the /lvmdata/ partition.

Remember, all this was done without unmounting the filesystem and all the data remained intact and accessible to users.

Migrating data to a SAN storage device:

Imagine you want to migrate all your data from your servers local hard drive to a SAN (Storage Area Network).

In our case the SAN device is /dev/sde

So, our job is to migrate all of our data from local storage (sdc1 and sdd1) to our SAN device (sde)

Using LVM you can easily accomplish this task. First we need to initialize our SAN storage using pvcreate.

pvcreate /dev/sde

Then we can extend our existing Volume Group (VG) on to our new SAN device using ‘vgextend’.

vgextend exampleVG /dev/sde

Now we need to migrate all the data from our local physical volume to our SAN PVs using the command ‘pvmove’

pvmove /dev/sdc1 /dev/sde
pvmove /dev/sdd1 /dev/sde

Finally, we can remove the local storage devices from our VG using the command ‘vgreduce’

vgreduce exampleVG /dev/sdc1 /dev/sdd1

Again, this was done in real time when the file system data remained online and fully accessible to the users.

The below shows the practical example of creation and extending an LVM volume

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-13054, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-13054, default 13054):
Using default value 13054

Command (m for help): p

Disk /dev/xvdk: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc77f72d6

Device Boot Start End Blocks Id System
/dev/xvdk1 1 13054 104856223+ 83 Linux

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/xvdk: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc77f72d6

Device Boot Start End Blocks Id System
/dev/xvdk1 1 13054 104856223+ 8e Linux LVM

Command (m for help): w
The partition table has been altered!

[root@nagios ~]# fdisk -l /dev/xvdk

Disk /dev/xvdk: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc77f72d6

Device Boot Start End Blocks Id System
/dev/xvdk1 1 13054 104856223+ 8e Linux LVM

[root@nagios ~]# pvcreate /dev/xvdk1
Writing physical volume data to disk “/dev/xvdk1”
Physical volume “/dev/xvdk1” successfully created

[root@nagios ~]# pvdisplay
“/dev/xvdk1” is a new physical volume of “100.00 GiB”
— NEW Physical volume —
PV Name /dev/xvdk1
VG Name
PV Size 100.00 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID CgYySs-W40i-zjOn-j1oY-uPk9-982l-fWd0PX

[root@nagios ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/xvdk1 lvm2 a– 100.00g 100.00g

[root@nagios ~]# vgcreate exampleVG /dev/xvdk1
Volume group “exampleVG” successfully created

[root@nagios ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/xvdk1 exampleVG lvm2 a– 100.00g 100.00g

[root@nagios ~]# pvdisplay
— Physical volume —
PV Name /dev/xvdk1
VG Name exampleVG
PV Size 100.00 GiB / not usable 2.66 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 25599
Free PE 25599
Allocated PE 0
PV UUID CgYySs-W40i-zjOn-j1oY-uPk9-982l-fWd0PX

[root@nagios ~]# lvcreate -n exampleLV -L 20G exampleVG
Logical volume “exampleLV” created

[root@nagios ~]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
exampleLV exampleVG -wi-a- 20.00g

[root@nagios ~]# lvdisplay
— Logical volume —
LV Name /dev/exampleVG/exampleLV
VG Name exampleVG
LV UUID xMd0My-l7Ez-dqYb-JM75-cKH3-YOfK-HSV1zx
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:0

[root@nagios ~]# vgdisplay
— Volume group —
VG Name exampleVG
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 100.00 GiB
PE Size 4.00 MiB
Total PE 25599
Alloc PE / Size 5120 / 20.00 GiB
Free PE / Size 20479 / 80.00 GiB
VG UUID 7qvBGR-QYFu-XP27-xx2G-iFUa-GXqX-VCK1zc

[root@nagios ~]# mkfs.ext4 /dev/exampleVG/exampleLV
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1310720 inodes, 5242880 blocks
262144 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@nagios ~]# mkdir /lvmdata
[root@nagios ~]# mount /dev/exampleVG/exampleLV /lvmdata/
[root@nagios ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvde1 5.8G 3.3G 2.3G 60% /
tmpfs 836M 0 836M 0% /dev/shm
/dev/xvdj1 40G 1.3G 37G 4% /drive
/dev/mapper/exampleVG-exampleLV
20G 172M 19G 1% /lvmdata

[root@nagios soj]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvde1 5.8G 3.3G 2.3G 60% /
tmpfs 836M 0 836M 0% /dev/shm
/dev/xvdj1 40G 1.3G 37G 4% /drive
/dev/mapper/exampleVG-exampleLV
20G 2.9G 16G 16% /lvmdata

[root@nagios soj]# lvextend -L +10G /dev/exampleVG/exampleLV
Extending logical volume exampleLV to 30.00 GiB
Logical volume exampleLV successfully resized

[root@nagios soj]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvde1 5.8G 3.3G 2.3G 60% /
tmpfs 836M 0 836M 0% /dev/shm
/dev/xvdj1 40G 1.3G 37G 4% /drive
/dev/mapper/exampleVG-exampleLV
20G 2.9G 16G 16% /lvmdata

[root@nagios soj]# resize2fs /dev/exampleVG/exampleLV
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/exampleVG/exampleLV is mounted on /lvmdata; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/exampleVG/exampleLV to 7864320 (4k) blocks.
The filesystem on /dev/exampleVG/exampleLV is now 7864320 blocks long.

[root@nagios soj]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvde1 5.8G 3.3G 2.3G 60% /
tmpfs 836M 0 836M 0% /dev/shm
/dev/xvdj1 40G 1.3G 37G 4% /drive
/dev/mapper/exampleVG-exampleLV
30G 2.9G 26G 11% /lvmdata

——————

[root@nagios soj]# pvdisplay
— Physical volume —
PV Name /dev/xvdk1
VG Name exampleVG
PV Size 100.00 GiB / not usable 2.66 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 25599
Free PE 17919
Allocated PE 7680
PV UUID CgYySs-W40i-zjOn-j1oY-uPk9-982l-fWd0PX

[root@nagios soj]# vgdisplay
— Volume group —
VG Name exampleVG
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 100.00 GiB
PE Size 4.00 MiB
Total PE 25599
Alloc PE / Size 7680 / 30.00 GiB
Free PE / Size 17919 / 70.00 GiB
VG UUID 7qvBGR-QYFu-XP27-xx2G-iFUa-GXqX-VCK1zc

[root@nagios soj]# lvdisplay
— Logical volume —
LV Name /dev/exampleVG/exampleLV
VG Name exampleVG
LV UUID xMd0My-l7Ez-dqYb-JM75-cKH3-YOfK-HSV1zx
LV Write Access read/write
LV Status available
# open 1
LV Size 30.00 GiB
Current LE 7680
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:0

[root@nagios soj]# pvs
PV VG Fmt Attr PSize PFree
/dev/xvdk1 exampleVG lvm2 a– 100.00g 70.00g

[root@nagios soj]# vgs
VG #PV #LV #SN Attr VSize VFree
exampleVG 1 1 0 wz–n- 100.00g 70.00g

[root@nagios soj]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
exampleLV exampleVG -wi-ao 30.00g
[root@nagios soj]#

========= Creating another partition – lvmvar =============

[root@nagios soj]# lvcreate -n varLV -L 5G exampleVG
Logical volume “varLV” created

[root@nagios soj]# mkfs.ext4 /dev/exampleVG/varLV
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1310720 blocks
65536 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 39 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

[root@nagios soj]# mkdir /lvmvar

[root@nagios soj]# mount /dev/exampleVG/varLV /lvmvar/

[root@nagios soj]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvde1 5.8G 3.3G 2.3G 60% /
tmpfs 836M 0 836M 0% /dev/shm
/dev/xvdj1 40G 1.3G 37G 4% /drive
/dev/mapper/exampleVG-exampleLV
30G 2.9G 26G 11% /lvmdata
/dev/mapper/exampleVG-varLV
5.0G 138M 4.6G 3% /lvmvar

[root@nagios soj]# lvdisplay
— Logical volume —
LV Name /dev/exampleVG/exampleLV
VG Name exampleVG
LV UUID xMd0My-l7Ez-dqYb-JM75-cKH3-YOfK-HSV1zx
LV Write Access read/write
LV Status available
# open 1
LV Size 30.00 GiB
Current LE 7680
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:0

— Logical volume —
LV Name /dev/exampleVG/varLV
VG Name exampleVG
LV UUID y8GibW-vbWh-M68Y-p9vv-AH4f-0FPQ-0cCVwh
LV Write Access read/write
LV Status available
# open 1
LV Size 5.00 GiB
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:1

Advertisements