Bradford Embedded

Debian Jessie and LVM Cache Exact Steps

The whole concept of using a pile of big disks as a RAID in order to get lots of storage and some redundancy has always intrigued me. Back in 2006, when I last built a home computer for myself, I bought a PCI-X battery backed RAID card and loaded that system up with four 400 GB disks (not small for the time). It was very nice to know that my data wouldn’t get lost due to an unexpected power loss and that the RAID card (with 256 MB of RAM itself) would do some nice caching for me to speed things up.

But now I’m a bit more thrifty ($300 for a RAID card is not in my budget!) and I’ve gotten addicted to the disk speeds of SSDs.

Hence, the introduction of md-cache into Linux and the lvmcache capability into the lvm2 tools is very interesting to me as now I can have a software RAID (without some of the nice features the hardware RAID had but with a $0 price tag) and use an SSD as a cache for the big slow disks!

There’s a few decent resources on the web saying how to build up your disks into a software RAID and then add dm-caching to it, but I wanted to put all my personal experiences in one place for my own reference.

You’ll need at least Linux 3.9 in order to enable dm-cache and you’ll need a relatively recent version of the lvm2 tools (I am using lvm2 v2.02.111 in Debian Jessie whose dependencies include a bunch of systemd things which may make backporting it to wheezy not super simple). You can, in theory, make this work on Debian Wheezy if you can backport (or simply package) a modern lvm2 tools package and use a backported 3.9 or newer Linux kernel.

My test installation virtual machine has 3 big disks (20GB each) and 1 smaller SSD (8GB). Each of the big disks are /dev/sda, /dev/sdb, and /dev/sdc while the SSD is /dev/sdd.

During install, I created 3 partitions on each of the big disks:

  1. sdX1: 256 MB partition for software RAID
  2. sdX2: 2 GB partition for swap
  3. sdX3: remaining space for software RAID

Then, setting up the software RAID in the Debian installer, combine all of the sdX1 partitions into a RAID1 and all of the sdX3 partitions into a RAID5.

Set up the sdX1 RAID1 as an ext4 file system and configure it to mount to /boot. Set up the sdX3 RAID5 to be used for LVM. Set up each of the sdX2 partitions (all 3, independently) as swap space (the swapping will take advantage of all the drives in parallel so no need to burden them with software RAID).

Set up the LVM space to split into two logical volumes, one for / and one for /home. Format each for ext4 and the proper mounted locations.

Continue with the Debian installer as normal. After the installer finishes, I found that grub was not installed correctly and I had to manually help it out a bit with:

$ grub-install --recheck /dev/sda
$ grub-install --recheck /dev/sdb
$ grub-install --recheck /dev/sdc

Manually check that your /etc/fstab is correct, too. Just in case.

Once you can boot into your system, then setup caching for both the / and /home logical volumes.

Setup /dev/sdd as a physical volume:

$ sudo pvcreate /dev/sdd

My volume group (which contains the / and /home lvs) is called vg0, so to add the /dev/sdd pv to vg0 you need to extend vg0:

$ sudo vgextend vg0 /dev/sdd

Now create a pair of 3.6 GB cache logical volumes and a pair of 32 MB cache metadata logical volumes (one cache and cache metadata for each lv you want to cache):

$ sudo lvcreate -n lvrootcache -L3.6G vg0 /dev/sdd
$ sudo lvcreate -n lvhomecache -L3.6G vg0 /dev/sdd
$ sudo lvcreate -n lvrootcachemeta -L32M vg0 /dev/sdd
$ sudo lvcreate -n lvhomecachemeta -L32M vg0 /dev/sdd

Turn your cache lvs and cachemeta lvs into cache pools (one cache and one set of metadata per cache pool, ideally you might want to put the metadata on a different physical SSD from the cache itself):

$ sudo lvconvert --type cache-pool --cachemode writeback --poolmetadata \
	vg0/lvhomecachemeta vg0/lvhomecache
$ sudo lvconvert --type cache-pool --cachemode writeback --poolmetadata \
	vg0/lvrootcachemeta vg0/lvrootcache

Lastly, configure each of the actual logical volumes (root and home) to be cached by each of the cache pools:

$ sudo lvconvert --type cache --cachepool vg0/lvrootcache vg0/lvroot
$ sudo lvconvert --type cache --cachepool vg0/lvhomecache vg0/lvhome

Now, before you boot up with this caching configuration, make sure your kernel actually supports dm-cache by ensuring it has CONFIG_DM_CACHE set to y or if a module that the module gets loaded during early boot! (Debian Jessie’s present 3.16.0-4 kernel on amd64 does not, so booting is fun!)

EDIT TO ADD

In addition to all of this configuration, you’ll also need the thin-provisioning-tools package and to ensure that your initramfs gets built with the proper modules included and the cache-check program. There’s a nice overview on the Debian forum which includes this script which you should place into the /etc/initramfs-tools/hooks/ directory so that when you run update-initramfs -u that the right things happen:

#!/bin/sh

PREREQ="lvm2"

prereqs()
{
    echo "$PREREQ"
}

case $1 in
prereqs)
    prereqs
    exit 0
    ;;
esac

if [ ! -x /usr/sbin/cache_check ]; then
    exit 0
fi

. /usr/share/initramfs-tools/hook-functions

copy_exec /usr/sbin/cache_check

manual_add_modules dm_cache dm_cache_mq

I’m also tracking some of my own notes on this work in a gist.

Published

26 March 2015