Home Page
Archive > Posts > Tags > RAID
Search:

Btrfs RAID1 inside LUKS encrypted volumes

I recently moved to Linux and have all my hard drives Luks encrypted, including the primary. I decided to convert my ext4 partitions to Btrfs recently, which I’m totally loving. I also decided to grab another nvme drive and use it as a RAID1 (mirror) drive against my primary drive, using Btrfs’ RAID mechanics. Below are the instructions to accomplish this.

Do note that this is for a situation where you already have a BTRFS volume and want to add a device as RAID1. This assumes you already have your system booting to the LUKS encrypted drive with the root being btrfs. Many modern Linux OS installers can do this for you automatically. Parts of these instructions can still be used in other situations.


  • Hopefully you also have a swap partition under the same LVM as your LUKS root (the Linux Mint installer does this by default), as we’ll be using it. If not, you’ll need to modify the instructions. This script resizes the swap partition and adds an “extra” partition to hold your drive key. This is required because a drive key cannot be loaded off your btrfs volume as both drives need to be unlocked first.
  • This should be ran from another operating system. I would recommend using Universal USB Installer to do this. It allows you to put multiple OS live cds on a USB key, including optional persistence.
  • Run the following script as root (you can use sudo). Make sure to fill in the variables section first. Or even better, run the script 1 line at a time to make sure there are no problems.

#!/bin/bash
#-----------------------------------Variables----------------------------------
#Current root drive
CurPart="nvme0n1p3" #The current drive partition in /dev. This example uses nvme disk #0 partition #3
CurCryptVol="vgmint" #What you named your LVM under LUKS
CurCryptRoot="root" #What you named your root partition under the LVM
CurCryptRootSubVol="/" #The path of the subvolume that is used as the root partition. For example, I use “@”
CurCryptSwap="swap_1" #What you named your swap partition under the LVM
CurCryptExtra="extra" #What you WANT to name your extra partition under the LVM
CurCryptExtraSize="100M" #How big you want your extra partition that will hold your key file
CurKeyPath="" #The path to a key file that will unlock both drives. If left blank then one will be created

#New drive
NewDrive="nvme1n1" #The new drive in /dev. This example uses nvme disk #1
NewPart="nvme1n1p3" #The new partition in /dev. You will be creating this with the cfdisk. This example uses nvme disk#1 partition#3
NewCryptName="raid1_crypt" #What we’ll name the root LUKS partition (no LVM)

#Other variables you do not need to set
CurMount="/mnt/primary"
ExtraMountPath="$CurMount/mnt/extra"
BtrfsReleasePath="kdave/btrfs-progs"
BtrfsReleaseFile="btrfs.box.static"
DriveKeyName="drivekey"

echo "---------------------------------Update BTRFS---------------------------------"
echo "Make sure you are using the latest btrfs-progs"
cd "$(dirname "$(which btrfs)")"
LATEST_RELEASE=$(curl -s "https://api.github.com/repos/$BtrfsReleasePath/releases/latest" | grep tag_name | cut -d \" -f4)
wget "https://github.com/$BtrfsReleasePath/releases/download/$LATEST_RELEASE/$BtrfsReleaseFile"
chmod +x "$BtrfsReleaseFile"

echo "Link all btrfs programs to btrfs.box.static. Rename old files as .old.FILENAME"
if ! [ -L ./btrfs ]; then
  for v in $(\ls btrfs*); do
    if [ "$v" != "$BtrfsReleaseFile" ]; then
      mv "$v" ".old.$v"
      ln -s "$BtrfsReleaseFile" "$v"
    fi
  done
fi

echo "--------------------------Current drive and key setup-------------------------"
echo "Mount the current root partition"
cryptsetup luksOpen "/dev/$CurPart" "$CurCryptVol"
vgchange -ay "$CurCryptVol"
mkdir -p "$CurMount"
mount -o "subvol=$CurCryptRootSubVol" "/dev/$CurCryptVol/$CurCryptRoot" "$CurMount"

echo "If the extra volume has not been created, then resize the swap and create it"
if ! [ -e "/dev/$CurCryptVol/$CurCryptExtra" ]; then
  lvremove -y "/dev/$CurCryptVol/$CurCryptSwap"

  lvcreate -n "$CurCryptExtra" -L "$CurCryptExtraSize" "$CurCryptVol"
  mkfs.ext4 "/dev/$CurCryptVol/$CurCryptExtra"

  lvcreate -n "$CurCryptSwap" -l 100%FREE "$CurCryptVol"
  mkswap "/dev/$CurCryptVol/$CurCryptSwap"
fi

echo "Make sure the key file exists, if it does not, either copy it (if given in $CurKeyPath) or create it"
mkdir -p "$ExtraMountPath"
mount "/dev/$CurCryptVol/$CurCryptExtra" "$ExtraMountPath"
if ! [ -e "$ExtraMountPath/$DriveKeyName" ]; then
  if [ "$CurKeyPath" != "" ]; then
    if ! [ -e "$CurKeyPath" ]; then
      echo "Not found: $CurKeyPath"
      exit 1
    fi
    cp "$CurKeyPath" "$ExtraMountPath/$DriveKeyName"
  else
    openssl rand -out "$ExtraMountPath/$DriveKeyName" 512
  fi
  chmod 400 "$ExtraMountPath/$DriveKeyName"
  chown root:root "$ExtraMountPath/$DriveKeyName"
fi

echo "Make sure the key file works on the current drive"
if cryptsetup --test-passphrase luksOpen --key-file "$ExtraMountPath/$DriveKeyName" "/dev/$CurPart" test; then
  echo "Keyfile successfully opened the LUKS partition."
  #cryptsetup luksClose test #This doesn’t seem to be needed
else
  echo "Adding keyfile to the LUKS partition"
  cryptsetup luksAddKey "/dev/$CurPart" "$ExtraMountPath/$DriveKeyName"
fi

echo "--------------------------------New drive setup-------------------------------"
echo "Use cfdisk to set the new disk as GPT and add partitions."
echo "Make sure to mark the partition you want to use for the raid disk as type “Linux Filesystem”."
echo "Also make it the same size as /dev/$CurPart to avoid errors"
cfdisk "/dev/$NewDrive"

echo "Encrypt the new partition"
cryptsetup luksFormat "/dev/$NewPart"

echo "Open the encrypted partition"
cryptsetup luksOpen "/dev/$NewPart" "$NewCryptName"

echo "Add the key to the partition"
cryptsetup luksAddKey "/dev/$NewPart" "$ExtraMountPath/$DriveKeyName"

echo "Add the new partition to the root btrfs file system"
btrfs device add "/dev/mapper/$NewCryptName" "$CurMount"

echo "Convert to RAID1"
btrfs balance start -dconvert=raid1 -mconvert=raid1 "$CurMount"

echo "Confirm both disks are in use"
btrfs filesystem usage "$CurMount"

echo "--------------------Booting script to load encrypted drives-------------------"
echo "Get the UUID of the second btrfs volume"
Drive2_UUID=$(lsblk -o UUID -d "/dev/$NewPart" | tail -n1)

echo "Create a script to open your second luks volumes before mounting the partition"
echo "Note: In some scenarios this may need to go into “scripts/local-premount” instead of “scripts/local-bottom”"
cat <<EOF > "$CurMount/etc/initramfs-tools/scripts/local-bottom/unlock_drive2"
#!/bin/sh
PREREQ=""

prereqs()
{
    echo "\$PREREQ"
}

case "\$1" in
    prereqs)
        prereqs
        exit 0
        ;;
esac

. /scripts/functions
cryptroot-unlock
vgchange -ay "$CurCryptVol"
mkdir -p /mnt/keyfile
mount "/dev/$CurCryptVol/$CurCryptExtra" /mnt/keyfile
cryptsetup luksOpen /dev/disk/by-uuid/$Drive2_UUID "$NewCryptName" "--key-file=/mnt/keyfile/$DriveKeyName"
umount /mnt/keyfile
rmdir /mnt/keyfile

mount -t btrfs -o "subvol=$CurCryptRootSubVol" "/dev/$CurCryptVol/$CurCryptRoot" /root

#If you are weird like me and /usr is stored elsewhere, here is where you would need to mount it.
#It cannot be done through your fstab in this setup.
#mount --bind /root/sub/sys/usr /root/usr

mount --bind /dev /root/dev
mount --bind /proc /root/proc
mount --bind /sys /root/sys
EOF

chmod 755 "$CurMount/etc/initramfs-tools/scripts/local-bottom/unlock_drive2"

echo "--------------------Setup booting from the root file system-------------------"
echo "Prepare a chroot environment"
for i in dev dev/pts proc sys run tmp; do
  mount -o bind /$i "$CurMount/$i"
done

echo "Run commands in the chroot environment to update initramfs and grub"
chroot "$CurMount" <<EOF
echo "Mount the other partitions (specifically for “boot” and “boot/efi”)"
mount -a

echo "Update initramfs and grub"
update-initramfs -u -k all
update-grub
EOF

echo "-----------------------------------Finish up----------------------------------"
echo "Reboot and pray"
reboot

Hardware performance speed tests

So I got a new computer back in April and have finally gotten around to doing some speed tests to see how different applications and settings affect performance/harddrive read speed.


The following is the (relevant) computer hardware configuration:
  • Motherboard: MSI Z87-GD65
  • CPU: Intel Core i7-4770K Haswell 3.5GHz
  • GPU: GIGABYTE GV-N770OC-4GD GeForce GTX 770 4GB
  • RAM: Crucial Ballistix Tactical 2*8GB
  • 2*Solid state hard drives (SDD): Crucial M500 480GB SATA 2.5" 7mm
  • 7200RPM hard drive (HDD): Seagate Barracuda 3TB ST3000DM001
  • Power Supply: RAIDMAX HYBRID 2 RX-730SS 730W
  • CPU Water Cooler: CORSAIR H100i
  • Case Fans: 2*Cooler Master MegaFlow 200, 200mm case fan

Test setup:

I started with a completely clean install of Windows 7 Ultimate N x64 to gather these numbers.

The first column is the boot time, from the time the start of the "Starting Windows" animation shows to when the user login screen shows up, so the BIOS is not included. I used a stopwatch to get these boot numbers (in seconds), so they are not particularly accurate.

The second and third columns are the time (in seconds) to run a "time md5sum" on cygwin64 on a 1.39GB file (1,503,196,839 bytes), on the solid state (SDD) and 7200RPM (HDD) drives respectively. They are taken immediately after boot so caching and other applications using resources are not variables. I generally did not worry about running the tests multiple times and taking lowest case numbers. The shown milliseconds fluctuations are within margin of error for software measurements due to context switches.


Results:

Boot times are affected between multiple steps, as seen below, but not too bad. The only thing that affected the MD5sum was adding the hardware mirror raid on the SSDs, which dropped the time of the md5 by half. So overall, antivirus and system encryption did not have any noticeable affect on the computer's performance (at least regarding IO on a single file and number crunching).


Numbers:
What was added Boot SSD HDD Notes
Initial installation 4 - -
NIC Drivers and Cygwin 7 4.664 8.393 I'm not sure why the boot time jump so much at this point. The initial number might have been a fluke.
All Windows updates + drivers + 6 monitors 14 4.618 8.393 The boot time jumped up a lot due to having to load all the monitors
Raid 1 mirror[Windows] on SSDs + no page file 17 4.618 8.393 This was removed once I realized Truecrypt could not be used on a dynamic disk (Windows software) RAID
Raid 1 mirror[hardware] on SSDs + no page file 17 2.246 8.408
Truecrypt System Volume Encryption (SSD Raid Only) 17-18 2.278 8.424
Antivirus 18 2.324 8.408 Kaspersky 2014