pk stuff

How to install btrfs in RAID1, with bcache, on Ubuntu.

To avoid problems, use the most recent kernel you can. I did all this on Ubuntu 14.04 LTS but with Linux kernel 3.17.
Earlier kernels (btrfs versions) reported all sorts of unpleasantness in dmesg. 3.17 works.

First, install bcache tools

sudo add-apt-repository ppa:g2p/storage
sudo apt-get update
sudo apt-get install bcache-tools

I tried a bunch of recipes that failed. Several suggested it was possible to migrate existing btrfs volumes to use bcache.
Nothing I found came close to working without completely rebuilding the btrfs volumes.

MAKE A BACKUP – THIS WILL ERASE EVERYTHING ON YOUR HARD DRIVE

So I did an 800+GB backup and started from scratch. This recipe worked:

My setup:

2 x 3TB Seagate HDD backing store – /dev/sdc & /dev/sdd
40GB partition on Samsung SSD cache – /dev/sda6

Create bcache devices (-B for backing devices HDD; -C for cache device SSD):

$ make-bcache –wipe-bcache –writeback -B /dev/sd[cd] -C /dev/sda6

Build RAID1 btrfs on bcache devices:

$ mkfs.btrfs -f -d raid1 -m raid1 /dev/bcache0 /dev/bcache1

Get device id needed for /etc/fstab:

$ blkid /dev/bcache0
/dev/bcache0: UUID=”eccb74e8-6cae-850a-4eac-d217b11e21a4″ UUID_SUB=”9da2d9dc-6636-9de4-4293-169894f750ba” TYPE=”btrfs”

Updating /etc/fstab:

UID=eccb74e8-6cae-850a-4eac-d217b11e21a4 /big        btrfs   defaults,subvolid=0  0   0

If necessary, regenerate initramfs:

$ update-initramfs -u -k all

And voila, it worked!

 

Handy links:

https://bcache.evilpiepirate.org/

https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices

https://bbs.archlinux.org/viewtopic.php?id=178230 & https://pastebin.com/ZwwZMd40

 

Troubleshooting steps for when things go wrong (and they likely will):

Wipe out file system id on cache device:

$ wipefs -a /dev/sda6

or on backing device:

$ wipefs -a /dev/sdd

More/different ways to wipe out file system info:

$ dd if=/dev/zero count=1 bs=1024 seek=1 of=/dev/sda6
$ dd if=/dev/zero count=1024 bs=1024 seek=1 of=/dev/sda6

I had to use gdisk to wipe GPT table from backing devices. ‘x’ for advanced, ‘z’ to zap partition table:

$ gdisk

Lots of reboots to synchronize bcache. There’s probably a better way, but this worked:

$ reboot

Display a bunch of useful block dev info:

$ lsblk -o NAME,MAJ:MIN,RM,SIZE,TYPE,FSTYPE,MOUNTPOINT,UUID

NAME MAJ:MIN RM SIZE TYPE FSTYPE MOUNTPOINT UUID
sda 8:0 0 232.9G disk
├─sda1 8:1 0 177.9G part btrfs 71cbb2af-7bd3-9d04-4800-1b1841c88ce1
├─sda2 8:2 0 1K part
├─sda5 8:5 0 15.5G part swap [SWAP] ed662fe6-0f76-9a1e-418c-ad26d15b9234
└─sda6 8:6 0 39.4G part
├─bcache0 251:0 0 2.7T disk btrfs eccb74e8-6cae-850a-4eac-d217b11e21a4
└─bcache1 251:1 0 2.7T disk btrfs eccb74e8-6cae-850a-4eac-d217b11e21a4
sdb 8:16 0 465.8G disk
sdc 8:32 0 2.7T disk
└─bcache0 251:0 0 2.7T disk btrfs eccb74e8-6cae-850a-4eac-d217b11e21a4
sdd 8:48 0 2.7T disk
└─bcache1 251:1 0 2.7T disk btrfs eccb74e8-6cae-850a-4eac-d217b11e21a4
sde 8:64 0 2.7T disk
├─sde1 8:65 0 1T part ext4 /mnt/ext4 693ac7fb-fc0c-b76d-4215-fe5c10df5a1a
└─sde2 8:66 0 1T part xfs 72f87cdb-8149-973a-4863-ca3c4063cba9

The SystemRescueCd has all sorts of useful tools to restore ailing Linux systems.
Check it out at: https://www.sysresccd.org

I recently replaced the RAID 1 drives in my CentOS 5 box and had to reinstall the grub MBR. As the drives are a couple of TB each, I thought I’d try GPT instead of the old MBR partitioning, since soon all drives will exceed MBR’s 2TB limit. Luckily GPT is baked into RH/CentOS kernels (unlike many others).
Restoring the MBR was trickier than I expected and I made a couple of mistakes along the way – wish I had found the following description sooner:
https://idolinux.blogspot.com/2009/07/reinstall-grub-bootloader-on-md0.html

In a nutshell run grub interactively and do the following:

# grub
grub> root (hd0,0)        
root (hd0,0)
 Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
setup (hd0)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd0)"...  16 sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.
grub> quit
# reboot