pk stuff

How to install btrfs in RAID1, with bcache, on Ubuntu.

To avoid problems, use the most recent kernel you can. I did all this on Ubuntu 14.04 LTS but with Linux kernel 3.17.
Earlier kernels (btrfs versions) reported all sorts of unpleasantness in dmesg. 3.17 works.

First, install bcache tools

sudo add-apt-repository ppa:g2p/storage
sudo apt-get update
sudo apt-get install bcache-tools

I tried a bunch of recipes that failed. Several suggested it was possible to migrate existing btrfs volumes to use bcache.
Nothing I found came close to working without completely rebuilding the btrfs volumes.

MAKE A BACKUP – THIS WILL ERASE EVERYTHING ON YOUR HARD DRIVE

So I did an 800+GB backup and started from scratch. This recipe worked:

My setup:

2 x 3TB Seagate HDD backing store – /dev/sdc & /dev/sdd
40GB partition on Samsung SSD cache – /dev/sda6

Create bcache devices (-B for backing devices HDD; -C for cache device SSD):

$ make-bcache –wipe-bcache –writeback -B /dev/sd[cd] -C /dev/sda6

Build RAID1 btrfs on bcache devices:

$ mkfs.btrfs -f -d raid1 -m raid1 /dev/bcache0 /dev/bcache1

Get device id needed for /etc/fstab:

$ blkid /dev/bcache0
/dev/bcache0: UUID=”eccb74e8-6cae-850a-4eac-d217b11e21a4″ UUID_SUB=”9da2d9dc-6636-9de4-4293-169894f750ba” TYPE=”btrfs”

Updating /etc/fstab:

UID=eccb74e8-6cae-850a-4eac-d217b11e21a4 /big        btrfs   defaults,subvolid=0  0   0

If necessary, regenerate initramfs:

$ update-initramfs -u -k all

And voila, it worked!

 

Handy links:

https://bcache.evilpiepirate.org/

https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices

https://bbs.archlinux.org/viewtopic.php?id=178230 & https://pastebin.com/ZwwZMd40

 

Troubleshooting steps for when things go wrong (and they likely will):

Wipe out file system id on cache device:

$ wipefs -a /dev/sda6

or on backing device:

$ wipefs -a /dev/sdd

More/different ways to wipe out file system info:

$ dd if=/dev/zero count=1 bs=1024 seek=1 of=/dev/sda6
$ dd if=/dev/zero count=1024 bs=1024 seek=1 of=/dev/sda6

I had to use gdisk to wipe GPT table from backing devices. ‘x’ for advanced, ‘z’ to zap partition table:

$ gdisk

Lots of reboots to synchronize bcache. There’s probably a better way, but this worked:

$ reboot

Display a bunch of useful block dev info:

$ lsblk -o NAME,MAJ:MIN,RM,SIZE,TYPE,FSTYPE,MOUNTPOINT,UUID

NAME MAJ:MIN RM SIZE TYPE FSTYPE MOUNTPOINT UUID
sda 8:0 0 232.9G disk
├─sda1 8:1 0 177.9G part btrfs 71cbb2af-7bd3-9d04-4800-1b1841c88ce1
├─sda2 8:2 0 1K part
├─sda5 8:5 0 15.5G part swap [SWAP] ed662fe6-0f76-9a1e-418c-ad26d15b9234
└─sda6 8:6 0 39.4G part
├─bcache0 251:0 0 2.7T disk btrfs eccb74e8-6cae-850a-4eac-d217b11e21a4
└─bcache1 251:1 0 2.7T disk btrfs eccb74e8-6cae-850a-4eac-d217b11e21a4
sdb 8:16 0 465.8G disk
sdc 8:32 0 2.7T disk
└─bcache0 251:0 0 2.7T disk btrfs eccb74e8-6cae-850a-4eac-d217b11e21a4
sdd 8:48 0 2.7T disk
└─bcache1 251:1 0 2.7T disk btrfs eccb74e8-6cae-850a-4eac-d217b11e21a4
sde 8:64 0 2.7T disk
├─sde1 8:65 0 1T part ext4 /mnt/ext4 693ac7fb-fc0c-b76d-4215-fe5c10df5a1a
└─sde2 8:66 0 1T part xfs 72f87cdb-8149-973a-4863-ca3c4063cba9

From Docker’s site:

Ubuntu Trusty 14.04 (LTS) (64-bit)

Ubuntu Trusty comes with a 3.13.0 Linux kernel, and a docker.io package which installs Docker 0.9.1 and all its prerequisites from Ubuntu’s repository.

Note: Ubuntu (and Debian) contain a much older KDE3/GNOME2 package called docker, so the package and the executable are called docker.io.

Installation

To install the latest Ubuntu package (may not be the latest Docker release):

$ sudo apt-get update
$ sudo apt-get install docker.io
$ sudo ln -sf /usr/bin/docker.io /usr/local/bin/docker
$ sudo sed -i '$acomplete -F _docker docker'/etc/bash_completion.d/docker.io
$ source /etc/bash_completion.d/docker.io

If you’d like to try the latest version of Docker:

First, check that your APT system can deal with https URLs: the file /usr/lib/apt/methods/httpsshould exist. If it doesn’t, you need to install the package apt-transport-https.

[-e /usr/lib/apt/methods/https ]||{
  apt-get update
  apt-get install apt-transport-https
}

Then, add the Docker repository key to your local keychain.

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

Add the Docker repository to your apt sources list, update and install the lxc-docker package.

You may receive a warning that the package isn’t trusted. Answer yes to continue installation.

$ sudo sh -c "echo deb https://get.docker.com/ubuntu docker main
> /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker

Note:

There is also a simple curl script available to help with this process.

$ curl -sSL https://get.docker.com/ubuntu/ | sudo sh

To verify that everything has worked as expected:

$ sudo docker run -i -t ubuntu /bin/bash

Which should download the ubuntu image, and then start bash in a container.

Docker inspect provides all sorts of useful information. Use with no format to get all values, then use –format to filter.

This example gets the complete container Id:

docker inspect –format='{{.Id}}’ $SHORT_CONTAINER_ID

 

 

While packaging minidlna in a Docker container, it became clear that IP Broadcasts (required by minidlna) are not bridged by default in Ubuntu 14.04.

Adding the following file (10-fix-bridge.conf) with the following to /etc/sysctl.d/ fixes this:

# allows broadcasts to reach Docker containers
#net.bridge.bridge-nf-call-arptables = 1
#net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 0

First, my ISP doesn’t support IPv6 so Linux works but DNS name lookups take forever. This helps.

Create a file named 10-no-ipv6.conf in /etc/sysctl.d/ with the following:

# No IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

 

 

Sometime I’m in a hurry and delete a VM’s disk image before removing it from libvirt/virsh, which leaves a lingering, dead reference to that VM.
The following fixes this:

sudo virsh –connect qemu:///system undefine vm_to_be_removed

Replace vm_to_be_removed with your VM name.

I had to make a few changes to /etc/default/grub to get Ubuntu 14.04 to boot with a working console. Initially, my monitor just went dark.

Changes with reasons:

GRUB_HIDDEN_TIMEOUT: if you want to see the Grub menu, this must be commented out.

GRUB_CMDLINE_LINUX: allows passing parameters to the linux startup command. Newer kernels (>3.5? or so) reinitialize the graphics card. My AMD Radeon something-or-other didn’t like this at all. nomodeset forces linux to use/keep BIOS graphics settings.

GRUB_GFXMODE: select resolution for Grub graphical starup. 1280×1024 works on my aging ViewSonic. This setting is for the Grub menu.

GRUB_GFXPAYLOAD_LINUX: specifying keep, causes Grub to keep using the previous resolution (from GRUB_GFXMODE) for the rest of the boot. 

NB: after making any changes, remember to update-grub (or nothing will happen).

Contents of my /etc/default/grub after making required changes:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=”splash”
GRUB_CMDLINE_LINUX=”nomodeset”
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD …)
#GRUB_BADRAM=”0x01234567,0xfefefefe,0x89abcdef,0xefefefef”

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo’
#GRUB_GFXMODE=640×480
GRUB_GFXMODE=1280×1024
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you don’t want GRUB to pass “root=UUID=xxx” parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY=”true”

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE=”480 440 1″

My ISP (ding, dong) is stupid, so I need to disable IPv6.

sudo vim /etc/sysctl.conf

Add these four lines:

# No IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Adding existing logical volumes to the list of virtual disks is non-obvious.
First add the VG to Proxmox using the “Storage|Add” UI. Once done, adding existing LV to Proxmox must be done via LVM command line tools.
A logical volume needs to have the correct naming, and (surprise!) an LVM tag set. For the correct tag (and LV name) format, use the UI to create a sample volume (VM|Hardware|Add|Hard Disk).
Note: LV names must fit a certain pattern. However, names like this seem to work: vm-110-timemachine.
The example below shows a pre-existing LV (vg1/vm-110-disk-2) without the “pve-vm-110” tag Proxmox requires.
Subsequent steps add the tag, after which the volume appears in Proxmox.

# lvs -o vg_name,lv_name,lv_tags
VG     LV              LV Tags
pve    data
pve    root
pve    swap
vg0    vm-110-disk-1   pve-vm-110
vg0    vm-110-disk-2   pve-vm-110
vg1    vm-110-disk-1   pve-vm-110
vg1    vm-110-disk-2


To see LV tags:

# lvs @pve-vm-110
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
vm-110-disk-1 vg0 -wi-ao--- 500.00g
vm-110-disk-2 vg0 -wi-ao--- 750.00g
vm-110-disk-1 vg1 -wi-ao--- 750.00g


To add missing tags:

# lvchange --addtag pve-vm-110 /dev/vg1/vm-110-disk-2


Refresh the Proxmox Storage UI to see the new LV.

There are plenty of broken jsfiddles for this, so to save time, here is some code that works: boostrap-scrollspy

or preview (or download) here