Archive for June, 2007

LVM Recovery Tale

June 30th, 2007 1 comment
LVM recovery tale quoted from Gnomedia Codeworks:

Over the weekend I had the worrying experience of losing my LVM settings and potentially all my data! A quick search on the web showed a confusing set of information, much of it for older versions of LVM and therefore rather suspect.

Well, I recovered all my data and it was really quite simple, so I’ve written up what I did in the hope that someone else, in a similar situation, will find it useful. It’s a scary thing, losing the whole hard disk and knowing that, in reality, its all there.

First the situation

I’ve got a small /boot partition as ext2, and a larger one for the root directory (2Gb, also ext2). The rest of the hard disks (nearly 120Gb) are assigned to a volume group called, descriptively, system (which is SuSE’s idea of a default name).
More accurately, they were supposed to be assigned to it. At first I had just added 60Gb to the volume group, it was my first use of LVM and I was hedging my bets. After 6 months of trouble free operation I decided to add another 60Gb of disk, which I did 3 months ago. Except that, although the physical and volume group managers all agreed that the volume group had 120Gb, the logical volume manager insisted that there was only 60Gb. I’d used Yast2 to create and add the volumes.
I tried every combination of commands I could think of to get the logical volume manager to recognize the additional space but it wouldn’t.
At the time, I was busy, so forgot about it, then last week I realised that I wanted to use the space so settled down to do something about it.

The problem

So, it seemed the best solution would be to remove the second partition that I had added (/dev/hdd1) from the physical volume manager and then add it back.
It wasn’t recognised so wouldn’t be missed, right?
pvremove /dev/hdd1 removed the label from /dev/hdd1 but also from /dev/hda7 (which was the original partition and full of data).
pvscan and pvs reported no physical volumes on the disk.
vgscan and vgs couldn’t find any volume groups.
lvscan and lvs were non-starters obviously.

The rather surreal thing was, the whole system kept on running quite nicely, X Server and KDE desktop and all, but I knew that as soon as I rebooted the system would be toast.

First I tried adding the partition back to the volume group system, but the system couldn’t find the ‘system’ group. I tried creating the physical volume again (pvcreate) but that told me that the volume already existed. It became clear that I would need to reboot and hope that the system sorted itself out, flushed the disks, resynced, whatever.

The solution

After rebooting the system wouldn’t come up, which is kind of what I had expected so I had to reboot from the SuSE Rescue disks. So now I had to think about how to recreate the physical volumes, volume group and logical volume and do it with the data intact. (I have daily backups but the thought of restoring the whole system, applications and data, was not too exciting, especially as I knew all the data was there and intact. With a ‘regular’ hard disk partition that had got lost I could scan the disk for potential disk partitions and restore them. But that wouldn’t work with LVM.

On a search through various sites, I found one that mentioned the importance of saving a copy of the volume group parameters to a file using vgcfgbackup. This file could then be used to restore the parameters later, assuming that the underlying physical structure hadn’t changed. Well, the physical layout hadn’t changed but unfortunately I hadn’t created a backup of the volume group parameters (the ‘descriptor area’ to use the technical term) so that didn’t seem to hopeful. I poked around in the /etc directory (I still had the ‘/’ partition remember, as that was on its own ext2 partition) and noticed that there was a /etc/lvm/backup/ directory and a /etc/lvm/archive/ directory. Further investigation and I found that these are automatically created by LVM whenever changes are made to the system.

Unfortunately, all the messing around I had done had created a non-working version of the system file and the archive files didn’t seem to be recent enough. But, I remembered that I had a backup of the system files (going back 6 months in fact) and so I dug out a copy of the /etc/lvm/backup/sysem file and used that.

Here is what I did: First find out the old UID’s of the partitions, this is in the /etc/lvm/backup/system file. They are quite long; make sure you get the UID for the physical volumes.
$pvcreate -u sdSD-2343-SD939-adIda2 /dev/hda6
$pvcreate -u dk33kd-929293nd-adfja298a /dev/hdd1
$vgcreate -v system /dev/hda7 /dev/hdd1
$vgcfgrestore -f /etc/lvm/backup/system system

and lo!, all data present and correct!

In fact, I just rebooted the system and was back where I had started with the additional benefit of an extra 60Gb of disk space, because now I had the extra partition properly included.

[Note: in the lines using pvcreate… above I could have used:
$ pvcreate -restorefile /etc/lvm/backup/system
to automatically find the ID’s but I hadn’t realized that at the time. Without the UID’s then the vgcfgrestore will not find the physical volumes that it needs to recreate the volume group.]

The lesson

Don’t panic!
Keep a safe copy of your /etc/lvm/ files!
Make sure that you have a Rescue disk that understands the LVM system!

Apart from the above disaster, which seems to have sorted itself out very easily, I have had no trouble with the LVM system. At first I was worried that if there was a failure it would lose everything. There is something very comforting about a simple ext2 (or FAT) partition in that I know it can just be hacked at the bit level and rebuilt. Something like LVM, which is logical volumes on top of volume groups on top of physical volumes is impossible to rebuild ‘by hand’ so I’m learning to trust technology a bit.

End quote

Categories: Uncategorized Tags:

Mount from Image with Multiple Partitions

June 30th, 2007 No comments

Quoted from Jeffrey Fernandez:

If you have played around with virtualisation technologies, specifically Xen, then you may have experienced creating a disk image. To create a disk image on Linux, I execute the following command:

dd if=/dev/zero of=OpenSuse-10.2.img bs=1024k count=0 seek=5000

This would have created for me an image which is approximately 5GB in size. And then I format it to an ext3 file-system with:

mkfs.ext3 OpenSuse-10.2.img

If I now run a file command against this image, it should tell me what type of file-system it is.

file OpenSuse-10.2.imgOpenSuse-10.2.img: Linux rev 1.0 ext3 filesystem data (large files)

Thats all fine when there is a single partition for your disk image. Recently I happened to encounter a problem whereby there was more than a single partition on an image.

Enter “Xen Virtual Machine Management”

I recently have been playing with the “Xen Virtual Machine Management” which is available in OpenSuse 10.2, and it totally rocks. It automates the process of creating you the configuration file for your Xen VM along with the disk images and the final VM without having you to type anything on the command line. Not that I don’t like the command line but I was just pointing our that OpenSuse developers have done a great job with the Xen Manager.

After helping you create the necessary configuration, you can start installing your VM through a little xterm window. This is not one of those “bootstrap” scripts working in the background, but a full OpenSuse Operating System install, just like how you would do when you install an Operating System on your desktop, except this is in text mode.

Now because you are creating a full Operating system installation for your VM, you can basically go through the installation process and create your partitions for your different mount points, select the required packages to be installed and so on. So in the end you end up with a disk image with multiple partitions in it. This is where I encountered the problem. Normally when you have a single partition as show in our earlier example, you can mount the image through a loop-back device like shown below:

mount -o loop OpenSuse-10.2.img mnt

But since the image which gets created during the installation process of the VM, has multiple partittions, a simple mount won’t work. This is what happens when I try to mount an image (hda is the image) which has multiple partitions:

mount -o loop hda mntmount: you must specify the filesystem type

And if I run the same file command on this image file, the following is the output:

hda: x86 boot sector; partition 1: ID=0x83, active, starthead 0, startsector 1060290, 16787925 sectors; partition 2: ID=0x83, starthead 254, startsector 17848215, 3116610 sectors

As you can see within the output, its lists two partitions for the image. We can look at the detailed listing by executing:

fdisk -lu hda

Which outputs:

You must set cylinders.
You can do this from the extra functions menu.

Disk hda: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot      Start         End      Blocks   Id  System
  hda1   *     1060290    17848214     8393962+  83  Linux
Partition 1 has different physical/logical endings:
     phys=(1023, 254, 63) logical=(1110, 254, 63)
  hda2        17848215    20964824     1558305   83  Linux
Partition 2 has different physical/logical beginnings (non-Linux?):
     phys=(1023, 254, 63) logical=(1111, 0, 1)
Partition 2 has different physical/logical endings:
     phys=(1023, 254, 63) logical=(1304, 254, 63)

The “u” flag within the fdisk command gives the partition tables sizes in sectors instead of cylinders. This is important for us as we will be using that information for some calculation. Also important to us is the “Bytes/sector” values

Calculate the Offset in “Bytes”

Now in order to mount the two partitions from the image file, we need to know the starting Byte from which each partition starts. To calculate this offset, we mulitply the (start_sector * sector_byte_size). So for the Partition 1 we have (1060290 * 512) = 542868480 and for Partition 2 we have (17848215 * 512) = 9138286080. I pretty well know that the Partition 1 was the root partition and Partition 2 was the Home partition so I will mount them into appropriate folder names.

Now we can use those “Start Byte” values to mount those partitions in two seperate locations:

mount -o loop,offset=542868480 hda rootmount/
mount -o loop,offset=9138286080 hda homemount/

Categories: Uncategorized Tags:

Disk Cloning and Backup

June 29th, 2007 No comments

The old standby to clone a partition:
dd if=/dev/hda1 of=/output/file
Apparently a newer tool, dcfldd, does a better job.

Much smaller though not an “image” copy:
tar zcvpf backup.tgz --exclude=dev --exclude=sys --exclude=proc --directory=/ .
And if you don’t want to create the intermediate backup file:
tar zcp --exclude=dev --exclude=sys --exclude=proc --exclude=/this-folder --directory=/ . | tar zx
Note: /sys, /proc and /dev are rebuilt as linux boots and in the no-backup version, you must exclude the destination folder (or the copy never ends).

Categories: Uncategorized Tags:

Migrate CentOS-4.4 to Xen

June 23rd, 2007 No comments

Boot Knoppix
In a root shell window:

vgchange -ay							# activate LVM

e2fsck -f /dev/vg00/lv00

resize2fs /dev/vg00/lv00 9G					# resize ext3 fs

lvreduce /dev/vg00/lv00 -L9G					# resize logical vol

Reboot host OS

mount /mnt/share				# mount nfs share

lvcreate -s -n backup -L9G /dev/vg00/lv00			# create lvm snapshot

mount /dev/vg00/backup /mnt/backup

tar zcvpf /mnt/share/app2-backup.tgz -directory=/mnt/backup .	# tar lv backup to nfs

On Xen host

lvcreate -n app2.swap -L2G vg0

mkswap /dev/vg0/app2.swap

lvcreate -n app2 -L9G vg0

mkfs.ext3 /dev/vg0/app2

mount /dev/vg0/app2 /mnt/app2

cd /mnt/app2

tar zxvf /share/app2.tgz

In the VM do the following

Copy /boot/* from another CentOS-4.4 VM

rename/remove /etc/sysconfig/hwconf

change entries in fstab to use scsi (sda) instead of lvm

To prepare /etc/xen config files

Copy and modify a working config, or read the Xen docs

To reboot VM automatically

Create a link to the appropriate /etc/xen/xxx.conf in /etc/xen/auto/
Categories: Uncategorized Tags:

A Mac-like Rails Environment

June 23rd, 2007 No comments

While I use eclipse and the single-click Ruby with Rails under Windows this may be more to your liking:

Categories: Uncategorized Tags:

Voxeo VoiceXML Tools Suggestions

June 14th, 2007 No comments

“Several customers have reported success with using the Voice Tools Project for Eclipse which includes the OpenMethods Visual Designer. Eclipse VTP can be found @

“Other than that, many of our customers are hand-coding their VoiceXML using a regular XML editor. Altova’s XML editor is top notch and provides some really good developer features such as auto-complete and document templates. “

“If you need help getting started, you might want to check out some of our RocketSoruce applications. These are fully-functional VoiceXML application complete with back-end database integration. You can get more information about RocketSource @

“As always, we are here to support you throughout your development process. If you have any additional questions please let us know. “

Categories: Uncategorized Tags: