Providing multiple Linux images

Author: L.S.Lowe. File: imageinstall. This update: 20111027. Part of Guide to the Local System.

The need for particular operating systems can come and go during the life of a PC or server, particularly for systems which are on a short update cycle like Fedora. To cope with this on our cluster of servers and desktop PCs, the base system is Fedora 12 or Fedora 15, but there are also available complete SL4 and SL5 sub-systems (SL is Scientific Linux, similar to CentOS and based on Red Hat Enterprise Linux). There's the opportunity to extend that to have a bigger set of different sub-systems, including 32-bit and 64-bit versions of the same system, if and when we want them.

The images can be separately bootable, in the old time-honoured dual-boot or multiple-boot fashion, or they can all be used from one of the systems, using chroot to give application virtualisation, as described in another page, or they could be used in a fully virtualised environment (though that precludes efficient sharing of other file-systems on the hard drive such as /tmp and /var/tmp/ and swap, and is not described here).

Partitioning the hard drive

For me, the usual hard drive is 500 GB at the time of writing (or in the case of a server, two smaller disks), and on arrival I partition this to have several small partitions including the OEM vendor diagnostics partition, a super-grub boot partition (see later section), a tiny partition for boot use by Windows if needed (see Windows section), a number of system partitions of 24 GB for both present and future installed systems, a swap partition to be used by any system, and a large (320GB) /share partition. The fdisk utility shows (Start and End in old-fashioned cylinders):

      Device Boot      Start         End      Blocks   Id  System
   /dev/sda1               1          11       88326   de  Dell Utility
   /dev/sda2              12          30      152617+  83  Linux
   /dev/sda3              31          50      160650    c  W95 FAT32 (LBA)
   /dev/sda4              51       60801   487982407+   5  Extended
   /dev/sda5              51        3050    24097468+  83  Linux
   /dev/sda6            3051        6050    24097468+  83  Linux
   /dev/sda7            6051        9050    24097468+  83  Linux
   /dev/sda8   *        9051       12050    24097468+  83  Linux
   /dev/sda9           12051       15050    24097468+  83  Linux
   /dev/sda10          15051       18050    24097468+  83  Linux
   /dev/sda11          18051       21050    24097468+  82  Linux swap / Solaris
   /dev/sda12          21051       60801   319299876   83  Linux

As I have a cluster of these things, I use an installation kickstart method and the partitioning of the hard drive is done as a %pre pre-installation script in the kickstart file, using sfdisk, if that script detects that the partitioning scheme hasn't been done yet. It's much easier doing that than trying to force kickstart anaconda to allocate the partitions itself in exactly the way you want it.

Installing the separate system images

For a single PC or server, the best way of providing the different systems is the obvious one: install each system using a normal linux installation technique, with a separate partition for each installation. I use a kickstart file and the relevant bits of that file for a particular installation were:

bootloader --location=partition --md5pass=$1$someMD5encryption
part / --fstype=ext3 --onpart=sda8
part swap --onpart=sda11

For a manual interactive installation, the equivalent would be to use a single chosen partition for the root (/) directory, choose the common swap partition for swap, and when asked to specify the location of the boot loader, specify it to be in the partition rather than in the MBR. The use of the latter for all the images you install implies the use of some other partition as the real system boot partition: see next section.

Aside: a reference example of installation-time grub command

As can be seen in an installation text window (ctl/alt/F5), RedHat Enterprise 6 anaconda uses the following grub command sequence to install GRUB 0.97 in the first sector of a partition, in this case sda6:

grub
grub> root (hd0,5)
grub> install --stage2=/boot/grub/stage2 /boot/grub/stage1 d (hd0,5) /boot/grub/stage2 p (hd0,5)/boot/grub/grub.conf

whereas for GRUB installed in the MBR, it's as follows:

grub
grub> root (hd0,5)
grub> install --stage2=/boot/grub/stage2 /boot/grub/stage1 d (hd0) /boot/grub/stage2 p (hd0,5)/boot/grub/grub.conf

Regarding the --stage2 option, the 0.97 grub manual says: "You must specify the option --stage2 in the grub shell, if you cannot unmount the filesystem where your stage2 file resides. The argument should be the file name in your operating system".

Setting up a super-grub / mastergrub

I have a super-grub or mastergrub partition of a few cylinders near the start of the hard disk (see Partitioning section). This linux-formatted partition contains a grub directory, and the grub.conf file has simple chain-loads to each of the available systems. Thus this super-grub doesn't need to know about the kernels or ramdisks of each system: each image partition contains its own grub loader in its sector 1, and the super-grub passes control to this. (Chain-loading is often used within grub to boot a system like Windows which doesn't know about grub, but it can be used equally to boot another grub bootloader).

This is the sane alternative to copying the kernels and ramdisks from multiple systems into the boot directory of one of the systems, as is often the advice on the web for multiple boot systems, and so avoids unnecessary complications, not least when the system on the chosen partition becomes obsolete and is to be replaced.

Here's a script which creates such a super-grub / mastergrub partition on sda2; for me this script runs as a %post post-installation script when installing the first-installed image on a particular PC or server. It installs a grub bootloader in the hard disk's MBR which will use the stage2 and grub.conf in the small super-grub partition. The grub.conf file contains chain-loads to various installed systems. Your requirements will certainly vary. (For me, I chose sda2 because sda1 often is a vendor-dependent diagnostic area. For most of my PCs, I don't list all the images in the grub.conf file, but instead provide a chroot technique to access the different images).

Note that this advice is relevant for Grub 0.97, and probably not Grub2 which I haven't checked out yet.

# L.S.Lowe: makesupergrub.sh, can be used as a %post kickstart script
bootdev=/dev/sda2
bootgrub='(hd0,1)' # the grub equivalent of /dev/sda2
bootmnt=/tmp/booty.bham
tune2fs -l $bootdev && exit 0
mkfs -t ext3 $bootdev || exit
echo Supergrub: made a file-system on boot device $bootdev
mkdir $bootmnt || exit 
mount $bootdev $bootmnt || exit
chmod 700 $bootmnt
mkdir -m 700 $bootmnt/grub || exit
cp -p /usr/share/grub/*/stage? $bootmnt/grub/ || exit
url=http://mirrors.kernel.org/fedora/releases/14/Fedora/x86_64/os/isolinux/memtest
wget -q -O $bootmnt/memtest $url 
url=http://www.ep.ph.bham.ac.uk/general/support/image-splash.xpm.gz
wget -q -O $bootmnt/grub/splash.xpm.gz $url 
splashgrub=$bootgrub/grub/splash.xpm.gz
arch=$(uname -m)
bootmd5='$1$SomeMD5encryptedPassword'
cat > $bootmnt/grub/grub.conf <<EOF
# Super grub configuration
timeout 2
default 0
splashimage $splashgrub
password --md5 $bootmd5
title Fedora 15 system $arch
      rootnoverify (hd0,6)
      chainloader +1
title Fedora 12 system $arch
      rootnoverify (hd0,7)
      chainloader +1
title SL5 system $arch
      rootnoverify (hd0,5)
      chainloader +1
title SL6 system $arch
      rootnoverify (hd0,4)
      chainloader +1
title Memory system test
      kernel $bootgrub/memtest
EOF
chmod 600 $bootmnt/grub/grub.conf
umount $bootmnt
rmdir $bootmnt
# grub cmd doesn't need the boot partition to be mounted
echo Supergrub: Calling grub cmd to make a super GRUB for boot
grub --batch --no-floppy <<EOF
root $bootgrub
install /grub/stage1 d (hd0) /grub/stage2 p /grub/grub.conf
EOF
echo Supergrub: Completed special supergrub initialisation

If you're not using kickstart and it's the first system ...

If you're not using kickstart and this is your first system on this PC, then setting up the supergrub during first installation can't be done as an automatic %post script. And if you have chosen to set up the boot loader in the system partition rather than the MBR, then booting would be tricky! Nevertheless you can get round this manually at the final stage of installation. When the installation procedure says it's complete, then type Ctrl-Alt-F2 to change from the installation graphic screen to a shell prompt screen, and copy the grub stage1 from your install partition (say /dev/sda8 in our example) to the MBR, carefully avoiding overwriting the partition table in the MBR by specifying bs=440. You could optionally precede that by keeping a copy of the original MBR, as follows:

         dd if=/dev/sda  of=/mnt/sysimage/root/mbr440.orig count=1 bs=440
         dd if=/dev/sda8 of=/dev/sda count=1 bs=440
         sync

You will then be able to boot directly into the system you have installed. You can later run the makesupergrub script above (as customised for your system) to make the super-grub.

(Note: although in my experience with RedHat systems, a grub-boot-loader written by legacy grub to a system partition first sector is relocatable to the MBR as shown above, a grub-boot-loader written into the MBR does not appear to be relocatable to a partition first sector, for whatever reason: not investigated!).

Setting up a super-grub / mastergrub: alternative method

When installing a multi-boot set of systems on a particular user's laptop, I decided to use the system-supplied grub-install command to create the supergrub, rather than my custom script above, and I decided to locate the grub stuff in a /home filesystem partition, shared by the multiple linux systems on this laptop, rather than having a partition of its own. The idea was that the /home partition would have a longer lifetime than the individual systems on the laptop, and so was a good place to locate the super-grub rather than within an individual system.

The command grub-install --root-directory=/home /dev/sda did the job for me, your experience may vary, and then I created the file /home/boot/grub/grub.conf containing the title, rootnoverify, and chainloader grub commands as required, similar to the custom script above. Note that grub-install is not an available command during installation (for Fedora, anyway), so could not be used in a %post section of a kickstart installation.

Distributing an image to other PCs and servers

Where you have a number of PCs and/or servers, you have the opportunity to install the systems on a separate off-air master PC or server, on different partitions, again using the normal linux installation technique. Then you can distribute that image (using rsync for example) to an unused partition on your cluster of PCs and servers, while those machines are up and running and being used for normal activities. When the user wants to make use of a new image on their PC, they can do this straightaway if you are using a chroot technique (above), or by doing a quick reboot at a convenient time if you are using the old dual-boot or multi-boot approach.

Making a distributed image bootable

If you are only going to use the distributed image via chroot, not boot, then skip this section!

There are several fixups that need to be applied after a distributed image has been synched to a new system, or pre-applied to the image before it's distributed, before the image can be booted:

For file /etc/fstab, any UUIDs need to be corrected for the new machine, or the distributed image's /etc/fstab has to be UUID-free (for example, set this up to use LABEL instead).
For file /boot/grub/grub.conf, the same applies (free of UUIDs).
For file in /etc/ssh, any specific key files should be deleted so that new ssh key files are generated on boot.
For file /etc/sysconfig/network, any specific HOSTNAME should be updated.
For file /etc/sysconfig/network-scripts/ifcfg-xxx, where xxx is the name assigned to the network interface (historically eth0), any HWADDR definition should be removed, and the DEVICE value should reflect the name that the new system will use. This applies for example to Fedora 15 which may use interface names other than eth0 for network interfaces it recognises.
Grub stage1 code may need to be set-up (see below).

One difference between installing a system and copying an image of a system (via rsync say) is that simply copying the files of the image leaves one thing uncopied: the boot loader stage1 code used to boot the system.

Here's a script which puts a grub stage1 in the first sector of a partition to allow that partition to be chainloader-booted (see previous section); this can be run as the final stage of preparing an image on a particular PC. You can do this with the image un-mounted:

#!/bin/sh
# L.S.Lowe: makegrubsector.sh
# Install a grub stage1 in the first sector of the designated partition.
# This is intended to be run after an image has been rsync'd on a partition.
# The image already contains the required kernel and /boot/grub files.
dev=$1
# (Various safety checks for particular systems could be added here)
case "$dev" in
/dev/sda5) gdev='(hd0,4)';; /dev/sda6) gdev='(hd0,5)';;
/dev/sda7) gdev='(hd0,6)';; /dev/sda8) gdev='(hd0,7)';;
*        ) echo Unscripted device $dev in $0; exit 3;;
esac
sync
grub --batch --no-floppy << EOF
root $gdev
install /boot/grub/stage1 d $gdev /boot/grub/stage2 p /boot/grub/grub.conf
EOF

Notes: in the above, we are using the /sbin/grub command from the native system. The above uses the stage1 and stage2 files from the $gdev device (producing the same sector 1 content as if the stage file specifications on the above install line are prefixed with $gdev). If you get difficulties, try refreshing the image's grub stage files first, eg to use the native system versions, try this first: cp -p --backup=numbered /usr/share/grub/*/stage? mountpoint/boot/grub/ . You can do the grub install with the corresponding file-system unmounted. The grub install embeds the name of the configuration file into the stage2 file you've specified. As always with grub, there is no problem with editing (or changing the block offsets of) the grub.conf file, but if you change (the block offsets of) the stage2 file, you need to re-do the above grub install. Copying a system image to another PC does not necessarily mean that it will successfully boot: the system might not support the hardware.

The large share partition and tmp areas

The large /share partition is tailored after the first system installation so that it contains (amongst others) a tmp and var/tmp subdirectory; these subdirectories will be bind-mounted over /tmp and /var/tmp in each image so that all the images benefit from this space, rather than requiring it on a per-image basis. For example, my /etc/fstab files contain lines like:

/share/tmp        /tmp              auto    bind         0 0
/share/var/tmp    /var/tmp          auto    bind         0 0

By making those different areas actually share a single partition, each one can ebb and flow up to the available size, without requiring to use logical volume manager to make them individually adjustable. In my setup the /home file-system is separately-hosted and is shared over the whole cluster, so doesn't need any local hard drive space, but in other circumstances it could make sense to make a /share/home and bind-mount that to /home.

Adding a Windows 7 system on a multi-boot PC

One member of our staff wanted to have Windows 7 as one of the multiple boots. As the PC's original Windows system had been obliterated, apart from the sticky label on the case ☺, this was a matter of adding a Windows system to an all-Linux PC, which is the reverse of the common situation that people often deal with. So for that PC I split the large sda12 partition into three 80 GiB partitions and one 64 GiB partition (sda12-sda15), using fdisk. Partition sda14 was to be for Windows 7 and sda15 for a D drive. I set the partition type for those to partitions to NTFS, still using fdisk. I backed-up my grub MBR sector code by copying it elsewhere:

   dd if=/dev/sda bs=440 count=1 of=/some/where/else

(In fact my /some/where/else was the first sector of my unmounted /dev/sda2 partition, but your setup may be different, so take care! A file in one of the Linux file-systems would have been as good).

Installing was straightforward, as it turned out. Windows 7 is normally installed into a primary partition. But Windows 7 is happy to install into a logical partition if the partition-type is known to Windows (eg 07, 0C) provided there is also a primary partition which also has a partition-type known to Windows, and which has a bit of space to put a bootmgr and boot files on; it's happy to format that partition if it's uninitialised. In my case a spare small 160 MB primary partition had been present from day 1.

So I booted from the Windows 7 installation DVD. It went through the following usual steps: Windows is loading files, Starting Windows, Set localisation, Install now, Setup is starting, EULA, Custom (advanced). Chose sda14 (Windows called it Partition 13) to install Windows on. It then proceeded as normal: copying files, reboots, Setup is installing your computer for first use, set username and password, time/date, Preparing your desktop. A look at the Computer view shows that the C: drive had 59 GiB free out of 80 GiB. The small boot partition didn't show up in that Computer view (which is fine).

The Windows installer had seen that my sda3 of 160 MB was defined as FAT32 (LBA) and was uninitialised, and without prompting had formatted it as NTFS, put a bootmgr file and a boot directory there, changed sda3 partition type from 0C to 07 (NTFS), and marked sda3 as bootable. (For this purpose the installer seemed rightly to disregard primary partitions whose partition type it did not recognise, such as vendor or linux types, and the installer wouldn't go ahead if there were no suitable primary partition). The Windows 7 system itself went to sda14 as explicitly requested.

A normal boot then took me into Windows, because the grub MBR code has been replaced by normal DOS/Windows code. I then rebooted from the F15 installation CD, and restored my grub code (440 bytes) to the MBR without destroying the partition table:

   dd if=/some/where/else bs=440 count=1 of=/dev/sda

(If I hadn't saved it, I would have had to re-create it using the grub incantation of a previous section). I then re-booted and chose my usual Linux system from the grub menu. At leisure, I added the Windows boot partition sda3 to my master-grub as a chain-loaded system. Job done!

Errors seen at initial boot:

         Error 13: Invalid or unsupported executable format

This message from grub stage1 occurs if the system image being chainloaded to does not (yet) have a proper boot setup, such as a grub stage1 record in the first sector of the partition.

L.S.Lowe
Birmingham Particle Physics Group