By Seán Boran (sean AT boran.com)
Why not RAID? | How it works | Configuring |
Known Problems | Recovery | Alternatives & Improvements | Acknowledgments
Minimum downtime and prevention of data loss is important for
most system administrators. The traditional solution is to use backups or RAID to cover
for disk failures. We describe an alternative for "cold mirroring" of system
disks it mounts a spare disk, copies files to the spare, installs a boot block and
copies over a new vfstab. This creates a fully updated bootable spare disk. The
administrator is notified of success/failure by syslog or email. This tool, called mirror_boot.sh, has been tested on several Solaris
If the system disks (/, /usr, /var file systems) are on RAID and for example the raid controller (or fiber cable) fails, you have a problem, unless the RAID is fully redundant. Also, cold mirroring is simpler, and software RAID can be difficult to recover when the system disk fails.
For some servers, I prefer to put system (and certain data) files on a "normal" disk and mirror to a second disk once or twice a week ("cold mirroring"). If the boot disk dies, we simply boot from the mirror disk. This solution is easier to understand, to recover from in a disaster scenario, and system disks can be more easily added/removed/changed.
In addition, files changed by accident can be recovered since the last mirror run, and deleted files can be recovered until the disk fills up and needs to be wiped clean. More details are provided below.
Each night the offline disk is mounted and synchronized with the primary disk. The script is called from the root cron nightly. It mounts the spare disk under /newroot, copies all file systems, installs a boot block and copies over a new vfstab. This creates a fully updated bootable spare disk. The results of the script are sent to the administrator via email (sample output is mirror_output.txt).
Start off by downloading mirror_boot.sh, browsing the source and reading this article completely. Then practice on a non-critical Solaris host.
We'll now walk through two examples, an x86 and a SPARC Solaris 8 box, both of which have system disks with three partitions: root, /var and swap.
The disks in our example are set up as follows:
|The primary and mirror disk do not have to be identical|
|a) The partitions on the secondary disk must be at least as big as the primary.
b) It is an advantage if the second disk is bigger, as it allows deleted files to be kept longer (the backup file systems don't need to be purged as often).
c) The file systems on the primary or secondary disk can in fact be spread out over several physical disks. The mirror script does not care, as it simply backs up each file system to an equivalent under /newroot.
1. Active (primary) system disk:
On x86: / c0d0s0, swap c0d0s1, /var c0d0s5
On SPARC: / c0t3d0s0, swap c0t3d0s1, /var c0t3d0s6
2. Backup (secondary) disk
On x86: / c0d1s0, swap c0d1s1, /var c0d1s5
On SPARC: / c0t1d0s0, swap c0t1d0s1, /var c0t1d0s6
a) Set up the secondary disk:
Fdisk second disk (x86 architecture only), e.g., "fdisk /dev/rdsk/c0d1p0"
Partition the second disk identically to the first (with format): e.g., create root, swap and /var partition with the same number of cylinders, then write the disk label and quit format.
Note: If the master and mirror disk are identical, the fmthard tool together with prtvtoc is much faster than manually using format. For example, if the master is target 3 and the second disk is target 1, and we wish to give it the disk label "mirror," then:
/usr/sbin/prtvtoc /dev/rdsk/c0d0s2 | /usr/sbin/fmthard -n mirror -s - /dev/rdsk/c0d1s2
/usr/sbin/prtvtoc /dev/rdsk/c0t3d0s2 | /usr/sbin/fmthard -n mirror -s - /dev/rdsk/c0t1d0s2
Create file systems on the second disk, e.g.,
on x86: newfs c0d1s0; newfs c0d1s5;
on SPARC: newfs c0t1d0s0; newfs c0t1d0s6;
b) Get the secondary disk online:
Add entries for the new disk in /etc/vfstab under /newroot, /newroot/var, etc.
Tip.. I've written a script to initialise this tool for a specific set of machines, you may wish to download and adapt it for your needs.
/etc/vfstab for our x86 example:
/dev/dsk/c0d0s1 - - swap - no -
swap - /tmp tmpfs - yes size=150m
/dev/dsk/c0d0s0 /dev/rdsk/c0d0s0 / ufs 1 no logging
/dev/dsk/c0d0s6 /dev/rdsk/c0d0s6 /var ufs 1 yes logging,nosuid,noatime
# Cold mirror backup disk
/dev/dsk/c0d1s0 /dev/rdsk/c0d1s0 /newroot ufs 1 no logging
/dev/dsk/c0d1s6 /dev/rdsk/c0d1s6 /newroot/var ufs 1 yes logging,nos uid,noatime
/etc/vfstab for our SPARC example:
/dev/dsk/c0t3d0s1 - - swap - no -
/dev/dsk/c0t3d0s0 /dev/rdsk/c0t3d0s0 / ufs 1 no logging
/dev/dsk/c0t3d0s6 /dev/rdsk/c0t3d0s6 /var ufs 1 no logging,noatime,nosuid
swap - /tmp tmpfs - yes size=120m
# Cold mirror backup disk
/dev/dsk/c0t1d0s0 /dev/rdsk/c0t1d0s0 /newroot ufs 1 no logging
/dev/dsk/c0t1d0s6 /dev/rdsk/c0t1d0s6 /newroot/var ufs 1 no logging,noatime,nosuid
mkdir /newroot; chmod 777 /newroot; mount /newroot;
/etc/vfstab.newroot for our x86 example:
swap - /tmp tmpfs - yes -
swap - /tmp tmpfs - yes size=150m
/dev/dsk/c0d1s0 /dev/rdsk/c0d1s0 / ufs 1 no logging
/dev/dsk/c0d1s6 /dev/rdsk/c0d1s6 /var ufs 1 yes logging,nosuid,noatime
/etc/vfstab.newroot for our SPARC example:
/dev/dsk/c0t1d0s1 - - swap - no -
swap - /tmp tmpfs - yes size=120m
/dev/dsk/c0t1d0s0 /dev/rdsk/c0t1d0s0 / ufs 1 no logging
/dev/dsk/c0t1d0s6 /dev/rdsk/c0t1d0s6 /var ufs 1 no logging,noatime,nosuid
Attention: On an x86 system with IDE disks, just copy vfstab to vfstab.newroot. The reason for this is that if the primary IDE fails, the system will not boot, so the backup disk will have to replace the master and hence will assume its device address. See the recovery section below for an example.
c) Configure the script /secure/mirror_boot.sh
stop_daemons='sh /etc/rc3.d/S99msql stop';
start_daemons='sh /etc/rc3.d/S99msql start';
#Otherwise, comment out the daemon stop/start lines:
# If we want to ignore tar archives:
d) Do a dry run to check configuration set DEBUG='1' and run the script. It will not copy over the file systems, but do everything else, so it allows you to check that everything has been set up correctly.
e) First real run set DEBUG='0' and let it go:
If you run it in background mode, then progress can be followed by running "df -k" now and again.
f) SPARC: Change the EEPROM so that the system will boot from the mirror automatically if the main disk fails.
eeprom boot-device="disk3 disk1"
g) Configure the mirroring to run regularly. For example, a root cron entry for each Wednesday at 4 a.m.. would be:
0 4 * * 3 /secure/mirror_boot.sh
It's all very well doing mirroring or backups, but it's useless if you haven't tested it in a disaster scenario.
1) Solaris SPARC with scsi disks
Example: "server1" has two scsi disks. The main one has target 3, the second (target 1) is mirrored each night from the first. If the main disk fails, proceed as follows to boot from the second disk.
a) Go to the "ok" prompt.
b) "boot disk1"
c) Log in as usual, check with "df" that you really are using the backup disk.
d) Disable the mirroring script in the root cron, until the primary disk is working again!
2) Solaris Intel with IDE disks
|Solaris on Intel x86 Architecture|
|Booting a Solaris PC has several key steps that are important to understand when
backing up/restoring boot disks
a) The PC BIOS decides what devices will be used for booting, in what order and whether certain devices are disabled.
b) The SCSI controller (if there is one) can be configured to insist on its own boot device.
c) The boot disk has been now selected and activated.
d) The Device Configuration Assistant (DCA) detects devices, makes them visible to the boot manager and chooses a root file system. If ESC is pressed when this is booting, devices can be changed as can default boot order, boot disks etc., via a menu-based interface. It is installed in the first few disk cylinders, in a separate partition (/boot).
e) Boot Manager: Allow options to be given to Solaris on booting, or using the boot interpreter.
f) Finally, trusty old Solaris starts up.
Example: "server1" has two IDE disks. The main one is the IDE master, the second (IDE slave) is mirrored each night from the first. If the main disk fails, proceed as follows to boot from the second disk. Important: /etc/vfstab must be identical on both disks for the following to work.
a) Switch off the machine, remove the faulty (master) IDE disk, change the jumpers on the slave IDE disk to be a master.
b) Switch on.
c) Confirm BIOS changes (some PCs such as Compaq will detect the disk removal and ask you to confirm the fact that the number of disks detected has changed).
d) Log in as usual, check with "df" that you really are using the backup disk.
e) Disable the mirroring script in the root cron, until the primary disk is working again!
3) Solaris Intel with scsi disks
Example: "server1" has two SCSI disks. The main one has target 0, the second, target 1, is mirrored each night from the first. If the main disk fails, proceed as follows to boot from the second disk:
a) Stop the PC, switch it back on and allow the bios to do its checks.
b) When the SCSI controller is checking the disks, Press Ctrl-A to enter the SCSI menu (the command needed to enter the SCSI menu depends on the controller).
c) Change the boot device to SCSI ID 1 (which is the ID of the second disk). Save changes, and exit, and it will probably insist on a reboot.
d) After the SCSI disk check, the "Device Configuration Assistant" (DCA) will start. Press Esc to enter its menu and F2 ("continue") about three times until the list of boot disks is shown. Select the backup disk (target 1) and press continue, then allow the boot process to continue.
e) Log in as usual, check with "df" that you really are using the backup disk.
f) Disable the mirroring script in the root cron, until the primary disk is working again!
g) To tell the DCA to permanently use the secondary disk for booting Solaris, the "setprop bootpath" line in /boot/solaris/bootenv.rc needs to point to the boot partition on the secondary disk. For example, root is on /dev/dsk/c0d1s0. Do "ls -l /dev/dsk/c0d1s0" on this and you'll get the path to the actual device.
Thanks to the following readers for their ideas and constructive feedback: Garry Garrett, Pavel, David Schweikert, Neil Collins.
Seán Boran is an IT security consultant based in Switzerland and the author of the online IT Security Cookbook.
08.Mar.01 Update TOC, known problems,
Future Improvements, fmthard
19.Apr.01 Update: dd, Minor fixes. touch /newroot/MIRROR and if /MIRROR exists, fail.
23.Oct.01 Update: fssnap, flarcreate, init script, fix links since SecurityPortal now dead.
20.Jan.02 Link fix.
20.Jun.03 Tip for cleaning /newroot