CAcert's critical servers require some protection against outage due to a single disk failure. Currently the way these servers are setup does not include provisions for RAID or mirroring at the hardware or kernel level. Therefore a simple mirroring scheme at the process level has been designed which can be retrofitted to any running server, while retaining its encryption qualities. The mirroring is not perfect, as it is driven by a script which is ran from cron at a suitable frequency: every 15 minutes is suggested, but this may be adjusted to accomodate the system's I/O load.

How to initialize the mirror disk on the webserver and signing server

Primary disk is /dev/hda, mirror disk /dev/hdc.

1. Copy the boot block from the primary disk to the mirror

Copy the boot block from the primary disk to the mirror, so we can use the mirror disk as a boot disk when necessary. There are two ways to accomplish this:

# dd if=/dev/hda of=/dev/hdc bs=512 count=1

# echo Copy mirror boot block including partition table data
# dd if=/dev/hdc of=/tmp/ccc bs=512 count=1
# echo Use master boot block as template for new boot block
# dd if=/dev/hda of=/tmp/aaa bs=512 count=1
# echo Copy partition table from mirror into master boot block template
# dd if=/tmp/ccc of=/tmp/aaa skip=446 seek=446 bs=1 count=66
# echo Write patched boot block back to mirror disk
# dd if=/tmp/aaa of=/dev/hdc bs=512 count=1
# echo Remove temporary files
# rm -f /tmp/aaa /tmp/ccc

Method (a) is fine at this point, because in step 2 we will (re-)partition the mirror disk. Method (b) is needed when you need to do this after step 2, i.e. when the existing partitioning needs to be preserved.

2. Partition the mirror disk exactly the same as the primary disk

Partition the mirror disk exactly the same as the primary disk; i.e display primary disk partioning with:

   # fdisk -l /dev/hda

then run

   # fdisk -H 255 /dev/hdc

and apply the same partitioning (-H 255 seems to be necessary on Debian Etch to get the same cylinder sizes for primary disk and mirror disk ...).

3. Setup encryption for mirror disk

 # dd if=/dev/urandom of=/dev/hdc5 bs=1024k count=100
   (i.e. measure speed for randomizing 100 MB of the disk)
 # dd if=/dev/urandom of=/dev/hdc5 bs=1024k
   (this will take a very long time, you can calculate from the
   size of the disk and the timing above how long; measured on has been 1.689 MB/sec, i.e. ~53 hours for 320 GB)
   (you can inspect the progress by a kill -USR1 pid-of-dd-process)
   (NOTE: there should be a faster way to do this, since the Debian
   installer can do it faster; need to investigate how it does it)
 # cryptsetup luksFormat /dev/hdc5
   (will prompt twice for passphrase, use the passphrase of hda5)
 # cryptsetup luksOpen /dev/hdc5 hdc5_crypt
   (will prompt again for passphrase)
 # vi /etc/crypttab
   add a line for hdc5_crypt similar to the line for hda5_crypt,
   so the result becomes:
          hda5_crypt /dev/hda5 none luks
          hdc5_crypt /dev/hdc5 none luks

4. Setup LVM for mirror disk

# pvcreate /dev/mapper/hdc5_crypt
# vgcreate mirror /dev/mapper/hdc5_crypt
# lvs
  (note the LSize of the LV root in VG www | signer)
# lvcreate -n root -L xxxG mirror
  (the number xxx should be identical to the LSize above)
# pvs; vgs; lvs
  (check results for completeness and consistency)
# mke2fs -j /dev/mapper/mirror-root
# mke2fs -j /dev/hdc1

5. Adjust fstab and rsync partitions

# mkdir /mirror
# vi /etc/fstab
  add two lines for the new partitions:
      /dev/hdc1                /mirror/boot  ext3  noauto  0  2
      /dev/mapper/mirror-root  /mirror       ext3  noauto  0  3
# mount /mirror
# mkdir /mirror/boot
# mount /mirror/boot
# rsync --verbose --archive --hard-links --delete-after --one-file-system / /mirror
# rsync --verbose --archive --hard-links --delete-after --one-file-system /boot /mirror
# umount /mirror/boot
# umount /mirror

6. Install script for systematic rsyncing of primary disk to mirror

Install MIRROR script in /root (see Appendix for script code).

Add the following line to /etc/crontab:
   */15  *  *  *  *  root /root/MIRROR >>/var/log/mirror 2>&1
to enable rsyncing every 15 minutes, a log will be kept in /var/log/mirror

7. Add script to control rotation of mirror log file

Install the following file named 'mirror' in /etc/logrotate.d:

/var/log/mirror {
   rotate 52

Appendix: MIRROR script

#  @(#)(CAcert) $Id: MIRROR,v 1.1 2009/12/29 19:51:59 root Exp $
# MIRROR - rsync / and /boot to mirror disk


# Log start of operation.

# Check lock or establish one (with automatic cleanup on exit)
# Note that race conditions are not an issue here, since this script
# will only be run at intervals of 15 minutes or more.
test -f ${LOCK} && exit 0
echo $$ >${LOCK}
trap "rm -f ${LOCK}" 0 1 2 3 15

# Check mirror file systems before mounting.
# Note that this also provides some protection against overlapping runs,
# since fsck will bail out for a mounted file system.
/sbin/fsck -p /dev/mapper/mirror-root || exit $?
/sbin/fsck -p /dev/hdc1 || exit $?
mount /mirror
mount /mirror/boot

# Mirror the root file system (which is encrypted) and the boot file system.
for FS in / /boot
        echo "rsyncing ${FS} to /mirror ..."
        rsync --verbose --archive --hard-links --delete-after --one-file-system \
                ${FS} /mirror

# Unmount mirror file systems to keep them offline as much as possible.
umount /mirror/boot
umount /mirror

Other notes


SystemAdministration/Procedures/DiskMirroring (last edited 2009-12-29 20:25:52 by WytzevanderRaay)