=== Introduction === CAcert's critical servers require some protection against outage due to a single disk failure. Currently the way these servers are setup does not include provisions for RAID or mirroring at the hardware or kernel level. Therefore a simple mirroring scheme at the process level has been designed which can be retrofitted to any running server, while retaining its encryption qualities. The mirroring is not perfect, as it is driven by a script which is ran from cron at a suitable frequency: every 15 minutes is suggested, but this may be adjusted to accomodate the system's I/O load. === How to initialize the mirror disk on the webserver and signing server === Primary disk is /dev/hda, mirror disk /dev/hdc. ==== 1. Copy the boot block from the primary disk to the mirror ==== Copy the boot block from the primary disk to the mirror, so we can use the mirror disk as a boot disk when necessary. There are two ways to accomplish this: (a) simple method, overwrites any existing partition data on mirror disk: {{{ # dd if=/dev/hda of=/dev/hdc bs=512 count=1 }}} (b) careful method, preserves existing partition data on mirror disk: {{{ # echo Copy mirror boot block including partition table data # dd if=/dev/hdc of=/tmp/ccc bs=512 count=1 # echo Use master boot block as template for new boot block # dd if=/dev/hda of=/tmp/aaa bs=512 count=1 # echo Copy partition table from mirror into master boot block template # dd if=/tmp/ccc of=/tmp/aaa skip=446 seek=446 bs=1 count=66 # echo Write patched boot block back to mirror disk # dd if=/tmp/aaa of=/dev/hdc bs=512 count=1 # echo Remove temporary files # rm -f /tmp/aaa /tmp/ccc }}} Method (a) is fine at this point, because in step 2 we will (re-)partition the mirror disk. Method (b) is needed when you need to do this after step 2, i.e. when the existing partitioning needs to be preserved. ==== 2. Partition the mirror disk exactly the same as the primary disk ==== Partition the mirror disk exactly the same as the primary disk; i.e display primary disk partioning with: {{{ # fdisk -l /dev/hda }}} then run {{{ # fdisk -H 255 /dev/hdc }}} and apply the same partitioning (-H 255 seems to be necessary on Debian Etch to get the same cylinder sizes for primary disk and mirror disk ...). ==== 3. Setup encryption for mirror disk ==== {{{ # dd if=/dev/urandom of=/dev/hdc5 bs=1024k count=100 (i.e. measure speed for randomizing 100 MB of the disk) # dd if=/dev/urandom of=/dev/hdc5 bs=1024k (this will take a very long time, you can calculate from the size of the disk and the timing above how long; measured on 172.28.50.6 has been 1.689 MB/sec, i.e. ~53 hours for 320 GB) (you can inspect the progress by a kill -USR1 pid-of-dd-process) (NOTE: there should be a faster way to do this, since the Debian installer can do it faster; need to investigate how it does it) # cryptsetup luksFormat /dev/hdc5 (will prompt twice for passphrase, use the passphrase of hda5) # cryptsetup luksOpen /dev/hdc5 hdc5_crypt (will prompt again for passphrase) # vi /etc/crypttab add a line for hdc5_crypt similar to the line for hda5_crypt, so the result becomes: hda5_crypt /dev/hda5 none luks hdc5_crypt /dev/hdc5 none luks }}} ==== 4. Setup LVM for mirror disk ==== {{{ # pvcreate /dev/mapper/hdc5_crypt # vgcreate mirror /dev/mapper/hdc5_crypt # lvs (note the LSize of the LV root in VG www | signer) # lvcreate -n root -L xxxG mirror (the number xxx should be identical to the LSize above) # pvs; vgs; lvs (check results for completeness and consistency) # mke2fs -j /dev/mapper/mirror-root # mke2fs -j /dev/hdc1 }}} ==== 5. Adjust fstab and rsync partitions ==== {{{ # mkdir /mirror # vi /etc/fstab add two lines for the new partitions: /dev/hdc1 /mirror/boot ext3 noauto 0 2 /dev/mapper/mirror-root /mirror ext3 noauto 0 3 # mount /mirror # mkdir /mirror/boot # mount /mirror/boot # rsync --verbose --archive --hard-links --delete-after --one-file-system / /mirror # rsync --verbose --archive --hard-links --delete-after --one-file-system /boot /mirror # umount /mirror/boot # umount /mirror }}} ==== 6. Install script for systematic rsyncing of primary disk to mirror ==== Install MIRROR script in /root (see Appendix for script code). {{{ Add the following line to /etc/crontab: */15 * * * * root /root/MIRROR >>/var/log/mirror 2>&1 to enable rsyncing every 15 minutes, a log will be kept in /var/log/mirror }}} ==== 7. Add script to control rotation of mirror log file ==== Install the following file named 'mirror' in /etc/logrotate.d: {{{ /var/log/mirror { weekly rotate 52 compress delaycompress } }}} ==== Appendix: MIRROR script ==== {{{ #! /bin/bash # @(#)(CAcert) $Id: MIRROR,v 1.1 2009/12/29 19:51:59 root Exp $ # MIRROR - rsync / and /boot to mirror disk LOCK=/var/lock/MIRROR.lock # Log start of operation. date # Check lock or establish one (with automatic cleanup on exit) # Note that race conditions are not an issue here, since this script # will only be run at intervals of 15 minutes or more. test -f ${LOCK} && exit 0 echo $$ >${LOCK} trap "rm -f ${LOCK}" 0 1 2 3 15 # Check mirror file systems before mounting. # Note that this also provides some protection against overlapping runs, # since fsck will bail out for a mounted file system. /sbin/fsck -p /dev/mapper/mirror-root || exit $? /sbin/fsck -p /dev/hdc1 || exit $? mount /mirror mount /mirror/boot # Mirror the root file system (which is encrypted) and the boot file system. for FS in / /boot do echo "rsyncing ${FS} to /mirror ..." rsync --verbose --archive --hard-links --delete-after --one-file-system \ ${FS} /mirror done # Unmount mirror file systems to keep them offline as much as possible. umount /mirror/boot umount /mirror }}} === Other notes === ---- CategoryProcedures