Systems - WEBDB
Basics
Purpose
The webdb server provides the web front-end to the world for CAcert's certificate services. It also runs the database with all subscriber data. It communicates with the signer server for submission of certificate signing requests and retrieval of signed certificates, certificate revocation lists and signed PGP keys. This is one of CAcert's critical servers, and is operated under the requirements of the CAcert Security Policy and Security Manual.
Physical Location
The signing server is located on physical machine webdb in the CAcert rack at BIT 2, Ede.
Physical Configuration
The webdb machine contains an Intel Pentium 4 processor running at 1700 MHz, 512 MB of RAM, one SATA disk of 160 GB and two PATA disks of 320 GB each.
See also SystemAdministration/EquipmentList
Connections
This system has two ethernet connections, one to the CAcert internal administrative network (172.28.50) and one to the internet switch provided by BIT.
The keyboard, mouse and video monitor connections are normally connected to the CAcert KVM server located elsewhere in the CAcert rack.
The system is also connected by a serial cable and/or USB cable to the signer server. The serial cable is a shielded and crossed RS-232 cable. The USB cable is a USB-Link cable based on the Prolific Technology Inc. PL2501 chipset, which can be found in USB 2.0 NET Link Cable from EdNet. A custom-developed software module named CommModule is used to transfer controlled data over this link between the webdb server and the signer server.
Applicable Documentation
CommModule - Documentation and Operator Manual (to be put online somewhere)
Administration
Critical System Administrator Team:
- Wytze van der Raay
- Mendel Mobach
- Stefan Kooman
Services
Listening services
Protocol |
Port |
Remarks |
HTTP |
TCP/80 |
web server for main CAcert application |
HTTPS |
TCP/443 |
web server for main CAcert application in secure mode |
SSH |
TCP/22 |
only from two hosts on internal admin network; remote system maintenance |
DNS
External names are:
- www.cacert.org secure.cacert.org tverify.cacert.org hlin.cacert.org
Internal names are:
- none
Connected Systems
Outbound network connections
Protocol |
Port |
Remarks |
DNS |
UDP/53 + TCP/53 |
DNS lookups by main CAcert application and utilities |
SMTP |
TCP/25 |
outgoing mail sent by main CAcert application |
WHOIS |
TCP/43 |
domain name lookup by main CAcert application |
HTTP |
TCP/80 |
web lookups, mainly for system updates |
NTP |
UDP/123 |
time synchronization with internet time servers |
boxbackup |
TCP/2201 |
only to backup.intern.cacert.org; for on-line backups |
Security
Privileged Access: Critical System Administrators
Other Access: CAcert support engineers
Software installation
The base OS for the signing server is a minimum install of Debian 4.0 (Etch). During the installation, LUKS volume encryption must be activated, as described in Disk Encryption.
After performing the base install, the set of installed packages needs to be adjusted to conform to the guideline defined by the pkg-conform script, which can be found at pkg-conform.
The boxbackup-client package is not available in the default debian repository, but can be installed as follows:
# edit /etc/apt/sources.list and add this line: deb http://www.backports.org/debian etch-backports main contrib non-free # apt-get update; apt-get update # apt-get install debian-backports-keyring # apt-get update # should now be free from errors # apt-get -t etch-backports install boxbackup-client
The only custom software to be installed is the main CAcert web application. The source for this can be downloaded from http://www.cacert.org/src-lic.php. For setting up the chroot environment additional tools and configuration files are available from http://svn.cacert.org/CAcert/SystemAdministration/webdb as explained below in more detail.
Software Configuration
All configuration files should be kept under RCS control wherever possible, i.e. when you want to modify a pristine configuration file, do this:
# mkdir RCS # vi configfile # ci -u configfile
Next time you want to change the configuration file, do this:
# co -l configfile # vi configfile # ci -u configfile
Boxbackup
To configure boxbackup-client (assuming a UID of 0x157112, check with server admin):
# bbackup-config /etc/boxbackup lazy 157112 SERVERNAME /var/lib/bbackupd /home /etc/ var
Send the resulting 157112-csr.pem to SERVERNAME administrator for signing and install two returned files. Then run
# bbackupd
chroot environment for main CAcert web application
The main CAcert web application is written in PHP and served by an Apache web server running in a restricted chroot environment for security reasons. This environment must be setup under the directory /home/cacert, and contains a minimal subset of the standard Debian tools required for running the service, plus of course the application-specific parts.
To setup the environment, use the mkchrootenv script which is found at mkchrootenv. Read the first lines of the script to check whether the variables defined there (ROOT, SPEC, SSL_CACERT) need to be adjusted. In particular the SPEC variable requires some attention: this points to a directory which should contain a number of configuration and other special files not easily found elsewhere. An initial cut of its content can be found at spec. The file spec/distro/www.tar.bz2 should be replaced by an up-to-date copy of the bzip2-compressed tar ball of the main CAcert application source code obtained from http://www.cacert.org/src-lic.php.
Running the mkchrootenv script will perform most of the necessary steps automatically, and even generate a set of fake certificates for a test server at the end (this requires some user interaction). When installing a production server, these fake certificates should obviously be replaced by the real ones.
Additional manual configuration
/etc/default/commmodule: needs to be setup with definitions for COMMROOT and COMMROLE e.g.:
COMMROOT=/home/cacert/www/CommModule COMMROLE=client
/etc/init.d/commmodule: needs to be copied from /home/cacert/www/CommModule/commmodule. After that, the init script should be enabled with:
/usr/sbin/update-rc.d commmodule defaults 90 20
NOTE: the full effect of step 1 and 2 can simply be achieved by:
cd /home/cacert/www/CommModule; ./commmodule setup
/etc/logrotate.d/apache: needs to be adjusted with pathnames reflecting chroot for apache to /home/cacert. Use dateext to keep stable names for old logfiles. Omit log file attributes from the create directive, so logrotate will use the same attributes as for the just rotated logfile (note that phperrors.log requires a different set of attributes than access.log or error.log). The following context diff shows the details:
@@ -1,14 +1,15 @@ -/var/log/apache/*.log { +/home/cacert/var/log/apache/*.log { weekly missingok rotate 52 compress + dateext delaycompress notifempty - create 640 root adm + create sharedscripts postrotate - if [ -f /var/run/apache.pid ]; then \ + if [ -f /home/cacert/var/run/apache.pid ]; then \ if [ -x /usr/sbin/invoke-rc.d ]; then \ invoke-rc.d apache reload > /dev/null; \ else \
/etc/postfix/main.cf: needs a couple of adjustments:
append_dot_mydomain = yes relayhost = # inet_interfaces = loopback-only (i.e. comment out or remove!) sender_canonical_maps = hash:/etc/postfix/sender_rewrite
while etc/postfix/sender_rewrite should contain:
www-data returns@cacert.org root returns@cacert.org
/etc/init.d/apache: needs to be adjusted for running the Apache webserver in a chroot environment. Reference svn.cacert.org/CAcert/SystemAdministration/webdb/apache for an example.
/etc/init.d/networking: needs to be extended with a hook to invoke a custom firewall script, like this:
@@ -74,6 +74,7 @@ case "$1" in start) process_options + bash /root/firewall.sh log_action_begin_msg "Configuring network interfaces" if ifup -a; then log_action_end_msg $?
NOTE: it will probably be better to merge this into /etc/network/interfaces rather than modifying a standard system script.
/etc/network/interfaces: needs to be configured with two additional IPv4 addresses for the equivalents of secure.cacert.org and tverify.cacert.org (in addition to www.cacert.org). For example:
# The primary network interface allow-hotplug eth1 iface eth1 inet static address 192.109.159.30 netmask 255.255.255.0 gateway 192.109.159.3 up ifup eth1:0 eth1:1 down ifdown eth1:0 eth1:1 iface eth1:0 inet static address 192.109.159.28 netmask 255.255.255.0 iface eth1:1 inet static address 192.109.159.29 netmask 255.255.255.0
/etc/ssh/sshd_config: should have PermitRootLogin, PasswordAuthentication and UsePAM turned off.
/etc/cron.d/php4: should be expanded with an extra cleanup job for the chroot environment:
@@ -5,3 +5,4 @@ # Look for and purge old sessions every 30 minutes 09,39 * * * * root [ -d /var/lib/php4 ] && find /var/lib/php4/ -type f -cmin +$(/usr/lib/php4/maxlifetime) -print0 | xargs -r -0 rm +09,39 * * * * root [ -d /home/cacert/var/lib/php4 ] && find /home/cacert/var/lib/php4/ -type f -cmin +$(/usr/sbin/chroot /home/cacert /usr/lib/php4/maxlifetime) -print0 | xargs -r -0 rm
/root/firewall.sh: is a custom firewall script for the webdb server.
/home/cacert/etc/apache/httpd.conf: needs to be adjusted when used for a test system setup. The following context diff shows the nature of the changes:
@@ -200,10 +200,10 @@ AddEncoding gzip .gz </IfModule> -NameVirtualHost 213.154.225.245:80 -NameVirtualHost 213.154.225.245:443 +NameVirtualHost 192.109.159.30:80 +NameVirtualHost 192.109.159.30:443 -<VirtualHost 213.154.225.245:80> +<VirtualHost 192.109.159.30:80> ServerAdmin cacert@cacert.org DocumentRoot /www/stamp ServerName stamp.cacert.org @@ -213,7 +213,7 @@ </VirtualHost> -<VirtualHost 213.154.225.245:443> +<VirtualHost 192.109.159.30:443> SSLEngine on SSLCertificateFile /etc/ssl/certs/cacert.crt SSLCertificateKeyFile /etc/ssl/private/cacert.pem @@ -226,12 +226,12 @@ RewriteRule .* - [F] </VirtualHost> -<VirtualHost 213.154.225.245:80> +<VirtualHost 192.109.159.30:80> ServerAdmin cacert@cacert.org DocumentRoot /www/www ScriptAlias /cgi-bin/ /www/cgi-bin/ -ServerName cacert.org -ServerAlias cacert.com *.cacert.com cacert.net *.cacert.net *.cacert.org freecert.org *.freecert.org +ServerName cacert1.it-sls.de +ServerAlias cacert.org cacert.com *.cacert.com cacert.net *.cacert.net *.cacert.org freecert.org *.freecert.org Redirect /revoke.crl http://crl.cacert.org/revoke.crl permanent Redirect /class3-revoke.crl http://crl.cacert.org/class3-revoke.crl permanent RewriteEngine On @@ -239,12 +239,12 @@ RewriteRule .* - [F] </VirtualHost> -<VirtualHost 213.154.225.245:443> +<VirtualHost 192.109.159.30:443> SSLEngine on SSLCertificateFile /etc/ssl/certs/cacert.crt SSLCertificateKeyFile /etc/ssl/private/cacert.pem -ServerName www.cacert.org +ServerName cacert1.it-sls.de DocumentRoot /www/www ScriptAlias /cgi-bin/ /www/cgi-bin/ Redirect /revoke.crl http://crl.cacert.org/revoke.crl permanent @@ -254,7 +254,7 @@ RewriteRule .* - [F] </VirtualHost> -<VirtualHost 213.154.225.246:443> +<VirtualHost 192.109.159.28:443> SSLEngine on SSLVerifyClient require SSLVerifyDepth 2 @@ -266,7 +266,7 @@ #SSLOCSPDefaultResponder http://ocsp.cacert.org/ SSLOptions +StdEnvVars -ServerName secure.cacert.org +ServerName secure1.it-sls.de DocumentRoot /www/www RewriteEngine On @@ -274,7 +274,7 @@ RewriteRule .* - [F] </VirtualHost> -<VirtualHost 213.154.225.247:443> +<VirtualHost 192.109.159.29:443> SSLEngine on ##OFF on Nov 18, 2009 -- see below##SSLVerifyClient require ##OFF on Nov 18, 2009 -- see below##SSLVerifyDepth 2 @@ -283,7 +283,7 @@ SSLCertificateKeyFile /etc/ssl/private/cacert.pem SSLOptions +StdEnvVars -ServerName secure.cacert.org +ServerName tverify1.it-sls.de DocumentRoot /www/tverify # tverify out of service on Nov 18, 2009: redirect to appropriate wiki page
/home/cacert/etc/group and /home/cacert/etc/passwd: need a change of uid/gid for chrapach to 33/33 to match www-data in the regular environment.
/home/cacert/www/includes/mysql.php: needs to be created as a copy of /home/cacert/www/includes/mysql.php.sample and edited to set the username and password for accessing the CAcert MySQL database, and set normalhostname, securehostname and tverify for a test system. For example:
@@ -16,15 +16,15 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ - $_SESSION['mconn'] = mysql_connect("127.0.0.1", "username", "password"); + $_SESSION['mconn'] = mysql_connect("127.0.0.1", "cacert", "klodder"); if ($_SESSION['mconn'] != FALSE) { - mysql_select_db("database"); + mysql_select_db("cacert"); $_SESSION['mconn'] = TRUE; } - $_SESSION['_config']['normalhostname'] = "www.cacert.org"; - $_SESSION['_config']['securehostname'] = "secure.cacert.org"; - $_SESSION['_config']['tverify'] = "tverify.cacert.org"; + $_SESSION['_config']['normalhostname'] = "cacert1.it-sls.de"; + $_SESSION['_config']['securehostname'] = "secure1.it-sls.de"; + $_SESSION['_config']['tverify'] = "tverify1.it-sls.de"; function sendmail($to, $subject, $message, $from, $replyto = "", $toname = "", $fromname = "", $errorsto = "returns@cacert.org", $extra="") {
Disk mirroring
Disk mirroring must be setup as described in Disk Mirroring, to ensure that a usable replacement disk is available in case the primary disk fails. Failover is not automatic, and requires system administrator intervention.
Cron jobs
The following CAcert-specific cron jobs are used on the server, in /etc/crontab:
/root/Mirror, to be run every 15 to 30 minutes, see Disk Mirroring
- /home/cacert/www/CommModule/logclean.sh, to be run daily or weekly
- cd /home/cacert/www/scripts; ./removedead.php, to be run every hour
- cd /home/cacert/www/scripts; ./warning.php, to be run every hour
- cd /home/cacert/www/scripts; ./updatesort.php, to be run every eight hours
wget -O /home/cacert/www/pages/index/feed.rss http://blog.cacert.org/feed/rss/, to be run every 5 minutes
- /root/updatecrls.sh, to be run every hour
Common Tasks
Logfile Inspection
Full Backup
Log File Extraction
Password Changes
Planned Changes
The base OS needs to be upgraded to Debian Lenny (status "stable") before 15 February 2010.
Changelog
All modifications to this system must be logged to the cacert-systemlog mailing list, which is primarily archived here.
Notes
Configuring normalhostname and securehostname in /home/cacert/www/includes/mysql.php
(Contributed by Markus Warg)
My "knowledge" about the two hostnames:
normalhostname is used for all regular operation on the http part of the page. normalhostname is even used for regular operation of all https parts of the page, if the user NOT logged in by certificate.
As you can see, the securehostname is linked under certificate login, so if the user wants to log in by crt, he is redirected to securehostname.
The code now seperates actions, if HTTP_HOST == securehostname and id=4, try to authenticate the user by crt, otherwise redirect to login page UNLESS he already logged in.
Both hosts point to the same document root, its just another virtual host to prevent the ssl renegotiation problem, I think. If you log in by crt, you will need to stay on securehost, because there is no session login, if you crt log in and manually switch from secure to www, you will be forced to log in.
If you log in and switch to securehost, you will be asked for an crt.