. '''To [[SystemAdministration/Systems|Systems Overview]]''' ---- = Systems - Signer = = Basics = == Purpose == The signing server signs appropriately formatted Certificate Signing Requests with one of CAcert's (sub)root keys. It also performs PGP signing of appropriately formatted PGP keys submitted by CAcert's users. This is one of CAcert's critical servers, and is operated under the requirements of the CAcert [[http://svn.cacert.org/CAcert/Policies/SecurityPolicy.html|Security Policy]] and [[SecurityManual|Security Manual]]. == Physical Location == The signing server is located on physical machine '''signer''' in the CAcert rack at BIT 2, Ede. == Physical Configuration == The '''signer''' machine has been replaced by a new server on September 11, 2009. The new server contains an Intel Dual Core Xeon E3113 processor (3.0 GHz, 6 MB, 1333 MHz FSB), 2 GB of RAM, and two hotplug SATA disks of 160 GB each. The machine broke in March 2020. The '''signer''' machine has been replaced by a new server on May 4, 2020. The new server contains an Intel Quad Core Xeon X3323 processor (2.5 GHz, 6 MB, 1333 MHz FSB), 2 GB of RAM. The original drives of the old signer had been used. See also [[SystemAdministration/EquipmentList]] == Connections == This system does '''not''' have any network connections. There is normally no keyboard, mouse or video monitor attached to it either. Only for maintenance purposes, a critical systems administrator may connect a crash-cart-based keyboard/mouse/video monitor to the system. The only connections of this system with the outside world is a serial cable and/or USB cable to the [[SystemAdministration/Systems/Webdb|webdb server]]. The serial cable is a shielded and crossed RS-232 cable. The USB cable is a USB-Link cable based on the Prolific Technology Inc. PL2501 chipset, which can be found in ''USB 2.0 NET Link Cable'' from ''Ed''''''Net''. A custom-developed software module named ''Comm''''''Module'' is used to transfer controlled data over this link between the signing server and the [[SystemAdministration/Systems/Webdb|webdb server]]. As of September 11, 2009, the USB cable connection is used instead of the serial cable. == Applicable Documentation == 1. Comm''''''Module - Documentation and Operator Manual (''to be put online somewhere''). [[http://svn.cacert.org/CAcert/CommModule/README| README of CommModule on CAcert SVN]] 1. [[http://svn.cacert.org/CAcert/Policies/SecurityPolicy.html|Security Policy]] 1. [[SecurityManual|Security Manual]] 1. [[SystemAdministration/Procedures/DiskEncryption|Disk Encryption]] 1. [[SystemAdministration/Procedures/DiskMirroring|Disk Mirroring]] 1. [[SystemAdministration/Procedures/DriveRetirement|Drive Retirement]] 1. [[SystemAdministration/Procedures/FullBackupRestore|Full Backup Restore]] 1. [[SystemAdministration/Procedures/PasswordManagement|Password Management]] 1. [[http://svn.cacert.org/CAcert/Software/CAcertCeremonyScript/CAcertSubrootsSysAdminInfo.odt|Installing subroot keys]] == Administration == Critical System Administrator Team: * Dirk Astrath = Services = == Listening services == Not applicable -- system has no network connections. === DNS === Not applicable -- system has no network connections. == Connected Systems == * [[SystemAdministration/Systems/Webdb|www.cacert.org]] through serial/USB cable as described above === Outbound network connections === Not applicable -- system has no network connections. = Security = Privileged Access: Critical System Administrators, only when physically present after connecting crash-cart-based keyboard/mouse/video monitor. Other Access: none. == Software installation == The base OS for the signing server is a minimum install of Debian 5.0 (Lenny). During the installation, LUKS volume encryption must be activated, as described in [[SystemAdministration/Procedures/DiskEncryption|Disk Encryption]]. The following packages need to be added after performing the base install: * apt-listchanges * apt-show-versions * hddtemp * hdparm * ipmitool * libdevice-serialport-perl * libdigest-sha1-perl * libfile-counterfile-perl * libxdelta2 * openssl * rcs * rsync * setserial * smartmontools * sysv-rc-conf * xdelta To add a package, use: {{{ # aptitude install packagename }}} The following packages need to be removed after performing the base install: * at * bc * bind9-host * dc * dictionaries-c * dnsutils * doc-debian * doc-linux-text * exim4 * exim4-base * exim4-config * exim4-daemon-l * file * finger * ftp * iamerican * ibritish * ispell * libbind9-0 * libdns22 * libevent1 * libgc1c2 * libgssapi2 * libidn11 * libisc11 * libisccc0 * libisccfg1 * liblockfile1 * liblwres9 * libmagic1 * libnfsidmap2 * libpci2 * libpcre3 * librpcsecgss3 * libsemanage1 * lsof * m4 * mailx * mime-support * mpack * mtools * mtr-tiny * mutt * ncurses-term * nfs-common * patch * pciutils * pidentd * policycoreutil * portmap * procmail * python * python2.4 * python2.4-mini * python-central * python-minimal * python-newt * python-selinux * python-semanag * python-support * readline-common * reportbug * selinux-policy * sharutils * strace * tcsh * telnet * texinfo * time * ucf * w3m * wamerican * whois To remove a package, use: {{{ # aptitude remove packagename_ }}} '''NOTE: need to check the above lists again for Debian Lenny''' The only custom software to be installed is the !CommModule server side, which is currently not found in the CVS or SVN tree ('''NOTE: add a reference once it is there'''). This is installed in /root/CommModule. == Software Configuration == All configuration files should be kept under RCS control wherever possible, i.e. when you want to modify a pristine configuration file, do this: {{{ # mkdir RCS # vi configfile # ci -u configfile }}} Next time you want to change the configuration file, do this: {{{ # co -l configfile # vi configfile # ci -u configfile }}} === X.509 signing configuration === Specific configuration files for the signing server are installed under the /etc/ssl directory. This directory contains a .cnf file for each combination of signing key and certificate type, and an associated subdirectory. The config file structure looks like: {{{ [ CAcert original root ] /etc/ssl/openssl-client.cnf /etc/ssl/openssl-client-codesign.cnf /etc/ssl/openssl-client-org.cnf /etc/ssl/openssl-ocsp.cnf /etc/ssl/openssl-server.cnf /etc/ssl/openssl-server-org.cnf /etc/ssl/openssl-subca.cnf /etc/ssl/openssl-timestamp.cnf [ CAcert original class3 subroot ] /etc/ssl/class3-client.cnf /etc/ssl/class3-client-codesign.cnf /etc/ssl/class3-client-org.cnf /etc/ssl/class3-ocsp.cnf /etc/ssl/class3-server.cnf /etc/ssl/class3-server-org.cnf /etc/ssl/class3-subca.cnf /etc/ssl/class3-timestamp.cnf [ CAcert new class1 subroot ] /etc/ssl/root3/client.cnf /etc/ssl/root3/client-codesign.cnf /etc/ssl/root3/client-org.cnf /etc/ssl/root3/ocsp.cnf /etc/ssl/root3/server.cnf /etc/ssl/root3/server-org.cnf /etc/ssl/root3/subca.cnf /etc/ssl/root3/timestamp.cnf [ CAcert new class3 subroot ] /etc/ssl/root4/client.cnf /etc/ssl/root4/client-codesign.cnf /etc/ssl/root4/client-org.cnf /etc/ssl/root4/ocsp.cnf /etc/ssl/root4/server.cnf /etc/ssl/root4/server-org.cnf /etc/ssl/root4/subca.cnf /etc/ssl/root4/timestamp.cnf }}} There are also subdirectories here containing the signing key, signer certfificate, next serial to be used, certificate index and issued certificates for each main key: {{{ [ CAcert original root ] /etc/ssl/CA/cacert.crt /etc/ssl/CA/cacert.pem /etc/ssl/CA/index.txt /etc/ssl/CA/index.txt.attr /etc/ssl/CA/serial /etc/ssl/CA/newcerts [ CAcert original class3 subroot ] /etc/ssl/class3/cacert.crt /etc/ssl/class3/cacert.pem /etc/ssl/class3/index.txt /etc/ssl/class3/index.txt.attr /etc/ssl/class3/serial /etc/ssl/class3/newcerts [ CAcert new class1 subroot ] /etc/ssl/root3/cacert.crt /etc/ssl/root3/cacert.pem /etc/ssl/root3/serial /etc/ssl/root3/index.txt /etc/ssl/root3/index.txt.attr /etc/ssl/root3/newcerts [ CAcert new class3 subroot ] /etc/ssl/root4/cacert.crt /etc/ssl/root4/cacert.pem /etc/ssl/root4/serial /etc/ssl/root4/index.txt /etc/ssl/root4/index.txt.attr /etc/ssl/root4/newcerts }}} The actual Certificate Revocation List for each signing key is kept in these files: {{{ /root/CommModule/revoke-root0.crl CAcert original root /root/CommModule/revoke-root1.crl CAcert original class3 subroot /root/CommModule/revoke-root3.crl CAcert new class1 subroot /root/CommModule/revoke-root4.crl CAcert new class3 subroot }}} A delta file mechanism is used to communicate updates of the CRLs to the webdb server. The following additional files are used by this mechanism: {{{ /root/CommModule/currentcrls /root/CommModule/currentcrls/[0134]/* /root/CommModule/delta[0134].diff }}} === PGP signing configuration === Configuration files for PGP signing by the signing server are installed in the /root/CommModule directory. This directory contains the secret and public key rimgs for the CAcert PGP signing entity ''gpg@cacert.org'': {{{ /root/CommModule/secring0.gpg /root/CommModule/pubring0.gpg }}} These files should not be confused with the files in /root/.gnupg, which contains only the public key needed for creating pgp-encrypted system backups. === Disk mirroring === Disk mirroring must be setup as described in [[SystemAdministration/Procedures/DiskMirroring|Disk Mirroring]], to ensure that a usable replacement disk is available in case the primary disk fails. Failover is not automatic, and requires system administrator intervention. === Cron jobs === The following cron jobs are used on the server: * /root/Mirror, to be run every 15 minutes, see [[SystemAdministration/Procedures/DiskMirroring|Disk Mirroring]] * /root/CommModule/logclean.sh, to be run daily or weekly (link to /etc/cron.daily) = Migration to new hardware = When migrating the server to new hardware, the following steps are needed: 1. verify reliable operation of new hardware 1. install base OS on new hardware as described in ''Software Installation'' above; do not forget to use '''new''' passwords for the encrypted file systems! 1. stop operation of signing server on old hardware 1. transfer the following directories from old hardware to new hardware: * /root/CommModule * /etc/ssl This should preferably by done by connecting disks of old hardware to new hardware. If it is necessary to copy the data to an intermediate medium, e.g. USB stick, the medium must be destroyed securely afterwards. 1. ensure that the required cron jobs are enabled 1. ensure that the startup script for the !CommModule is enabled 1. put new hardware in operation 1. decommission disks of old hardware as documented in [[SystemAdministration/Procedures/DriveRetirement|Drive Retirement]] 1. schedule a new off-site backup of the signing server as documented in [[SystemAdministration/Procedures/FullBackupRestore|Full Backup Restore]] = Common Tasks = == Time Synchronisation == The signing server is not attached to the Internet and thus has no way to verify its time externally. But the [[SystemAdministration/Systems/Webdb|webdb server]] has an external time synchronisation mechanism. Therefore a side task of the Comm''''''Module software is to transport current time information from the webdb server to the signing server on every cycle. It stores this time information in a shell script /root/CommModule/timesync.sh on the signing server. The signing server should be inspected at least once every three months, and time synchronisation must be effected manually by the system administrator like this: 1. inspect whether /root/CommModule/timesync.sh is updated regularly 1. inspect whether /root/CommModule/timesync.sh contains correct time 1. issue time synchronization command like this {{{ # date; sh /root/CommModule/timesync.sh; date }}} 1. record the amount of time correction applied and report this Note that the time transmission protocol currently only works when there is no queue of outstanding signing requests. The implication of this is that before running the ''timesync.sh'' script, e.g. after restarting the signing server, one should wait for the queue of outstanding signing requests to be fully processed, and only after that the ''timesync.sh'' file can be relied upon. Hence steps 1 and 2 in the above recipe are mandatory! == Full Backup == Since this system does not have any suitable connections for taking automatic on-line backups, only full offsite backups are kept. The procedure for this is described in [[SystemAdministration/Procedures/FullBackupRestore|Full Backup Restore]]. == Log File Extraction == TBD == Password Changes == Please check [[SystemAdministration/Procedures/PasswordManagement|Password Management]]. == Changelog == All modifications to this system must be logged to the cacert-systemlog mailing list, which is primarily archived [[https://lists.cacert.org/wws/arc/cacert-systemlog|here]]. = Changes Needed = Critical system administration has identified a number of performance / scaling issues in the current implementation which will need to be addressed in the medium term. == Unbounded directory growth == The following directories are growing linearly with the number of certificates issued in each category, since every issued certificate is saved in there: ||directory||purpose||size in June 2009||data in June 2009|| ||/etc/ssl/CA/newcerts||certs issued from original root||7 MB||2 GB|| ||/etc/ssl/class3/newcerts||certs issued from original class3 subroot||1 MB||0.26 GB|| ||/etc/ssl/root3/newcerts||certs issued from new class1 subroot||-||-|| ||/etc/ssl/root4/newcerts||certs issued from new class3 subroot||-||-|| == Incomplete cleanup of work directories == For each signing request, a new work directory is created, whose name is based on an incrementing counter: {{{ /root/CommModule/work/NNN/MMM }}} where NNN = counter / 1000, and MMM = counter % 1000. On completion of the request, an ''unlink'' is done of the work directory. This should be an ''rmdir'' instead, and some checking might be necessary to verify whether the work directory is empty at the end of the work cycle. In June 2009, we found about 115.000 unused work directories lying around on the server. == Critics == BerndEckenfels: First of all, sorry for using this official page, but since Moin has no discussion pages, and I dont really know where to discuss this, let me add some points: ''wytze: no problem, we can have some discussion here'' Bernd: I propose to trust the webdb host less and minimize attack surfaces on the system: 1. For example the timesync should work differently: the signing server maintains it's own clock and compares it to the WebDB's timestamp in the !CommServer protocol. Once the difference gets too big, it refuses to work (and requires adjustment of the clock). It may adjust the time if smaller drifts are present, but I think it is better to not invest the time to make this work. Especially since stepping time while signing is very critical. Maybe it is better to have an external Timestamping Service generate singned timestamps in the CSR packets from webdb to signhost? ''wytze: the above seems to assume that the signing server has more reliable time than the webdb server -- this is normally not the case, the signing server has only its internal clock (which drifts over time), while the webdb server is synchronized through NTP. The only exception would be when the webdb server was attacked to manipulate its time. In the latter case the signing server would be undisturbed, continuing to use its (somewhat drifting) time. With the manual time synchronisation as documented there is no problem -- the sysadmin will check validity and changes are only applied at quiet moments. Stopping the time server automatically on detecting a time difference is NOT an option: it would cause many outages at unpredictable moments and require many expensive visits to the data centre by at least two people.'' ''Bernd: well, I guess the policy should define how critical time skew in the signature is. But yes I agree having a manual sync process is fine. What is contained in timesync, is it using adjtime file?'' 2. The documentation is not really complete, so forgive me when I make falso asumptions, but I think it is a absolute critical violation to run the !CommModule or any of the maintenance jobs as root. ''wytze: Actually, this is not an issue at all on the signing server, because it is totally isolated (with the exception of the serial link). The only real secrets on the signing server are the private keys, and they need to be readable by the !CommModule -- whether it does that running as root or otherwise is immaterial (and one could even argue that running as root reduces the attack surfacce.'' ''Bernd: from my point of few the CommModule does not need to modify system files, and it also does not need to read the most valuable asset on that server. I would run the communication handler as an unpriveledged user and do the signing in a totally different user context. This can be done by a file system or fifo based interface between signer loop and comm module handler. Least priveldge principle.'' 3. You should publish a list of actual installed packages, only. ''wytze: this is a work in progress ... once we have completed the installation of the new signing server, the list of package which 'probably should not be installed' can go away. Please take into account that there is virtually no room for experimentation here -- the signing server is locked up in the data centre and can only be inspected or modified at great cost.'' 4. How are CRL entries generated? (I.e. why do they flow from sign to webdb and not the other way around?) ''wytze: When you want to revoke a certificate, the webdb server will submit the revoke request to the signer, which will update the CRL with a new entry and sign the CRL. Obviously that signature can only be made by the signer. This is the way openssl works, I don't know whether that could be done differently as you suggest.'' ''Bernd: ok thats true. One possible thing would be to only sent hashes to the sign server, but I guess its cleaner to have a full openssl sign operation on this server.'' 5. why dont you publish the tripwire/integrit Hash database monthly for external review ''wytze: First of all: we are not running tripwire or integrit at this moment, but I agree that it is probably a good idea to do so. However, we would not be able to extract the database monthly except by going to the data centre each month and do it manually. That is too expensive. On the other hand, we are looking into a way to get syslog information out of the signer server channeled into a central logging server over a serial cable; this would allow us to get some integrity information out of the signer, and publish it automatically in some fashion.'' ''Bernd: maybe on a bi-monthly schedule or something. I learned it is a good method to catch uncontroled emergency fixes and accidential policy violations.'' ---- CategorySystems