Team Reports 2015

Team Leaders are encouraged to present a report for their team. (alphabetic order)





Audit Team

In FY 2014/2015 the audit team has undertaken the following actions:

Incident Types in FY 2014 / 2015



Process Fault


Data Privacy Breach


To build a strong audit capability, further internal Auditors are welcome to participate in the Audit Team.

Benedikt Heintel
CAcert Lead Auditor

Critical System Administrator Team Report July 2014 - June 2015

Hardware changes

After the many hardware changes made in the previous reporting year, no further changes were necessary in the past year. There were no failing components, and only one site visit was performed.

On-site activity

The log of visits to the hosting facility shows the following "on site" activities:

Actually, the firewall OS upgrade was done from outside the server room, and might as well have been performed from home, but since it was the first time we did this, we decided to limit the risks by doing it this way.

The total number of visits (1) was much lower than the number in previous years (7 in 2013/2014, and also 7 in 2012/2013), which shows the relative stability of the critical systems.

The equipment is getting older and older though, so we shouldn't be surprised to see more failures / visits in the future due to hardware problems.

Off-site activity

All other (i.e. most!) system administration work has been performed remotely. Issues directly affecting the operation of the webdb server continue to be logged to the mailing list (archived at ) with headings like "configuration change webdb server", "security upgrades webdb server" or " checkin notification". This logging is also used for changes to all other services like DNS, OCSP etc. under critical-admin management. A total of 187 messages were posted on this mailing list during the year.

Webdb server

While the upgrade of the main server environment from Debian "Squeeze" to "Wheezy" could be performed in the previous reporting year, the upgrade of the chroot environment in which the CAcert application code is running to the same OS level, was delayed until August 28, 2014. This was due to some issues with the CAcert application code which had to be resolved by the Software Assessment team first.

In the mean time, it is time again for another OS upgrade, as Debian "Wheezy" has gone to "oldstable" status, while "Jessy" is the current Debian stable release. We expect to be able to do this upgrade later this year.

Other maintenance work on the webdb server during the reporting period involved:

thus making a total of 88 critical admin interventions for this server (previous year: 97).

DNS service

The DNS service has been continued in more or less the same configuration as the previous year. One noteworthy change was the addition of a TLSA record for and This supports effective use of the DNSSEC/TLSA Validator browser plugin available from CZ.NIC Labs.

Maintenance activities for this server boiled down to:

thus making a total of 19 critical admin interventions for this server (previous year: 46).

OCSP service

The OCSP service has been continued in more or less the same configuration as the previous year.

Maintenance activities for the OCSP service boiled down to:

thus making a total of 6 critical admin interventions for this server (previous year: 7).

CRL service

The CRL service has been continued in more or less the same fashion as the previous year.

Maintenance activities for the CRL service boiled down to:

this making a total of 4 critical admin intervention for this server.

Last year we reported that the availability of the CRL service had been stabilized during that year. While this is still true, we want to repeat the driving negative factors here:

Note that we are routinely pushing out over 150 GB of data *per day* from just this server. This is an artificial limit, imposed by the the bandwidth that has been donated to CAcert by BIT. Without this bandwidth limit, the amount of data pushed out would grow even worse. While our server can handle this, CAcert does not have the finances to pay for such an extravagant usage of bandwidth.

The most fundamental method of attacking the problem is still open though, as it entails some fundamental changes in the operation of the CAcert signing server:

Backup service

The boxbackup server has also been continued unchanged, with maintenance activities consisting of a number of smaller interventions:

thus making a total of 53 critical admin interventions for this server (previous year: 22).


The new dual firewall has performed flawlessly during the reporting year. One major software upgrade was performed in November 2014, by upgrading the OS on both firewalls to OpenBSD 5.6. Thanks to the dual firewall design, this upgrade could be performed with zero downtime for CAcert users and services.

Maintenance of the firewall has boiled down to three key components:

thus making a total of 17 critical admin interventions besides the OS upgrade for the new firewall


Our primary external monitoring remained based on the use of a private server of a critical team member, with the limitations implied by that. A new offer made for a well-connected VM has not materialized yet.

Infrastructure support

Some support has been given to the infrastructure team for enabling IPv6 support.

Software Assessment Team support

We continued to support the Software Assessment Team by maintaining a test server (on a virtual machine) which looks as closely as possible to the production webdb server. A second similar test server is also maintained for special critical system tests and preparation of major software upgrades. This second test server is normally the first one to be upgraded when new OS upgrades or security patches are to be installed.

The patch process developed by the Software Assessment Team has resulted again in a significant number (30) of successful patch updates to the production server (previous year: 58).

Events team support

From time to time the events team wants to inform CAcert members about important events like Assurer Training Events and the like. These mailings are performed by adding a custom script to the webdb server and running it against the current database. Based on arbitration, such scripts are prepared by the events team and handed over to the critical admin team for installation and execution. 5 cases were handled in the past year.

One huge mailing was also executed by the critical admin team, for informing the CAcert membership about the CCA policy changes:

Some smaller mailings were performed as wll with regard to organizational assurers and an incorrect CCA being displayed on the website for some time.

Interaction with other teams

From time to time the critical admin team also receives requests from other CAcert teams like Support and Arbitration, which we try to handle as quickly as possible. The total number of e-mails processed or generated by the critical admin team during the reporting year amounts to around 1000.

Team changes

There were no team changes in the past year.


Plans for the coming year include:

Wytze van der Raay, Mendel Mobach, Martin Simons
Critical System Administrator Team


Management of CATS and the Assurer Challenge

The past year also saw a very stable and essentially unchanged productive CATS. At least the french language for the user interface has now been activated, though there's still some discussion going on about improvements in the corresponding bugtracker case.

On the development system the user interface and the Assurer Challenge have been translated (at least to a good percentage) to czech language. The user interface translation has still to be activated (one of the things quite on top of my ToDo list), and user interface, as well as Assurer Challenge need reviews by others before they can be transferred to the production system.

Translating the Assurer Challenge to czech required some changes in the software, since czech language uses quite some characters not covered by the ISO-8859-1 character set, which currently used by CATS. The quick solution was to use HTML encoded unicode to store the questionaire data. This was easy to implement and should allow comperatively easy migration, once software and database are ready to handle unicode natively.

Since early summer this year, issuing of CATS certificates has grinded to halt, because the very limited amount of manpower in support currently is needed for more important jobs than verifying Assurer status of the applicants. An interface to the main database that allows verification of Assurer status is (AFAIK) still not implemented.

Several new tests, variants of a "Data Privacy Quiz", have been created on the testserver, but still contain only a handful of questions.

The usual statistics, from July 2014 to June 2015:

So, the number of tests (started and passed) dropped by about 30% compared to last year, but are stil above the level of 2013. Probably the main reason is the lower number of new Assurers which can be seen in the CAcert statistics.

The number of users who tried at least once but never passed the Challenge has increased (in absolute numbers!), but is still small compared to the users that passed. This needs some monitoring, but I won't consider it as alarming yet.

Plans for next year

Bernhard Fröhlich



Early this year the Infrastructure Team lead changed from Mario Lipinski to Jan Dittberner, Board approved this change in January. Thanks to Mario for all his work in the past.

Jan did a major rework of the documentation in the Wiki in February and wrote an announcement to the system administrators lists about this work in progress. During the work on this task it became obvious that many systems are in an outdated state and lacking proper documentation. This has not changed very much yet.

Unfortunatelly Jan did not find the time to continue the tasks announced in February, namely replacing the old email and webmail containers, sending a infrastructure team census and enabling DKIM/DMARC.

The team would need more people with enough time to maintain all systems in a professional way.

New Root & Escrow Project (NRE)

Organisation Assurance Team

Policy Group

For your information:

During the second part of 2014 all policies were set to policy status. During the remaining part of the business year, multiple policies were discussed, including PoP changes, a privacy web policy, appeals and relations of officer roles. Non of them reached a status where a policy was changed.


For your information:

Alexander Bahlo, Officer for Public Relations, was recalled by board on January, 11th, 2015 with motion m20150111.3. Unfortunately board did not give any reason why having decided like this. Also, there was no discussion between board and Alexander before this motion, so this decision came very sudden - especially because the communication of this motion was done only by january 30th, 2015 21:19 CET. On this week-end the FOSDEM in Brussel took place, where I (Alexander) participated, so I couldn't get this information before the end of the event. While other board members and involved people were attending as well, nobody talked to me (Alexander) about this motion. While this is, of course, totally legal, this is no good behaviour.

Software Development Team

Support Team