Incident Reports

This wiki page contains security incident reports in chronological order.

1. 20140407 Heartbleed OpenSSL.

Overview of actions

2014-04-07 23:19 CEST


iang reports Heartbleed bug on cacert-sysadm mailing list

2014-04-08 0:10 CEST


CAcert critical team checks OpenSSL version on webdb and signer and confirms that these critical machines are not vulnerable

2014-04-08 0:20 CEST


golem reports about bug

2014-04-08 0:36 CEST


CAcert personnel gets aware that something was up with OpenSSL

2014-04-08 3:30 CEST


start to seriously discuss the issue

2014-04-08 3:50 CEST


As it was already checked that the root key could not have been affected and the central systems were safe as well, it was agreed that a double tracked strategy should be followed, as two things were likewise important:

  1. recovery strategy for CAcert systems

    • investigate what infrastructure systems were affected

    • update them

    • get new keys for them

  2. informing our members

    • to prepare information about root keys being safe, but member probably were likely to be affected, anyway, and what the status on our systems is

    • via blog post

    • via mail to all members (this was later narrowed down)

    • for this the authorization of an Arbitrator was needed

    • maybe other information strategies

2014-04-08 8:50


At this time already was done:

2014-04-08 10:50


CAcert critical team reports detailed versions of OpenSSL for all CAcert critical machines and confirms that they are not vulnerable

2014-04-09 10:45


At this time was already done

This was mostly done by a handful of people - during their free time (at night).


(all times of when systems were fixed should be added)

2014-04-07 23:19

bug reported on cacert-sysadm by iang

2014-04-08 00:10

critical team confirms that webdb and signer are not vulnerable

2014-04-08 00:20

bug is announced by golem

2014-04-08 00:36

CAcert personnel gets aware that something was up with OpenSSL

2014-04-08 ???

persons starte to take it as something possibly serious, and to look at "their" systems

2014-04-08 3:30

VoIP session initiated to discuss the issue.

2014-04-08 3:50

decisions on reaction strategy

2014-04-08 ???

contact with critical, verification about status of central systems

2014-04-08 4:10

dispute send to arbitration

2014-04-08 5:00 fixed, but no new cert

2014-04-08 6:05

arbitration case a20140408.1 (previously transfered to Wiki) picked up by A and started

2014-04-08 7:00

internal auditor is informed

2014-04-08 7:25

ruling in arbitration case to inform members with possibly affected server certificates or had the "general announcement flag" set via scripted mail (goes to software, critical, board, support)

2014-04-08 7:45

support team provided with more details

2014-04-08 8:00

mail to PR team lead

2014-04-08 8:10

blog post is send

2014-04-08 8:10

tweet sent

2014-04-08 8:20

infrastructure team for systems not covered by present persons are informed and asked to also upgrade their systems

2014-04-08 8:45

mail an

2014-04-08 10:20 informed with german post

2014-04-08 10:45

status report to open board list

2014-04-08 10:50

critical team confirms that none of critical systems are vulnerable

2014-04-08 17:30

bug 1265 for scripted mails to members

2014-04-08 17:45

coding started

2014-04-09 5:30

first review for script, transfered to testsystem

2014-04-09 5:35

script started on testsystem

2014-04-09 6:45

press release send out

2014-04-09 8:40

first test report for script (was updated later)

2014-04-09 9:35

second review for script

2014-04-09 9:35

second test for script

2014-04-09 9:45

changes to text approved by CM of a20140408.1

2014-04-09 10:00

mail from software to critical to execute script referencing a20140408.1

2014-04-09 10:45

A of a20140408.1 approves execution request (with reference to ruling)

2014-04-09 10:45

script executed on production system, mails are going out

2014-04-09 21:00 receives new cert

2. 20140828 CAcert website showing incomplete names

Overview of actions

2014-08-28 17:02:53 CEST

CAcert critical sysadmin team completes upgrade of CAcert chroot application environment to Debian Wheezy on the main webserver, after having installed the patch for, deemed to be the last blocking issue for this upgrade. The webserver is started in the new upgraded environment, but a backup of the old environment is kept online just in case.

2014-08-28 18:10:01 CEST

A well-known CAcert user sends e-mail to, reporting that the last name and username in the account of this user are showing up as empty. The user is requesting clarification, and is hinting that the problem might have to do with the recent change by the critical sysadmin team.

2014-08-28 around 20:00 CEST opens case s20140828.90 - Last name gone

2014-08-28 20:19 CEST tries to phone critical sysadmin teamleader on his mobile phone, but gets no response. A voice mail is left. (Unfortunately, no attempt was made to phone the critical sysadmin teamleader on his landline -- which would have resulted in a response in this case --, or to phone another critical sysadmin team member).

2014-08-28 20:30 CEST

Support sends e-mail to stating "we do have a problem with the database. All entries with special characters are no visible.", and including a copy of the e-mail of the original problem reporter.

2014-08-29 08:41 CEST

Critical sysadmin teamleader checks mobile phone voice mail, and starts investigating the reported problem. His initial suspicion is that some php5 environment settings concerning character encoding may have changed between the old and new CAcert chroot application environment. After some tests on the cacert2 test server (which has a copy of the production software environment), this turns out not quite to be the case. But further investigation reveals that the php5 function htlmentities() has different behaviour between PHP 5.3 (used in the old environment) and PHP 5.4 (used in the new environment) when its "encoding" parameter is left unspecified. This function is used by the CAcert application function sanitizeHTML(). It turns out that invoking this function with a string containing non-ascii characters results in an empty string as the return value.

2014-08-29 09:46:41 CEST

Critical team shuts down the main webserver, and re-activates the old CAcert chroot application environment which was a kept as a backup after the upgrade on 2014-08-28.

2014-08-28 09:46:54 CEST

Main webserver is running again with Debian Squeeze-based CAcert chroot application environment. The "dropping names" issue is resolved.

2014-08-29 10:11

Critical team reports bug #1301 to software development, including a detailed analysis of the problem, and including a suggested bug fix (i.e. explicitly specifying the "decoding" parameter for htmlentities()).

2014-08-29 later that day

Problem is picked up by software development. Further progress can be monitored in

Impact analysis

From 2014-08-28 17:02:53 CEST until 2014-08-29 09:41 CEST the production server has been running with a software configuration causing incomplete names to be displayed to website users. Two groups of users were affected by this:

  1. users with non-ascii characters in one or more of their name fields
  2. users trying to assure a user in group #1

For the first group of users we can state that the only damage done will have been some confusion for the user -- suddenly seeing (part of) his/her name dropped makes a bad impression. However it cannot result in bad values being recorded in the CAcert database. Note that only the display of these names was affected, the name values used internally were not affected at all.

For the second group of users, the issue can be more severe: an assurer may have been presented with an incomplete name of the user which he/she was attempting to assure. In the worst case this may have caused an assurer to refuse to perform such an assurance. We have investigated how many assurances have actually been recorded in the system in the vulnerable time period. This resulted in the identification of some 11 assurances. For these assurances the true names have been checked. Fortunately, none of these true names contained any non-ascii characters, so they were not affected by the bug.

Summarizing we think that we can conclude that the integrity of the CAcert database, and particularly the assurance part, has not been damaged by this incident.