* Case Number: a20140408.1 * Status: closed * Claimant: Michael T (as Organization Admin for CAcert) * Claimant2: Ron C (joined into case at 2014-04-11) * Respondent: CAcert * initial Case Manager: MartinGummi * Case Manager: MartinGummi * Arbitrator: EvaStöwe * Date of arbitration start: 2014-04-08 * Date of ruling: 2014-04-08 * Case closed: 2016-06-30 * Complaint: Notify members of Heartbleed * Relief: allow a mass mail to be send to inform affected members about heartbleed Before: Arbitrator EvaStöwe (A), Respondent: CAcert (R), Claimant: Michael T (C) Ron C (C2), Case: a20140408.1 The dispute of C2 was joined into this case at 2014-04-11 because it was a response to the mass-mail which was send to inform the members based on the ruling. == History Log == . 2014-04-08 (issue.c.o): case [s20140408.5] . 2014-04-08 (iCM): added to wiki, request for CM / A . 2014-04-08 (A): I'll take care about this case as A and MartinGummi will be CM . 2014-04-08 (A): send init mail to C . 2014-04-08 (A): informed internal auditor about the bug, affection of CAcert, done and planned actions and the case (with the comment that it was done under emergency conditions and may have been flaws . 2014-04-08 (A): ruling sent to C (also as software TL), critical admins, support, board(private) and CM . 2014-04-08 (CM): contacted internal-support-list with further informations . 2014-04-08 (A, CM, C, and two other persons): prepared a blog post that was posted by the CM . 2014-04-08 (A): contacted infrastructure personal that was not already working on fixing the system (not posting as A) . 2014-04-08 (A): notified cacert@lists.cacertorg (not posting as arbitrator to not confuse people even more) . 2014-04-08 (svn-admin, A, CM): mail exchange that SVN was fixed, svn fingerprint posted on blog (not explicitly posting as A/CM) . (infrastructure): thank you for checking some servers; updated openssl packages on SVM, created new private key for Apache httpd, issued new cert; restarted the SVN container, so that ssl binaries are used; updated openssl packages on infrastructure host . (A): thanks (infrastructure) for fast reaction, asks for fingerprint of new certificate for blog post . (infrastructure): provides fingerprints . 2014-04-08 (A): informs cacert-de@lists.cacertorg (not naming role of arbitrator to not confuse people even more) . 2014-04-08 (A): infomred board about actions taken . 2014-04-08 (CM): posted link to blog on cacert irc channels . 2014-04-08 (A, infrastructure admin): conversation about details for actions required, CATS, IRC, Mailinglist fixed (not posting as arbitrator) . (infrastructure): asks for advise on special/non-default openssl parameters/config to use; considers to alter used key size for new certs . (A): provides configuration proposal per external chat . (infrastructure): finished with CATS, IRC, Mailinglists-server, sends CSRs; Question about tests on testserver . (A): Thanks (infrastructure), answers question to test server . 2014-04-08 (A,CM): provide mail-template to software team (pad) . 2014-04-08/09 (software): adjust, review, test script - mail-template was changed a little bit, with ok from CM (see bugtracker bug 1265) . 2014-04-09 (software): provides script to criticals with execution request . 2014-04-09 (A): approves execution request . 2014-04-09 (critical): execution of script started . 2014-04-09 (ciritcal): reports rejection of some mail because spam-filters considering one of the mail addresses mentioned in the mail as too nee . 2014-04-09 (DRO): asks if there were any emergency actions done not covered by an arbitration case . 2014-04-09 (A): to DRO as far as known there were not . 2014-04-09 (DRO): wants to take care that everything is done via process, in correct roles and docuemtned correctly . 2014-04-09 (A): confirms to DRO, that a lot of care is taken that everything is done via process and overviewed by arbitration, everybody tries to document as much as possible, but main focus was to act quickly, as much as possible was noted for documentation; explains further details about who acted in what roles . 2014-04-10 (critical): after script running for 28, only half the mails send, increased delay of sending, as server was hardly idel, delivery problems is at 20% . 2014-04-10 (A): informs internal Auditor about sending of mails (by forwarding earlier mail from crit) . 2014-04-11 (critical): reports completion of sending of mails . 2014-04-11 (C2): files "second dispute" . 2014-04-11 (A): informs other arbitrators about intention to merge new dispute into a20140408.1 . 2014-04-11 (A): picks up case, as no other arbitrator objected . 2014-04-11 (A): init mail including information about merge of the case send to C2 . 2014-04-11 (A): informs C about merge and new claiant . 2014-04-11 (C2): clarifies that he doesn't intent to file a lawsuit but that he believes other would do so . 2014-04-11 (A): thanks C2 for clarification, explains backgrounds of mail and further details . 2014-04-12 (C): asks about other other claimant and his intentions . 2014-04-12 (A): answers C via voice-channel . 2014-04-13 (board): issues motion [[https://community.cacert.org/board/motions.php?motion=|m20140413.3]] to thank all involved members for activities to fix the heartbleed issue . 2014-04-16 (A): asks (critical) how much returns there were because of running into spam filters [could be relevant for comparable situations] . 2014-04-21 (critical): hard to give exact value, but less than 10% , cause was mostly none existing adresses or mis-configured servers or thelike . 2014-05-01 (A): asks critical and infrastructure team for further details for documentation of activities . 2014-05-03 (critical): answers questions . 2016-06-23 (A): thanks involved members . 2016-06-30 (A): closed case == Private Part == * '''Link to Arbitration case [[Arbitrations/priv/a20140408.1|a20140408.1 (Private Part)]], Access for (CM) + (A) only''' ## ==> INCLUDE SECTION BOT <> ## <== INCLUDE SECTION EOT ==== EOT Private Part ==== == original Dispute == {{{ Hi, Just now the Heartbleed attack was discovered http://heartbleed.com/. This a) affects some of our infrastructure systems so users of those should know that they might need to change their credentials b) probably affects a lot of our server certificate users so they should be notified how they should react (also because they need to comply with the CCA and protect their keys, so we should inform them how they can do that) I hereby file dispute against CAcert to notify members of the potential issue. I accepted DRP and CCA. }}} === second dispute filed by C2 recieved and merged into this case at 2014-04-11 === {{{ > Expect a lawsuit. }}} This was a response to the mail send in this case. This "dispute" resolved itself easily later. The claimant had no plans to file a lawsuit but wanted us to warn that others may file a lawsuit. No further discovery, action or decision necessary, here. == Discovery == A critical bug within openssl (heartbleed) was discovered. The critical systems from CAcert were not affected. But some of the infrastructure servers were affected. For those servers the openssl-version and the certificates have to be replaced. Also a lot of our users could be affected. As our principles state that we care about the security of our members, CAcert should inform the possibly affected members about * the bug itself * that it did not affect our critical server * which of our servers wer affected * that they themselves may be affected * how to resolve the issue on their own servers Beside of placing this information internally at many places (different mailing lists, blog, PR-post), this is critical to send mails to the primary email addresses of the members. This should be done for all possibly affected members, which are all who have issued a server certificate in the time frame where the bug was within openssl. There are also members who flagged their account with a request to be informed about general topics. This topic is critical enough so that those members also should be informed, as they quite likely at least want to know that our critical systems were not affected. == Ruling == {{{ I hereby come to the following ruling: * Software team should provide a script that allows to send a mail to all current members, who have either have had an active server-certificate since 2011-12-01 or have activated the "general announcement"-flag. (This is not issued since then, but active at this date or later.) * The arbitrator or the case manager should take care, that a mail-template to inform those members about the Heartbleed Bug and in what regard they may be affected, is provided. * As soon as both (script and mail-template) are available they should be send to critical team, which should execute the script to send the mail to said users. * Support should be aware, that there may be a lot of mails and returns because of this action. 2014-04-08, Cologne }}} == Execution == . 2014-04-08 (A): ruling sent to C (also as software TL), critical admins, support, board(private) and CM . 2014-04-08 (CM): contacted internal-support-list with further informations . 2014-04-08 (A,CM): provide mail-template to software team (pad) . 2014-04-08/09 (software): adjust, review, test script - mail-template was changed a little bit, with ok from CM (see bugtracker bug 1265) . 2014-04-09 (software): provides script to criticals with execution request . 2014-04-09 (A): approves execution request . 2014-04-09 (critical): execution started . 2014-04-09 (ciritcal): reports rejection of some mail because spam-filters considering one of the mail addresses mentioned in the mail as too nee (needs to be updated, mail.sending comleted) == Execution details == Mostly documented in history log. Further information can be found at: * [[https://blog.cacert.org/2014/04/openssl-heartbleed-bug/|blog post]] including English, German version. (Comparable to text approved by A/CM later) * [[https://lists.cacert.org/wws/arc/cacert-board/2014-04/msg00011.html|report to board]] * [[https://bugs.cacert.org/view.php?id=1265|bug-1265]] bugtracker entry for mail-script * [[https://blog.cacert.org/wp-content/uploads/2014/04/CAcert-PM-Heartbleed-en.pdf|CAcert-PM-Heartbleed-EN.pdf]] press release English * Thank you motion [[https://community.cacert.org/board/motions.php?motion=|m20140413.3]] from board [[SecurityManual/IncidentReports|Further details of emergency action and other links in this context]] The script is posted in the following mail, this includes the texts which were send: [[https://lists.cacert.org/wws/arc/cacert-systemlog/2014-04/msg00003.html|Mail from cacert-systemlog]] === Request for executing mail script from software assessor team to critical team === {{{ Dear critical, Please execute the ruling of a20140408.1 and execute the script scripts/send_heartbleed.php in branch bug-1265 (Revision e5c83c7a2d97b5990bc1ebc5c941638a33233dce). Cf. https://git.cacert.org/gitweb/?p=cacert-devel.git;a=blob;f=scripts/send_heartbleed.php;h=6bf0f5f8fee63bbd66cccc50975162f704d5f1f2;hb=e5c83c7a2d97b5990bc1ebc5c941638a33233dce Report results and status updates in the bugtracker at https://bugs.cacert.org/view.php?id=1265 Please perform this operation ASAP. }}} === Report from critical team about mail sending === {{{ The script has been running from April 9, 10:45 until April 10, 18:37 CEST. A total of 168977 messages has been sent out, for a total userid base of 290146 entries. According to the postfix mail statistics, a total of 170213 e-mails were sent during this period (including regular webdb service mails). For 22414 e-mails out of these delivery problems were reported. At this moment (April 11, 11:30 CEST) there are still some 3700 e-mails queued for possible delivery later (the regular queue size is more like 50 - 100 e-mails). For future mass-mailings like this, it is recommended to increase the sending rate from 1 msg/second to 10 msg/second, so the available server resources are used better and the mailing can complete within a number of hours rather than days. We applied this change for this mail shot after 28 hours (during which around 99000 msgs were sent out), and after that, the remaining shot of some 70.000 mails completed within 4 hours. }}} * further about mailng issues {{{ It's difficult to give an exact percentage, but from browsing through the error messages it looks like spam filtering at the mail server level is only a minor cause (less than 10%) for the delivery failures we've experienced. Much more common are mail addresses which don't exist anymore, mis-configured mail servers and similar issues. Greylisting has contributed a bit to slowing down the message delivery, but it's not a real problem. }}} == Similiar Cases == none known ---- . CategoryArbitration . CategoryArbCaseSystemTasks