Audit/CommunityReport20090623

The June 2009 report to the Community on Audit progress. This is the final report from myself, Iang. See also: all others. Previous.

0. Termination as Auditor

I resigned as auditor 20090612.
The reasons & motives are here and do not need to be repeated here in this report.
1. there is some speculation that the resignation is over one thing, and we just have to fix that one simple thing. Or one person is to blame and we have to find him and mob him.
2. This is wrong. There were many signs, and all of them pointed in the same direction. If they weren't seen, then that is part of the problem.
It is important to realise that in my opinion, the CAcert community is in good shape.
However, CAcert needs to focus on its number one objective: Audit. It has not done that. Instead, it has been distracted.
The future Auditor:
1. Finding another auditor is not so much of a problem. I know a couple of possibilities.
2. Doing the work to meet audit criteria is a big problem.
3. Indeed, for most of my time as Auditor I did relatively little auditing (and most of that was concentrated in the last 6 months) and a lot of policy preparation and management advice.
4. This core problems remain the case, whatever and whoever the title. Auditing is not hard once the work is done.
5. The tricky part is in knowing what the work is. The boring part is doing it.
6. CAcert should concentrate on that work, and not get distracted on finding and burning another Auditor. The Auditor is not the good fairy with the magic wand, whose presence makes it all happen. Your presence and your actions make it happen.
I strongly suggest that before any auditor is negotiated, CAcert identifies the programme of work that needs to be done. Then, has as fair a shot at doing that. Once a good faith effort has been made, you can then talk to a new auditor, say this is what went wrong last time, this is what we did to fix it, and now we think we are ready.
What to do?
1. There is the AuditToDo list.
2. Ask. I am always willing to suggest what needs to be done.
3. There is also this series of reports tracking the history and the current state!
That said, read on.

1. Systems

Critical Systems recieved one visit before termination.

1.1 Review of Critical Systems

I visited Netherlands during week 20090504-09 for the purpose of "visit #1".
Checks:
1. Physical access control was checked. This was not a drama because I have seen it many times, and not a lot has changed. But the need still existed to check it against a policy.
2. Inventory and cabling was checked. Some things noted, later fixed.
3. Discussion on Key Persons List, waiting on Board to respond.
4. Discussed OS security patching and updates; this is something that requires more attention.
5. reviewed firewall. Doco looks to match situation.
My prediction was that the review over systems would take around 3 visits.
1. Visit #2 was cancelled at extremely short notice, which I regret and apologise for!

1.2. New Roots

I investigated and quizzed the systems team on the location and status of the new roots.
1. No testing is reported.
2. No availability is reported.
In general, for the systems side and subroots, this worked out well. The systems team were able to answer every question.
I also checked up on the status of the top-level root. This was more of an issue:
1. The top-level root was created back in November 2008 (with the sub-roots).
2. The intention was to place them into escrow with notary(s). This did not happen (for probably good reasons).
3. The top-level root is stored in a sealed envelope in a safe, alongside a similar envelope with encryption keys.
This raises an issue of protection of the top-level root.
1. In effect, the top-level root is not protected by encryption nor by dual control.
2. The physical protection is reasonable.
3. There is no particular reason to believe there is a compromise, indeed it is extremely unlikely.
What we might conclude is that there is a paper failure to protect the root. Which is what the audit is about.
I don't think this is as serious as it sounds, this may be one to write off as "experience".
There is no easy way to protect off-line roots, because it by definition creates a physical lump that needs to be accessible at some notice.
There is no real cause for alarm. There is plenty of time to sort this out, due to the other factors.

1.3. Team Participation

a new Systems Administrator has been welcomed to the critical team: welcome to Stefan.
a new Access Engineer has been welcomed to the access team: welcome to Bas.
Join the cacert-sysadm list and hang around there if you can help.
All teams are interested in help!
The new Security Policy describes the background check and process (through Arbitration).
1. is making a difference as there is now a formal process to bring in new people. No longer do we not know what to do, which was a problem that dogged all previous efforts.

1.4 Software

From the last report, we have a finding that the software is uncertain. It should be rewritten from scratch.
1. Obviously this is a huge undertaking.
2. The reason I believe that CAcert can do this is because it already climbed similar mountains.
3. A proposal was kicked around to finance hacking CAcert software at the HAR2009 hacking camp. In short, for lots of reasons, this proposal is probably not going to happen.
Hopefully the software team that was formed at Innsbruck will create a basis for a new software set. I've heard some good signs of progress, but this always takes time.
Since then, some software fixes have been done to the existing software:
1. The CAP forms have now been fixed up to include the magic incancation: "I agree to the CAcert Community Agreement."
2. A patch has been written to add the incantation to the certificate request pages, but this is still wip.
3. Scripts have been written to mailout to Assurers for the ATE process. The challenge is now to turn this ad hoc script into a feature to be added to the system.
4. the further challenge is to "control" the Members who have not agreed as yet to CCA, and do not know about it. Probably by turning the ad hoc script into a mailout to those Members notifying them of the CCA.
Another problem turned up: apparently the software PHP version is no longer supported.

2. Policies, Business

Notification of the CCA to our Members is remains a critical hole.

2.1 The CAcert Community Agreement

Two important rollout steps remain with the CAcert Community Agreement:

Members need to be notified.

Certificate creation needs the "I Agree".

2.2 The CPS

The CPS is in my opinion ready to go to DRAFT.
1. Unfortunately we lack champions to push it through the policy group.
This is one of the symptoms. Asking around for someone to push the CPS did not succeed. Please note, this is not a hard task.
After my resignation, the board supported a motion to pass CPS to DRAFT.
1. I was unsure how to interpret this action at first, but it turns out that this is an experiment based on a good policy group suggestion initiated by Philipp Dunkel. He was responding positively to my comments that policies were too slow.
2. I had forgotten that I even commented on how the Board needs a tool to meet its responsibilities in passing policies.
3. Alejandro has picked up the spirit of Philipp's proposal and pushed the CPS through the Board list.
4. This is good stuff. However I did not see a real consensus on this motion to policy group. For example, Teus thought it not necessary, and the board members could always do it on the policy group.
5. Out of an abundance of caution, because of the Policy on Policy, I suggest policy group now look at passing this exception.

2.2 Security Policy

Security Policy got a bit of a tryout in the visit #1. It worked well.
SP understanding was checked with around half of the Systems Administration and Access Engineer team.
1. (The other team members will understand the expectations here.)
The SP is looking like a success, probably due to the hard work put in by the team to make it their own policy.
1. That team that put in the work on this document: Pat, Philipp Dunkell, Wytze, Teus.
2. Others have helped. Especially, others are helping on the Security Manual, the living face of SP.
Access Engineers are a late incorporation into SP. No problems have been spotted, during visit #1 or other times.
The background check procedure has been done now twice to incorporate two new team members. Welcome to Stefan and Bas.
1. Although I did not verify it, the process is now sufficiently robust and self-governing to be reliable.
One set of ad hoc scripts were run according to Security Policy.
1. This required a combined team from Assurance/Events, Software, Assessment, Arbitration, Systems.
2. They have now figured out how to send emails out in one particular circumstance: an ATE in an area.
3. This is particularly important because the emails were sent out by an ad hoc script under Security Policy
4. In the past, some emails have been sent out bypassing SP, or in absence of SP. That is dangerous, because an Arbitration might be filed, and the Arbitrator might be minded to take the policies seriously.
There is one big hole in SP: that of Application.
1. Early expectations were that the System Administrators manage the whole application.
2. There is an open question as to whether this is plausible. It may be too much work, or it may be too complex.
3. One suggestion from Wytze is that we create a role "Application Engineer" or similar.
4. This role would be responsible for patching the app.
5. Presumably: the access to the systems for managing the app would have to be quite clearly controlled. E.g., users.
6. These are early thoughts.

2.3 Assurance

The Assurance Policy has been put on the website at http://www.cacert.org/policy/AssurancePolicy.php which is its final long-term home.
Exceptions to standard Assurance are now ruled by the full policy, so if there is anything you want to do in the future, get cracking on those subsidiary policies:
1. The Exceptions: TTP, Super-Assurer, Junior, etc. Coming soon to a Policy debate near you.
2. Code-signing: CPS says an Assurer can get code-signing. There is a work-in-progress document on the wiki. I have not looked at it.
3. TTP: the Remote Assurance Policy remains a work-in-progress. It has somewhat stalled as policy group cannot quite agree on how to do it. Possibly it needs fresh eyes.
4. Junior-Assurer and Junion-Member -- Ted and others on policy group have been looking at that, after the miniTOP.
5. TVerify -- there is a work-in-progress policy on this with Guillaume.
6. Super-Assurer -- is dead. However, Board retains the abilty to issue more Experience Points. This should be documented, but it is sufficiently covered in the AP as to not require additional doco.
Assurance has been reviewed.
1. This involved going to as many Assurance Events as I could find.
2. I attended ATEs: Innsbruck, Prague, Budapest, Paris, London and Munich.
3. I also recently attended non-ATE events in Ede, NL and Vienna.
4. In the lifetime of AP, I've also attended non-ATE events in Vienna (again) and San Diego.
5. This gives enough of a statistical sample to draw a conclusion over all Assurance.
At the last date in Munich, 16th May, we held our mini-TOP on Assurance. Very successful meeting.
1. Minutes are posted.
2. This was primarily financed by Audit budget for 432.20.
I presented the review of the Assurance practice over Individuals.
1. See Minutes.
2. Since then, we can add: ATEs in Germany and two in Netherlands. Very successful, very needed to fill in the red.
3. Hint: think about an ATE in your country.
The goal given to Assurance for next year is: Assurance needs to be Self-verifying.
1. The reason for this is that it is too expensive to review normally. Consider the 2300 or so Assurers and the many locations!
2. See the minutes for background.
3. Each of those above tasks feeds directly into Assurance's goal.
To reach this goal, Assurance has a big programme to get through:
1. More systems patches to fix more CCA / Assurance bugs.
2. We need to ATE the whole world, so as to move the information downwards. You can help! Contact events@ to ask how.
3. Formalise the ATE process, essentially documenting and improving the process that was developed by the team.
4. Make mutual assurance *routine* so as to spread the knowledge horizontally.
5. Investigate the idea of Senior Assurer to help with downwards information.
6. Explore ways to limit the Experience Points to a declaration that the Assurance was conducted to AP.
7. We are also exploring how we formalise the statement made by the Assurer, especially over another Assurer, in a way that we can rely upon it for audit. That was hinted above when the co-Auditor makes a CAcert Assurer Reliable Statement.
If Audit were to "opine" (write the opinion) over Assurance, it would probably pass.
1. There are good motives for this approach. Assurance and Systems are two very separate and diverse businesses. They are clearly linked by the Assurance Statement.
2. The Assurance process can now be a role model for other organisations.
3. Potentially with an Audited Assurance, other CAs could make use of the results. This opens up more opportunities for the community, and feeds directly into the mission of helping the members to secure themselves and preserve their privacy.
4. This option was discussed but was not encouraged by the board of CAcert.
what actual positive steps have been taken?
1. Patches have been installed into the system to fix some of the bugs. Thanks to the team. CAP form is now ok. certs are next, then after that, the mailout.
2. Junior Assurer & Member Policy has been started.
ATE has now been formalised by the Assurance team.
1. Co-auditors observe the assurances by being assured.
2. Each Assurer is checked for understanding of Assurance Policy.
3. The Co-auditor works from a sheet of things to check, against each Assurer.
4. At the end of the event, the Co-auditor makes a CAcert Assurer Reliable Statement that the assurances were conducted according to Assurance Policy.
5. This statement may be relied upon by the community and by Audit in much the same that an Assurance and a certificate is relied upon.
6. For those where an Auditor cannot go, we can rely on the report of the Event Organisor. Within this report, there needs to be checks that the Assurance is done according to AP, and a statement to that effect in the report to the Events coordinator.
7. You can help! Contact events@c.o for info on how to run an ATE.
8. There is wip doco on this.
Overall, Assurance is in good shape.
1. The Assurers are moving up to a new plateau of professionalism.
2. The team is formed.
3. The team has a task list, and is working through it.
4. The task list links upwards directly to (Audit) priorities and community needs.
5. A successful rollout is behind them, the wind is in their sales!

2.5 Other Policy Areas

Arbitration recently solved a big problem by creating a new list where all disputes are forwarded to.
1. A backlog had developed.
2. Lack of oversite was causing lack of solution.
3. By forwarding all disputes to a new list with visibility to all Arbitrators, we could then see the scope and nature of the problem.
4. Within days, the backlog was controlled.
5. Many thanks to Nicholas Bebout and Alejandro Mery for making it happen. This is how a community works: Members see problems, think about them, and figure out ways to fix them.
We all welcome Hans to the circle of Arbitrators.
1. Hans is also an Access Engineer, so we have an interesting conflict of interest there.
2. I see no problem with this; as informed observers, this should be a conflict that we can manage and manage well.

3. Audit Thinking

3.1 The Management Assertion

A Management Assertion was written by Philipp Dunkel.
1. In his appointed role as audit liason. It took a couple of months to work through it, even though it is a short document.
2. This was reviewed by Alejandro Mery at Innsbruck and Alejandro proposed to the board.
It received some criticism from Teus Hagen over the liability aspects.
1. His interpretation was discussed with me 20090508. A modified text was proposed to board, and then accepted.
Some context: The management assertion is the subject of Audit.
1. That is, it is what the Auditor looks at and renders an opinion over. If it is in the management assertion, then it should be covered, if not, then not.
2. In theory at least. One major influence is the DRC or David Ross Criteria, which control many of the detailed issues. These are stated as included.
3. Another influence that the Auditor brings in is experience and common sense. For example, DRC does not require one to disclaim liabilities to the general public; that was a common sense thing that I brought in.
4. Then, there are a wide variety of standards and practices for audit and so forth.
5. Some audits are also conducted with the end-reader or customer in mind. In this case, this audit was specifically intended for Mozilla, and the end-users of Mozilla software. They have a policy which was included in our thoughts. For example, validation of email addresses and domains was subject of some controversy and decisions, with Mozilla in mind.
So it is an important document.
1. It is written as a declaration by management. But, it applies to all Members.
2. It is written with an implication of CAcert Inc. Yet it applies to the Community.
3. There may be legalistic differences, but these are meaningless, or should be made meaningless.
4. Community is us, Arbitration is our forum, and we are all in this together. Opinions to the contrary should be treated with skepticism.
That all said, the Managent Assertion is written. Don't worry about it too much for now.

3.2 Assurance

With the completed Assurance review, it was theoretically possible to do an opinion over Assurance of Individuals. Curiously, CAcert's board did not support that idea.
It may be -- this is complete speculation -- that the board hoped for Organisation Assurance as well.
1. This is fantasy. Management and/or the OAs do not have the capacity to deal with OA at an auditable level.
2. There is a list of things that need to be done to fix OA. To my knowledge, none of these have been addressed.
3. OA is a rare beast. We are a community of individuals, where we put in our efforts to secure ourselves. But this natural law breaks completely when talking about organisations. We are 2300 (c.f., CATS) and growing and gaining in confidence. Organisations are none of that, at least that I see.
This whole focus on OA is a search for the perfect and a battle against the good. I see this as a distraction from the audit.
It is often stated that we cannot survive without assured organisations to help us.
1. Nonsense: I've met a hundred individuals that have helped us greatly. The individuals got us to where we are today.
2. I've met a few organisations that have helped us: Paul Data. BIT. Tunix. HCC. But, it was really the individuals in the organisations that helped us. IMHO.

3.3. Outstanding Problems for CAcert to look at

Offline (disaster recovery) Backups look problematic:
1. accessibility is unsure.
2. they are snapshots, not dynamic (no updates).
3. there is an online backup in the same place but for DR this is ruled out.
4. although we did not get a chance to work through them all, I do not have the feeling that the disaster recovery aspects of the backup system would survive audit.
5. a further problem is that the current model assumes an in-country approach. This is not good for an international group. Juridical attacks are quite common in some quarters of the world, and CAcert has to take its data seriously.
6. Backups has been before the board.
the whole disaster recovery issue has been indicated to the board, but no attention granted as far as I know.
1. Disaster recovery needs a Key Persons List. Has been before the board.
Systems access and Application access.
1. There is a lack of confidence in the updating of software / patching.
2. Our earlier thoughts were that the Systems Administrators would do the patching. This is being re-thought.
3. One view is that the Applications Engineer (or approximately the Software Assessment team) do this job.
4. This will require: tightly controlled access to the application, and reworking of the Security Policy.
5. Unfortunately this debate and work were not completed. This is a big outstanding issue.
6. This has not been discussed with the board.
While not a problem, the teams have little "fat" and cannot survive lean periods. All technical teams need more people. IMO.
Access Engineers are not really formed into a proper team, per se. There is confusion as to who is the leader, who is the coordinator, who is responsible.
Support.
1. We desperately need more support engineers. I know work is going on in this area.
2. To some extent, I can be blamed for this because I frequently said that we have to concentrate on systems administration and software, and leave support until later.
3. Unfortunately this was wrong because support is draining those very areas. So we need more support to ease the pressure.
4. Recently, a lot of Arbitration was released and unblocked by some simple administrative changes. This looks like good work, thanks to Alejandro.
Softare.
1. we need a new software system and a new team. This is a big project.
2. the current system is not confidence-inspiring. If you want to pass audit, you need to be confidence inspiring.
3. Thanks to a few brave efforts, there are now some patches making their way into the system. This is good, we can knock off the critical things, one by one, and keep the business alive. Until the new system comes online.
4. more work on patches is needed.
5. Some work is required to fix problems in software that isn't right-now-critical: multiple email checks and multiple names.
root keys are an issue.
1. protection is an issue.
2. Documentation is short.
3. No testing has been done.
4. It probably needs a re-think. It has been indicated to the board.

3.3.a DPA / Access Engineers

DPA issues are being worked on in Netherlands.
It is suggested that Oophaga becomes the responsible party.
1. this is probably a good use of Oophaga.
This has the drawback that it clashes with the security model of CAcert
1. Security Model relies upon the principle that the Access Engineers must not have access to the data.
2. This only works if there is no plausible back-door.
One can make comments that Oophaga do not need the data or do not want the data. This misses the point; they can get the data.
1. It's the law. What are you going to say against that? Which person in CAcert is ready to stand up and argue that point?
2. This places too much of a strain on the people. They can and will breach the Security Policy under these conditions. A Policy, and a law, is only as strong as the reasonableness contained in its foundations.
Which becomes a problem when the Access Engineers can quietly go in and provide the data.
It has been commented that the DPA proposal is being prepared by knowledgeable lawyers, and is entirely reasonable under the law. That also misses the point: of course lawyers are interested primarily in being inside the law.
1. There remains however a business, and the business of CAcert is security and privacy of the members.
2. Does this proposal consider the business?
In effect, there is now no longer a strong control against this weakness. Arbitration will not be able to protect against this, nor dual control, nor Security Policy.
One way to fix this is to move the Access Engineers move from Oophaga to CAcert.
1. That is, the team reports to the Board of CAcert, not the Board of Oophaga.
2. Access Engineers are already under CAcert's policies for working practices.
3. (We should also think about a team leader for the Access Engineers.)
4. This takes the technical capability away from Oophaga to access the data.
5. Which means that Arbitration can oversight any requests, and deal with them. Governance is restored.
6. Governance is restored. Any access to data would have to be resolved, and the privacy and security of the data can be ensured.
Of course we have to see the final proposal to the board before anything can be done by CAcert, or the proposal can be accepted.

3.4 Outstanding Work

In order to close out this audit cycle:

Publish this last report.
Submit some final expenses, around 200 euros.
Maybe review the DRC and bring them up to date. That's a distinct maybe, it depends on whether it is worthwhile.

4. Admin

4.1 Budget

Audit budget expenses snafu. During the last month, an audit budget problem turned up:
1. Additional costs were added without notification to myself.
2. Almost certainly these were valid expenses for CAcert.
3. But: managing budgets requires knowing about them. (Although I did not agree with this approach, CAcert's board insisted that I manage the Audit project, which means managing the budget, which means knowing all the transactions.)
4. Therefore, and considering that Audit's budget was already running short, the reasonable solution was to transfer these costs outside Audit completely.
5. There is no board decision on this issue to my knowledge.
Audit Budget / retainer.
1. I opened discussions on the issue of CAcert's lateness versus fixed retainers. The retainers are important to me as a source of real income.
2. The position of the board was that the only acceptable option was to seek additional funding from NLnet.
3. I advised against that position because (a) funding always takes time, (b) funding is risky, and (c) NLnet already rejected a similar proposal, citing the exisitng funding.
4. I instead advised to either add a phase or tranche, or negotiate a modification of milestone. These options were not accepted by the board.
5. The board did negotiate an 11th hour boost of new 'expenses' money from NLnet, but declined to address the 'retainer' issue.
Audit Budget Costs / Expenses.
1. As reported last time, Audit expenses were running towards the brick wall.
2. Costs would have covered Systems Visit #2, but not the anticipated Systems Visit #3.
3. Some juggling is required to move some costs from "expenses" to "work" subaccounts. I propose to move ATE work, as this is really work that we have (later) decided will be CAcert's responsibility to future Audits.
Current situation is summarised at AuditBudget.
1. This spreadsheet has been up there since New Year and is a fairly good guide.
2. It's a little out of date, I'll get to that soon.
3. More detailed is the internal spreadsheet, but that is huge.

5. Conclusion

As mentioned, this is the last report from myself.
It is longer, primarily because the last 2 months were packed solid with important new events.
CAcert achieved great things with the ATE rollout and the slow-but-steady rise of the critical systems team.
However, audit required many other things to be done, and these are properly before the board.
1. Only the CPS has received some attention.

Previous. all.

CategoryAudit

Audit/CommunityReport20090623 (last edited 2009-10-11 15:45:09 by UlrichSchroeter)