Assurance Quality Assurance (sic!)
Although the system of CAcert Assurance is built to protect against any one failing, there will always be problems. For example, a given Assurer might not know enough or a given assurance might not work out as carefully as normal. It is desirable to find ways to keep the influence and frequency of these anomalies as low as possible.
As CAcert is a heterogeneous community of contributers, there exist different ideas as to how this goal can be achieved. This page collects all the ideas that might arise to ensure the quality of the assurances.
At the moment, a User becomes an Assurer when they have gained 100 points. As this can be done by simply being assured by enough people, this is not a reliable indicator that the new Assurer knows in turn how to conduct the process.
Once a User gets to a certain number of points (100 or close to it), she should sit and pass the AssurerChallenge AssuranceTest. On completion of the AssurerChallenge, and on gaining 100 assurance points, she can become an Assurer.
Pro's: clear conditions, can be fully automated, tests knowledge, well-understood signal
Con's: too static, takes work to keep it fresh, centrally organized, does not test practice
Another view on the AssurerChallenge is to consider it less as a test and more as an interactive education program. The potential Assurer is asked a question, s/he tries to answer it and then the correct answer including an explanation why this answer is correct and the other answers are false is shown to him/her. So ideally s/he should develop a feeling of where the problems are and what to consider.
If s/he feels confident after answering several questions correctly s/he may make the "real" test.
The test can not exclude someone who is willing to cheat. It cannot ensure that everyone passing it is a good Assurer. But it can help those who are willing to become ones. And, it sets a minimum standard or baseline, which is a signal well-understood by the world at large.
Assurance and Experience have been mixed up in the past. From the Assurance Policy (POLICY), there will be two systems for each, being Assurance Points and Experience Points. Assurance points are only given for actual assurances done on the individual; Experience points reflect the number of Assurances an Assurer has done.
Pro's: separation of confusing signals, better incentives, leads way to Member training
Con's: requires changes to system, Assurer loses the easy points, and has to work for them
Delayed Assurance Points
At the moment, every Assurer gets 2 points immediately for each assurance. That means, that the (old) points from 100-150 reflect how many assurances someone already did as a form of experience-level. But if somebody is doing something wrong and nobody corrects the process, the assurer will continue with the bad habits. He will get experienced, but experienced doing it wrong.
Proposal: if you assure somebody with less than 50 points, you won't get immediately 2 points, but 1 point, when this assured person reaches 50 points itself, and another point when the person reaches 100 points. Why? This turns the scale from 100-150 from an pure experience level to a kind of likelihood, that you are a good assurer, because with each point you get, at least one other assurer assured the same person as you.
Pro's: confirms/rewards the quality of assurance, aligns the points to the results.
Con's: some people may never get to 50, so points will never be awarded.
If you want to assure somebody, it sometimes happens that the information in the CAcert database disagrees with the information on your own CAP form. Normally, you would then not complete the assurance process for that User. But sometimes the User has already gained sufficient points! This raises questions as to the quality of the assurance process carried out by prior assurers.
Proposal: Surface the mismatch by putting it into the CAcert DB with a kind of "Complaint" button. Some possible effects for each assurer of pressing the big red button could be:
- lose 2 or 4 (or more) points
- gain 1 point on a separate "bad-assurance scale"
- file a dispute and have the Arbitrator select an action.
(Or all of them.) An Assurer pressing the "Complaint" button has to be able to prove his complaint with the CAP form of course, if somebody asks. If you divide the "bad-assurer scale" by the number of assurances, you get a kind of quality value for each Assurer.
Pro's: eases the contacting of CAcert
Con's: might be abused
User Feedback & Evaluation
Sometimes assurances are done in a rushed fashion or sometimes the User knows that something is wrong. It would be useful to draw the User's observations and knowledge into the process.
Proposal: In order to get the points from an assurance, the assured person gives a rating (like 0-4) to the assurer. This could reflect how careful she/he thinks the assurer has checked her/his identity. The average rating of each assurer has an influence (as an factor e.g.), how much points he/she can give. This can even go down to zero. This is also good for newcomers, who should start with zero, so they have to do some "good" assurances before they can issue points. Based on these values, the new assurer can build up a quality ranking as a motivation.
Pro's: gives Users more to think about, draws them into the process, lets them reveal clear problems
Con's: might be ignored / filled out wrongly, if it starts nerving them, so the quality of the feedback might be worse than expected; there needs to be an independent and convenient way for the user to provide the feedback.
Who can see the feedback? Points could be revealed to the Assurer, comments could be sent to the Quality Assurance department, averages could be posted on the site? How is it processed?
Once the 100 points are gained, there is no real sense that an assurer knows what to do (see Assurer Test above).
Proposal: For each new Assurer, appoint a experienced Senior Assurer as a mentor to guide the Junior Assurer through the first N assurances.
Pros: Improves the web of trust, teaches the tricks of the trade, gives new assurers someone to ask lots of questions.
Cons: Assurers will need to work more in pairs.
Please add further ideas and pro's or con's.