CAcert Regression Test
We are currently building up a regression test system, to automatically test the functionality of the CAcert website.
Regression tests against the production system must be authorized by at least 2 people from the core-team, and monitored by at least one core-team member. Regression code that runs against the production system must not be commited to the SVN, or otherwise publizised
There is a new regression testing system that we should try out: Selenium
Where to start?
You can get our current regression tests from a subversion repostory at: http://svn.cacert.org/CAcert/Regression/
A live site running the tests software is available at http://test.chost.de/Regression/ - username and password is both cacert.
The tests are currently not testing the main CAcert website but https://www.test1.cacert.at/
Several tests have been written so far. The testing framework we use is SimpleTest. Basically a test is a script simulating the user/admin/etc input to CAcert web site and checking the results.
Please protect your test installation and framework from indexing by a search engine. You can do this by using a "robots.txt" in your web directory.
Tests Already Implemented
- front page available via http/https?
- contact page available via http/https?
- Root Cert fingerprints correct via http/https?
- first line of Root Cert (PEM) correct via http/https?
- (Changing language to de_DE possibe via http/https?) not testable testserver
- is wiki.cacert.org online?
- is blog.cacert.org online?
- is bugs.cacert.org online?
- logon with empty password possible?
- logon with empty email address possible?
- logon with wrong password possible?
- normal logon possible?
- logout possible?
- lost password page available?
- empty email and date of birth catched?
- empty date of birth catched?
- lost password step 2 available?
- empty lost password answers catched?
- correct answers but no new password catched?
- setting new password using lost password questions possible?
- empty password detected?
- missing lost password questions/answers detected?
- missing date of birth detected?
- missing email detected?
- missing last 2 lost password questions/answers detected?
- simple passwords detected?
- different passwords detected?
- using already registered emails addresses possible?
- joining possible?
- login possible? (link in verification email is automatically followed)
- login possible?
- adding new email possible?
- does verification of new emails work? (link in verification email is automatically followed)
- deletion of email addresses possible?
- login possible?
- setting user details do test values possible?
- changing password possible?
- bad passwords (short, simple) catched?
- assuring a user
- revoking a user assurance by admin
The following scenarios should be covered by the Regression Test system. If you start working on a scenario, please add your name besides the scenario on this page, to make sure, that somebody else isn't doing the same thing at the same time.
Please see UseCases
- adding DB access to compare/simulate user behaviour (mail probes...) - bluec says that this should be handles using small scripts behind the test email addresses. Its already working more or less.
- Creation of the same user
- creation of a "random named" user
- deletion of the previous user by an admin.
- login of an unknown user
- adding a domain for a user
- creation of a certificate with a properly crafted CSR
- creation of a certificate with a domain name not validated in the user account
- login with a wrong password
- login with a password of another account
- creation of a user with a very long name
- creation of a user with special UTF-8 characters
- creation of a user with invalid UTF-8 characters
- issuing certificates with UTF-8 characters
- issuing wildcard certificates
- issuing wildcard certificates for subdomains
- issuing wildcard certificates with www.*.domain.com
- issuing too many points
- issuing certificates with CSRs with broken Signature
- issuing a class1 certificate
- issuing a class3 certificate
- issuing a certificate, modifying the server time into the future, past the certificate expiry time, verifying the certificate with OCSP
- issuing a certificate, modifying the server time into the past, verify the certificate with OCSP
Organisation account test
creating a new organisation (admin) => DN, domain names, organisation admins
- creating an email certificate and testing DN values
- creating a server certificate and testing DN values
At the beginning, we started collecting the requirements we have for a CAcert regression-test framework:
- Remote testing
- Transaction oriented
Then we evaluated the following software packages:
This is the currently used framework.
Looks good. Python based
Bash. It´s just a shell, and not very useful for regression testing
Not HTTPS capable
Only works for CGI, forget it
Needs python module xml.dom.ext
More for load/stress testing
Test::Builder, WWW::Mechanize, Perl
Didn´t work properly
Not tested yet
Great for MS Windows Automation