= CAcert Regression Test = We are currently building up a regression test system, to automatically test the functionality of the CAcert website. /!\ Regression tests against the production system must be authorized by at least 2 people from the core-team, and monitored by at least one core-team member. /!\ Regression code that runs against the production system must not be commited to the SVN, or otherwise publizised There is a new regression testing system that we should try out: [[http://www.openqa.org/selenium|Selenium]] == Where to start? == You can get our current regression tests from a subversion repostory at: http://svn.cacert.org/CAcert/Regression/ A live site running the tests software is available at http://test.chost.de/Regression/ - username and password is both cacert. The tests are currently not testing the main CAcert website but https://www.test1.cacert.at/ Several tests have been written so far. The testing framework we use is SimpleTest. Basically a test is a script simulating the user/admin/etc input to CAcert web site and checking the results. Please protect your test installation and framework from indexing by a search engine. You can do this by using a "robots.txt" in your web directory. == Tests Already Implemented == === general-website.php === * front page available via http/https? * contact page available via http/https? * Root Cert fingerprints correct via http/https? * first line of Root Cert (PEM) correct via http/https? * (Changing language to de_DE possibe via http/https?) not testable testserver * is wiki.cacert.org online? * is blog.cacert.org online? * is bugs.cacert.org online? === login.php === * logon with empty password possible? * logon with empty email address possible? * logon with wrong password possible? * normal logon possible? * logout possible? === lostpassword.php === * lost password page available? * empty email and date of birth catched? * empty date of birth catched? * lost password step 2 available? * empty lost password answers catched? * correct answers but no new password catched? * setting new password using lost password questions possible? === join.php === * empty password detected? * missing lost password questions/answers detected? * missing date of birth detected? * missing email detected? * missing last 2 lost password questions/answers detected? * simple passwords detected? * different passwords detected? * using already registered emails addresses possible? * joining possible? * login possible? (link in verification email is automatically followed) === newemail.php === * login possible? * adding new email possible? * does verification of new emails work? (link in verification email is automatically followed) * deletion of email addresses possible? === change-details-password.php === * login possible? * setting user details do test values possible? * changing password possible? * bad passwords (short, simple) catched? === additional tests === * assuring a user * revoking a user assurance by admin == Planned Tests == The following scenarios should be covered by the Regression Test system. If you start working on a scenario, please add your name besides the scenario on this page, to make sure, that somebody else isn't doing the same thing at the same time. Please see UseCases * adding DB access to compare/simulate user behaviour (mail probes...) - bluec says that this should be handles using small scripts behind the test email addresses. Its already working more or less. * Creation of the same user * creation of a "random named" user * deletion of the previous user by an admin. * login of an unknown user * adding a domain for a user * creation of a certificate with a properly crafted CSR * creation of a certificate with a domain name not validated in the user account * ... * login with a wrong password * login with a password of another account * creation of a user with a very long name * creation of a user with special UTF-8 characters * creation of a user with invalid UTF-8 characters * issuing certificates with UTF-8 characters * issuing wildcard certificates * issuing wildcard certificates for subdomains * issuing wildcard certificates with www.*.domain.com * issuing too many points * issuing certificates with CSRs with broken Signature * issuing a class1 certificate * issuing a class3 certificate * issuing a certificate, modifying the server time into the future, past the certificate expiry time, verifying the certificate with OCSP * issuing a certificate, modifying the server time into the past, verify the certificate with OCSP * ... '''Organisation account test''' * creating a new organisation (admin) => DN, domain names, organisation admins * creating an email certificate and testing DN values * creating a server certificate and testing DN values = History = == Requirements == At the beginning, we started collecting the requirements we have for a CAcert regression-test framework: Requirements: * Remote testing * HTTPS * Login * Cookies * Transaction oriented == Frameworks evaluated == Then we evaluated the following software packages: ||[[http://simpletest.org/|SimpleTest]] ||This is the currently used framework. || ||[[http://freshmeat.net/projects/funkload/|FunkLoad]] ||Looks good. Python based || ||http://www.bash.org/ ||Bash. It´s just a shell, and not very useful for regression testing || ||[[http://search.cpan.org/~adiraj/ApacheBench-0.63/lib/HTTPD/Bench/ApacheBench.pm|HTTP::ApacheBench]] ||Not HTTPS capable || ||[[http://search.cpan.org/~mshiltonj/CGI-Test-0.104/Test.pm|CGI::Test]] ||Only works for CGI, forget it || ||Puffin ||Needs python module xml.dom.ext || ||[[http://jakarta.apache.org/jmeter/|JMeter]] ||More for load/stress testing || ||Test::Builder, WWW::Mechanize, Perl || ||[[http://guillaume.romagny.free.fr/testcacert|Junit+HTTPUnit]] ||Didn´t work properly || ||[[http://testng.org/doc/|TestNG]] ||Not tested yet || ||[[http://www.autoitscript.com/autoit3/|AutoIt]] ||Great for MS Windows Automation ||