I agree that a test suite is something that we need. I'll add this to our TODO list. As always, code contributions are always welcome.
See below and attached, a copy of what I sent april '04. It's not great, but it's a start...
Rob ---- Hi Ken & Rob Given that there's currently no regression test for cyrus, I thought it might be worth creating one. The basic idea being that the tests are kept up to date, and any new features get a basic set of tests. Also anytime a bug is discovered, a test is added that shows the bug, and left to make sure it doesn't come back in the future. We're willing to fund the starting of this project (geared towards the most recent patches for our needs), but hope CMU will keep it up to date in the future as well. I'm not quite sure how any of this would work (both funding and maintenance wise), but maybe you can make some suggestions? Anyway, I think there should be a generic imap component, and cyrus specific component, in that it does some tests that rely on cyrus and/or it's utilities (eg check location of spool directory and meta files, corrupt a mailbox and run reconstruct, check quota and quota fixing, etc) I think it's probably best to write the script in perl, and here's a couple of modules that should help: http://search.cpan.org/~robm/Mail-IMAPTalk-1.00/ http://search.cpan.org/~lhoward/Net-LMTP-0.02/ I tend to like creating two main types of tests: 1. A very structured set that test very specific set of actions. I tend to make it so each 'batch' of tests is a function and is independent, and that it leaves everything exactly how it was beforehand so you can easily run the same test suite over and over, or a particular test function. eg sub TestBasicFolder { $Imap->create('user.test1') || die "Could not create user.test1"; my $ACL = $Imap->getacl('user.test1') || die "Could not get user.test1 ACL 1"; $Imap->setacl('user.test1', 'admin', 'lrswipcda') || die "Could not set user.test1 ACL"; $ACL = $Imap->getacl('user.test1') || die "Could not get user.test1 ACL 2"; ... some more tests ... $Imap->delete('user.test1') || die "Could not delete user.test1"; } 2. Ones that pick a bunch of random commands from a set and executes them, somehow keeping internally what it expects the state to be and making sure that it is. eg something like: sub TestRandomFolderCreateDelete { my @Folders; for (1 .. 100000) { my $Rand = rand(); # Create a random folder if ($Rand < 0.7) { my $Folder = "user." . RandomFolderName(int(rand(20))); $Imap->create() || die "Could not create folder $Folder: $@"; push @Folders, $Folder; # Delete a random folder } elsif ($Rand < 0.9) { my $Folder = splice @Folders, int(rand(@Folders));, 1; $Imap->delete($Folder) || die "Could not delete folder $Folder: $@"; # Check mailbox list } else { my %List = map { $_->[2] => 1 } $Imap->list('*', '*'); for (@Folders) { delete $List{$_} || die "Folder $_ not found"; } !%List || die "Not all folders found"; } } for (@Folders) { $Imap->delete($_) || die "Could not delete folder $Folder: $@"; } my @List = $Imap->list('*', '*'); !@List || die "Unexpectedly found folders"; } I find tests like this quite important to find the more obscure bugs. Once it does find a bug, I then find that I try differnet values of srand() at the start (which then makes rand generate the same sequence each time the program is run) until the bug is reproducible so I can track it down. So basic things I can think of: 1. Login as 'administrator' user and create a bunch of user.* mailboxes and sub-folders and listing 2. Test deleting some folders (eg test problem of deleting user.adminxxx folder name) and listing 3. Set/get/myrights tests ACL's on the mailboxes (eg try read only, post only, sharing other users folder, etc) 4. Set quotas on the mailboxes (different quota roots, etc) 5. Login as a user for one of the mailboxes. Test different ACL's set above (eg try read only, post only, sharing other users folder, etc) 6. Appending messages to a mailbox 7. Reading messages (and seen flag) 8. Setting/getting other flags 9. Delivering email via LMTP (and of course + addressing) 10. LMTP new wildcard addressing 11. POP access to a mailbox 12. POP access to sub-folder 13. Group ACLs 14. Sieve actions A couple of tricky things: 1. User logins. Since these are separate to the actual mailbox names, I'm not sure how to easily handle this... maybe force usage of a saslauthd daemon that always authorises all username/password combinations... 2. imapd.conf will need to be changed and then the server restarted from the code To get the ball rolling, I looked at some existing code I had, and started the outlines of how I think things might work. I've attached the script, which currently has 3 tests which I've just tested at home and verified that it works. Rob
Attachment:
cyrus_regression_test.pl
Description: Binary data
---- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
