Hi,
Andrew Laurence wrote:
Has anyone evaluated the new "fixed" locking in NFSv4 for use with
Cyrus? I attended a talk by Sun's Spencer Shepler at USENIX '05 in
which he spent a good deal of time on NFSv4 (supposedly) fixes the
locking shortcomings and can be used as a native file system.
http://blogs.sun.com/shepler/date/20050407
Just curious if anyone has taken it for a drive with Cyrus?
Since the use of NFSv4 was mentioned again in the HA thread, I thought
I'd give this a shot by using the imaptest-utility from dovecot (the
only imap stress-testing tool I can think of, I've stressed replication
with it too) and NFSv4 mounts on RedHat.
Maybe it does not show any errors... maybe it does: better then no
testing at all. Probably a far too simple test and a lot to tweak.
It would be very interesting to see how and if this works with NetApp
filers for instance, instead of a RH server. Or on Solaris. I have a
feeling that this might show different results.
I exported the filesystem simply with:
/usr/sbin/exportfs -o rw,fsid=0,no_root_squash haver:/data/vol2
and also tried with the secure_locks option added to it, and mounted on
the client with:
mount -t nfs4 gerst:/ mnt
so nothing really special. (Suggestions for further tests welcome!)
I tried insecure_locks too, eventually, and that seems to result in a
similar situation to NFSv3, and apparently secure_locks are default.
The cyrus install is cyrus-imapd-2.3.7-3 from Simon/Invoca with mostly
default options: so skiplist is the default for all databases. I changed
to "flushseenstate: 0" though, because imaptest with a local filesystem
gave me errors if I didn't (!).
With an ext3 filesystem I just get a normal output from imaptest:
[root@haver dovecot-1.0.rc7]# ./imaptest
Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
0 43 41 40 39 23 30 26 28 25 0
0 35 36 36 37 17 26 24 39 33 0
0 33 33 34 33 21 26 25 45 37 0
0 38 39 38 38 11 23 24 43 34 0
0 34 33 32 33 21 28 28 43 36 0
If I mv the spool to the NFSv4 mount, and start cyrus with that
partition I see a lot of errors, unfortunally:
[root@haver dovecot-1.0.rc7]# ./imaptest
Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
0 20 20 20 20 1 3 1 0 0 0
Error: STORE failed: s NO System I/O error
Error: APPEND failed: a NO System I/O error
Error: STORE failed: s NO System I/O error
Error: STORE failed: s NO System I/O error
Error: EXPUNGE failed: e NO System I/O error
Error: EXPUNGE failed: e NO System I/O error
Error: STORE (\Deleted) failed: sd NO System I/O error
Error: STORE failed: s NO System I/O error
Error: STORE failed: s NO System I/O error
Error: STORE failed: s NO System I/O error
Error: STORE (\Deleted) failed: sd NO System I/O error
Error: STORE (\Deleted) failed: sd NO System I/O error
Error: APPEND failed: a NO System I/O error
Error: APPEND failed: a NO System I/O error
Error: APPEND failed: a NO System I/O error
Error: APPEND failed: a NO System I/O error
0 16 16 16 16 9 6 2 5 0 0
Error: STORE failed: s NO System I/O error
Error: STORE failed: s NO System I/O error
0 2 2 2 2 5 1 0 0 0 0
Error: APPEND failed: a NO System I/O error
0 1 1 1 1 1 0 0 1 0 0
0 0 0 0 0 1 1 0 0 0 0
0 0 0 0 0 0 1 0 0 0 0
0 4 4 3 3 1 0 1 4 4 0
0 3 3 4 4 2 2 3 3 3 0
and in the logs a couple of these:
Nov 24 23:00:37 haver imap[12315]: IOERROR: locking header for
user.test: Input/output error
Nov 24 23:00:37 haver imap[12312]: IOERROR: locking index for user.test:
Input/output error
Just a simple test so far, but I'm afraid this doesn't work "out of the
box"...
To show that NFSv4 *is* different from NFSv3, NFSv3 (or NFSv4 with
insecure_locking) gives me:
[root@haver dovecot-1.0.rc7]# ./imaptest
Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
0 22 22 22 22 1 2 1 3 2 0
0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0
...
0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0
- 0. stalled for 27 secs in STORE
- 1. stalled for 27 secs in APPEND
- 2. stalled for 27 secs in STORE
...
That isn't really working at all.
Curious if someone knows anything else to test, or has more options for
testing than I have.
Paul
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
