I upgraded to 3.0.1 last night and it still doesn't seem as stable as
2.0.9. Things I have bumped into since the upgrade:
1) I've had unfsd lock up hard when exporting the volume, it couldn't be
"kill -9"-ed. This happened just after a spurious disconnect (see 2).
2) Seeing random disconnects/timeouts between the servers that are on
the same switch (this was happening with 2.0.x as well, though, so not
sure what's going on). This is where the file clobbering/corruption used
to occur that causes contents of one file to be replaced with contents
of a different file, when the files are open. I HAVEN'T observed
clobbering with 3.0.1 (yet at least - it wasn't a particularly frequent
occurrence, but the chances of it were high on shared libraries during a
big yum update when glfs is rootfs), but the disconnects still happen
occassionally, usually under heavy-ish load.
My main concern here is that open file self-healing may cover up the
underlying bug that causes the clobbering, and possibly make it occur in
even more heisenbuggy ways.
ssh sessions to both servers don't show any
problems/disconnections/dropouts at the same time as the disconnects on
glfs happen. Is there a setting to set how many heartbeat packets have
to be lost before the disconnect is initiated?
This is the sort of thing I see in the logs:
[2010-01-26 07:36:56] N [server-protocol.c:6780:notify] server:
10.2.0.13:1010 disconnected
[2010-01-26 07:36:56] N [server-protocol.c:6780:notify] server:
10.2.0.13:1013 disconnected
[2010-01-26 07:36:56] N [server-helpers.c:849:server_connection_destroy]
server: destroyed connection of
thor.winterhearth.co.uk-11823-2010/01/26-05:29:32:239464-home2
[2010-01-26 07:37:25] E [saved-frames.c:165:saved_frames_unwind] home3:
forced unwinding frame type(1) op(SETATTR)
[2010-01-26 07:37:25] E [saved-frames.c:165:saved_frames_unwind] home3:
forced unwinding frame type(1) op(SETXATTR)
[2010-01-26 07:37:25] E [saved-frames.c:165:saved_frames_unwind] home3:
forced unwinding frame type(2) op(PING)
[2010-01-26 07:37:25] N [client-protocol.c:6973:notify] home3: disconnected
[2010-01-26 07:38:19] E
[client-protocol.c:415:client_ping_timer_expired] home3: Server
10.2.0.13:6997 has not responded in the last 42 seconds, disconnecting.
[2010-01-26 07:38:19] E [saved-frames.c:165:saved_frames_unwind] home3:
forced unwinding frame type(2) op(SETVOLUME)
[2010-01-26 07:38:19] E [saved-frames.c:165:saved_frames_unwind] home3:
forced unwinding frame type(2) op(SETVOLUME)
[2010-01-26 08:06:17] N [server-protocol.c:5811:mop_setvolume] server:
accepted client from 10.2.0.13:1018
[2010-01-26 08:06:17] N [server-protocol.c:5811:mop_setvolume] server:
accepted client from 10.2.0.13:1017
[2010-01-26 08:06:17] N [client-protocol.c:6225:client_setvolume_cbk]
home3: Connected to 10.2.0.13:6997, attached to remote volume 'home3'.
[2010-01-26 08:06:17] N [client-protocol.c:6225:client_setvolume_cbk]
home3: Connected to 10.2.0.13:6997, attached to remote volume 'home3'.
3) Something that started off as not being able to ssh in using public
keys turned out to be due to my home directory somehow acquiring 777
permissions. I certainly didn't do it, so at a guess it's a file
corruption issue, possibly during an unclean shutdown. Further, I've
found that / directory (I'm running glusterfs root on this cluster) had
permissions 777, too, which seems to have happened at the same time as
the home directory getting 777 permissions. If sendmail and ssh weren't
failing to work properly because of this, it's possible I wouldn't have
noticed. It's potentially quite a concerning problem, even if it is
caused by an unclean shutdown (put it this way - I've never seen it
happen on any other file system).
4) This looks potentially a bit concerning:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5633 root 15 0 25.8g 119m 1532 S 36.7 3.0 36:25.42
/usr/sbin/glusterfs --log-level=NONE --log-file=/dev/null
--disable-direct-io-mode --volfile=/etc/glusterfs.root/root2.vol
/mnt/newroot
This is the rootfs daemon. 25.8GB of virtual address space mapped?
Surely that can't be right, even if the resident size looks reasonably sane.
Worse - it's growing by about 100MB/minute during heavy compiling on the
system. I've just tried to test the nvidia driver installer to see if
that old bug report I filed is still valid, and it doesn't seem to get
anywhere (just makes glusterfsd and gcc use CPU time but doesn't ever
finish - which is certainly a different fail case from 2.0.9 - that at
least finishes the compile stage).
The virtual memory bloat is rather reminiscent of the memory
fragmentation/leak problem that was fixed on 2.0.x branch a while back
that was arising when shared libraries were on glusterfs. A bit leaked
every time a shared library call was made. A regression, perhaps? Wasn't
there a memory consumption sanity check added to the test suite after
this was fixed last time?
Other glfs daemons are exhibiting similar behaviour:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5633 root 15 0 26.1g 119m 1532 S 0.7 3.0 37:57.01
/usr/sbin/glusterfs --log-level=NONE --log-file=/dev/null
--disable-direct-io-mode --volfile=/etc/glusterfs.root/root2.vol
/mnt/newroot
12037 root 15 0 24.8g 68m 1072 S 0.0 1.7 3:21.41
/usr/sbin/glusterfs --log-level=NORMAL
--volfile=/etc/glusterfs/shared.vol /shared
11977 root 15 0 24.8g 67m 1092 S 0.7 1.7 3:59.11
/usr/sbin/glusterfs --log-level=NORMAL --disable-direct-io-mode
--volfile=/etc/glusterfs/home.vol /home
11915 root 15 0 24.9g 32m 972 S 0.0 0.8 0:21.65
/usr/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/boot.vol
/boot
The home, shared and boot volumes don't have any shared libraries on
them, and 24.9GB of virtual memory mapped for the /boot volume which is
backed with a 250MB file system also seems a bit excessive.
Gordan
