Hi,
I've attached the relevant section of a client log after noticing load
problems due to the glusterfs process after an upgrade from 1.3.8 to
2.0.1.
The first warning about ulimit -n is due to the selinux policy not
allowing it by default (selinux is enforcing and will stay that way).
This denial has also been seen on the server, could it be related to
the issue?
avc: denied { mmap_zero } for pid=3155 comm="glusterfs2"
scontext=system_u:system_r:mount_t:s0
tcontext=system_u:system_r:mount_t:s0 tclass=memprotect
Any help would be appreciated.
Matthias
--
Clean custom Red Hat Linux rpm packages : http://freshrpms.net/
Fedora release 10 (Cambridge) - Linux kernel
2.6.27.21-170.2.56.fc10.x86_64 Load : 0.16 0.16 0.13
================================================================================
Version : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-26 12:07:06
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL --volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID : 3136
System name : Linux
Nodename : foo
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686
Given volfile:
+------------------------------------------------------------------------------+
1: volume foo
2: type protocol/client
3: option transport-type tcp
4: option remote-host file01.priv # can be IP or hostname
5: option remote-port 6997
6: option remote-subvolume foo-remote
7: end-volume
8:
9: volume foo-readahead
10: type performance/read-ahead
11: option page-count 4
12: subvolumes foo
13: end-volume
14:
15: volume foo-writebehind
16: type performance/write-behind
17: option cache-size 512KB
18: subvolumes foo-readahead
19: end-volume
20:
21: volume foo-iothreads
22: type performance/io-threads
23: option autoscaling on
24: option min-threads 16
25: option max-threads 256
26: subvolumes foo-writebehind
27: end-volume
28:
29: volume foo-iocache
30: type performance/io-cache
31: option cache-size 64MB
32: option page-size 128KB
33: option cache-timeout 2
34: subvolumes foo-iothreads
35: end-volume
36:
+------------------------------------------------------------------------------+
[2009-05-26 12:07:06] W [client-protocol.c:6012:init] foo: WARNING: Failed to set 'ulimit -n 1M': Permission denied
[2009-05-26 12:07:06] N [glusterfsd.c:1152:main] glusterfs: Successfully started
[2009-05-26 12:07:06] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'.
[2009-05-26 12:07:06] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'.
[2009-05-26 12:08:22] W [fuse-bridge.c:1365:fuse_create_cbk] glusterfs-fuse: 118: /cust/a => -1 (Read-only file system)
[2009-05-27 00:41:29] E [socket.c:601:__socket_proto_state_machine] foo: unable to allocate IO buffer for peer 192.168.168.231:6997
[2009-05-27 00:41:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:30] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'.
[2009-05-27 00:41:33] W [fuse-bridge.c:884:fuse_err_cbk] glusterfs-fuse: 4101773: FLUSH() ERR => -1 (Invalid argument)
[2009-05-27 00:41:47] E [socket.c:601:__socket_proto_state_machine] foo: unable to allocate IO buffer for peer 192.168.168.231:6997
[2009-05-27 00:41:47] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:47] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:47] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:47] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:47] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:41:47] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'.
[2009-05-27 00:41:51] W [fuse-bridge.c:884:fuse_err_cbk] glusterfs-fuse: 4104453: FLUSH() ERR => -1 (Invalid argument)
[2009-05-27 00:46:30] E [socket.c:601:__socket_proto_state_machine] foo: unable to allocate IO buffer for peer 192.168.168.231:6997
[2009-05-27 00:46:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:30] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:30] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'.
[2009-05-27 00:46:43] W [fuse-bridge.c:884:fuse_err_cbk] glusterfs-fuse: 4122215: FLUSH() ERR => -1 (Invalid argument)
[2009-05-27 00:46:56] E [socket.c:601:__socket_proto_state_machine] foo: unable to allocate IO buffer for peer 192.168.168.231:6997
[2009-05-27 00:46:56] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:56] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:56] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:56] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:56] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:46:56] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'.
[2009-05-27 00:46:59] W [fuse-bridge.c:884:fuse_err_cbk] glusterfs-fuse: 4124195: FLUSH() ERR => -1 (Invalid argument)
[2009-05-27 00:47:54] E [socket.c:601:__socket_proto_state_machine] foo: unable to allocate IO buffer for peer 192.168.168.231:6997
[2009-05-27 00:47:54] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:47:54] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:47:54] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:47:54] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:47:54] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:47:54] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'.
[2009-05-27 00:48:56] W [fuse-bridge.c:884:fuse_err_cbk] glusterfs-fuse: 4133136: FLUSH() ERR => -1 (Invalid argument)
[2009-05-27 00:50:07] E [socket.c:601:__socket_proto_state_machine] foo: unable to allocate IO buffer for peer 192.168.168.231:6997
[2009-05-27 00:50:07] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:50:07] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:50:07] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:50:07] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:50:07] E [saved-frames.c:165:saved_frames_unwind] foo: forced unwinding frame type(1) op(READ)
[2009-05-27 00:50:07] N [client-protocol.c:5557:client_setvolume_cbk] foo: Connected to 192.168.168.231:6997, attached to remote volume 'foo-remote'.
[2009-05-27 00:50:14] W [fuse-bridge.c:884:fuse_err_cbk] glusterfs-fuse: 4146311: FLUSH() ERR => -1 (Invalid argument)
[2009-05-27 01:18:25] W [socket.c:1319:socket_init] trans: disabling non-blocking IO
[2009-05-27 01:18:25] W [socket.c:1319:socket_init] trans: disabling non-blocking IO
================================================================================
Version : glusterfs 2.0.1 built on May 26 2009 05:23:52
[...]
