On Sat, Nov 29, 2014 at 11:23:07PM +0100, Heiko Schröter wrote: > sorry, at least for the next 18 month i'am stuck with 3.2.5 (Lifetime of > project). > But maybe someone from the dev team could give a hint where to start > debugging with these error messages. > It seems to be related to the "rpc XDR" encoding mechanism. The only thing > i've found so far is that it might be a "LOCALE" probelm. I checked the > locale settings. They where indeed different. > So recompiled everything with correct locale but still get the error. > "strace" gives me no real clue. If there is an issue with the locale, recompiling will likely not be helping. You would need to set the locale for the running processes in their environment. The easiest is probably to add 'export LC_ALL=C' in the startup scripts (maybe more, see 'man locale'). Other than that, the locale that is set in the environment of where the 'gluster' commandline is started may be important too, depending on where the error originates. Potentially related: - http://review.gluster.org/8299 - https://bugzilla.redhat.com/1117951 HTH, Niels > > Is there a way to debug or trace the gluster RPC XDR encoding mechanism ? > > Heiko > > > [2014-11-29 23:10:52.912545] W [rpc-common.c:38:xdr_serialize_generic] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_op_send_cli_response+0x97) > [0x7f801881d7e7] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_submit_reply+0x65) > [0x7f801880f695] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_serialize_reply+0x56) > [0x7f801880f576]))) 0-xdr: XDR encoding failed > [2014-11-29 23:10:52.912556] E > [glusterd-utils.c:404:glusterd_serialize_reply] 0-: Failed to encode message > [2014-11-29 23:10:52.912561] E [glusterd-utils.c:446:glusterd_submit_reply] > 0-: Failed to serialize reply > > > strace end: > ..... > read(5, "# /etc/services\n#\n# Network serv"..., 4096) = 4096 > read(5, " private\t77/tcp\t\t\t\t# any private"..., 4096) = 4096 > read(5, "e\nemfis-cntl\t141/udp\nimap\t\t143/t"..., 4096) = 4096 > read(5, "dialog\t360/tcp\t\t\t\t# scoi2odialog"..., 4096) = 4096 > read(5, "\t\tdqs313_intercell\ncryptoadmin\t6"..., 4096) = 4096 > read(5, "# Citrix ICA Client\nica\t\t1494/ud"..., 4096) = 4096 > read(5, "05/udp\nlstp\t\t2559/tcp\t\t\t# \nlstp\t"..., 4096) = 4096 > read(5, "t-pmp\t\t5351/udp\ndns-llq\t\t5352/tc"..., 4096) = 4096 > read(5, "p\t\t\t# OpenPGP HTTP Keyserver\nhkp"..., 4096) = 3373 > read(5, "", 4096) = 0 > close(5) = 0 > munmap(0x7f8cf229a000, 4096) = 0 > socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 5 > setsockopt(5, SOL_SOCKET, SO_RCVBUF, [524288], 4) = 0 > setsockopt(5, SOL_SOCKET, SO_SNDBUF, [524288], 4) = 0 > setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 > fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) > fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0 > setsockopt(5, SOL_TCP, TCP_KEEPIDLE, [20], 4) = 0 > setsockopt(5, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0 > bind(5, {sa_family=AF_INET, sin_port=htons(1023), > sin_addr=inet_addr("0.0.0.0")}, 16) = 0 > connect(5, {sa_family=AF_INET, sin_port=htons(24007), > sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in > progress) > epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLPRI|EPOLLOUT, {u32=5, u64=5}}) > = 0 > mmap(NULL, 8392704, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f8cef018000 > mprotect(0x7f8cef018000, 4096, PROT_NONE) = 0 > clone(child_stack=0x7f8cef817ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, > parent_tidptr=0x7f8cef8189d0, tls=0x7f8cef818700, > child_tidptr=0x7f8cef8189d0) = 2840 > mmap(NULL, 8392704, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f8cee817000 > mprotect(0x7f8cee817000, 4096, PROT_NONE) = 0 > clone(child_stack=0x7f8cef016ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, > parent_tidptr=0x7f8cef0179d0, tls=0x7f8cef017700, > child_tidptr=0x7f8cef0179d0) = 2841 > epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 257, -1) = 1 > getsockopt(5, SOL_SOCKET, SO_ERROR, [0], [4]) = 0 > getsockname(5, {sa_family=AF_INET, sin_port=htons(1023), > sin_addr=inet_addr("127.0.0.1")}, [16]) = 0 > futex(0x621ea4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x621e60, 2) = 1 > epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN|EPOLLPRI, {u32=5, u64=5}}) = 0 > epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 257, -1) = 1 > readv(5, [{"\200\2\0\30", 4}], 1) = 4 > readv(5, [{"\0\0\0\1\0\0\0\1", 8}], 1) = 8 > readv(5, > [{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\0"..., > 131088}], 1) = 131088 > stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0 > write(4, "[2014-11-29 23:21:41.942744] I ["..., 118) = 118 > fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 > mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f8cf229a000 > write(1, "replace-brick failed to start\n", 30replace-brick failed to start > ) = 30 > futex(0x621e24, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x621de0, 2) = 1 > epoll_wait(3, <unfinished ...> > +++ exited with 1 +++ > > > > > > > Am 28.11.2014 um 12:50 schrieb Atin Mukherjee: > >3.2.5 is too old, can you please upgrade your cluster to recent version > >of glusterfs bits and try it out? > > > >~Atin > > > >On 11/28/2014 05:17 PM, Heiko Schröter wrote: > >>Unable to set cli op > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://supercolony.gluster.org/mailman/listinfo/gluster-users
Attachment:
pgpKT0I8CIpjO.pgp
Description: PGP signature
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
