Re: Replace Brick fails (gluster 3.2.5)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sat, Nov 29, 2014 at 11:23:07PM +0100, Heiko Schröter wrote:
> sorry, at least for the next 18 month i'am stuck with 3.2.5 (Lifetime of
> project).
> But maybe someone from the dev team could give a hint where to start
> debugging with these error messages.
> It seems to be related to the "rpc XDR" encoding mechanism. The only thing
> i've found so far is that it might be a "LOCALE" probelm. I checked the
> locale settings. They where indeed different.
> So recompiled everything with correct locale but still get the error.
> "strace" gives me no real clue.

If there is an issue with the locale, recompiling will likely not be
helping. You would need to set the locale for the running processes in
their environment. The easiest is probably to add 'export LC_ALL=C' in
the startup scripts (maybe more, see 'man locale').

Other than that, the locale that is set in the environment of where the
'gluster' commandline is started may be important too, depending on
where the error originates.

Potentially related:
- http://review.gluster.org/8299
- https://bugzilla.redhat.com/1117951

HTH,
Niels

> 
> Is there a way to debug or trace the gluster RPC XDR encoding mechanism ?
> 
> Heiko
> 
> 
> [2014-11-29 23:10:52.912545] W [rpc-common.c:38:xdr_serialize_generic] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_op_send_cli_response+0x97)
> [0x7f801881d7e7] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_submit_reply+0x65)
> [0x7f801880f695] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_serialize_reply+0x56)
> [0x7f801880f576]))) 0-xdr: XDR encoding failed
> [2014-11-29 23:10:52.912556] E
> [glusterd-utils.c:404:glusterd_serialize_reply] 0-: Failed to encode message
> [2014-11-29 23:10:52.912561] E [glusterd-utils.c:446:glusterd_submit_reply]
> 0-: Failed to serialize reply
> 
> 
> strace end:
> .....
> read(5, "# /etc/services\n#\n# Network serv"..., 4096) = 4096
> read(5, " private\t77/tcp\t\t\t\t# any private"..., 4096) = 4096
> read(5, "e\nemfis-cntl\t141/udp\nimap\t\t143/t"..., 4096) = 4096
> read(5, "dialog\t360/tcp\t\t\t\t# scoi2odialog"..., 4096) = 4096
> read(5, "\t\tdqs313_intercell\ncryptoadmin\t6"..., 4096) = 4096
> read(5, "# Citrix ICA Client\nica\t\t1494/ud"..., 4096) = 4096
> read(5, "05/udp\nlstp\t\t2559/tcp\t\t\t# \nlstp\t"..., 4096) = 4096
> read(5, "t-pmp\t\t5351/udp\ndns-llq\t\t5352/tc"..., 4096) = 4096
> read(5, "p\t\t\t# OpenPGP HTTP Keyserver\nhkp"..., 4096) = 3373
> read(5, "", 4096)                       = 0
> close(5)                                = 0
> munmap(0x7f8cf229a000, 4096)            = 0
> socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 5
> setsockopt(5, SOL_SOCKET, SO_RCVBUF, [524288], 4) = 0
> setsockopt(5, SOL_SOCKET, SO_SNDBUF, [524288], 4) = 0
> setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
> fcntl(5, F_GETFL)                       = 0x2 (flags O_RDWR)
> fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
> setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
> setsockopt(5, SOL_TCP, TCP_KEEPIDLE, [20], 4) = 0
> setsockopt(5, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
> bind(5, {sa_family=AF_INET, sin_port=htons(1023),
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> connect(5, {sa_family=AF_INET, sin_port=htons(24007),
> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in
> progress)
> epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLPRI|EPOLLOUT, {u32=5, u64=5}})
> = 0
> mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f8cef018000
> mprotect(0x7f8cef018000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7f8cef817ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
> parent_tidptr=0x7f8cef8189d0, tls=0x7f8cef818700,
> child_tidptr=0x7f8cef8189d0) = 2840
> mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f8cee817000
> mprotect(0x7f8cee817000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7f8cef016ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
> parent_tidptr=0x7f8cef0179d0, tls=0x7f8cef017700,
> child_tidptr=0x7f8cef0179d0) = 2841
> epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 257, -1) = 1
> getsockopt(5, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
> getsockname(5, {sa_family=AF_INET, sin_port=htons(1023),
> sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
> futex(0x621ea4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x621e60, 2) = 1
> epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN|EPOLLPRI, {u32=5, u64=5}}) = 0
> epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 257, -1) = 1
> readv(5, [{"\200\2\0\30", 4}], 1)       = 4
> readv(5, [{"\0\0\0\1\0\0\0\1", 8}], 1)  = 8
> readv(5,
> [{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\0"...,
> 131088}], 1) = 131088
> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
> write(4, "[2014-11-29 23:21:41.942744] I ["..., 118) = 118
> fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f8cf229a000
> write(1, "replace-brick failed to start\n", 30replace-brick failed to start
> ) = 30
> futex(0x621e24, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x621de0, 2) = 1
> epoll_wait(3,  <unfinished ...>
> +++ exited with 1 +++
> 
> 
> 
> 
> 
> 
> Am 28.11.2014 um 12:50 schrieb Atin Mukherjee:
> >3.2.5 is too old, can you please upgrade your cluster to recent version
> >of glusterfs bits and try it out?
> >
> >~Atin
> >
> >On 11/28/2014 05:17 PM, Heiko Schröter wrote:
> >>Unable to set cli op
> >
> >
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

Attachment: pgpKT0I8CIpjO.pgp
Description: PGP signature

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux