This happens when a portmap request and RPC reconfigure fails on a client. The client should connect to glusterd to get the brick port, and then reconnect to the brick using the port. But this fails (for some reason), leaving the client connected to glusterd instead of the brick. When the client now tries to do a data IO operation, it sends the RPCs to GlusterD instead of the brick, leading to the error logs you are seeing. The error seen just indicates that GlusterD doesn't support the IO or FOP RPCs for the brick. Try remounting the client. That should help the client correct to the bricks correctly. On Sat, Sep 17, 2016 at 2:07 AM, Vijay Bellur <vbellur@xxxxxxxxxx> wrote: > Have you checked the brick logs to see if there's anything unusual there? > > Regards, > Vijay > > On Thu, Sep 15, 2016 at 5:24 PM, Danny Lee <dannyl@xxxxxx> wrote: >> Hi, >> >> Environment: >> Gluster Version: 3.8.3 >> Operating System: CentOS Linux 7 (Core) >> Kernel: Linux 3.10.0-327.28.3.el7.x86_64 >> Architecture: x86-64 >> Replicated 3-Node Volume >> ~400GB of around a million files >> >> Description of Problem: >> One of the brick dies. The only suspect log I see is in the >> etc-glusterfs-glusterd.vol.log (shown below). Trying to get an idea of why >> the brick died and how it could be prevented in the future. >> >> During this time, I was forcing replication (find . | xargs stat on the >> mount). There were some services starting up as well that was using the >> gluster mount. >> >> [2016-09-13 20:01:50.033369] W [socket.c:590:__socket_rwv] 0-management: >> readv on /var/run/gluster/cfc57a83cf77779864900aa08380be93.socket failed (No >> data available) >> [2016-09-13 20:01:50.033830] I [MSGID: 106005] >> [glusterd-handler.c:5050:__glusterd_brick_rpc_notify] 0-management: Brick >> 172.17.32.28:/usr/local/volname/local-data/mirrored-data has disconnected >> from glusterd. >> [2016-09-13 20:01:50.121316] W [rpcsvc.c:265:rpcsvc_program_actor] >> 0-rpc-service: RPC program not available (req 1298437 330) for >> 172.17.32.28:49146 >> [2016-09-13 20:01:50.121339] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] >> 0-rpcsvc: rpc actor failed to complete successfully >> [2016-09-13 20:01:50.121383] W [rpcsvc.c:265:rpcsvc_program_actor] >> 0-rpc-service: RPC program not available (req 1298437 330) for >> 172.17.32.28:49146 >> [2016-09-13 20:01:50.121392] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] >> 0-rpcsvc: rpc actor failed to complete successfully >> The message "I [MSGID: 106005] >> [glusterd-handler.c:5050:__glusterd_brick_rpc_notify] 0-management: Brick >> 172.17.32.28:/usr/local/volname/local-data/mirrored-data has disconnected >> from glusterd." repeated 34 times between [2016-09-13 20:01:50.033830] and >> [2016-09-13 20:03:40.010862] >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@xxxxxxxxxxx >> http://www.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
