----- Original Message ----- > From: "Sam McLeod" <mailinglists@xxxxxxxxxxx> > To: gluster-users@xxxxxxxxxxx > Sent: Friday, September 15, 2017 6:42:13 AM > Subject: 0-client_t: null client [Invalid argument] & high CPU usage (Gluster 3.12) > > Howdy, > > I'm setting up several gluster 3.12 clusters running on CentOS 7 and have > having issues with glusterd.log and glustershd.log both being filled with > errors relating to null client errors and client-callback functions. > > They seem to be related to high CPU usage across the nodes although I don't > have a way of confirming that (suggestions welcomed!). > > > in /var/log/glusterfs/glusterd.log: > > csvc_request_init+0x7f) [0x7f382007b93f] > -->/lib64/libglusterfs.so.0(gf_client_ref+0x179) [0x7f3820315e59] ) > 0-client_t: null client [Invalid argument] > [2017-09-15 00:54:14.454022] E [client_t.c:324:gf_client_ref] > (-->/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf8) [0x7f382007e7e8] > -->/lib64/libgfrpc.so.0(rpcsvc_request_init+0x7f) [0x7f382007b93f] > -->/lib64/libglusterfs.so.0(gf_client_ref+0x179) [0x7f3820315e59] ) > 0-client_t: null client [Invalid argument] This issue of spurious logging is fixed in v3.12.1. Thanks to Nithya for bringing this issue to my notice. The issue of high cpu usage seems to be a different issue provided logging itself is not driving cpu usage. > > > This is repeated _thousands_ of times and is especially noisey when any node > is running gluster volume set <volname> <option> <value>. > > and I'm unsure if it's related but in /var/log/glusterfs/glustershd.log: > > [2017-09-15 00:36:21.654242] W [MSGID: 114010] > [client-callback.c:28:client_cbk_fetchspec] 0-my_volume_name-client-0: this > function should not be called > > > --- > > > Cluster configuration: > > Gluster 3.12 > CentOS 7.4 > Replica 3, Arbiter 1 > NFS disabled (using Kubernetes with the FUSE client) > Each node is 8 Xeon E5-2660 with 16GB RAM virtualised on XenServer 7.2 > > > root@int-gluster-03:~ # gluster get > glusterd state dumped to /var/run/gluster/glusterd_state_20170915_110532 > > [Global] > MYUUID: 0b42ffb2-217a-4db6-96bf-cf304a0fa1ae > op-version: 31200 > > [Global options] > cluster.brick-multiplex: enable > > [Peers] > Peer1.primary_hostname: int-gluster-02.fqdn.here > Peer1.uuid: e614686d-0654-43c9-90ca-42bcbeda3255 > Peer1.state: Peer in Cluster > Peer1.connected: Connected > Peer1.othernames: > Peer2.primary_hostname: int-gluster-01.fqdn.here > Peer2.uuid: 9b0c82ef-329d-4bd5-92fc-95e2e90204a6 > Peer2.state: Peer in Cluster > Peer2.connected: Connected > Peer2.othernames: > > (Then volume options are listed) > > > --- > > > Volume configuration: > > root@int-gluster-03:~ # gluster volume info my_volume_name > > Volume Name: my_volume_name > Type: Replicate > Volume ID: 6574a963-3210-404b-97e2-bcff0fa9f4c9 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: int-gluster-01.fqdn.here:/mnt/gluster-storage/my_volume_name > Brick2: int-gluster-02.fqdn.here:/mnt/gluster-storage/my_volume_name > Brick3: int-gluster-03.fqdn.here:/mnt/gluster-storage/my_volume_name > Options Reconfigured: > performance.stat-prefetch: true > performance.parallel-readdir: true > performance.client-io-threads: true > network.ping-timeout: 5 > diagnostics.client-log-level: WARNING > diagnostics.brick-log-level: WARNING > cluster.readdir-optimize: true > cluster.lookup-optimize: true > transport.address-family: inet > nfs.disable: on > cluster.brick-multiplex: enable > > > -- > Sam McLeod > @s_mcleod > https://smcleod.net > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
