On Fri, May 29, 2020 at 1:28 PM jifeng-call <17607319886@xxxxxxx> wrote:
Hi All,
I have 6 servers that form a glusterfs 2x3 distributed replication volume, the details are as follows:[root@node1 ~]# gluster volume infoVolume Name: ksvd_volType: Distributed-ReplicateVolume ID: c9848daa-b06f-4f82-a2f8-1b425b8e869cStatus: StartedSnapshot Count: 0Number of Bricks: 2 x 3 = 6Transport-type: tcpBricks:Brick1: node1:/export/kylin/ksvdBrick2: node2:/export/kylin/ksvdBrick3: node3:/export/kylin/ksvdBrick4: node4:/export/kylin/ksvdBrick5: node5:/export/kylin/ksvdBrick6: node6:/export/kylin/ksvdOptions Reconfigured:diagnostics.client-log-level: DEBUGdiagnostics.brick-log-level: DEBUGperformance.client-io-threads: offnfs.disable: ontransport.address-family: inetcluster.quorum-type: autoRestart 6 servers at a time, the server automatically mounts the ksvd_vol volume to the / home / kylin-ksvd directory during the startup process：[root@node1 ~]# df -h | grep ksvd_volnode1:/ksvd_vol 4.5T 362G 4.1T 9% /home/kylin-ksvdFailed to create a file in the glusterfs mount directory of node3, the log shows the error as follows (the files created by other servers are normal)：
Possibly the layout setxattr failed which should not have happened given it is a replicated volume. You can check log on client side and brick side for the parent directory for more information.
Post remount, a fresh triggers which heals the layout missing before and create goes in.
[2020-05-29 05:55:03.065656] E [MSGID: 109011] [dht-common.c:8683:dht_create] 0-ksvd_vol-dht: no subvolume in layout for path=/.ksvd.time.405fa8a6-575d-4474-a97f-a107cbf1c673.18512[2020-05-29 05:55:03.065719] W [fuse-bridge.c:2122:fuse_create_cbk] 0-glusterfs-fuse: 2454790: /.ksvd.time.405fa8a6-575d-4474-a97f-a107cbf1c673.18512 => -1 (输入/输出错误)[2020-05-29 05:55:03.680303] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696[2020-05-29 05:55:04.687456] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 491618322[2020-05-29 05:55:04.688612] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696[2020-05-29 05:55:05.694446] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696[2020-05-29 05:55:05.830555] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 491618322[2020-05-29 05:55:06.700423] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696[2020-05-29 05:55:07.706536] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696[2020-05-29 05:55:07.833049] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 491618322[2020-05-29 05:55:08.712128] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696The message "W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696" repeated 2 times between [2020-05-29 05:55:08.712128] and [2020-05-29 05:55:09.718541]The pathinfo information in the / home / kylin-ksvd directory is displayed as follows:[root@node3 kylin-ksvd]# getfattr -d -m . -n trusted.glusterfs.pathinfo /home/kylin-ksvd/getfattr: Removing leading '/' from absolute path names# file: home/kylin-ksvd/trusted.glusterfs.pathinfo="((<DISTRIBUTE:ksvd_vol-dht> (<REPLICATE:ksvd_vol-replicate-0> <POSIX(/export/kylin/ksvd):node2:/export/kylin/ksvd/> <POSIX(/export/kylin/ksvd):node3:/export/kylin/ksvd/> <POSIX(/export/kylin/ksvd):node1:/export/kylin/ksvd/>) (<REPLICATE:ksvd_vol-replicate-1> <POSIX(/export/kylin/ksvd):node5:/export/kylin/ksvd/> <POSIX(/export/kylin/ksvd):node6:/export/kylin/ksvd/> <POSIX(/export/kylin/ksvd):node4:/export/kylin/ksvd/>)) (ksvd_vol-dht-layout (ksvd_vol-replicate-0 0 0) (ksvd_vol-replicate-1 3539976838 4294967295)))"It can be seen from the above information that ksvd_vol-dht-layout is missing the interval 0 to 3539976837After umount, remounting returned to normal ... What is the reason?Best regards________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
