Krutika Dhananjay <kdhananj@xxxxxxxxxx> wrote: > Could you provide the trace logs you had added with this failure? A bit different since I had tossed out the previous trace logs. The file that cannot heal is gfid dae15ff0-0b16-4ca0-b27d-734f9d2290f8. You can see that the sweep completes, but the file is not reported as healed. [afr-self-heald.c:676:afr_shd_full_healer] 0-patchy-replicate-0: starting full sweep on subvol patchy-client-0 [afr-self-heald.c:676:afr_shd_full_healer] 0-patchy-replicate-0: starting full sweep on subvol patchy-client-1 [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ... [client-rpc-fops.c:1525:client3_3_inodelk_cbk] 0-patchy-client-1: remote operation failed: Resource temporarily unavailable [client-rpc-fops.c:1525:client3_3_inodelk_cbk] 0-patchy-client-0: remote operation failed: Resource temporarily unavailable [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_UNLCK ... [afr-self-heal-data.c:754:afr_selfheal_data] 0-patchy-replicate-0: calling afr_selfheal_tryinodelk inode->gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0:self-heal, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0:self-heal, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ... [afr-self-heal-data.c:757:afr_selfheal_data] 0-patchy-replicate-0: calling afr_selfheal_tryinodelk inode->gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 => 2 [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ... [afr-self-heal-data.c:754:afr_selfheal_data] 0-patchy-replicate-0: calling afr_selfheal_tryinodelk inode->gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0:self-heal, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0:self-heal, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ... [afr-self-heal-data.c:757:afr_selfheal_data] 0-patchy-replicate-0: calling afr_selfheal_tryinodelk inode->gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 => 2 [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ... [afr-self-heal-common.c:476:afr_log_selfheal] 0-patchy-replicate-0: Completed data selfheal on 994f616e-b6a8-4026-a1b2-176cb8220a18. source=1 sinks=0 [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0:self-heal, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0:self-heal, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ... [afr-self-heal-metadata.c:54:__afr_selfheal_metadata_do] 0-patchy-replicate-0: performing metadata selfheal on 994f616e-b6a8-4026-a1b2-176cb8220a18 [afr-self-heal-common.c:476:afr_log_selfheal] 0-patchy-replicate-0: Completed data selfheal on beb51df4-9670-4b77-8a59-7d0ae7f84b28. source=0 sinks=1 [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0:self-heal, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0:self-heal, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ... [afr-self-heal-metadata.c:54:__afr_selfheal_metadata_do] 0-patchy-replicate-0: performing metadata selfheal on beb51df4-9670-4b77-8a59-7d0ae7f84b28 [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ... [afr-self-heal-common.c:476:afr_log_selfheal] 0-patchy-replicate-0: Completed metadata selfheal on 994f616e-b6a8-4026-a1b2-176cb8220a18. source=1 sinks=0 [client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ... [client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ... [afr-self-heal-common.c:476:afr_log_selfheal] 0-patchy-replicate-0: Completed metadata selfheal on beb51df4-9670-4b77-8a59-7d0ae7f84b28. source=0 sinks=1 [afr-self-heald.c:686:afr_shd_full_healer] 0-patchy-replicate-0: finished full sweep on subvol patchy-client-1 [afr-self-heald.c:686:afr_shd_full_healer] 0-patchy-replicate-0: finished full sweep on subvol patchy-client-0 -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz manu@xxxxxxxxxx _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-devel
