Re: Snapshot auto-delete unmount problem

Gary Lloyd <g.lloyd@xxxxxxxxxxx> · Wed, 31 May 2017 14:46:39 +0100

Here is the glusterd.log file as it has just happened with a manual removal of a snapshot.The lvs exist even after the snapshot gets removed. I'm also running samba on these with shadowcopy. 

Thanks

Gary Lloyd________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
+44 1782 733063
________________________________________________

On 31 May 2017 at 12:38, Mohammed Rafi K C <rkavunga@xxxxxxxxxx> wrote:

    Can you give us more logs on this issue. Also by any chance did
      somebody unmount the lv's in any cases ?

    Regards
    Rafi KC

    On 05/31/2017 03:00 PM, Gary Lloyd
      wrote:

        Hi I am having a problem deleting snapshots, gluster is
          failing to unmount them. I am running centos 7.3 with
          gluster-3.10.2-1

        here is some log output:

          [2017-05-31 09:21:39.961371] W [MSGID: 106057]
            [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find]
            0-management: Snap volume
331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snaps-331ec972f90d494d8a86dd4f69d718b7-brick1-b
            not found [Invalid argument]
          [2017-05-31 09:21:51.520811] W [MSGID: 106112]
            [glusterd-snapshot.c:8128:glusterd_handle_snap_limit]
            0-management: Soft-limit (value = 27) of volume shares1 is
            reached. Deleting snapshot Snap_GMT-2017.05.31-09.20.04.
          [2017-05-31 09:21:51.531729] E [MSGID: 106095]
            [glusterd-snapshot-utils.c:3359:glusterd_umount]
            0-management: umounting
            /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1
            failed (No such file or directory) [No such file or
            directory]
          [2017-05-31 09:22:00.540373] E [MSGID: 106038]
            [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove]
            0-management: umount failed for path
            /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1
            (brick:
            /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b):
            No such file or directory.
          [2017-05-31 09:22:02.442048] W [MSGID: 106033]
            [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove]
            0-management: Failed to rmdir:
            /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/, err:
            Directory not empty. More than one glusterd running on this
            node. [Directory not empty]
          [2017-05-31 09:22:02.443336] W [MSGID: 106039]
            [glusterd-snapshot-utils.c:55:glusterd_snapobject_delete]
            0-management: Failed destroying lockof snap
            Snap_GMT-2017.05.31-09.20.04
          [2017-05-31 09:22:02.444038] I [MSGID: 106144]
            [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing
            brick
            /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b
            on port 49157

        Can anyone help ?

        Thanks

                Gary Lloyd
                  ________________________________________________

                    I.T.
                        Systems:Keele University

                      Finance &
                        IT Directorate
                    Keele:Staffs:IC1
                        Building:ST5 5NB:UK

                      ________________________________________________

      _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

[2017-05-28 02:48:02.172739] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume c77d369f14784b04857c59c96755ac97.glust01-lc.run-gluster-snaps-c77d369f14784b04857c59c96755ac97-brick2-b not found [Invalid argument]
[2017-05-28 09:00:03.038971] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/db2695d36fc3400d97984c4141a8d2f5/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-28 09:00:03.039025] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume db2695d36fc3400d97984c4141a8d2f5.glust01-lc.run-gluster-snaps-db2695d36fc3400d97984c4141a8d2f5-brick2-b not found [Invalid argument]
[2017-05-28 11:00:02.812555] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/2deeaa2d1c0741f5ad8eb2c0bda1e452/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-28 11:00:02.812611] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 2deeaa2d1c0741f5ad8eb2c0bda1e452.glust01-lc.run-gluster-snaps-2deeaa2d1c0741f5ad8eb2c0bda1e452-brick2-b not found [Invalid argument]
[2017-05-28 13:00:04.953488] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/5b14d311e3b947518520eacc03210841/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-28 13:00:04.953540] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 5b14d311e3b947518520eacc03210841.glust01-lc.run-gluster-snaps-5b14d311e3b947518520eacc03210841-brick2-b not found [Invalid argument]
[2017-05-28 15:00:03.236386] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/a795995808a445589abb2114dac57e8d/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-28 15:00:03.236445] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume a795995808a445589abb2114dac57e8d.glust01-lc.run-gluster-snaps-a795995808a445589abb2114dac57e8d-brick2-b not found [Invalid argument]
[2017-05-28 17:00:02.734250] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/cf2c06d2e79546adaaa3f4ad1fa5ea54/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-28 17:00:02.734324] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume cf2c06d2e79546adaaa3f4ad1fa5ea54.glust01-lc.run-gluster-snaps-cf2c06d2e79546adaaa3f4ad1fa5ea54-brick2-b not found [Invalid argument]
[2017-05-29 09:00:05.389311] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/a0faf802c0ab4024a7305032a04ff80f/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-29 09:00:05.389417] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume a0faf802c0ab4024a7305032a04ff80f.glust01-lc.run-gluster-snaps-a0faf802c0ab4024a7305032a04ff80f-brick2-b not found [Invalid argument]
[2017-05-29 11:00:05.898815] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/eda0679715dd42549e6f37bcd5176344/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-29 11:00:05.898897] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume eda0679715dd42549e6f37bcd5176344.glust01-lc.run-gluster-snaps-eda0679715dd42549e6f37bcd5176344-brick2-b not found [Invalid argument]
[2017-05-29 13:00:05.662774] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/ec832a972dda403db2e37fb87c442415/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-29 13:00:05.662876] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume ec832a972dda403db2e37fb87c442415.glust01-lc.run-gluster-snaps-ec832a972dda403db2e37fb87c442415-brick2-b not found [Invalid argument]
[2017-05-29 15:00:05.620300] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/57a0149fa85649a9b52b5a630aab50d7/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-29 15:00:05.620389] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 57a0149fa85649a9b52b5a630aab50d7.glust01-lc.run-gluster-snaps-57a0149fa85649a9b52b5a630aab50d7-brick2-b not found [Invalid argument]
[2017-05-29 17:00:05.589324] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/a6c87a5a99be43f196c682a5cc2ddb3b/brick2/b to existing process for /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b
[2017-05-29 17:00:05.589409] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume a6c87a5a99be43f196c682a5cc2ddb3b.glust01-lc.run-gluster-snaps-a6c87a5a99be43f196c682a5cc2ddb3b-brick2-b not found [Invalid argument]
[2017-05-30 07:44:02.344190] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.26-13.26.51 for decommission.
[2017-05-30 07:44:02.353656] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:44:02.353656] and [2017-05-30 07:44:08.362024]
[2017-05-30 07:44:11.362154] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2 (brick: /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/brick2/b): No such file or directory.
[2017-05-30 07:44:11.573999] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/c77d369f14784b04857c59c96755ac97/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:44:21.196685] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.26-17.00.01 for decommission.
[2017-05-30 07:44:21.201067] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/fe0931e00cbd44559e4774a34d8d63af/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 07:44:30.210141] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/fe0931e00cbd44559e4774a34d8d63af/brick2 (brick: /run/gluster/snaps/fe0931e00cbd44559e4774a34d8d63af/brick2/b): No such file or directory.
[2017-05-30 07:44:30.418549] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/fe0931e00cbd44559e4774a34d8d63af/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:44:40.109738] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.27-09.00.01 for decommission.
[2017-05-30 07:44:40.115404] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/1db574c3f832422aba62da8373067e46/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/1db574c3f832422aba62da8373067e46/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:44:40.115404] and [2017-05-30 07:44:46.124932]
[2017-05-30 07:44:49.125059] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/1db574c3f832422aba62da8373067e46/brick2 (brick: /run/gluster/snaps/1db574c3f832422aba62da8373067e46/brick2/b): No such file or directory.
[2017-05-30 07:44:49.646194] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/1db574c3f832422aba62da8373067e46/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:44:49.647474] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/1db574c3f832422aba62da8373067e46/brick2/b on port 49154
[2017-05-30 07:44:59.317103] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.27-11.00.01 for decommission.
[2017-05-30 07:44:59.321571] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/df913bc1db304c6e83e4f90acd6d8214/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/df913bc1db304c6e83e4f90acd6d8214/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:44:59.321571] and [2017-05-30 07:45:05.331895]
[2017-05-30 07:45:08.332029] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/df913bc1db304c6e83e4f90acd6d8214/brick2 (brick: /run/gluster/snaps/df913bc1db304c6e83e4f90acd6d8214/brick2/b): No such file or directory.
[2017-05-30 07:45:08.554373] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/df913bc1db304c6e83e4f90acd6d8214/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:45:08.555455] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/df913bc1db304c6e83e4f90acd6d8214/brick2/b on port 49154
[2017-05-30 07:45:17.930129] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.27-13.00.01 for decommission.
[2017-05-30 07:45:17.934404] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/8905cdd1363449129ef3d2ef9965ea09/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/8905cdd1363449129ef3d2ef9965ea09/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:45:17.934404] and [2017-05-30 07:45:23.943033]
[2017-05-30 07:45:26.943150] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/8905cdd1363449129ef3d2ef9965ea09/brick2 (brick: /run/gluster/snaps/8905cdd1363449129ef3d2ef9965ea09/brick2/b): No such file or directory.
[2017-05-30 07:45:27.508946] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/8905cdd1363449129ef3d2ef9965ea09/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:45:27.510319] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/8905cdd1363449129ef3d2ef9965ea09/brick2/b on port 49154
[2017-05-30 07:45:37.019966] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.27-15.00.01 for decommission.
[2017-05-30 07:45:37.024371] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/088b4a2b4df741c08f37ce3543d152c3/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/088b4a2b4df741c08f37ce3543d152c3/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:45:37.024371] and [2017-05-30 07:45:43.034927]
[2017-05-30 07:45:46.035054] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/088b4a2b4df741c08f37ce3543d152c3/brick2 (brick: /run/gluster/snaps/088b4a2b4df741c08f37ce3543d152c3/brick2/b): No such file or directory.
[2017-05-30 07:45:46.261533] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/088b4a2b4df741c08f37ce3543d152c3/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:45:46.262713] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/088b4a2b4df741c08f37ce3543d152c3/brick2/b on port 49154
[2017-05-30 07:45:55.636473] I [socket.c:3507:socket_submit_request] 0-socket.management: not connected (priv->connected = -1)
[2017-05-30 07:45:55.636498] W [rpcsvc.c:1116:rpcsvc_callback_submit] 0-rpcsvc: transmission of rpc-request failed
[2017-05-30 07:45:55.647530] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.27-17.00.01 for decommission.
[2017-05-30 07:45:55.651268] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ef7001f097e847e28ce4830a1d67d916/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ef7001f097e847e28ce4830a1d67d916/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:45:55.651268] and [2017-05-30 07:46:01.661326]
[2017-05-30 07:46:04.661478] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/ef7001f097e847e28ce4830a1d67d916/brick2 (brick: /run/gluster/snaps/ef7001f097e847e28ce4830a1d67d916/brick2/b): No such file or directory.
[2017-05-30 07:46:04.914385] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/ef7001f097e847e28ce4830a1d67d916/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:46:04.915738] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/ef7001f097e847e28ce4830a1d67d916/brick2/b on port 49154
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/fe0931e00cbd44559e4774a34d8d63af/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:44:21.201067] and [2017-05-30 07:44:27.210014]
[2017-05-30 07:46:14.212951] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.28-09.00.01 for decommission.
[2017-05-30 07:46:14.218874] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/db2695d36fc3400d97984c4141a8d2f5/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 07:46:23.228066] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/db2695d36fc3400d97984c4141a8d2f5/brick2 (brick: /run/gluster/snaps/db2695d36fc3400d97984c4141a8d2f5/brick2/b): No such file or directory.
[2017-05-30 07:46:24.375274] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/db2695d36fc3400d97984c4141a8d2f5/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:46:24.376637] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/db2695d36fc3400d97984c4141a8d2f5/brick2/b on port 49154
[2017-05-30 07:46:33.787044] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.28-11.00.01 for decommission.
[2017-05-30 07:46:33.790827] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/2deeaa2d1c0741f5ad8eb2c0bda1e452/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/2deeaa2d1c0741f5ad8eb2c0bda1e452/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:46:33.790827] and [2017-05-30 07:46:39.799503]
[2017-05-30 07:46:42.799675] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/2deeaa2d1c0741f5ad8eb2c0bda1e452/brick2 (brick: /run/gluster/snaps/2deeaa2d1c0741f5ad8eb2c0bda1e452/brick2/b): No such file or directory.
[2017-05-30 07:46:43.199633] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/2deeaa2d1c0741f5ad8eb2c0bda1e452/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:46:43.200752] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/2deeaa2d1c0741f5ad8eb2c0bda1e452/brick2/b on port 49154
[2017-05-30 07:46:52.597636] I [socket.c:3507:socket_submit_request] 0-socket.management: not connected (priv->connected = -1)
[2017-05-30 07:46:52.597661] W [rpcsvc.c:1116:rpcsvc_callback_submit] 0-rpcsvc: transmission of rpc-request failed
[2017-05-30 07:46:52.608923] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.28-13.00.01 for decommission.
[2017-05-30 07:46:52.613402] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/5b14d311e3b947518520eacc03210841/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/5b14d311e3b947518520eacc03210841/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:46:52.613402] and [2017-05-30 07:46:58.622148]
[2017-05-30 07:47:01.622287] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/5b14d311e3b947518520eacc03210841/brick2 (brick: /run/gluster/snaps/5b14d311e3b947518520eacc03210841/brick2/b): No such file or directory.
[2017-05-30 07:47:02.001382] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/5b14d311e3b947518520eacc03210841/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:47:02.002534] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/5b14d311e3b947518520eacc03210841/brick2/b on port 49154
[2017-05-30 07:47:11.525611] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.28-15.00.01 for decommission.
[2017-05-30 07:47:11.531262] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a795995808a445589abb2114dac57e8d/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a795995808a445589abb2114dac57e8d/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:47:11.531262] and [2017-05-30 07:47:17.540143]
[2017-05-30 07:47:20.540273] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/a795995808a445589abb2114dac57e8d/brick2 (brick: /run/gluster/snaps/a795995808a445589abb2114dac57e8d/brick2/b): No such file or directory.
[2017-05-30 07:47:20.828246] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/a795995808a445589abb2114dac57e8d/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:47:20.829666] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/a795995808a445589abb2114dac57e8d/brick2/b on port 49154
[2017-05-30 07:47:30.128213] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.28-17.00.01 for decommission.
[2017-05-30 07:47:30.132036] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/cf2c06d2e79546adaaa3f4ad1fa5ea54/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/cf2c06d2e79546adaaa3f4ad1fa5ea54/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:47:30.132036] and [2017-05-30 07:47:36.140696]
[2017-05-30 07:47:39.140841] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/cf2c06d2e79546adaaa3f4ad1fa5ea54/brick2 (brick: /run/gluster/snaps/cf2c06d2e79546adaaa3f4ad1fa5ea54/brick2/b): No such file or directory.
[2017-05-30 07:47:39.462593] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/cf2c06d2e79546adaaa3f4ad1fa5ea54/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:47:39.463568] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/cf2c06d2e79546adaaa3f4ad1fa5ea54/brick2/b on port 49154
[2017-05-30 07:47:48.827243] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.29-09.00.01 for decommission.
[2017-05-30 07:47:48.831854] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a0faf802c0ab4024a7305032a04ff80f/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a0faf802c0ab4024a7305032a04ff80f/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:47:48.831854] and [2017-05-30 07:47:54.840715]
[2017-05-30 07:47:57.840846] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/a0faf802c0ab4024a7305032a04ff80f/brick2 (brick: /run/gluster/snaps/a0faf802c0ab4024a7305032a04ff80f/brick2/b): No such file or directory.
[2017-05-30 07:47:58.784138] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/a0faf802c0ab4024a7305032a04ff80f/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:47:58.785323] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/a0faf802c0ab4024a7305032a04ff80f/brick2/b on port 49154
[2017-05-30 07:48:08.445936] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.29-11.00.01 for decommission.
[2017-05-30 07:48:08.452284] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/eda0679715dd42549e6f37bcd5176344/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/db2695d36fc3400d97984c4141a8d2f5/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:46:14.218874] and [2017-05-30 07:46:20.227950]
[2017-05-30 07:48:11.457038] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/eda0679715dd42549e6f37bcd5176344/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 07:48:17.461442] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/eda0679715dd42549e6f37bcd5176344/brick2 (brick: /run/gluster/snaps/eda0679715dd42549e6f37bcd5176344/brick2/b): No such file or directory.
[2017-05-30 07:48:18.024190] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/eda0679715dd42549e6f37bcd5176344/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:48:18.025753] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/eda0679715dd42549e6f37bcd5176344/brick2/b on port 49154
[2017-05-30 07:48:27.889370] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.29-13.00.01 for decommission.
[2017-05-30 07:48:27.893539] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ec832a972dda403db2e37fb87c442415/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ec832a972dda403db2e37fb87c442415/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:48:27.893539] and [2017-05-30 07:48:33.903108]
[2017-05-30 07:48:36.903241] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/ec832a972dda403db2e37fb87c442415/brick2 (brick: /run/gluster/snaps/ec832a972dda403db2e37fb87c442415/brick2/b): No such file or directory.
[2017-05-30 07:48:37.092407] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/ec832a972dda403db2e37fb87c442415/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:48:37.093492] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/ec832a972dda403db2e37fb87c442415/brick2/b on port 49154
[2017-05-30 07:48:46.322286] I [socket.c:3507:socket_submit_request] 0-socket.management: not connected (priv->connected = -1)
[2017-05-30 07:48:46.322308] W [rpcsvc.c:1116:rpcsvc_callback_submit] 0-rpcsvc: transmission of rpc-request failed
[2017-05-30 07:48:46.334219] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.29-15.00.01 for decommission.
[2017-05-30 07:48:46.338050] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/57a0149fa85649a9b52b5a630aab50d7/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/57a0149fa85649a9b52b5a630aab50d7/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:48:46.338050] and [2017-05-30 07:48:52.347146]
[2017-05-30 07:48:55.347337] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/57a0149fa85649a9b52b5a630aab50d7/brick2 (brick: /run/gluster/snaps/57a0149fa85649a9b52b5a630aab50d7/brick2/b): No such file or directory.
[2017-05-30 07:48:55.587527] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/57a0149fa85649a9b52b5a630aab50d7/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:48:55.588737] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/57a0149fa85649a9b52b5a630aab50d7/brick2/b on port 49154
[2017-05-30 07:49:05.426268] I [socket.c:3507:socket_submit_request] 0-socket.management: not connected (priv->connected = -1)
[2017-05-30 07:49:05.426318] W [rpcsvc.c:1116:rpcsvc_callback_submit] 0-rpcsvc: transmission of rpc-request failed
[2017-05-30 07:49:05.438063] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.29-17.00.01 for decommission.
[2017-05-30 07:49:05.443668] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a6c87a5a99be43f196c682a5cc2ddb3b/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a6c87a5a99be43f196c682a5cc2ddb3b/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 07:49:05.443668] and [2017-05-30 07:49:11.453750]
[2017-05-30 07:49:14.453871] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/a6c87a5a99be43f196c682a5cc2ddb3b/brick2 (brick: /run/gluster/snaps/a6c87a5a99be43f196c682a5cc2ddb3b/brick2/b): No such file or directory.
[2017-05-30 07:49:14.702164] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/a6c87a5a99be43f196c682a5cc2ddb3b/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 07:49:14.703355] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/a6c87a5a99be43f196c682a5cc2ddb3b/brick2/b on port 49154
[2017-05-30 07:49:14.703723] E [MSGID: 106057] [glusterd-handshake.c:94:get_snap_volname_and_volinfo] 0-management: Failed to fetch snap Scheduled-shares-hourly-shares1_GMT-2017.05.29-17.00.01 [Invalid argument]
[2017-05-30 07:49:14.703742] E [MSGID: 106063] [glusterd-handshake.c:274:build_volfile_path] 0-management: Failed to get snap volinfo from path (/snaps/Scheduled-shares-hourly-shares1_GMT-2017.05.29-17.00.01/a6c87a5a99be43f196c682a5cc2ddb3b)
[2017-05-30 07:49:15.752993] E [MSGID: 106057] [glusterd-handshake.c:94:get_snap_volname_and_volinfo] 0-management: Failed to fetch snap Scheduled-shares-hourly-shares1_GMT-2017.05.29-17.00.01 [Invalid argument]
[2017-05-30 07:49:15.753038] E [MSGID: 106063] [glusterd-handshake.c:274:build_volfile_path] 0-management: Failed to get snap volinfo from path (/snaps/Scheduled-shares-hourly-shares1_GMT-2017.05.29-17.00.01/a6c87a5a99be43f196c682a5cc2ddb3b)
[2017-05-30 07:49:36.029382] W [socket.c:593:__socket_rwv] 0-management: readv on /var/run/gluster/187460d26abbb83ae43340a792e33477.socket failed (No data available)
[2017-05-30 07:49:36.029433] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
[2017-05-30 07:48:14.461291] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/eda0679715dd42549e6f37bcd5176344/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 07:49:36.029455] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
[2017-05-30 08:19:23.211612] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 08:19:23.234407] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 08:19:23.234545] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b has disconnected from glusterd.
[2017-05-30 08:19:23.234792] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 06f33eb03f48404a9e2d9106bb106b8a.glust01-lc.run-gluster-snaps-06f33eb03f48404a9e2d9106bb106b8a-brick2-b not found [Invalid argument]
[2017-05-30 08:19:23.328587] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b on port 49158
[2017-05-30 08:19:29.473127] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/73a9629c3f8248d88b3c252835c2e740/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:29.473234] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 73a9629c3f8248d88b3c252835c2e740.glust01-lc.run-gluster-snaps-73a9629c3f8248d88b3c252835c2e740-brick2-b not found [Invalid argument]
[2017-05-30 08:19:32.875665] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/32c460d1a6804c6195c3e95535180482/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:32.875720] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 32c460d1a6804c6195c3e95535180482.glust01-lc.run-gluster-snaps-32c460d1a6804c6195c3e95535180482-brick2-b not found [Invalid argument]
[2017-05-30 08:19:35.553720] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/a48cebbf65444bfebaacc6781366c827/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:35.553771] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume a48cebbf65444bfebaacc6781366c827.glust01-lc.run-gluster-snaps-a48cebbf65444bfebaacc6781366c827-brick2-b not found [Invalid argument]
[2017-05-30 08:19:38.298432] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/2f13b81adf054233a3144e090aba2a8f/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:38.298489] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 2f13b81adf054233a3144e090aba2a8f.glust01-lc.run-gluster-snaps-2f13b81adf054233a3144e090aba2a8f-brick2-b not found [Invalid argument]
[2017-05-30 08:19:40.980214] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/a8663967c17d4c84a0a3311b3fdfa102/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:40.980274] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume a8663967c17d4c84a0a3311b3fdfa102.glust01-lc.run-gluster-snaps-a8663967c17d4c84a0a3311b3fdfa102-brick2-b not found [Invalid argument]
[2017-05-30 08:19:43.880077] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/39acd7c9410342d594ba9eb9d65044c8/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:43.880133] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 39acd7c9410342d594ba9eb9d65044c8.glust01-lc.run-gluster-snaps-39acd7c9410342d594ba9eb9d65044c8-brick2-b not found [Invalid argument]
[2017-05-30 08:19:46.564330] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/8687c8a516bc483e9f84a1e66c1a9178/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:46.564385] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 8687c8a516bc483e9f84a1e66c1a9178.glust01-lc.run-gluster-snaps-8687c8a516bc483e9f84a1e66c1a9178-brick2-b not found [Invalid argument]
[2017-05-30 08:19:50.084853] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/c4447980075346cbb6a3ebfd1bbfd0d5/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:50.084908] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume c4447980075346cbb6a3ebfd1bbfd0d5.glust01-lc.run-gluster-snaps-c4447980075346cbb6a3ebfd1bbfd0d5-brick2-b not found [Invalid argument]
[2017-05-30 08:19:53.014885] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/c5f15b12a51c42ffa42f6ef94b84b77f/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:53.014972] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume c5f15b12a51c42ffa42f6ef94b84b77f.glust01-lc.run-gluster-snaps-c5f15b12a51c42ffa42f6ef94b84b77f-brick2-b not found [Invalid argument]
[2017-05-30 08:19:55.706382] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/1efe104bd4054ae5a261ace7a007e951/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:55.706433] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 1efe104bd4054ae5a261ace7a007e951.glust01-lc.run-gluster-snaps-1efe104bd4054ae5a261ace7a007e951-brick2-b not found [Invalid argument]
[2017-05-30 08:19:59.983599] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/9889754298064d86ba559bd37ce1feb4/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:19:59.983667] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 9889754298064d86ba559bd37ce1feb4.glust01-lc.run-gluster-snaps-9889754298064d86ba559bd37ce1feb4-brick2-b not found [Invalid argument]
[2017-05-30 08:20:03.136464] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/65a2550f72844cf19e1d0cb5d3fe5c65/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:03.136522] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 65a2550f72844cf19e1d0cb5d3fe5c65.glust01-lc.run-gluster-snaps-65a2550f72844cf19e1d0cb5d3fe5c65-brick2-b not found [Invalid argument]
[2017-05-30 08:20:05.805854] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/52096fb5008d437b9d50d4810768311c/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:05.805932] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 52096fb5008d437b9d50d4810768311c.glust01-lc.run-gluster-snaps-52096fb5008d437b9d50d4810768311c-brick2-b not found [Invalid argument]
[2017-05-30 08:20:08.633859] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/2706ab2b5bc741a594de20d81d9f8452/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:08.633950] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 2706ab2b5bc741a594de20d81d9f8452.glust01-lc.run-gluster-snaps-2706ab2b5bc741a594de20d81d9f8452-brick2-b not found [Invalid argument]
[2017-05-30 08:20:11.634315] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/051da4af41cf4a29b5df07b5016cc7cc/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:11.634383] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 051da4af41cf4a29b5df07b5016cc7cc.glust01-lc.run-gluster-snaps-051da4af41cf4a29b5df07b5016cc7cc-brick2-b not found [Invalid argument]
[2017-05-30 08:20:14.611525] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/b4dd63609d9d45cdad3e48b50b7114e8/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:14.611579] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume b4dd63609d9d45cdad3e48b50b7114e8.glust01-lc.run-gluster-snaps-b4dd63609d9d45cdad3e48b50b7114e8-brick2-b not found [Invalid argument]
[2017-05-30 08:20:17.398659] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/aa38c9cd175d43c694ccf62d2f0b234a/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:17.398738] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume aa38c9cd175d43c694ccf62d2f0b234a.glust01-lc.run-gluster-snaps-aa38c9cd175d43c694ccf62d2f0b234a-brick2-b not found [Invalid argument]
[2017-05-30 08:20:20.803099] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/5784667f83764ce9970bcb00f748a763/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:20.803186] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 5784667f83764ce9970bcb00f748a763.glust01-lc.run-gluster-snaps-5784667f83764ce9970bcb00f748a763-brick2-b not found [Invalid argument]
[2017-05-30 08:20:23.591584] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/f87362c8b98b467d8265e8d622e2d942/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:23.591637] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume f87362c8b98b467d8265e8d622e2d942.glust01-lc.run-gluster-snaps-f87362c8b98b467d8265e8d622e2d942-brick2-b not found [Invalid argument]
[2017-05-30 08:20:27.421564] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/4457cdf2d3e14338b0a958547db7fa12/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:27.421640] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 4457cdf2d3e14338b0a958547db7fa12.glust01-lc.run-gluster-snaps-4457cdf2d3e14338b0a958547db7fa12-brick2-b not found [Invalid argument]
[2017-05-30 08:20:30.363138] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/ad555b9d9c1c43e0a3f5a2bcb3920b85/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:30.363241] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume ad555b9d9c1c43e0a3f5a2bcb3920b85.glust01-lc.run-gluster-snaps-ad555b9d9c1c43e0a3f5a2bcb3920b85-brick2-b not found [Invalid argument]
[2017-05-30 08:20:33.301084] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/9fde855054c241ce85be177bad6efd87/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:33.301145] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 9fde855054c241ce85be177bad6efd87.glust01-lc.run-gluster-snaps-9fde855054c241ce85be177bad6efd87-brick2-b not found [Invalid argument]
[2017-05-30 08:20:36.128662] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/844b78400ab4482aa172d8c5b59c689f/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:36.128725] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 844b78400ab4482aa172d8c5b59c689f.glust01-lc.run-gluster-snaps-844b78400ab4482aa172d8c5b59c689f-brick2-b not found [Invalid argument]
[2017-05-30 08:20:38.812086] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/a6040b3c00b7449a9a4dac0a80b55d9d/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:38.812166] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume a6040b3c00b7449a9a4dac0a80b55d9d.glust01-lc.run-gluster-snaps-a6040b3c00b7449a9a4dac0a80b55d9d-brick2-b not found [Invalid argument]
[2017-05-30 08:20:41.935254] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/2e53105cd40843609d3a71a1d0046ec8/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:41.935380] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 2e53105cd40843609d3a71a1d0046ec8.glust01-lc.run-gluster-snaps-2e53105cd40843609d3a71a1d0046ec8-brick2-b not found [Invalid argument]
[2017-05-30 08:20:44.673487] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/22d8d112d4614d2cb5a0702b4556e90c/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:44.673556] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 22d8d112d4614d2cb5a0702b4556e90c.glust01-lc.run-gluster-snaps-22d8d112d4614d2cb5a0702b4556e90c-brick2-b not found [Invalid argument]
[2017-05-30 08:20:47.442292] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/757b3b384d714f34bc9b82e799971187/brick2/b to existing process for /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b
[2017-05-30 08:20:47.442376] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 757b3b384d714f34bc9b82e799971187.glust01-lc.run-gluster-snaps-757b3b384d714f34bc9b82e799971187-brick2-b not found [Invalid argument]
[2017-05-30 08:20:48.454552] W [MSGID: 106112] [glusterd-snapshot.c:8128:glusterd_handle_snap_limit] 0-management: Soft-limit (value = 27) of volume shares1 is reached. Deleting snapshot Snap_GMT-2017.05.30-08.19.21.
[2017-05-30 08:20:48.463440] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:20:48.463440] and [2017-05-30 08:20:54.472876]
[2017-05-30 08:20:57.472993] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2 (brick: /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b): No such file or directory.
[2017-05-30 08:20:58.440710] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:20:58.441604] W [MSGID: 106039] [glusterd-snapshot-utils.c:55:glusterd_snapobject_delete] 0-management: Failed destroying lockof snap Snap_GMT-2017.05.30-08.19.21
[2017-05-30 08:20:58.442364] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/06f33eb03f48404a9e2d9106bb106b8a/brick2/b on port 49158
[2017-05-30 08:21:19.848116] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/3970bb21c91443779af062a2a3aa0356/brick2/b to existing process for /run/gluster/snaps/73a9629c3f8248d88b3c252835c2e740/brick2/b
[2017-05-30 08:21:19.848201] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 3970bb21c91443779af062a2a3aa0356.glust01-lc.run-gluster-snaps-3970bb21c91443779af062a2a3aa0356-brick2-b not found [Invalid argument]
[2017-05-30 08:21:21.194470] W [MSGID: 106112] [glusterd-snapshot.c:8128:glusterd_handle_snap_limit] 0-management: Soft-limit (value = 27) of volume shares1 is reached. Deleting snapshot Snap_GMT-2017.05.30-08.19.28.
[2017-05-30 08:21:21.203736] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/73a9629c3f8248d88b3c252835c2e740/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/73a9629c3f8248d88b3c252835c2e740/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:21:21.203736] and [2017-05-30 08:21:27.213970]
[2017-05-30 08:21:30.214132] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/73a9629c3f8248d88b3c252835c2e740/brick2 (brick: /run/gluster/snaps/73a9629c3f8248d88b3c252835c2e740/brick2/b): No such file or directory.
[2017-05-30 08:21:30.434312] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/73a9629c3f8248d88b3c252835c2e740/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:21:30.435321] W [MSGID: 106039] [glusterd-snapshot-utils.c:55:glusterd_snapobject_delete] 0-management: Failed destroying lockof snap Snap_GMT-2017.05.30-08.19.28
[2017-05-30 08:21:30.436912] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/73a9629c3f8248d88b3c252835c2e740/brick2/b on port 49158
[2017-05-30 08:27:40.913776] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.31 for decommission.
[2017-05-30 08:27:40.922853] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/32c460d1a6804c6195c3e95535180482/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 08:27:49.932260] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/32c460d1a6804c6195c3e95535180482/brick2 (brick: /run/gluster/snaps/32c460d1a6804c6195c3e95535180482/brick2/b): No such file or directory.
[2017-05-30 08:27:50.287456] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/32c460d1a6804c6195c3e95535180482/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:27:50.288584] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/32c460d1a6804c6195c3e95535180482/brick2/b on port 49158
[2017-05-30 08:27:59.520038] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.34 for decommission.
[2017-05-30 08:27:59.524275] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a48cebbf65444bfebaacc6781366c827/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a48cebbf65444bfebaacc6781366c827/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:27:59.524275] and [2017-05-30 08:28:05.535939]
[2017-05-30 08:28:08.536073] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/a48cebbf65444bfebaacc6781366c827/brick2 (brick: /run/gluster/snaps/a48cebbf65444bfebaacc6781366c827/brick2/b): No such file or directory.
[2017-05-30 08:28:08.850570] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/a48cebbf65444bfebaacc6781366c827/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:28:08.851625] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/a48cebbf65444bfebaacc6781366c827/brick2/b on port 49158
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/32c460d1a6804c6195c3e95535180482/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:27:40.922853] and [2017-05-30 08:27:46.932118]
[2017-05-30 08:28:18.216473] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.37 for decommission.
[2017-05-30 08:28:18.220989] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/2f13b81adf054233a3144e090aba2a8f/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 08:28:27.229237] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/2f13b81adf054233a3144e090aba2a8f/brick2 (brick: /run/gluster/snaps/2f13b81adf054233a3144e090aba2a8f/brick2/b): No such file or directory.
[2017-05-30 08:28:27.659379] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/2f13b81adf054233a3144e090aba2a8f/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:28:27.660404] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/2f13b81adf054233a3144e090aba2a8f/brick2/b on port 49158
[2017-05-30 08:28:37.272157] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.39 for decommission.
[2017-05-30 08:28:37.276873] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a8663967c17d4c84a0a3311b3fdfa102/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a8663967c17d4c84a0a3311b3fdfa102/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:28:37.276873] and [2017-05-30 08:28:43.285444]
[2017-05-30 08:28:46.285569] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/a8663967c17d4c84a0a3311b3fdfa102/brick2 (brick: /run/gluster/snaps/a8663967c17d4c84a0a3311b3fdfa102/brick2/b): No such file or directory.
[2017-05-30 08:28:46.673134] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/a8663967c17d4c84a0a3311b3fdfa102/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:28:46.674324] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/a8663967c17d4c84a0a3311b3fdfa102/brick2/b on port 49158
[2017-05-30 08:28:55.938809] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.42 for decommission.
[2017-05-30 08:28:55.943042] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/39acd7c9410342d594ba9eb9d65044c8/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/39acd7c9410342d594ba9eb9d65044c8/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:28:55.943042] and [2017-05-30 08:29:01.953421]
[2017-05-30 08:29:04.953541] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/39acd7c9410342d594ba9eb9d65044c8/brick2 (brick: /run/gluster/snaps/39acd7c9410342d594ba9eb9d65044c8/brick2/b): No such file or directory.
[2017-05-30 08:29:05.141080] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/39acd7c9410342d594ba9eb9d65044c8/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:29:05.142269] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/39acd7c9410342d594ba9eb9d65044c8/brick2/b on port 49158
[2017-05-30 08:29:14.438133] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.45 for decommission.
[2017-05-30 08:29:14.442177] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/8687c8a516bc483e9f84a1e66c1a9178/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/8687c8a516bc483e9f84a1e66c1a9178/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:29:14.442177] and [2017-05-30 08:29:20.450340]
[2017-05-30 08:29:23.450461] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/8687c8a516bc483e9f84a1e66c1a9178/brick2 (brick: /run/gluster/snaps/8687c8a516bc483e9f84a1e66c1a9178/brick2/b): No such file or directory.
[2017-05-30 08:29:23.565552] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/8687c8a516bc483e9f84a1e66c1a9178/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:29:23.566631] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/8687c8a516bc483e9f84a1e66c1a9178/brick2/b on port 49158
[2017-05-30 08:29:32.932065] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.48 for decommission.
[2017-05-30 08:29:32.936123] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/c4447980075346cbb6a3ebfd1bbfd0d5/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/c4447980075346cbb6a3ebfd1bbfd0d5/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:29:32.936123] and [2017-05-30 08:29:38.944891]
[2017-05-30 08:29:41.945070] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/c4447980075346cbb6a3ebfd1bbfd0d5/brick2 (brick: /run/gluster/snaps/c4447980075346cbb6a3ebfd1bbfd0d5/brick2/b): No such file or directory.
[2017-05-30 08:29:42.200849] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/c4447980075346cbb6a3ebfd1bbfd0d5/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:29:42.202127] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/c4447980075346cbb6a3ebfd1bbfd0d5/brick2/b on port 49158
[2017-05-30 08:29:51.397570] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.51 for decommission.
[2017-05-30 08:29:51.402122] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/c5f15b12a51c42ffa42f6ef94b84b77f/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/c5f15b12a51c42ffa42f6ef94b84b77f/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:29:51.402122] and [2017-05-30 08:29:57.410724]
[2017-05-30 08:30:00.410841] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/c5f15b12a51c42ffa42f6ef94b84b77f/brick2 (brick: /run/gluster/snaps/c5f15b12a51c42ffa42f6ef94b84b77f/brick2/b): No such file or directory.
[2017-05-30 08:30:00.910251] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/c5f15b12a51c42ffa42f6ef94b84b77f/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:30:00.911390] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/c5f15b12a51c42ffa42f6ef94b84b77f/brick2/b on port 49158
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/2f13b81adf054233a3144e090aba2a8f/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:28:18.220989] and [2017-05-30 08:28:24.229103]
[2017-05-30 08:30:10.307058] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.54 for decommission.
[2017-05-30 08:30:10.313138] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/1efe104bd4054ae5a261ace7a007e951/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 08:30:19.322508] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/1efe104bd4054ae5a261ace7a007e951/brick2 (brick: /run/gluster/snaps/1efe104bd4054ae5a261ace7a007e951/brick2/b): No such file or directory.
[2017-05-30 08:30:19.732766] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/1efe104bd4054ae5a261ace7a007e951/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:30:19.734510] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/1efe104bd4054ae5a261ace7a007e951/brick2/b on port 49158
[2017-05-30 08:30:29.330698] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.19.58 for decommission.
[2017-05-30 08:30:29.335526] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/9889754298064d86ba559bd37ce1feb4/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/9889754298064d86ba559bd37ce1feb4/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:30:29.335526] and [2017-05-30 08:30:35.344084]
[2017-05-30 08:30:38.344283] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/9889754298064d86ba559bd37ce1feb4/brick2 (brick: /run/gluster/snaps/9889754298064d86ba559bd37ce1feb4/brick2/b): No such file or directory.
[2017-05-30 08:30:38.562465] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/9889754298064d86ba559bd37ce1feb4/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:30:38.563713] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/9889754298064d86ba559bd37ce1feb4/brick2/b on port 49158
[2017-05-30 08:30:48.469593] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.01 for decommission.
[2017-05-30 08:30:48.473898] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/65a2550f72844cf19e1d0cb5d3fe5c65/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/65a2550f72844cf19e1d0cb5d3fe5c65/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:30:48.473898] and [2017-05-30 08:30:54.482259]
[2017-05-30 08:30:57.482399] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/65a2550f72844cf19e1d0cb5d3fe5c65/brick2 (brick: /run/gluster/snaps/65a2550f72844cf19e1d0cb5d3fe5c65/brick2/b): No such file or directory.
[2017-05-30 08:30:57.768336] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/65a2550f72844cf19e1d0cb5d3fe5c65/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:30:57.769522] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/65a2550f72844cf19e1d0cb5d3fe5c65/brick2/b on port 49158
[2017-05-30 08:31:07.020243] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.04 for decommission.
[2017-05-30 08:31:07.025432] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/52096fb5008d437b9d50d4810768311c/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/52096fb5008d437b9d50d4810768311c/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:31:07.025432] and [2017-05-30 08:31:13.033742]
[2017-05-30 08:31:16.033876] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/52096fb5008d437b9d50d4810768311c/brick2 (brick: /run/gluster/snaps/52096fb5008d437b9d50d4810768311c/brick2/b): No such file or directory.
[2017-05-30 08:31:16.284911] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/52096fb5008d437b9d50d4810768311c/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:31:16.286183] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/52096fb5008d437b9d50d4810768311c/brick2/b on port 49158
[2017-05-30 08:31:25.875412] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.07 for decommission.
[2017-05-30 08:31:25.879446] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/2706ab2b5bc741a594de20d81d9f8452/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/2706ab2b5bc741a594de20d81d9f8452/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:31:25.879446] and [2017-05-30 08:31:31.887487]
[2017-05-30 08:31:34.887605] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/2706ab2b5bc741a594de20d81d9f8452/brick2 (brick: /run/gluster/snaps/2706ab2b5bc741a594de20d81d9f8452/brick2/b): No such file or directory.
[2017-05-30 08:31:35.078317] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/2706ab2b5bc741a594de20d81d9f8452/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:31:35.079349] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/2706ab2b5bc741a594de20d81d9f8452/brick2/b on port 49158
[2017-05-30 08:31:44.935470] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.10 for decommission.
[2017-05-30 08:31:44.940746] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/051da4af41cf4a29b5df07b5016cc7cc/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/051da4af41cf4a29b5df07b5016cc7cc/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:31:44.940746] and [2017-05-30 08:31:50.948559]
[2017-05-30 08:31:53.948689] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/051da4af41cf4a29b5df07b5016cc7cc/brick2 (brick: /run/gluster/snaps/051da4af41cf4a29b5df07b5016cc7cc/brick2/b): No such file or directory.
[2017-05-30 08:31:54.243957] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/051da4af41cf4a29b5df07b5016cc7cc/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:31:54.245047] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/051da4af41cf4a29b5df07b5016cc7cc/brick2/b on port 49158
[2017-05-30 08:32:03.508907] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.13 for decommission.
[2017-05-30 08:32:03.513379] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b4dd63609d9d45cdad3e48b50b7114e8/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/1efe104bd4054ae5a261ace7a007e951/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:30:10.313138] and [2017-05-30 08:30:16.322361]
[2017-05-30 08:32:06.517801] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b4dd63609d9d45cdad3e48b50b7114e8/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 08:32:09.521626] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b4dd63609d9d45cdad3e48b50b7114e8/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 08:32:12.521763] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/b4dd63609d9d45cdad3e48b50b7114e8/brick2 (brick: /run/gluster/snaps/b4dd63609d9d45cdad3e48b50b7114e8/brick2/b): No such file or directory.
[2017-05-30 08:32:12.960807] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/b4dd63609d9d45cdad3e48b50b7114e8/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:32:12.962163] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/b4dd63609d9d45cdad3e48b50b7114e8/brick2/b on port 49158
[2017-05-30 08:32:22.290935] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.16 for decommission.
[2017-05-30 08:32:22.294847] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/aa38c9cd175d43c694ccf62d2f0b234a/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/aa38c9cd175d43c694ccf62d2f0b234a/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:32:22.294847] and [2017-05-30 08:32:28.302533]
[2017-05-30 08:32:31.302651] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/aa38c9cd175d43c694ccf62d2f0b234a/brick2 (brick: /run/gluster/snaps/aa38c9cd175d43c694ccf62d2f0b234a/brick2/b): No such file or directory.
[2017-05-30 08:32:31.523236] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/aa38c9cd175d43c694ccf62d2f0b234a/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:32:31.524317] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/aa38c9cd175d43c694ccf62d2f0b234a/brick2/b on port 49158
[2017-05-30 08:32:41.267240] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.19 for decommission.
[2017-05-30 08:32:41.271267] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/5784667f83764ce9970bcb00f748a763/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/5784667f83764ce9970bcb00f748a763/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:32:41.271267] and [2017-05-30 08:32:47.279316]
[2017-05-30 08:32:50.279424] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/5784667f83764ce9970bcb00f748a763/brick2 (brick: /run/gluster/snaps/5784667f83764ce9970bcb00f748a763/brick2/b): No such file or directory.
[2017-05-30 08:32:50.558355] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/5784667f83764ce9970bcb00f748a763/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:32:50.559495] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/5784667f83764ce9970bcb00f748a763/brick2/b on port 49158
[2017-05-30 08:33:00.175651] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.22 for decommission.
[2017-05-30 08:33:00.179927] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/f87362c8b98b467d8265e8d622e2d942/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/f87362c8b98b467d8265e8d622e2d942/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:33:00.179927] and [2017-05-30 08:33:06.188801]
[2017-05-30 08:33:09.188928] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/f87362c8b98b467d8265e8d622e2d942/brick2 (brick: /run/gluster/snaps/f87362c8b98b467d8265e8d622e2d942/brick2/b): No such file or directory.
[2017-05-30 08:33:09.694087] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/f87362c8b98b467d8265e8d622e2d942/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:33:09.695303] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/f87362c8b98b467d8265e8d622e2d942/brick2/b on port 49158
[2017-05-30 08:33:18.900485] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.25 for decommission.
[2017-05-30 08:33:18.904901] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/4457cdf2d3e14338b0a958547db7fa12/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/4457cdf2d3e14338b0a958547db7fa12/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:33:18.904901] and [2017-05-30 08:33:24.913641]
[2017-05-30 08:33:27.913764] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/4457cdf2d3e14338b0a958547db7fa12/brick2 (brick: /run/gluster/snaps/4457cdf2d3e14338b0a958547db7fa12/brick2/b): No such file or directory.
[2017-05-30 08:33:28.180398] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/4457cdf2d3e14338b0a958547db7fa12/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:33:28.181462] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/4457cdf2d3e14338b0a958547db7fa12/brick2/b on port 49158
[2017-05-30 08:33:37.427623] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.29 for decommission.
[2017-05-30 08:33:37.431475] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ad555b9d9c1c43e0a3f5a2bcb3920b85/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ad555b9d9c1c43e0a3f5a2bcb3920b85/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:33:37.431475] and [2017-05-30 08:33:43.439711]
[2017-05-30 08:33:46.439847] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/ad555b9d9c1c43e0a3f5a2bcb3920b85/brick2 (brick: /run/gluster/snaps/ad555b9d9c1c43e0a3f5a2bcb3920b85/brick2/b): No such file or directory.
[2017-05-30 08:33:46.700791] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/ad555b9d9c1c43e0a3f5a2bcb3920b85/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:33:46.702060] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/ad555b9d9c1c43e0a3f5a2bcb3920b85/brick2/b on port 49158
[2017-05-30 08:33:56.059661] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.32 for decommission.
[2017-05-30 08:33:56.063300] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/9fde855054c241ce85be177bad6efd87/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/9fde855054c241ce85be177bad6efd87/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:33:56.063300] and [2017-05-30 08:34:02.071688]
[2017-05-30 08:34:05.071797] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/9fde855054c241ce85be177bad6efd87/brick2 (brick: /run/gluster/snaps/9fde855054c241ce85be177bad6efd87/brick2/b): No such file or directory.
[2017-05-30 08:34:05.776309] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/9fde855054c241ce85be177bad6efd87/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:34:05.777795] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/9fde855054c241ce85be177bad6efd87/brick2/b on port 49158
[2017-05-30 08:34:15.094109] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.34 for decommission.
[2017-05-30 08:34:15.099159] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/844b78400ab4482aa172d8c5b59c689f/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 08:34:24.107397] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/844b78400ab4482aa172d8c5b59c689f/brick2 (brick: /run/gluster/snaps/844b78400ab4482aa172d8c5b59c689f/brick2/b): No such file or directory.
[2017-05-30 08:34:24.332440] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/844b78400ab4482aa172d8c5b59c689f/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:34:24.333658] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/844b78400ab4482aa172d8c5b59c689f/brick2/b on port 49158
[2017-05-30 08:34:34.322197] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.37 for decommission.
[2017-05-30 08:34:34.327261] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a6040b3c00b7449a9a4dac0a80b55d9d/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a6040b3c00b7449a9a4dac0a80b55d9d/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:34:34.327261] and [2017-05-30 08:34:40.336206]
[2017-05-30 08:34:43.336336] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/a6040b3c00b7449a9a4dac0a80b55d9d/brick2 (brick: /run/gluster/snaps/a6040b3c00b7449a9a4dac0a80b55d9d/brick2/b): No such file or directory.
[2017-05-30 08:34:43.639834] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/a6040b3c00b7449a9a4dac0a80b55d9d/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:34:43.641019] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/a6040b3c00b7449a9a4dac0a80b55d9d/brick2/b on port 49158
[2017-05-30 08:34:53.067055] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.40 for decommission.
[2017-05-30 08:34:53.071009] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/2e53105cd40843609d3a71a1d0046ec8/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/2e53105cd40843609d3a71a1d0046ec8/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:34:53.071009] and [2017-05-30 08:34:59.078860]
[2017-05-30 08:35:02.079012] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/2e53105cd40843609d3a71a1d0046ec8/brick2 (brick: /run/gluster/snaps/2e53105cd40843609d3a71a1d0046ec8/brick2/b): No such file or directory.
[2017-05-30 08:35:02.394024] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/2e53105cd40843609d3a71a1d0046ec8/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:35:02.395082] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/2e53105cd40843609d3a71a1d0046ec8/brick2/b on port 49158
[2017-05-30 08:35:11.678007] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.43 for decommission.
[2017-05-30 08:35:11.683722] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/22d8d112d4614d2cb5a0702b4556e90c/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/22d8d112d4614d2cb5a0702b4556e90c/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:35:11.683722] and [2017-05-30 08:35:17.692434]
[2017-05-30 08:35:20.692594] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/22d8d112d4614d2cb5a0702b4556e90c/brick2 (brick: /run/gluster/snaps/22d8d112d4614d2cb5a0702b4556e90c/brick2/b): No such file or directory.
[2017-05-30 08:35:21.052567] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/22d8d112d4614d2cb5a0702b4556e90c/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:35:21.053913] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/22d8d112d4614d2cb5a0702b4556e90c/brick2/b on port 49158
[2017-05-30 08:35:30.648641] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.20.46 for decommission.
[2017-05-30 08:35:30.652816] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/757b3b384d714f34bc9b82e799971187/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/757b3b384d714f34bc9b82e799971187/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:35:30.652816] and [2017-05-30 08:35:36.661217]
[2017-05-30 08:35:39.661378] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/757b3b384d714f34bc9b82e799971187/brick2 (brick: /run/gluster/snaps/757b3b384d714f34bc9b82e799971187/brick2/b): No such file or directory.
[2017-05-30 08:35:39.961124] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/757b3b384d714f34bc9b82e799971187/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-30 08:35:39.962189] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/757b3b384d714f34bc9b82e799971187/brick2/b on port 49158
[2017-05-30 08:35:43.983094] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.21.18 for decommission.
[2017-05-30 08:35:43.987765] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/3970bb21c91443779af062a2a3aa0356/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-30 08:35:47.150442] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/3970bb21c91443779af062a2a3aa0356/brick2/b on port 49158
[2017-05-30 08:35:47.150570] W [socket.c:593:__socket_rwv] 0-management: readv on /var/run/gluster/107fb789df994535afb53ac625970dd4.socket failed (No data available)
[2017-05-30 08:35:47.150588] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
[2017-05-30 08:35:47.150651] W [socket.c:593:__socket_rwv] 0-socket.management: writev on 160.5.13.90:49040 failed (Broken pipe)
[2017-05-30 08:35:47.150660] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/844b78400ab4482aa172d8c5b59c689f/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-30 08:34:15.099159] and [2017-05-30 08:34:21.107281]
[2017-05-30 08:35:47.150605] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
[2017-05-30 08:46:35.642529] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f505d712dc5] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x7f505edaafd5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x7f505edaadfb] ) 0-: received signum (15), shutting down
[2017-05-30 08:47:24.223258] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-05-30 08:47:24.576958] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-05-30 08:47:24.577012] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory
[2017-05-30 08:47:24.655688] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.10.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2017-05-30 08:47:24.655758] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2017-05-30 08:47:24.655796] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-05-30 08:47:24.655814] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-05-30 08:47:24.724244] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory]
[2017-05-30 08:47:24.754591] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31000
[2017-05-30 08:47:25.192561] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-30 08:47:25.208364] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-30 08:47:25.208422] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-05-30 08:47:25.208462] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 08:47:25.212235] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 08:47:25.212222] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-05-30 08:47:25.237661] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 877c6f79-0201-4c41-986e-d2a065911ef8
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option transport.address-family inet
  8:     option rpc-auth-allow-insecure on
  9:     option event-threads 1
 10:     option ping-timeout 0
 11:     option transport.socket.read-fail-log off
 12:     option transport.socket.keepalive-interval 2
 13:     option transport.socket.keepalive-time 10
 14:     option transport-type rdma
 15:     option working-directory /var/lib/glusterd
 16: end-volume
 17:  
+------------------------------------------------------------------------------+
[2017-05-30 08:47:25.238924] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-30 08:47:27.606381] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04, host: glust01-li, port: 0
[2017-05-30 08:47:27.615501] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-05-30 08:47:27.627524] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-05-30 08:47:27.627945] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-05-30 08:47:27.628590] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-05-30 08:47:27.670790] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-05-30 08:47:27.670829] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-05-30 08:47:27.670909] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-05-30 08:47:28.677441] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-05-30 08:47:28.677714] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-05-30 08:47:28.677739] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-05-30 08:47:28.677782] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-05-30 08:47:28.677956] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-05-30 08:47:28.677986] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-05-30 08:47:28.678027] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-05-30 08:47:28.678189] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-05-30 08:47:28.678204] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-05-30 08:47:28.692943] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 08:47:28.704934] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 08:47:28.722487] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 08:47:28.722682] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-30 08:47:29.729370] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-30 08:47:30.750325] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-30 08:47:30.750699] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:47:30.750751] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 08:47:30.755688] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65, host: 10.8.28.2, port: 0
[2017-05-30 08:47:30.760743] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 08:47:30.760780] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 08:47:30.765481] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:47:30.766631] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 08:47:30.767232] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/shares1/b has disconnected from glusterd.
[2017-05-30 08:47:30.767480] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 08:47:30.767609] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/config/b has disconnected from glusterd.
[2017-05-30 08:47:30.767819] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 08:47:30.767960] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/shared_storage/b has disconnected from glusterd.
[2017-05-30 08:47:30.768162] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 08:47:30.769895] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-30 08:47:30.773469] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-30 08:47:30.791276] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:47:30.792682] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 08:47:30.792725] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 08:47:30.796103] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glust01-li (0), ret: 0, op_ret: 0
[2017-05-30 08:47:30.802869] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 08:47:30.803983] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 08:47:30.804011] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 08:47:30.807193] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.8.28.2 (0), ret: 0, op_ret: 0
[2017-05-30 08:47:30.836846] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:47:30.836897] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 08:47:30.841616] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:47:30.841655] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 08:47:30.841674] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 08:47:30.845471] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 08:47:31.027152] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-vol_config on port 49156
[2017-05-30 08:47:31.027454] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-shares1 on port 49155
[2017-05-30 08:47:31.212197] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shared_storage/b on port 49154
[2017-05-30 08:47:31.230721] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/config/b on port 49153
[2017-05-30 08:47:31.273811] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shares1/b on port 49152
[2017-05-30 08:50:59.701828] W [socket.c:593:__socket_rwv] 0-management: readv on 160.5.5.90:24007 failed (No data available)
[2017-05-30 08:50:59.701874] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <glust01-li> (<64390a1d-af72-4f10-be41-e23d4f881e04>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-05-30 08:50:59.701993] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol shares1 not held
[2017-05-30 08:50:59.702004] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for shares1
[2017-05-30 08:50:59.702019] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol vol_config not held
[2017-05-30 08:50:59.702025] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_config
[2017-05-30 08:50:59.702037] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol vol_shared_storage not held
[2017-05-30 08:50:59.702053] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_shared_storage
[2017-05-30 08:51:53.205908] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-30 08:51:53.211902] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:52:13.384817] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 08:52:13.389462] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glust01-li (0), ret: 0, op_ret: 0
[2017-05-30 08:52:13.395936] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:52:13.395976] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 08:52:16.544692] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:52:16.572678] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04, host: glust01-li, port: 0
[2017-05-30 08:52:16.576720] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:52:16.576749] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 08:52:16.611900] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 08:52:13.384866] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 08:53:24.794519] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 08:53:24.826704] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 08:53:24.827304] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b has disconnected from glusterd.
[2017-05-30 08:53:24.827575] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume c8405ce657904bcb9602437df2aaaacc.glust01-lc.run-gluster-snaps-c8405ce657904bcb9602437df2aaaacc-brick2-b not found [Invalid argument]
[2017-05-30 08:53:24.924366] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b on port 49157
[2017-05-30 08:54:45.176340] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b to existing process for /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b
[2017-05-30 08:54:45.176403] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 5dae480fc375494aa1199ff40ff051b9.glust01-lc.run-gluster-snaps-5dae480fc375494aa1199ff40ff051b9-brick2-b not found [Invalid argument]
[2017-05-30 09:00:05.091814] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b to existing process for /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b
[2017-05-30 09:00:05.091879] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume b9e1bf3fe9244f4ebd0bd563a74058aa.glust01-lc.run-gluster-snaps-b9e1bf3fe9244f4ebd0bd563a74058aa-brick2-b not found [Invalid argument]
[2017-05-30 09:03:49.027330] W [socket.c:593:__socket_rwv] 0-management: readv on 10.8.28.2:24007 failed (No data available)
[2017-05-30 09:03:49.027387] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <10.8.28.2> (<62fa28ac-7ab1-4f51-bc4d-45ea0005fb65>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-05-30 09:03:49.027558] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol shares1 not held
[2017-05-30 09:03:49.027570] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for shares1
[2017-05-30 09:03:49.027587] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol vol_config not held
[2017-05-30 09:03:49.027593] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_config
[2017-05-30 09:03:49.027608] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol vol_shared_storage not held
[2017-05-30 09:03:49.027614] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_shared_storage
[2017-05-30 09:04:23.404927] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-30 09:04:23.420745] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 09:04:30.440670] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 09:04:30.440710] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 09:04:30.444395] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.8.28.2 (0), ret: 0, op_ret: 0
[2017-05-30 09:04:30.450997] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 09:04:30.451025] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 09:04:31.484877] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 09:04:32.565732] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65, host: 10.8.28.2, port: 0
[2017-05-30 09:04:32.569086] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 09:04:32.569114] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 09:04:32.575281] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 09:47:33.953271] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <glust01-li> (<64390a1d-af72-4f10-be41-e23d4f881e04>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-05-30 09:47:33.953416] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol shares1 not held
[2017-05-30 09:47:33.953441] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for shares1
[2017-05-30 09:47:33.953456] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol vol_config not held
[2017-05-30 09:47:33.953463] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_config
[2017-05-30 09:47:33.953477] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f8c97a59eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f8c97a63dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f8c97b1941a] ) 0-management: Lock for vol vol_shared_storage not held
[2017-05-30 09:47:33.953483] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_shared_storage
[2017-05-30 09:48:15.752272] E [socket.c:2310:socket_connect_finish] 0-management: connection to 160.5.5.90:24007 failed (Connection refused)
[2017-05-30 09:48:18.732834] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-30 09:48:19.796975] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 09:48:19.797602] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 09:48:19.797628] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 09:48:19.801118] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glust01-li (0), ret: 0, op_ret: 0
[2017-05-30 09:48:19.808751] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04, host: glust01-li, port: 0
[2017-05-30 09:48:19.812241] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 09:48:19.812281] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 09:48:20.777954] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 09:48:20.778007] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 09:48:20.785865] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 09:48:20.797816] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 11:00:03.231971] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/ecf94a72083d4803980307db1b923772/brick2/b to existing process for /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b
[2017-05-30 11:00:03.232036] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume ecf94a72083d4803980307db1b923772.glust01-lc.run-gluster-snaps-ecf94a72083d4803980307db1b923772-brick2-b not found [Invalid argument]
[2017-05-30 12:08:40.610324] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f8ca1d9fdc5] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x7f8ca3437fd5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x7f8ca3437dfb] ) 0-: received signum (15), shutting down
[2017-05-30 13:53:13.128536] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-05-30 13:53:13.291644] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-05-30 13:53:13.291790] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory
[2017-05-30 13:53:13.379458] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.10.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2017-05-30 13:53:13.379484] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2017-05-30 13:53:13.379529] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-05-30 13:53:13.379537] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-05-30 13:53:13.411697] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory]
[2017-05-30 13:53:13.438953] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31000
[2017-05-30 13:53:13.821175] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-30 13:53:13.835751] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-30 13:53:13.835818] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-05-30 13:53:13.835856] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 13:53:13.841675] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 13:53:13.841666] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-05-30 13:53:14.191980] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 877c6f79-0201-4c41-986e-d2a065911ef8
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option transport.address-family inet
  8:     option rpc-auth-allow-insecure on
  9:     option event-threads 1
 10:     option ping-timeout 0
 11:     option transport.socket.read-fail-log off
 12:     option transport.socket.keepalive-interval 2
 13:     option transport.socket.keepalive-time 10
 14:     option transport-type rdma
 15:     option working-directory /var/lib/glusterd
 16: end-volume
 17:  
+------------------------------------------------------------------------------+
[2017-05-30 13:53:16.025090] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-30 13:54:11.367678] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04, host: glust01-li, port: 0
[2017-05-30 13:54:11.372298] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-05-30 13:54:11.395077] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-05-30 13:54:11.395469] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-05-30 13:54:11.396091] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-05-30 13:54:11.437290] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-05-30 13:54:11.437319] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-05-30 13:54:11.437362] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-05-30 13:54:12.441385] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-05-30 13:54:12.441673] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-05-30 13:54:12.441696] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-05-30 13:54:12.441736] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-05-30 13:54:12.441939] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-05-30 13:54:12.441953] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-05-30 13:54:12.441988] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-05-30 13:54:12.442176] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-05-30 13:54:12.442189] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-05-30 13:54:12.459704] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 13:54:12.463515] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 13:54:12.476832] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 13:54:12.486159] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 13:54:12.503287] I [glusterd-utils.c:5222:find_compat_brick_in_vol] 0-management: cleaning up dead brick glust01-lc:/run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b
[2017-05-30 13:54:12.505024] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-30 13:54:12.522615] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b to existing process for /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b
[2017-05-30 13:54:12.522674] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume b9e1bf3fe9244f4ebd0bd563a74058aa.glust01-lc.run-gluster-snaps-b9e1bf3fe9244f4ebd0bd563a74058aa-brick2-b not found [Invalid argument]
[2017-05-30 13:54:12.522726] I [glusterd-utils.c:4961:send_attach_req] 0-management: not connected yet
[2017-05-30 13:54:12.522917] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 13:54:12.523015] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65, host: 10.8.28.2, port: 0
[2017-05-30 13:54:12.529131] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 13:54:12.529375] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/shares1/b has disconnected from glusterd.
[2017-05-30 13:54:12.529533] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-05-30 13:54:12.529572] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-05-30 13:54:12.531136] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 4155
[2017-05-30 13:54:13.531388] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-05-30 13:54:13.531475] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-05-30 13:54:13.535905] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-05-30 13:54:13.535937] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-05-30 13:54:13.536008] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-05-30 13:54:13.536021] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-05-30 13:54:13.536086] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-05-30 13:54:13.536099] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-05-30 13:54:13.536263] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /bricks/shares1/b
[2017-05-30 13:54:13.536282] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shares1/b on port 49152
[2017-05-30 13:54:13.536434] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /bricks/config/b
[2017-05-30 13:54:13.536446] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/config/b on port 49153
[2017-05-30 13:54:13.536551] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /bricks/shared_storage/b
[2017-05-30 13:54:13.536561] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shared_storage/b on port 49154
[2017-05-30 13:54:13.536625] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b
[2017-05-30 13:54:13.536634] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b on port 49155
[2017-05-30 13:54:13.536692] I [glusterd-utils.c:5437:glusterd_brick_start] 0-management: discovered already-running brick /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b
[2017-05-30 13:54:13.536703] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b on port 49156
[2017-05-30 13:54:13.536814] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b to existing process for /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b
[2017-05-30 13:54:13.536838] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume b9e1bf3fe9244f4ebd0bd563a74058aa.glust01-lc.run-gluster-snaps-b9e1bf3fe9244f4ebd0bd563a74058aa-brick2-b not found [Invalid argument]
[2017-05-30 13:54:13.536858] I [glusterd-utils.c:4961:send_attach_req] 0-management: not connected yet
[2017-05-30 13:54:13.537168] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 13:54:13.537318] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/config/b has disconnected from glusterd.
[2017-05-30 13:54:13.537538] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 13:54:13.537686] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/shared_storage/b has disconnected from glusterd.
[2017-05-30 13:54:13.537917] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 13:54:13.538030] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b has disconnected from glusterd.
[2017-05-30 13:54:13.538251] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-30 13:54:13.538374] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b has disconnected from glusterd.
[2017-05-30 13:54:13.538570] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 13:54:13.570916] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 5dae480fc375494aa1199ff40ff051b9.glust01-lc.run-gluster-snaps-5dae480fc375494aa1199ff40ff051b9-brick2-b not found [Invalid argument]
[2017-05-30 13:54:13.600749] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume c8405ce657904bcb9602437df2aaaacc.glust01-lc.run-gluster-snaps-c8405ce657904bcb9602437df2aaaacc-brick2-b not found [Invalid argument]
[2017-05-30 13:54:13.610921] I [glusterd-utils.c:4961:send_attach_req] 0-management: not connected yet
[2017-05-30 13:54:13.625409] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-30 13:54:13.632575] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 13:54:13.634450] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 13:54:13.634501] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 13:54:13.638056] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glust01-li (0), ret: 0, op_ret: 0
[2017-05-30 13:54:13.645092] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 13:54:13.645117] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 13:54:13.648294] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-30 13:54:13.649539] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-30 13:54:13.657558] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 13:54:13.658802] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 13:54:13.658828] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-30 13:54:13.661846] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.8.28.2 (0), ret: 0, op_ret: 0
[2017-05-30 13:54:13.668642] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 13:54:13.668680] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-30 13:54:13.672862] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-30 13:54:14.115113] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shared_storage/b on port 49154
[2017-05-30 13:54:14.138205] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/config/b on port 49153
[2017-05-30 13:54:14.176533] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b on port 49155
[2017-05-30 13:54:14.177048] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b on port 49156
[2017-05-30 13:54:14.274381] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shares1/b on port 49152
[2017-05-30 13:54:14.536972] I [glusterd-utils.c:4961:send_attach_req] 0-management: not connected yet
[2017-05-30 13:54:14.611085] I [glusterd-utils.c:4961:send_attach_req] 0-management: not connected yet
[2017-05-30 13:54:15.537170] I [glusterd-utils.c:4961:send_attach_req] 0-management: not connected yet
[2017-05-30 13:54:15.611261] I [glusterd-utils.c:4961:send_attach_req] 0-management: not connected yet
[2017-05-30 13:54:16.548587] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/ecf94a72083d4803980307db1b923772/brick2/b to existing process for /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b
[2017-05-30 13:54:16.548634] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume ecf94a72083d4803980307db1b923772.glust01-lc.run-gluster-snaps-ecf94a72083d4803980307db1b923772-brick2-b not found [Invalid argument]
[2017-05-30 13:54:16.548891] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-30 13:54:17.552636] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-30 13:54:18.559099] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-30 13:54:18.586657] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-vol_config on port 49158
[2017-05-30 13:54:18.587354] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-shares1 on port 49157
[2017-05-30 15:00:03.411106] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/90a7dcba418947fd86f9418816c53ce4/brick2/b to existing process for /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b
[2017-05-30 15:00:03.411156] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 90a7dcba418947fd86f9418816c53ce4.glust01-lc.run-gluster-snaps-90a7dcba418947fd86f9418816c53ce4-brick2-b not found [Invalid argument]
[2017-05-30 17:00:03.355094] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/3c7065d9c7254fdd848dff55554dda95/brick2/b to existing process for /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b
[2017-05-30 17:00:03.355151] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 3c7065d9c7254fdd848dff55554dda95.glust01-lc.run-gluster-snaps-3c7065d9c7254fdd848dff55554dda95-brick2-b not found [Invalid argument]
[2017-05-31 08:10:55.249564] E [MSGID: 106027] [glusterd-snapshot.c:4027:glusterd_handle_snapshot_list] 0-management: Volume (Snap_GMT-2017.05.30-08.53.22) does not exist [Invalid argument]
[2017-05-31 08:10:55.249592] W [MSGID: 106037] [glusterd-snapshot.c:9510:glusterd_handle_snapshot_fn] 0-management: Snapshot list failed
[2017-05-31 08:23:49.279643] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b on port 49155
[2017-05-31 08:23:49.279769] W [socket.c:593:__socket_rwv] 0-management: readv on /var/run/gluster/abcc2921e86aa90a2f97de473fb191e2.socket failed (No data available)
[2017-05-31 08:23:49.280485] W [glusterd-handler.c:5684:__glusterd_brick_rpc_notify] 0-management: got disconnect from stale rpc on /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b
[2017-05-31 08:23:49.281023] W [socket.c:593:__socket_rwv] 0-socket.management: writev on 160.5.13.90:49068 failed (Broken pipe)
[2017-05-31 08:23:49.281036] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 08:24:26.131286] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b on port 49156
[2017-05-31 08:23:49.280957] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/c8405ce657904bcb9602437df2aaaacc/brick2/b on port 49155
[2017-05-31 08:36:21.061810] I [glusterd-utils.c:5222:find_compat_brick_in_vol] 0-management: cleaning up dead brick glust01-lc:/run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b
[2017-05-31 08:36:21.061908] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2/b to existing process for /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b
[2017-05-31 08:36:21.061941] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume d341ad1b177e46618b31375c823a976c.glust01-lc.run-gluster-snaps-d341ad1b177e46618b31375c823a976c-brick2-b not found [Invalid argument]
[2017-05-31 08:37:14.257972] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2/b on port 49156
[2017-05-31 08:37:35.110231] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2/b to existing process for /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b
[2017-05-31 08:37:35.110277] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume d341ad1b177e46618b31375c823a976c.glust01-lc.run-gluster-snaps-d341ad1b177e46618b31375c823a976c-brick2-b not found [Invalid argument]
[2017-05-31 08:38:07.946846] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.53.22 for decommission.
[2017-05-31 08:38:18.472650] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2/b on port 49156
[2017-05-31 08:38:38.320765] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2/b to existing process for /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b
[2017-05-31 08:38:43.014134] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.30-08.54.43 for decommission.
[2017-05-31 08:38:43.018206] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 08:38:43.018206] and [2017-05-31 08:38:49.024673]
[2017-05-31 08:38:52.024801] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2 (brick: /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/brick2/b): No such file or directory.
[2017-05-31 08:38:52.806855] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/5dae480fc375494aa1199ff40ff051b9/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 08:38:38.320833] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume d341ad1b177e46618b31375c823a976c.glust01-lc.run-gluster-snaps-d341ad1b177e46618b31375c823a976c-brick2-b not found [Invalid argument]
[2017-05-31 08:43:44.858375] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-08.36.17 for decommission.
[2017-05-31 08:43:44.866674] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 08:43:53.873584] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2 (brick: /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2/b): No such file or directory.
[2017-05-31 08:43:56.377867] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 08:43:56.379073] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2/b on port 49156
[2017-05-31 08:43:56.379293] E [MSGID: 106057] [glusterd-handshake.c:94:get_snap_volname_and_volinfo] 0-management: Failed to fetch snap Snap_GMT-2017.05.31-08.36.17 [Invalid argument]
[2017-05-31 08:43:56.379314] E [MSGID: 106063] [glusterd-handshake.c:274:build_volfile_path] 0-management: Failed to get snap volinfo from path (/snaps/Snap_GMT-2017.05.31-08.36.17/d341ad1b177e46618b31375c823a976c)
[2017-05-31 08:45:03.710391] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/ce0a041142274a1692c4c985700e141c/brick2/b to existing process for /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b
[2017-05-31 08:45:03.710448] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume ce0a041142274a1692c4c985700e141c.glust01-lc.run-gluster-snaps-ce0a041142274a1692c4c985700e141c-brick2-b not found [Invalid argument]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/d341ad1b177e46618b31375c823a976c/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 08:43:44.866674] and [2017-05-31 08:43:50.873452]
[2017-05-31 08:46:40.886599] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/ce0a041142274a1692c4c985700e141c/brick2/b on port 49156
[2017-05-31 08:47:06.034041] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-08.45.00 for decommission.
[2017-05-31 08:47:06.041362] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ce0a041142274a1692c4c985700e141c/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 08:47:15.047927] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/ce0a041142274a1692c4c985700e141c/brick2 (brick: /run/gluster/snaps/ce0a041142274a1692c4c985700e141c/brick2/b): No such file or directory.
[2017-05-31 08:47:17.184150] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/ce0a041142274a1692c4c985700e141c/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ce0a041142274a1692c4c985700e141c/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 08:47:06.041362] and [2017-05-31 08:47:12.047826]
[2017-05-31 08:51:22.557505] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/63f36c41e4dc463bb5fc74a54b50e39d/brick2/b to existing process for /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b
[2017-05-31 08:51:22.557559] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 63f36c41e4dc463bb5fc74a54b50e39d.glust01-lc.run-gluster-snaps-63f36c41e4dc463bb5fc74a54b50e39d-brick2-b not found [Invalid argument]
[2017-05-31 08:52:41.623031] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-08.51.18 for decommission.
[2017-05-31 08:52:41.629615] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/63f36c41e4dc463bb5fc74a54b50e39d/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 08:52:50.636514] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/63f36c41e4dc463bb5fc74a54b50e39d/brick2 (brick: /run/gluster/snaps/63f36c41e4dc463bb5fc74a54b50e39d/brick2/b): No such file or directory.
[2017-05-31 08:52:53.602885] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/63f36c41e4dc463bb5fc74a54b50e39d/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 08:52:53.626274] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/63f36c41e4dc463bb5fc74a54b50e39d/brick2/b on port 49156
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/63f36c41e4dc463bb5fc74a54b50e39d/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 08:52:41.629615] and [2017-05-31 08:52:47.636376]
[2017-05-31 08:56:07.417255] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2017-05-31 08:56:07.417255] and [2017-05-31 08:56:07.418863]
[2017-05-31 08:58:42.275427] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.30-09.00.01 for decommission.
[2017-05-31 08:58:42.284302] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 08:58:51.291865] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2 (brick: /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b): No such file or directory.
[2017-05-31 08:58:51.731826] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 08:58:51.733314] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2/b on port 49156
[2017-05-31 08:59:01.065831] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.30-11.00.01 for decommission.
[2017-05-31 08:59:01.068785] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ecf94a72083d4803980307db1b923772/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ecf94a72083d4803980307db1b923772/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 08:59:01.068785] and [2017-05-31 08:59:07.080314]
[2017-05-31 08:59:10.080442] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/ecf94a72083d4803980307db1b923772/brick2 (brick: /run/gluster/snaps/ecf94a72083d4803980307db1b923772/brick2/b): No such file or directory.
[2017-05-31 08:59:11.150770] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/ecf94a72083d4803980307db1b923772/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 08:59:11.152802] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/ecf94a72083d4803980307db1b923772/brick2/b on port 49156
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b9e1bf3fe9244f4ebd0bd563a74058aa/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 08:58:42.284302] and [2017-05-31 08:58:48.291734]
[2017-05-31 08:59:21.874750] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.30-15.00.01 for decommission.
[2017-05-31 08:59:21.879725] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/90a7dcba418947fd86f9418816c53ce4/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 08:59:30.887946] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/90a7dcba418947fd86f9418816c53ce4/brick2 (brick: /run/gluster/snaps/90a7dcba418947fd86f9418816c53ce4/brick2/b): No such file or directory.
[2017-05-31 08:59:34.473014] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/90a7dcba418947fd86f9418816c53ce4/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 08:59:34.474240] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/90a7dcba418947fd86f9418816c53ce4/brick2/b on port 49156
[2017-05-31 08:59:46.316184] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Scheduled-shares-hourly-shares1_GMT-2017.05.30-17.00.01 for decommission.
[2017-05-31 08:59:46.319204] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/3c7065d9c7254fdd848dff55554dda95/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/3c7065d9c7254fdd848dff55554dda95/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 08:59:46.319204] and [2017-05-31 08:59:52.325761]
[2017-05-31 08:59:55.325894] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/3c7065d9c7254fdd848dff55554dda95/brick2 (brick: /run/gluster/snaps/3c7065d9c7254fdd848dff55554dda95/brick2/b): No such file or directory.
[2017-05-31 08:59:57.123980] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/3c7065d9c7254fdd848dff55554dda95/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 08:59:57.125280] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/3c7065d9c7254fdd848dff55554dda95/brick2/b on port 49156
[2017-05-31 09:00:33.839207] W [socket.c:593:__socket_rwv] 0-management: readv on /var/run/gluster/269fed4d97a74f091fad6d755da921f6.socket failed (No data available)
[2017-05-31 09:00:33.839669] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/90a7dcba418947fd86f9418816c53ce4/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 08:59:21.879725] and [2017-05-31 08:59:27.887730]
The message "W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found" repeated 13 times between [2017-05-31 09:00:33.839669] and [2017-05-31 09:01:20.386750]
[2017-05-31 09:01:23.387276] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
The message "W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found" repeated 39 times between [2017-05-31 09:01:23.387276] and [2017-05-31 09:03:20.406100]
[2017-05-31 09:03:23.406673] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
The message "W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found" repeated 39 times between [2017-05-31 09:03:23.406673] and [2017-05-31 09:05:20.429594]
[2017-05-31 09:05:23.430144] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
The message "W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found" repeated 39 times between [2017-05-31 09:05:23.430144] and [2017-05-31 09:07:20.450915]
[2017-05-31 09:07:23.451589] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
The message "W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found" repeated 39 times between [2017-05-31 09:07:23.451589] and [2017-05-31 09:09:20.471392]
[2017-05-31 09:09:23.471908] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
The message "W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found" repeated 7 times between [2017-05-31 09:09:23.471908] and [2017-05-31 09:09:44.476929]
[2017-05-31 09:09:44.845401] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fd5423aadc5] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x7fd543a42fd5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x7fd543a42dfb] ) 0-: received signum (15), shutting down
[2017-05-31 09:10:20.135695] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-05-31 09:10:20.393159] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-05-31 09:10:20.393214] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory
[2017-05-31 09:10:20.494297] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.10.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2017-05-31 09:10:20.494360] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2017-05-31 09:10:20.494401] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-05-31 09:10:20.494410] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-05-31 09:10:20.591219] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory]
[2017-05-31 09:10:20.616516] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31000
[2017-05-31 09:10:21.199345] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-31 09:10:21.219244] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-31 09:10:21.219321] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-05-31 09:10:21.219375] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:10:21.227231] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:10:21.227217] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-05-31 09:10:21.288743] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 877c6f79-0201-4c41-986e-d2a065911ef8
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option transport.address-family inet
  8:     option rpc-auth-allow-insecure on
  9:     option event-threads 1
 10:     option ping-timeout 0
 11:     option transport.socket.read-fail-log off
 12:     option transport.socket.keepalive-interval 2
 13:     option transport.socket.keepalive-time 10
 14:     option transport-type rdma
 15:     option working-directory /var/lib/glusterd
 16: end-volume
 17:  
+------------------------------------------------------------------------------+
[2017-05-31 09:10:21.290197] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-31 09:10:26.382251] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65, host: 10.8.28.2, port: 0
[2017-05-31 09:10:26.387251] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-05-31 09:10:26.399638] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-05-31 09:10:26.400051] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-05-31 09:10:26.400556] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-05-31 09:10:26.424890] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-05-31 09:10:26.424927] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-05-31 09:10:26.424982] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-05-31 09:10:27.428800] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-05-31 09:10:27.429050] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-05-31 09:10:27.429078] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-05-31 09:10:27.429125] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-05-31 09:10:27.429274] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-05-31 09:10:27.429294] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-05-31 09:10:27.429339] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-05-31 09:10:27.429506] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-05-31 09:10:27.429520] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-05-31 09:10:27.454793] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:10:27.470174] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:10:27.481632] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:10:27.481854] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-31 09:10:28.494968] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-31 09:10:29.524665] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-31 09:10:29.525040] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:10:29.525097] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:10:29.530575] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:10:29.530652] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04, host: glust01-li, port: 0
[2017-05-31 09:10:29.535160] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:10:29.535316] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/shares1/b has disconnected from glusterd.
[2017-05-31 09:10:29.535562] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:10:29.535685] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/config/b has disconnected from glusterd.
[2017-05-31 09:10:29.535902] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:10:29.536023] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/shared_storage/b has disconnected from glusterd.
[2017-05-31 09:10:29.536434] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:10:29.538578] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-31 09:10:29.542436] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:10:29.542492] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:10:29.546044] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:10:29.547950] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:10:29.548002] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:10:29.551429] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.8.28.2 (0), ret: 0, op_ret: 0
[2017-05-31 09:10:29.573357] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:10:29.573384] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:10:29.576866] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-31 09:10:29.580283] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:10:29.594550] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:10:29.595771] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:10:29.595818] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:10:29.599721] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glust01-li (0), ret: 0, op_ret: 0
[2017-05-31 09:10:29.615403] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:10:29.615431] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:10:29.618569] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:10:29.826429] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-vol_config on port 49156
[2017-05-31 09:10:29.826636] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-shares1 on port 49155
[2017-05-31 09:10:30.414383] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shared_storage/b on port 49154
[2017-05-31 09:10:30.442971] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shares1/b on port 49152
[2017-05-31 09:10:30.500694] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/config/b on port 49153
[2017-05-31 09:11:18.644021] W [socket.c:593:__socket_rwv] 0-management: readv on 160.5.5.90:24007 failed (No data available)
[2017-05-31 09:11:18.644063] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <glust01-li> (<64390a1d-af72-4f10-be41-e23d4f881e04>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-05-31 09:11:18.644184] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7fe4504deeda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7fe4504e8dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7fe45059e41a] ) 0-management: Lock for vol shares1 not held
[2017-05-31 09:11:18.644197] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for shares1
[2017-05-31 09:11:18.644215] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7fe4504deeda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7fe4504e8dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7fe45059e41a] ) 0-management: Lock for vol vol_config not held
[2017-05-31 09:11:18.644221] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_config
[2017-05-31 09:11:18.644254] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7fe4504deeda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7fe4504e8dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7fe45059e41a] ) 0-management: Lock for vol vol_shared_storage not held
[2017-05-31 09:11:18.644262] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_shared_storage
[2017-05-31 09:11:51.718967] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-31 09:11:51.724102] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:12:00.232640] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:12:00.232707] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:12:00.236926] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glust01-li (0), ret: 0, op_ret: 0
[2017-05-31 09:12:00.244311] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:12:00.244338] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:12:00.250189] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:12:03.315535] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04, host: glust01-li, port: 0
[2017-05-31 09:12:03.320676] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:12:03.320705] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:12:03.333432] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:12:20.563359] W [socket.c:593:__socket_rwv] 0-management: readv on 10.8.28.2:24007 failed (No data available)
[2017-05-31 09:12:20.563393] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <10.8.28.2> (<62fa28ac-7ab1-4f51-bc4d-45ea0005fb65>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-05-31 09:12:20.563487] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7fe4504deeda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7fe4504e8dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7fe45059e41a] ) 0-management: Lock for vol shares1 not held
[2017-05-31 09:12:20.563498] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for shares1
[2017-05-31 09:12:20.563513] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7fe4504deeda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7fe4504e8dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7fe45059e41a] ) 0-management: Lock for vol vol_config not held
[2017-05-31 09:12:20.563519] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_config
[2017-05-31 09:12:20.563532] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7fe4504deeda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7fe4504e8dc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7fe45059e41a] ) 0-management: Lock for vol vol_shared_storage not held
[2017-05-31 09:12:20.563547] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_shared_storage
[2017-05-31 09:12:52.517373] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-31 09:12:52.524323] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:13:02.248788] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:13:02.248851] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:13:02.253519] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.8.28.2 (0), ret: 0, op_ret: 0
[2017-05-31 09:13:02.259626] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:13:02.259663] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:13:02.264637] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:13:02.279989] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65, host: 10.8.28.2, port: 0
[2017-05-31 09:13:02.283054] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:13:02.283080] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:13:05.318648] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:15:51.475753] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:15:51.500238] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:15:51.500801] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/run/gluster/snaps/7f948ff368f24db2beb9271c788a910c/brick2/b has disconnected from glusterd.
[2017-05-31 09:15:51.501025] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 7f948ff368f24db2beb9271c788a910c.glust01-lc.run-gluster-snaps-7f948ff368f24db2beb9271c788a910c-brick2-b not found [Invalid argument]
[2017-05-31 09:15:51.832257] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/7f948ff368f24db2beb9271c788a910c/brick2/b on port 49157
[2017-05-31 09:16:53.153516] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2017-05-31 09:16:53.153516] and [2017-05-31 09:16:53.155230]
[2017-05-31 09:18:41.631419] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.15.49 for decommission.
[2017-05-31 09:18:41.794939] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/7f948ff368f24db2beb9271c788a910c/brick2/b on port 49157
[2017-05-31 09:18:41.795056] W [socket.c:593:__socket_rwv] 0-management: readv on /var/run/gluster/a4bd792a90fc0b4bc2cf97ef3f89af88.socket failed (No data available)
[2017-05-31 09:18:41.795404] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
[2017-05-31 09:18:41.795483] W [socket.c:593:__socket_rwv] 0-socket.management: writev on 160.5.13.90:49144 failed (Broken pipe)
[2017-05-31 09:18:41.795493] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:20:06.294085] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:20:06.318305] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:20:06.318487] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b has disconnected from glusterd.
[2017-05-31 09:20:06.318810] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 4f980da64dec424ba0b48d6d36c4c54e.glust01-lc.run-gluster-snaps-4f980da64dec424ba0b48d6d36c4c54e-brick2-b not found [Invalid argument]
[2017-05-31 09:20:06.498841] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b on port 49157
[2017-05-31 09:20:11.865047] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/733ab2ade14d4104bfa930e728058c2a/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:11.865098] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 733ab2ade14d4104bfa930e728058c2a.glust01-lc.run-gluster-snaps-733ab2ade14d4104bfa930e728058c2a-brick2-b not found [Invalid argument]
[2017-05-31 09:20:15.345964] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/b0b2b9a548b5418089c12846b0767fd5/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:15.346021] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume b0b2b9a548b5418089c12846b0767fd5.glust01-lc.run-gluster-snaps-b0b2b9a548b5418089c12846b0767fd5-brick2-b not found [Invalid argument]
[2017-05-31 09:20:18.358522] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/b8667226c39e493cb3c27b7ddc1ed8ac/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:18.358577] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume b8667226c39e493cb3c27b7ddc1ed8ac.glust01-lc.run-gluster-snaps-b8667226c39e493cb3c27b7ddc1ed8ac-brick2-b not found [Invalid argument]
[2017-05-31 09:18:41.795423] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
[2017-05-31 09:20:21.254422] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/ff23d1d684a942b3a33126d4e5eae529/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:21.254493] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume ff23d1d684a942b3a33126d4e5eae529.glust01-lc.run-gluster-snaps-ff23d1d684a942b3a33126d4e5eae529-brick2-b not found [Invalid argument]
[2017-05-31 09:20:24.440411] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/d4f75749aab64695b2750bd040bd458f/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:24.440488] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume d4f75749aab64695b2750bd040bd458f.glust01-lc.run-gluster-snaps-d4f75749aab64695b2750bd040bd458f-brick2-b not found [Invalid argument]
[2017-05-31 09:20:27.114205] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/cce4d991c7f043b4921958193eb5a6d7/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:27.114264] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume cce4d991c7f043b4921958193eb5a6d7.glust01-lc.run-gluster-snaps-cce4d991c7f043b4921958193eb5a6d7-brick2-b not found [Invalid argument]
[2017-05-31 09:20:29.351923] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/063d9351a2d9421fa086a696d8453910/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:29.351973] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 063d9351a2d9421fa086a696d8453910.glust01-lc.run-gluster-snaps-063d9351a2d9421fa086a696d8453910-brick2-b not found [Invalid argument]
[2017-05-31 09:20:31.987280] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/735b016503bf449db0d93ebbdd6ee1b1/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:31.987327] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 735b016503bf449db0d93ebbdd6ee1b1.glust01-lc.run-gluster-snaps-735b016503bf449db0d93ebbdd6ee1b1-brick2-b not found [Invalid argument]
[2017-05-31 09:20:34.515426] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/089ae1ef4d664c39958c732ec7550d22/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:34.515480] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 089ae1ef4d664c39958c732ec7550d22.glust01-lc.run-gluster-snaps-089ae1ef4d664c39958c732ec7550d22-brick2-b not found [Invalid argument]
[2017-05-31 09:20:37.656195] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/6ec584fe2a10450fb54dde996a8fae76/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:37.656243] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 6ec584fe2a10450fb54dde996a8fae76.glust01-lc.run-gluster-snaps-6ec584fe2a10450fb54dde996a8fae76-brick2-b not found [Invalid argument]
[2017-05-31 09:20:40.440402] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/d1fa16d1e81248ecb78f203c3f652835/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:40.440467] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume d1fa16d1e81248ecb78f203c3f652835.glust01-lc.run-gluster-snaps-d1fa16d1e81248ecb78f203c3f652835-brick2-b not found [Invalid argument]
[2017-05-31 09:20:43.288643] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/df8a7f325b0343338e977ecf285117d0/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:43.288723] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume df8a7f325b0343338e977ecf285117d0.glust01-lc.run-gluster-snaps-df8a7f325b0343338e977ecf285117d0-brick2-b not found [Invalid argument]
[2017-05-31 09:20:46.514427] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/191fb0f349ca42f0ab0a63728a2de6d9/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:46.514505] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 191fb0f349ca42f0ab0a63728a2de6d9.glust01-lc.run-gluster-snaps-191fb0f349ca42f0ab0a63728a2de6d9-brick2-b not found [Invalid argument]
[2017-05-31 09:20:49.341164] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/8a95a01778f14a82b50001611dc4a55f/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:49.341222] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 8a95a01778f14a82b50001611dc4a55f.glust01-lc.run-gluster-snaps-8a95a01778f14a82b50001611dc4a55f-brick2-b not found [Invalid argument]
[2017-05-31 09:20:52.031314] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/463ee23ad0d64cd0b17ad241104036ad/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:52.031375] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 463ee23ad0d64cd0b17ad241104036ad.glust01-lc.run-gluster-snaps-463ee23ad0d64cd0b17ad241104036ad-brick2-b not found [Invalid argument]
[2017-05-31 09:20:54.780842] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/249a86d137fb42168f405493a033f7d2/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:54.780907] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 249a86d137fb42168f405493a033f7d2.glust01-lc.run-gluster-snaps-249a86d137fb42168f405493a033f7d2-brick2-b not found [Invalid argument]
[2017-05-31 09:20:58.633626] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/a60658012b48411fa95629b42ab23ff7/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:20:58.633694] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume a60658012b48411fa95629b42ab23ff7.glust01-lc.run-gluster-snaps-a60658012b48411fa95629b42ab23ff7-brick2-b not found [Invalid argument]
[2017-05-31 09:21:03.137223] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/cbfe670c9b7640c586f6f35fe98c9609/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:03.137285] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume cbfe670c9b7640c586f6f35fe98c9609.glust01-lc.run-gluster-snaps-cbfe670c9b7640c586f6f35fe98c9609-brick2-b not found [Invalid argument]
[2017-05-31 09:21:06.045036] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/8e2cc779e8834b72ab256fabadde10d6/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:06.045094] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 8e2cc779e8834b72ab256fabadde10d6.glust01-lc.run-gluster-snaps-8e2cc779e8834b72ab256fabadde10d6-brick2-b not found [Invalid argument]
[2017-05-31 09:21:08.845274] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/f2fad6bd93f849d89db4e24e73e16640/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:08.845330] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume f2fad6bd93f849d89db4e24e73e16640.glust01-lc.run-gluster-snaps-f2fad6bd93f849d89db4e24e73e16640-brick2-b not found [Invalid argument]
[2017-05-31 09:21:12.253143] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/0be3a9addc684f63966771f770c0724b/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:12.253211] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 0be3a9addc684f63966771f770c0724b.glust01-lc.run-gluster-snaps-0be3a9addc684f63966771f770c0724b-brick2-b not found [Invalid argument]
[2017-05-31 09:21:15.411677] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/e28edaee99444c4387b88fd52c361675/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:15.411767] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume e28edaee99444c4387b88fd52c361675.glust01-lc.run-gluster-snaps-e28edaee99444c4387b88fd52c361675-brick2-b not found [Invalid argument]
[2017-05-31 09:21:25.795694] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/1a88ec5f3a9b4fc8b8d824194b8a2124/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:25.795805] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 1a88ec5f3a9b4fc8b8d824194b8a2124.glust01-lc.run-gluster-snaps-1a88ec5f3a9b4fc8b8d824194b8a2124-brick2-b not found [Invalid argument]
[2017-05-31 09:21:29.440327] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/b592abfede1a4e98903c4dfcc4ad9e0e/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:29.440404] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume b592abfede1a4e98903c4dfcc4ad9e0e.glust01-lc.run-gluster-snaps-b592abfede1a4e98903c4dfcc4ad9e0e-brick2-b not found [Invalid argument]
[2017-05-31 09:21:32.535201] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/1a65ba423d31484d83c49ce5ae1bc482/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:32.535282] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 1a65ba423d31484d83c49ce5ae1bc482.glust01-lc.run-gluster-snaps-1a65ba423d31484d83c49ce5ae1bc482-brick2-b not found [Invalid argument]
[2017-05-31 09:21:35.680199] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/57b4dcafd74e4547a6403c34a1508b01/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:35.680273] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 57b4dcafd74e4547a6403c34a1508b01.glust01-lc.run-gluster-snaps-57b4dcafd74e4547a6403c34a1508b01-brick2-b not found [Invalid argument]
[2017-05-31 09:21:39.018157] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/331ec972f90d494d8a86dd4f69d718b7/brick2/b to existing process for /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b
[2017-05-31 09:21:39.018226] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 331ec972f90d494d8a86dd4f69d718b7.glust01-lc.run-gluster-snaps-331ec972f90d494d8a86dd4f69d718b7-brick2-b not found [Invalid argument]
[2017-05-31 09:21:40.103028] W [MSGID: 106112] [glusterd-snapshot.c:8128:glusterd_handle_snap_limit] 0-management: Soft-limit (value = 27) of volume shares1 is reached. Deleting snapshot Snap_GMT-2017.05.31-09.20.04.
[2017-05-31 09:21:40.112189] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:21:40.112189] and [2017-05-31 09:21:46.121366]
[2017-05-31 09:21:49.121503] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2 (brick: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b): No such file or directory.
[2017-05-31 09:21:51.514098] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:21:51.514921] W [MSGID: 106039] [glusterd-snapshot-utils.c:55:glusterd_snapobject_delete] 0-management: Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04
[2017-05-31 09:21:51.516019] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick2/b on port 49157
[2017-05-31 09:23:55.108194] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.10 for decommission.
[2017-05-31 09:23:55.118137] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/733ab2ade14d4104bfa930e728058c2a/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 09:24:04.126829] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/733ab2ade14d4104bfa930e728058c2a/brick2 (brick: /run/gluster/snaps/733ab2ade14d4104bfa930e728058c2a/brick2/b): No such file or directory.
[2017-05-31 09:24:06.784982] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/733ab2ade14d4104bfa930e728058c2a/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:24:06.786829] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/733ab2ade14d4104bfa930e728058c2a/brick2/b on port 49157
[2017-05-31 09:24:17.969898] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.14 for decommission.
[2017-05-31 09:24:17.974817] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b0b2b9a548b5418089c12846b0767fd5/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/733ab2ade14d4104bfa930e728058c2a/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:23:55.118137] and [2017-05-31 09:24:01.126709]
[2017-05-31 09:24:20.978968] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b0b2b9a548b5418089c12846b0767fd5/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 09:24:26.983830] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/b0b2b9a548b5418089c12846b0767fd5/brick2 (brick: /run/gluster/snaps/b0b2b9a548b5418089c12846b0767fd5/brick2/b): No such file or directory.
[2017-05-31 09:24:28.618946] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/b0b2b9a548b5418089c12846b0767fd5/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:24:28.620010] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/b0b2b9a548b5418089c12846b0767fd5/brick2/b on port 49157
[2017-05-31 09:24:39.838419] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.17 for decommission.
[2017-05-31 09:24:39.842526] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b8667226c39e493cb3c27b7ddc1ed8ac/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b8667226c39e493cb3c27b7ddc1ed8ac/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:24:39.842526] and [2017-05-31 09:24:45.850307]
[2017-05-31 09:24:48.850455] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/b8667226c39e493cb3c27b7ddc1ed8ac/brick2 (brick: /run/gluster/snaps/b8667226c39e493cb3c27b7ddc1ed8ac/brick2/b): No such file or directory.
[2017-05-31 09:24:50.986264] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/b8667226c39e493cb3c27b7ddc1ed8ac/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:24:50.987493] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/b8667226c39e493cb3c27b7ddc1ed8ac/brick2/b on port 49157
[2017-05-31 09:25:02.103752] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.20 for decommission.
[2017-05-31 09:25:02.110810] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ff23d1d684a942b3a33126d4e5eae529/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/ff23d1d684a942b3a33126d4e5eae529/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:25:02.110810] and [2017-05-31 09:25:08.120006]
[2017-05-31 09:25:11.120156] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/ff23d1d684a942b3a33126d4e5eae529/brick2 (brick: /run/gluster/snaps/ff23d1d684a942b3a33126d4e5eae529/brick2/b): No such file or directory.
[2017-05-31 09:25:14.345569] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/ff23d1d684a942b3a33126d4e5eae529/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:25:14.346875] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/ff23d1d684a942b3a33126d4e5eae529/brick2/b on port 49157
[2017-05-31 09:25:26.084446] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.23 for decommission.
[2017-05-31 09:25:26.088929] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/d4f75749aab64695b2750bd040bd458f/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/d4f75749aab64695b2750bd040bd458f/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:25:26.088929] and [2017-05-31 09:25:32.096425]
[2017-05-31 09:25:35.096563] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/d4f75749aab64695b2750bd040bd458f/brick2 (brick: /run/gluster/snaps/d4f75749aab64695b2750bd040bd458f/brick2/b): No such file or directory.
[2017-05-31 09:25:37.038421] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/d4f75749aab64695b2750bd040bd458f/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:25:37.040596] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/d4f75749aab64695b2750bd040bd458f/brick2/b on port 49157
[2017-05-31 09:25:48.701642] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.26 for decommission.
[2017-05-31 09:25:48.705344] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/cce4d991c7f043b4921958193eb5a6d7/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/cce4d991c7f043b4921958193eb5a6d7/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:25:48.705344] and [2017-05-31 09:25:54.714081]
[2017-05-31 09:25:57.714208] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/cce4d991c7f043b4921958193eb5a6d7/brick2 (brick: /run/gluster/snaps/cce4d991c7f043b4921958193eb5a6d7/brick2/b): No such file or directory.
[2017-05-31 09:25:59.971857] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/cce4d991c7f043b4921958193eb5a6d7/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:25:59.973378] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/cce4d991c7f043b4921958193eb5a6d7/brick2/b on port 49157
[2017-05-31 09:26:10.919932] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.28 for decommission.
[2017-05-31 09:26:10.925472] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/063d9351a2d9421fa086a696d8453910/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/063d9351a2d9421fa086a696d8453910/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:26:10.925472] and [2017-05-31 09:26:16.933760]
[2017-05-31 09:26:19.933908] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/063d9351a2d9421fa086a696d8453910/brick2 (brick: /run/gluster/snaps/063d9351a2d9421fa086a696d8453910/brick2/b): No such file or directory.
[2017-05-31 09:24:23.983680] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b0b2b9a548b5418089c12846b0767fd5/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 09:26:22.255792] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/063d9351a2d9421fa086a696d8453910/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:26:22.257112] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/063d9351a2d9421fa086a696d8453910/brick2/b on port 49157
[2017-05-31 09:26:33.375884] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.31 for decommission.
[2017-05-31 09:26:33.380133] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/735b016503bf449db0d93ebbdd6ee1b1/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 09:26:42.389362] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/735b016503bf449db0d93ebbdd6ee1b1/brick2 (brick: /run/gluster/snaps/735b016503bf449db0d93ebbdd6ee1b1/brick2/b): No such file or directory.
[2017-05-31 09:26:45.798202] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/735b016503bf449db0d93ebbdd6ee1b1/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:26:45.799288] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/735b016503bf449db0d93ebbdd6ee1b1/brick2/b on port 49157
[2017-05-31 09:26:57.079167] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.33 for decommission.
[2017-05-31 09:26:57.083576] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/089ae1ef4d664c39958c732ec7550d22/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/089ae1ef4d664c39958c732ec7550d22/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:26:57.083576] and [2017-05-31 09:27:03.093044]
[2017-05-31 09:27:06.093132] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/089ae1ef4d664c39958c732ec7550d22/brick2 (brick: /run/gluster/snaps/089ae1ef4d664c39958c732ec7550d22/brick2/b): No such file or directory.
[2017-05-31 09:27:08.174184] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/089ae1ef4d664c39958c732ec7550d22/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:27:08.175578] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/089ae1ef4d664c39958c732ec7550d22/brick2/b on port 49157
[2017-05-31 09:27:19.318951] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.36 for decommission.
[2017-05-31 09:27:19.323068] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/6ec584fe2a10450fb54dde996a8fae76/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/6ec584fe2a10450fb54dde996a8fae76/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:27:19.323068] and [2017-05-31 09:27:25.331166]
[2017-05-31 09:27:28.331290] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/6ec584fe2a10450fb54dde996a8fae76/brick2 (brick: /run/gluster/snaps/6ec584fe2a10450fb54dde996a8fae76/brick2/b): No such file or directory.
[2017-05-31 09:27:30.607750] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/6ec584fe2a10450fb54dde996a8fae76/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:27:30.608808] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/6ec584fe2a10450fb54dde996a8fae76/brick2/b on port 49157
[2017-05-31 09:27:41.621651] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.39 for decommission.
[2017-05-31 09:27:41.626375] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/d1fa16d1e81248ecb78f203c3f652835/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/d1fa16d1e81248ecb78f203c3f652835/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:27:41.626375] and [2017-05-31 09:27:47.634351]
[2017-05-31 09:27:50.634818] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/d1fa16d1e81248ecb78f203c3f652835/brick2 (brick: /run/gluster/snaps/d1fa16d1e81248ecb78f203c3f652835/brick2/b): No such file or directory.
[2017-05-31 09:27:52.666797] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/d1fa16d1e81248ecb78f203c3f652835/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:27:52.668225] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/d1fa16d1e81248ecb78f203c3f652835/brick2/b on port 49157
[2017-05-31 09:28:03.862079] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.41 for decommission.
[2017-05-31 09:28:03.866860] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/df8a7f325b0343338e977ecf285117d0/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/df8a7f325b0343338e977ecf285117d0/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:28:03.866860] and [2017-05-31 09:28:09.874281]
[2017-05-31 09:28:12.874400] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/df8a7f325b0343338e977ecf285117d0/brick2 (brick: /run/gluster/snaps/df8a7f325b0343338e977ecf285117d0/brick2/b): No such file or directory.
[2017-05-31 09:28:15.634286] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/df8a7f325b0343338e977ecf285117d0/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:28:15.635999] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/df8a7f325b0343338e977ecf285117d0/brick2/b on port 49157
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/735b016503bf449db0d93ebbdd6ee1b1/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:26:33.380133] and [2017-05-31 09:26:39.389221]
[2017-05-31 09:28:26.794976] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.45 for decommission.
[2017-05-31 09:28:26.798799] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/191fb0f349ca42f0ab0a63728a2de6d9/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 09:28:35.807362] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/191fb0f349ca42f0ab0a63728a2de6d9/brick2 (brick: /run/gluster/snaps/191fb0f349ca42f0ab0a63728a2de6d9/brick2/b): No such file or directory.
[2017-05-31 09:28:38.210657] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/191fb0f349ca42f0ab0a63728a2de6d9/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:28:38.212336] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/191fb0f349ca42f0ab0a63728a2de6d9/brick2/b on port 49157
[2017-05-31 09:28:49.425410] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.48 for decommission.
[2017-05-31 09:28:49.429213] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/8a95a01778f14a82b50001611dc4a55f/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/8a95a01778f14a82b50001611dc4a55f/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:28:49.429213] and [2017-05-31 09:28:55.436629]
[2017-05-31 09:28:58.436750] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/8a95a01778f14a82b50001611dc4a55f/brick2 (brick: /run/gluster/snaps/8a95a01778f14a82b50001611dc4a55f/brick2/b): No such file or directory.
[2017-05-31 09:29:00.010583] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/8a95a01778f14a82b50001611dc4a55f/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:29:00.011650] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/8a95a01778f14a82b50001611dc4a55f/brick2/b on port 49157
[2017-05-31 09:29:10.989006] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.50 for decommission.
[2017-05-31 09:29:10.994138] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/463ee23ad0d64cd0b17ad241104036ad/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/463ee23ad0d64cd0b17ad241104036ad/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:29:10.994138] and [2017-05-31 09:29:17.002140]
[2017-05-31 09:29:20.002262] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/463ee23ad0d64cd0b17ad241104036ad/brick2 (brick: /run/gluster/snaps/463ee23ad0d64cd0b17ad241104036ad/brick2/b): No such file or directory.
[2017-05-31 09:29:22.236497] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/463ee23ad0d64cd0b17ad241104036ad/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:29:22.237725] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/463ee23ad0d64cd0b17ad241104036ad/brick2/b on port 49157
[2017-05-31 09:29:33.857567] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.53 for decommission.
[2017-05-31 09:29:33.862120] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/249a86d137fb42168f405493a033f7d2/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/249a86d137fb42168f405493a033f7d2/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:29:33.862120] and [2017-05-31 09:29:39.869420]
[2017-05-31 09:29:42.869556] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/249a86d137fb42168f405493a033f7d2/brick2 (brick: /run/gluster/snaps/249a86d137fb42168f405493a033f7d2/brick2/b): No such file or directory.
[2017-05-31 09:29:45.395236] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/249a86d137fb42168f405493a033f7d2/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:29:45.396245] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/249a86d137fb42168f405493a033f7d2/brick2/b on port 49157
[2017-05-31 09:29:56.365504] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.20.57 for decommission.
[2017-05-31 09:29:56.369155] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a60658012b48411fa95629b42ab23ff7/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/a60658012b48411fa95629b42ab23ff7/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:29:56.369155] and [2017-05-31 09:30:02.378557]
[2017-05-31 09:30:05.378715] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/a60658012b48411fa95629b42ab23ff7/brick2 (brick: /run/gluster/snaps/a60658012b48411fa95629b42ab23ff7/brick2/b): No such file or directory.
[2017-05-31 09:30:06.979287] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/a60658012b48411fa95629b42ab23ff7/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:30:06.980847] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/a60658012b48411fa95629b42ab23ff7/brick2/b on port 49157
[2017-05-31 09:30:18.562210] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.01 for decommission.
[2017-05-31 09:30:18.566704] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/cbfe670c9b7640c586f6f35fe98c9609/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/191fb0f349ca42f0ab0a63728a2de6d9/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:28:26.798799] and [2017-05-31 09:28:32.807236]
[2017-05-31 09:30:21.570647] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/cbfe670c9b7640c586f6f35fe98c9609/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 09:30:27.574795] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/cbfe670c9b7640c586f6f35fe98c9609/brick2 (brick: /run/gluster/snaps/cbfe670c9b7640c586f6f35fe98c9609/brick2/b): No such file or directory.
[2017-05-31 09:30:29.313531] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/cbfe670c9b7640c586f6f35fe98c9609/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:30:29.314621] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/cbfe670c9b7640c586f6f35fe98c9609/brick2/b on port 49157
[2017-05-31 09:30:40.192230] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.04 for decommission.
[2017-05-31 09:30:40.196143] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/8e2cc779e8834b72ab256fabadde10d6/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/8e2cc779e8834b72ab256fabadde10d6/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:30:40.196143] and [2017-05-31 09:30:46.203903]
[2017-05-31 09:30:49.204031] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/8e2cc779e8834b72ab256fabadde10d6/brick2 (brick: /run/gluster/snaps/8e2cc779e8834b72ab256fabadde10d6/brick2/b): No such file or directory.
[2017-05-31 09:30:50.771281] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/8e2cc779e8834b72ab256fabadde10d6/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:30:50.772447] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/8e2cc779e8834b72ab256fabadde10d6/brick2/b on port 49157
[2017-05-31 09:31:02.179212] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.07 for decommission.
[2017-05-31 09:31:02.185861] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/f2fad6bd93f849d89db4e24e73e16640/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/f2fad6bd93f849d89db4e24e73e16640/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:31:02.185861] and [2017-05-31 09:31:08.194443]
[2017-05-31 09:31:11.194588] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/f2fad6bd93f849d89db4e24e73e16640/brick2 (brick: /run/gluster/snaps/f2fad6bd93f849d89db4e24e73e16640/brick2/b): No such file or directory.
[2017-05-31 09:31:15.539859] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/f2fad6bd93f849d89db4e24e73e16640/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:31:15.541113] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/f2fad6bd93f849d89db4e24e73e16640/brick2/b on port 49157
[2017-05-31 09:31:27.223510] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.10 for decommission.
[2017-05-31 09:31:27.228415] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/0be3a9addc684f63966771f770c0724b/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/0be3a9addc684f63966771f770c0724b/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:31:27.228415] and [2017-05-31 09:31:33.236121]
[2017-05-31 09:31:36.236253] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/0be3a9addc684f63966771f770c0724b/brick2 (brick: /run/gluster/snaps/0be3a9addc684f63966771f770c0724b/brick2/b): No such file or directory.
[2017-05-31 09:31:38.115437] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/0be3a9addc684f63966771f770c0724b/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:31:38.116815] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/0be3a9addc684f63966771f770c0724b/brick2/b on port 49157
[2017-05-31 09:31:49.169232] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.14 for decommission.
[2017-05-31 09:31:49.173226] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/e28edaee99444c4387b88fd52c361675/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/e28edaee99444c4387b88fd52c361675/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:31:49.173226] and [2017-05-31 09:31:55.183049]
[2017-05-31 09:31:58.183219] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/e28edaee99444c4387b88fd52c361675/brick2 (brick: /run/gluster/snaps/e28edaee99444c4387b88fd52c361675/brick2/b): No such file or directory.
[2017-05-31 09:32:00.241448] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/e28edaee99444c4387b88fd52c361675/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:32:00.242772] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/e28edaee99444c4387b88fd52c361675/brick2/b on port 49157
[2017-05-31 09:32:11.460854] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.24 for decommission.
[2017-05-31 09:32:11.466148] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/1a88ec5f3a9b4fc8b8d824194b8a2124/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 09:30:24.574638] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/cbfe670c9b7640c586f6f35fe98c9609/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/1a88ec5f3a9b4fc8b8d824194b8a2124/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:32:11.466148] and [2017-05-31 09:32:17.476076]
[2017-05-31 09:32:20.476216] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/1a88ec5f3a9b4fc8b8d824194b8a2124/brick2 (brick: /run/gluster/snaps/1a88ec5f3a9b4fc8b8d824194b8a2124/brick2/b): No such file or directory.
[2017-05-31 09:32:22.416744] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/1a88ec5f3a9b4fc8b8d824194b8a2124/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:32:22.417867] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/1a88ec5f3a9b4fc8b8d824194b8a2124/brick2/b on port 49157
[2017-05-31 09:32:33.598495] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.28 for decommission.
[2017-05-31 09:32:33.603353] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b592abfede1a4e98903c4dfcc4ad9e0e/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/b592abfede1a4e98903c4dfcc4ad9e0e/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:32:33.603353] and [2017-05-31 09:32:39.611048]
[2017-05-31 09:32:42.611174] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/b592abfede1a4e98903c4dfcc4ad9e0e/brick2 (brick: /run/gluster/snaps/b592abfede1a4e98903c4dfcc4ad9e0e/brick2/b): No such file or directory.
[2017-05-31 09:32:46.142655] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/b592abfede1a4e98903c4dfcc4ad9e0e/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:32:46.143801] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/b592abfede1a4e98903c4dfcc4ad9e0e/brick2/b on port 49157
[2017-05-31 09:32:57.313452] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.31 for decommission.
[2017-05-31 09:32:57.317137] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/1a65ba423d31484d83c49ce5ae1bc482/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/1a65ba423d31484d83c49ce5ae1bc482/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:32:57.317137] and [2017-05-31 09:33:03.326906]
[2017-05-31 09:33:06.327041] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/1a65ba423d31484d83c49ce5ae1bc482/brick2 (brick: /run/gluster/snaps/1a65ba423d31484d83c49ce5ae1bc482/brick2/b): No such file or directory.
[2017-05-31 09:33:08.918328] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/1a65ba423d31484d83c49ce5ae1bc482/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:33:08.920070] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/1a65ba423d31484d83c49ce5ae1bc482/brick2/b on port 49157
[2017-05-31 09:33:20.684577] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.34 for decommission.
[2017-05-31 09:33:20.688356] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/57b4dcafd74e4547a6403c34a1508b01/brick2 failed (No such file or directory) [No such file or directory]
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/57b4dcafd74e4547a6403c34a1508b01/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 09:33:20.688356] and [2017-05-31 09:33:26.696450]
[2017-05-31 09:33:29.696671] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/57b4dcafd74e4547a6403c34a1508b01/brick2 (brick: /run/gluster/snaps/57b4dcafd74e4547a6403c34a1508b01/brick2/b): No such file or directory.
[2017-05-31 09:33:31.602271] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/57b4dcafd74e4547a6403c34a1508b01/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 09:33:31.603783] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/57b4dcafd74e4547a6403c34a1508b01/brick2/b on port 49157
[2017-05-31 09:33:43.073725] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.21.37 for decommission.
[2017-05-31 09:33:43.077499] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/331ec972f90d494d8a86dd4f69d718b7/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 09:33:46.251035] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/331ec972f90d494d8a86dd4f69d718b7/brick2/b on port 49157
[2017-05-31 09:33:46.251569] W [socket.c:593:__socket_rwv] 0-management: readv on /var/run/gluster/2ee019d6b2c6ea43047f21e5eece2188.socket failed (No data available)
[2017-05-31 09:33:46.251591] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
[2017-05-31 09:33:46.251662] W [socket.c:593:__socket_rwv] 0-socket.management: writev on 160.5.13.90:49144 failed (Broken pipe)
[2017-05-31 09:33:46.251672] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:33:46.251620] W [MSGID: 106057] [glusterd-snapshot-utils.c:379:glusterd_snap_volinfo_find_by_volume_id] 0-management: Snap volume not found
[2017-05-31 09:33:46.251758] E [MSGID: 106057] [glusterd-handshake.c:94:get_snap_volname_and_volinfo] 0-management: Failed to fetch snap Snap_GMT-2017.05.31-09.21.37 [Invalid argument]
[2017-05-31 09:33:46.251781] E [MSGID: 106063] [glusterd-handshake.c:274:build_volfile_path] 0-management: Failed to get snap volinfo from path (/snaps/Snap_GMT-2017.05.31-09.21.37/331ec972f90d494d8a86dd4f69d718b7)
[2017-05-31 09:33:47.464566] E [MSGID: 106057] [glusterd-handshake.c:94:get_snap_volname_and_volinfo] 0-management: Failed to fetch snap Snap_GMT-2017.05.31-09.21.37 [Invalid argument]
[2017-05-31 09:33:47.464594] E [MSGID: 106063] [glusterd-handshake.c:274:build_volfile_path] 0-management: Failed to get snap volinfo from path (/snaps/Snap_GMT-2017.05.31-09.21.37/331ec972f90d494d8a86dd4f69d718b7)
[2017-05-31 09:33:48.465272] E [MSGID: 106057] [glusterd-handshake.c:94:get_snap_volname_and_volinfo] 0-management: Failed to fetch snap Snap_GMT-2017.05.31-09.21.37 [Invalid argument]
[2017-05-31 09:33:48.465314] E [MSGID: 106063] [glusterd-handshake.c:274:build_volfile_path] 0-management: Failed to get snap volinfo from path (/snaps/Snap_GMT-2017.05.31-09.21.37/331ec972f90d494d8a86dd4f69d718b7)
[2017-05-31 09:35:05.133683] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fe45a824dc5] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x7fe45bebcfd5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x7fe45bebcdfb] ) 0-: received signum (15), shutting down
[2017-05-31 09:35:46.137355] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-05-31 09:35:46.366310] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-05-31 09:35:46.366362] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory
[2017-05-31 09:35:46.472385] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.10.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2017-05-31 09:35:46.472422] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2017-05-31 09:35:46.472459] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-05-31 09:35:46.472468] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-05-31 09:35:46.521769] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory]
[2017-05-31 09:35:46.546113] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31000
[2017-05-31 09:35:46.944294] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-31 09:35:46.957046] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-31 09:35:46.957117] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-05-31 09:35:46.957162] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:35:46.966005] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:35:46.965995] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-05-31 09:35:47.015756] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 877c6f79-0201-4c41-986e-d2a065911ef8
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option transport.address-family inet
  8:     option rpc-auth-allow-insecure on
  9:     option event-threads 1
 10:     option ping-timeout 0
 11:     option transport.socket.read-fail-log off
 12:     option transport.socket.keepalive-interval 2
 13:     option transport.socket.keepalive-time 10
 14:     option transport-type rdma
 15:     option working-directory /var/lib/glusterd
 16: end-volume
 17:  
+------------------------------------------------------------------------------+
[2017-05-31 09:35:47.018060] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-31 09:35:47.154012] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04, host: glust01-li, port: 0
[2017-05-31 09:35:47.158156] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:35:47.158191] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:35:47.161895] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-05-31 09:35:47.169359] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-05-31 09:35:47.169519] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-05-31 09:35:47.170100] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-05-31 09:35:47.173330] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-05-31 09:35:47.173356] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-05-31 09:35:47.173401] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-05-31 09:35:48.181613] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-05-31 09:35:48.181969] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-05-31 09:35:48.181994] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-05-31 09:35:48.186337] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-05-31 09:35:48.186521] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-05-31 09:35:48.186539] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-05-31 09:35:48.186582] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-05-31 09:35:48.186735] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-05-31 09:35:48.186749] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-05-31 09:35:48.197486] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:35:48.214918] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:35:48.217998] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:35:48.218255] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-31 09:35:49.222917] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-31 09:35:50.237231] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-05-31 09:35:50.237633] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:35:50.237993] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65, host: 10.8.28.2, port: 0
[2017-05-31 09:35:50.243448] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:35:50.245414] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/shares1/b has disconnected from glusterd.
[2017-05-31 09:35:50.245738] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:35:50.245900] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/config/b has disconnected from glusterd.
[2017-05-31 09:35:50.246188] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:35:50.246340] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/bricks/shared_storage/b has disconnected from glusterd.
[2017-05-31 09:35:50.247269] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-31 09:35:50.250974] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:35:50.262879] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:35:50.264490] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:35:50.264546] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:35:50.268044] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glust01-li (0), ret: 0, op_ret: 0
[2017-05-31 09:35:50.284264] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:35:50.284300] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:35:50.287473] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:35:50.287588] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:35:50.287610] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:35:50.306761] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-31 09:35:50.330726] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:35:50.331795] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:35:50.331871] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:35:50.353609] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.8.28.2 (0), ret: 0, op_ret: 0
[2017-05-31 09:35:50.363722] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:35:50.363752] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:35:50.371265] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:35:50.551154] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-shares1 on port 49155
[2017-05-31 09:35:50.551242] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-vol_config on port 49156
[2017-05-31 09:35:50.879393] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shared_storage/b on port 49154
[2017-05-31 09:35:50.886438] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/config/b on port 49153
[2017-05-31 09:35:51.283486] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /bricks/shares1/b on port 49152
[2017-05-31 09:36:35.573344] W [socket.c:593:__socket_rwv] 0-management: readv on 10.8.28.2:24007 failed (No data available)
[2017-05-31 09:36:35.573397] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <10.8.28.2> (<62fa28ac-7ab1-4f51-bc4d-45ea0005fb65>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-05-31 09:36:35.573548] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f2c875a1eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f2c875abdc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f2c8766141a] ) 0-management: Lock for vol shares1 not held
[2017-05-31 09:36:35.573565] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for shares1
[2017-05-31 09:36:35.573590] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f2c875a1eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f2c875abdc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f2c8766141a] ) 0-management: Lock for vol vol_config not held
[2017-05-31 09:36:35.573601] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_config
[2017-05-31 09:36:35.573624] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f2c875a1eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f2c875abdc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f2c8766141a] ) 0-management: Lock for vol vol_shared_storage not held
[2017-05-31 09:36:35.573634] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_shared_storage
[2017-05-31 09:37:08.306367] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-31 09:37:08.313603] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:37:17.487648] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:37:17.487710] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:37:17.491689] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.8.28.2 (0), ret: 0, op_ret: 0
[2017-05-31 09:37:20.508086] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:37:20.508139] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:37:20.560982] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:37:20.582897] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65, host: 10.8.28.2, port: 0
[2017-05-31 09:37:20.590060] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:37:20.590087] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:37:20.598100] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 62fa28ac-7ab1-4f51-bc4d-45ea0005fb65
[2017-05-31 09:40:43.977784] W [socket.c:593:__socket_rwv] 0-management: readv on 160.5.5.90:24007 failed (No data available)
[2017-05-31 09:40:43.977829] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <glust01-li> (<64390a1d-af72-4f10-be41-e23d4f881e04>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-05-31 09:40:43.977936] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f2c875a1eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f2c875abdc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f2c8766141a] ) 0-management: Lock for vol shares1 not held
[2017-05-31 09:40:43.977948] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for shares1
[2017-05-31 09:40:43.977964] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f2c875a1eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f2c875abdc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f2c8766141a] ) 0-management: Lock for vol vol_config not held
[2017-05-31 09:40:43.977971] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_config
[2017-05-31 09:40:43.977984] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x20eda) [0x7f2c875a1eda] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0x2adc8) [0x7f2c875abdc8] -->/usr/lib64/glusterfs/3.10.2/xlator/mgmt/glusterd.so(+0xe041a) [0x7f2c8766141a] ) 0-management: Lock for vol vol_shared_storage not held
[2017-05-31 09:40:43.977991] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_shared_storage
[2017-05-31 09:41:25.519064] E [socket.c:2310:socket_connect_finish] 0-management: connection to 160.5.5.90:24007 failed (Connection refused)
[2017-05-31 09:41:27.801681] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000
[2017-05-31 09:41:27.810444] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:41:27.810654] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:41:27.810675] I [MSGID: 106086] [glusterd-snapshot.c:9703:glusterd_update_missed_snap_entry] 0-management: Duplicate entry. Not updating
[2017-05-31 09:41:27.813695] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glust01-li (0), ret: 0, op_ret: 0
[2017-05-31 09:41:27.822659] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04, host: glust01-li, port: 0
[2017-05-31 09:41:27.826238] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:41:27.826274] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:41:27.831062] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:41:27.831114] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-05-31 09:41:27.839162] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:41:30.742844] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 64390a1d-af72-4f10-be41-e23d4f881e04
[2017-05-31 09:43:29.746663] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 09:43:29.769377] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-05-31 09:43:29.769905] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick glust01-lc:/run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2/b has disconnected from glusterd.
[2017-05-31 09:43:29.772667] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 33f06f9cbcfb405bbc605a14d28e3cc7.glust01-lc.run-gluster-snaps-33f06f9cbcfb405bbc605a14d28e3cc7-brick2-b not found [Invalid argument]
[2017-05-31 09:43:29.919289] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2/b on port 49157
[2017-05-31 11:00:03.977413] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/1e3b14374bde4604a78b4f89890d368c/brick2/b to existing process for /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2/b
[2017-05-31 11:00:03.977480] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 1e3b14374bde4604a78b4f89890d368c.glust01-lc.run-gluster-snaps-1e3b14374bde4604a78b4f89890d368c-brick2-b not found [Invalid argument]
[2017-05-31 13:00:03.705215] I [glusterd-utils.c:5051:attach_brick] 0-management: add brick /run/gluster/snaps/3d575da9bd2d4c99a0990024e5432971/brick2/b to existing process for /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2/b
[2017-05-31 13:00:03.705267] W [MSGID: 106057] [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: Snap volume 3d575da9bd2d4c99a0990024e5432971.glust01-lc.run-gluster-snaps-3d575da9bd2d4c99a0990024e5432971-brick2-b not found [Invalid argument]
[2017-05-31 13:43:16.407739] I [MSGID: 106091] [glusterd-snapshot.c:6350:glusterd_snapshot_remove_commit] 0-management: Successfully marked snap Snap_GMT-2017.05.31-09.43.24 for decommission.
[2017-05-31 13:43:16.446905] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2 failed (No such file or directory) [No such file or directory]
[2017-05-31 13:43:25.453731] E [MSGID: 106038] [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2 (brick: /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2/b): No such file or directory.
[2017-05-31 13:43:27.095090] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: Failed to rmdir: /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/, err: Directory not empty. More than one glusterd running on this node. [Directory not empty]
[2017-05-31 13:43:27.096718] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2/b on port 49157
The message "E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting /run/gluster/snaps/33f06f9cbcfb405bbc605a14d28e3cc7/brick2 failed (No such file or directory) [No such file or directory]" repeated 2 times between [2017-05-31 13:43:16.446905] and [2017-05-31 13:43:22.453606]
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users