Best regards, Olaf Op ma 28 mrt. 2022 om 10:01 schreef Peter Schmidt < peterschmidt18351@xxxxxxxxxx>: > > Hi Olaf, > > I tried running "gluster volume start hdd force" but sadly it did not > change anything. > > the raid rebuild has finished now and everything seems to be fine: > md6 : active raid6 sdu1[2] sdx1[5] sds1[0] sdt1[1] sdz1[7] sdv1[3] sdw1[4] > sdaa1[8] sdy1[6] > 68364119040 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9] > [UUUUUUUUU] > bitmap: 0/73 pages [0KB], 65536KB chunk > > Best regards > Peter > > 25.03.2022, 12:36, "Olaf Buitelaar" <olaf.buitelaar@xxxxxxxxx>: > > Hi Peter, > > I see your raid array is rebuilding, could it be your xfs needs a repair, > using xfs_repair? > did you try running gluster v hdd start force? > > Kind regards, > > Olaf > > > Op do 24 mrt. 2022 om 15:54 schreef Peter Schmidt < > peterschmidt18351@xxxxxxxxxx>: > > Hello everyone, > > I'm running an oVirt cluster on top of a distributed-replicate gluster > volume and one of the bricks cannot be mounted anymore from my oVirt hosts. > This morning I also noticed a stack trace and a spike in TCP connections on > one of the three gluster nodes (storage2), which I have attached at the end > of this mail. Only this particular brick on storage2 seems to be causing > trouble: > *Brick storage2:/data/glusterfs/hdd/brick3/brick* > *Status: Transport endpoint is not connected* > > I don't know what's causing this or how to resolve this issue. I would > appreciate it if someone could take a look at my logs and point me in the > right direction. If any additional logs are required, please let me know. > Thank you in advance! > > Operating system on all hosts: Centos 7.9.2009 > oVirt version: 4.3.10.4-1 > Gluster versions: > - storage1: 6.10-1 > - storage2: 6.7-1 > - storage3: 6.7-1 > > #################################### > # brick is not connected/mounted on the oVirt hosts > > *[xlator.protocol.client.hdd-client-7.priv]* > *fd.0.remote_fd = -1* > *------ = ------* > *granted-posix-lock[0] = owner = 9d673ffe323e25cd, cmd = F_SETLK fl_type = > F_RDLCK, fl_start = 100, fl_end = 100, user_flock: l_type = F_RDLCK, > l_start = 100, l_len = 1* > *granted-posix-lock[1] = owner = 9d673ffe323e25cd, cmd = F_SETLK fl_type = > F_RDLCK, fl_start = 101, fl_end = 101, user_flock: l_type = F_RDLCK, > l_start = 101, l_len = 1* > *------ = ------* > *connected = 0* > *total_bytes_read = 11383136800* > *ping_timeout = 10* > *total_bytes_written = 16699851552* > *ping_msgs_sent = 1* > *msgs_sent = 2* > > #################################### > # mount log from one of the oVirt hosts > # the IP 172.22.102.142 corresponds to my gluster node "storage2" > # the port 49154 corresponds to the brick > storage2:/data/glusterfs/hdd/brick3/brick > > *[2022-03-24 10:59:28.138178] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] > 0-hdd-client-7: socket disconnected* > *[2022-03-24 10:59:38.142698] I [rpc-clnt.c:2028:rpc_clnt_reconfig] > 0-hdd-client-7: changing port to 49154 (from 0)* > *The message "I [MSGID: 114018] [client.c:2331:client_rpc_notify] > 0-hdd-client-7: disconnected from hdd-client-7. Client process will keep > trying to connect to glusterd until brick's port is available" repeated 4 > times between [2022-03-24 10:58:04.114741] and [2022-03-24 10:59:28.137380]* > *The message "W [MSGID: 114032] > [client-handshake.c:1546:client_dump_version_cbk] 0-hdd-client-7: received > RPC status error [Transport endpoint is not connected]" repeated 4 times > between [2022-03-24 10:58:04.115169] and [2022-03-24 10:59:28.138052]* > *[2022-03-24 10:59:49.143217] C > [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-hdd-client-7: server > 172.22.102.142:49154 <http://172.22.102.142:49154/> has not responded in > the last 10 seconds, disconnecting.* > *[2022-03-24 10:59:49.143838] I [MSGID: 114018] > [client.c:2331:client_rpc_notify] 0-hdd-client-7: disconnected from > hdd-client-7. Client process will keep trying to connect to glusterd until > brick's port is available* > *[2022-03-24 10:59:49.144540] E [rpc-clnt.c:346:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f6724643adb] (--> > /lib64/libgfrpc.so.0(+0xd7e4)[0x7f67243ea7e4] (--> > /lib64/libgfrpc.so.0(+0xd8fe)[0x7f67243ea8fe] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f67243eb987] (--> > /lib64/libgfrpc.so.0(+0xf518)[0x7f67243ec518] ))))) 0-hdd-client-7: forced > unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2022-03-24 > 10:59:38.145208 (xid=0x861)* > *[2022-03-24 10:59:49.144557] W [MSGID: 114032] > [client-handshake.c:1546:client_dump_version_cbk] 0-hdd-client-7: received > RPC status error [Transport endpoint is not connected]* > *[2022-03-24 10:59:49.144653] E [rpc-clnt.c:346:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f6724643adb] (--> > /lib64/libgfrpc.so.0(+0xd7e4)[0x7f67243ea7e4] (--> > /lib64/libgfrpc.so.0(+0xd8fe)[0x7f67243ea8fe] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f67243eb987] (--> > /lib64/libgfrpc.so.0(+0xf518)[0x7f67243ec518] ))))) 0-hdd-client-7: forced > unwinding frame type(GF-DUMP) op(NULL(2)) called at 2022-03-24 > 10:59:38.145218 (xid=0x862)* > *[2022-03-24 10:59:49.144665] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] > 0-hdd-client-7: socket disconnected* > > #################################### > # netcat/telnet to the brick's port of storage2 are working > > *[root@storage1 <root@storage1> ~]# netcat -z -v 172.22.102.142 49154* > *Connection to 172.22.102.142 49154 port [tcp/*] succeeded!* > > *[root@storage3 <root@storage3> ~]# netcat -z -v 172.22.102.142 49154* > *Connection to 172.22.102.142 49154 port [tcp/*] succeeded!* > > *[root@ovirthost1 <root@ovirthost1> /var/log/glusterfs]# netcat -z -v > 172.22.102.142 49154* > *Connection to 172.22.102.142 49154 port [tcp/*] succeeded!* > > #################################### > # gluster peer status - all gluster peers are connected > *[root@storage3 <root@storage3> ~]# gluster peer status* > *Number of Peers: 2* > > *Hostname: storage1* > *Uuid: 055e79c2-b1ff-4a82-9296-205d6877904e* > *State: Peer in Cluster (Connected)* > > *Hostname: storage2* > *Uuid: d7adcb92-2e71-41a9-80d4-13180ee673cf* > *State: Peer in Cluster (Connected)* > > #################################### > # Configuration of the volume > *Volume Name: hdd* > *Type: Distributed-Replicate* > *Volume ID: 1b47c2f8-5024-4b85-aa7f-a3f767bb076c* > *Status: Started* > *Snapshot Count: 0* > *Number of Bricks: 4 x 3 = 12* > *Transport-type: tcp* > *Bricks:* > *Brick1: storage1:/data/glusterfs/hdd/brick1/brick* > *Brick2: storage2:/data/glusterfs/hdd/brick1/brick* > *Brick3: storage3:/data/glusterfs/hdd/brick1/brick* > *Brick4: storage1:/data/glusterfs/hdd/brick2/brick* > *Brick5: storage2:/data/glusterfs/hdd/brick2/brick* > *Brick6: storage3:/data/glusterfs/hdd/brick2/brick* > *Brick7: storage1:/data/glusterfs/hdd/brick3/brick* > *Brick8: storage2:/data/glusterfs/hdd/brick3/brick* > *Brick9: storage3:/data/glusterfs/hdd/brick3/brick* > *Brick10: storage1:/data/glusterfs/hdd/brick4/brick* > *Brick11: storage2:/data/glusterfs/hdd/brick4/brick* > *Brick12: storage3:/data/glusterfs/hdd/brick4/brick* > *Options Reconfigured:* > *storage.owner-gid: 36* > *storage.owner-uid: 36* > *server.event-threads: 4* > *client.event-threads: 4* > *cluster.choose-local: off* > *user.cifs: off* > *features.shard: on* > *cluster.shd-wait-qlength: 10000* > *cluster.shd-max-threads: 8* > *cluster.locking-scheme: granular* > *cluster.data-self-heal-algorithm: full* > *cluster.server-quorum-type: server* > *cluster.eager-lock: enable* > *network.remote-dio: enable* > *performance.low-prio-threads: 32* > *performance.io-cache: off* > *performance.read-ahead: off* > *performance.quick-read: off* > *auth.allow: ** > *network.ping-timeout: 10* > *cluster.quorum-type: auto* > *transport.address-family: inet* > *nfs.disable: on* > *performance.client-io-threads: on* > > #################################### > # gluster volume status. The brick running on port 49154 is supposedly > online > > *Status of volume: hdd* > *Gluster process TCP Port RDMA Port Online > Pid* > > *------------------------------------------------------------------------------* > *Brick storage1:/data/gluste* > *rfs/hdd/brick1/brick 49158 0 Y > 9142* > *Brick storage2:/data/gluste* > *rfs/hdd/brick1/brick 49152 0 Y > 115896* > *Brick storage3:/data/gluste* > *rfs/hdd/brick1/brick 49158 0 Y > 131775* > *Brick storage1:/data/gluste* > *rfs/hdd/brick2/brick 49159 0 Y > 9151* > *Brick storage2:/data/gluste* > *rfs/hdd/brick2/brick 49153 0 Y > 115904* > *Brick storage3:/data/gluste* > *rfs/hdd/brick2/brick 49159 0 Y > 131783* > *Brick storage1:/data/gluste* > *rfs/hdd/brick3/brick 49160 0 Y > 9163* > *Brick storage2:/data/gluste* > *rfs/hdd/brick3/brick 49154 0 Y > 115913* > *Brick storage3:/data/gluste* > *rfs/hdd/brick3/brick 49160 0 Y > 131792* > *Brick storage1:/data/gluste* > *rfs/hdd/brick4/brick 49161 0 Y > 9170* > *Brick storage2:/data/gluste* > *rfs/hdd/brick4/brick 49155 0 Y > 115923* > *Brick storage3:/data/gluste* > *rfs/hdd/brick4/brick 49161 0 Y > 131800* > *Self-heal Daemon on localhost N/A N/A Y > 170468* > *Self-heal Daemon on storage3 N/A N/A Y > 132263* > *Self-heal Daemon on storage1 N/A N/A Y > 9512* > > *Task Status of Volume hdd* > > *------------------------------------------------------------------------------* > *There are no active volume tasks* > > #################################### > # gluster volume heal hdd info split-brain. All bricks are connected and > showing no entries (0), except for brick3 on storage2 > *Brick storage2:/data/glusterfs/hdd/brick3/brick* > *Status: Transport endpoint is not connected* > *Number of entries in split-brain: -* > > #################################### > # gluster volume heal hdd info. Only brick3 seems to be affected and it > has lots of entries. brick3 on storage2 is not connected > > *Brick storage1:/data/glusterfs/hdd/brick3/brick* > > */538befbf-ffa7-4a8c-8827-cee679d589f4/images/615fa020-9737-4b83-a3c1-a61e32400d59/f4917758-deae-4a62-bf4d-5b9a95a7db5b* > *<gfid:f3d0b19a-2544-48c5-90b7-addd561113bc>* > */.shard/753a8a81-bd06-4c8c-9515-d54123f6fe4d.1* > */.shard/c7f5f88f-dc85-4645-9178-c7df8e46a99d.83* > > */538befbf-ffa7-4a8c-8827-cee679d589f4/images/bc4362e6-cd43-4ab8-b8fa-0ea72405b7da/ea9c0e7c-d2c7-43c8-b19f-7a3076cc6743* > */.shard/dc46e963-2b68-4802-9537-42f25ea97ae2.10872* > */.shard/dc46e963-2b68-4802-9537-42f25ea97ae2.1901* > > */538befbf-ffa7-4a8c-8827-cee679d589f4/images/e48e80fb-d42f-47a4-9a56-07fd7ad868b3/31fd839f-85bf-4c42-ac0e-7055d903df40* > */.shard/82700f9b-c7e0-4568-a565-64c9a770449f.223* > */.shard/82700f9b-c7e0-4568-a565-64c9a770449f.243* > */.shard/dc46e963-2b68-4802-9537-42f25ea97ae2.10696* > */.shard/dc46e963-2b68-4802-9537-42f25ea97ae2.10902* > *..* > *Status: Connected* > *Number of entries: 664* > > *Brick storage2:/data/glusterfs/hdd/brick3/brick* > *Status: Transport endpoint is not connected* > *Number of entries: -* > > *Brick storage3:/data/glusterfs/hdd/brick3/brick* > > */538befbf-ffa7-4a8c-8827-cee679d589f4/images/615fa020-9737-4b83-a3c1-a61e32400d59/f4917758-deae-4a62-bf4d-5b9a95a7db5b* > *<gfid:f3d0b19a-2544-48c5-90b7-addd561113bc>* > */.shard/753a8a81-bd06-4c8c-9515-d54123f6fe4d.1* > *..* > *Status: Connected* > *Number of entries: 664* > > #################################### > # /data/glusterfs/hdd/brick3 on storage2 is running inside of a software > RAID > > *md6 : active raid6 sdac1[6] sdz1[3] sdx1[1] sdad1[7] sdaa1[4] sdy1[2] > sdw1[0] sdab1[5] sdae1[8]* > * 68364119040 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9] > [UUUUUUUUU]* > * [============>........] check = 64.4% (6290736128/9766302720) > finish=3220.5min speed=17985K/sec* > * bitmap: 10/73 pages [40KB], 65536KB chunk* > > #################################### > # glfsheal-hdd.log on storage2 > > *[2022-03-24 10:15:33.238884] I [MSGID: 114046] > [client-handshake.c:1106:client_setvolume_cbk] 0-hdd-client-10: Connected > to hdd-client-10, attached to remote volume > '/data/glusterfs/hdd/brick4/brick'.* > *[2022-03-24 10:15:33.238931] I [MSGID: 108002] > [afr-common.c:5607:afr_notify] 0-hdd-replicate-3: Client-quorum is met* > *[2022-03-24 10:15:33.241616] I [MSGID: 114046] > [client-handshake.c:1106:client_setvolume_cbk] 0-hdd-client-11: Connected > to hdd-client-11, attached to remote volume > '/data/glusterfs/hdd/brick4/brick'.* > *[2022-03-24 10:15:44.078651] C > [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-hdd-client-7: server > 172.22.102.142:49154 <http://172.22.102.142:49154/> has not responded in > the last 10 seconds, disconnecting.* > *[2022-03-24 10:15:44.078891] I [MSGID: 114018] > [client.c:2331:client_rpc_notify] 0-hdd-client-7: disconnected from > hdd-client-7. Client process will keep trying to connect to glusterd until > brick's port is available* > *[2022-03-24 10:15:44.079954] E [rpc-clnt.c:346:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc6c0cadadb] (--> > /lib64/libgfrpc.so.0(+0xd7e4)[0x7fc6c019f7e4] (--> > /lib64/libgfrpc.so.0(+0xd8fe)[0x7fc6c019f8fe] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7fc6c01a0987] (--> > /lib64/libgfrpc.so.0(+0xf518)[0x7fc6c01a1518] ))))) 0-hdd-client-7: forced > unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2022-03-24 > 10:15:33.209640 (xid=0x5)* > *[2022-03-24 10:15:44.080008] W [MSGID: 114032] > [client-handshake.c:1547:client_dump_version_cbk] 0-hdd-client-7: received > RPC status error [Transport endpoint is not connected]* > *[2022-03-24 10:15:44.080526] E [rpc-clnt.c:346:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc6c0cadadb] (--> > /lib64/libgfrpc.so.0(+0xd7e4)[0x7fc6c019f7e4] (--> > /lib64/libgfrpc.so.0(+0xd8fe)[0x7fc6c019f8fe] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7fc6c01a0987] (--> > /lib64/libgfrpc.so.0(+0xf518)[0x7fc6c01a1518] ))))) 0-hdd-client-7: forced > unwinding frame type(GF-DUMP) op(NULL(2)) called at 2022-03-24 > 10:15:33.209655 (xid=0x6)* > *[2022-03-24 10:15:44.080574] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] > 0-hdd-client-7: socket disconnected* > > #################################### > # stack trace on storage2 that happened this morning > > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr000:115974 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr000 D ffff9b91b8951070 0 > 115974 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>] > _xfs_log_force_lsn+0x2d1/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr001:121353 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr001 D ffff9b9b7d4dac80 0 > 121353 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>] > _xfs_log_force_lsn+0x2d1/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr002:121354 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr002 D ffff9b9b7d75ac80 0 > 121354 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>] > _xfs_log_force_lsn+0x2d1/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr003:121355 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr003 D ffff9b9b7d51ac80 0 > 121355 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>] > schedule_timeout+0x221/0x2d0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ? > ttwu_do_wakeup+0x19/0xe0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ? > ttwu_do_activate+0x6f/0x80* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ? > try_to_wake_up+0x190/0x390* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>] > wait_for_completion+0xfd/0x140* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>] > flush_work+0x10a/0x1b0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ? > move_linked_works+0x90/0x90* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>] > xlog_cil_force_lsn+0x8a/0x210 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>] > _xfs_log_force_lsn+0x74/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa15bcb1f>] ? > filemap_fdatawait_range+0x1f/0x30* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ? > down_read+0x12/0x40* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr004:121356 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr004 D ffff9b9b7d75ac80 0 > 121356 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>] > schedule_timeout+0x221/0x2d0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ? > ttwu_do_wakeup+0x19/0xe0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ? > ttwu_do_activate+0x6f/0x80* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ? > try_to_wake_up+0x190/0x390* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>] > wait_for_completion+0xfd/0x140* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>] > flush_work+0x10a/0x1b0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ? > move_linked_works+0x90/0x90* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>] > xlog_cil_force_lsn+0x8a/0x210 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>] > _xfs_log_force_lsn+0x74/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa15bcb1f>] ? > filemap_fdatawait_range+0x1f/0x30* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ? > down_read+0x12/0x40* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr005:153774 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr005 D ffff9b9b7d61ac80 0 > 153774 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>] > schedule_timeout+0x221/0x2d0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ? > ttwu_do_wakeup+0x19/0xe0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ? > ttwu_do_activate+0x6f/0x80* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ? > try_to_wake_up+0x190/0x390* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>] > wait_for_completion+0xfd/0x140* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>] > flush_work+0x10a/0x1b0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ? > move_linked_works+0x90/0x90* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>] > xlog_cil_force_lsn+0x8a/0x210 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167335b>] ? > getxattr+0x11b/0x180* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>] > _xfs_log_force_lsn+0x74/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ? > down_read+0x12/0x40* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr006:153775 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr006 D ffff9b9b7d49ac80 0 > 153775 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>] > schedule_timeout+0x221/0x2d0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ? > ttwu_do_wakeup+0x19/0xe0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ? > ttwu_do_activate+0x6f/0x80* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ? > try_to_wake_up+0x190/0x390* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>] > wait_for_completion+0xfd/0x140* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>] > flush_work+0x10a/0x1b0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ? > move_linked_works+0x90/0x90* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>] > xlog_cil_force_lsn+0x8a/0x210 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167335b>] ? > getxattr+0x11b/0x180* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>] > _xfs_log_force_lsn+0x74/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ? > down_read+0x12/0x40* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr007:153776 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr007 D ffff9b9958c962a0 0 > 153776 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>] > schedule_timeout+0x221/0x2d0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d7782>] ? > check_preempt_curr+0x92/0xa0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ? > ttwu_do_wakeup+0x19/0xe0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ? > try_to_wake_up+0x190/0x390* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>] > wait_for_completion+0xfd/0x140* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>] > flush_work+0x10a/0x1b0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ? > move_linked_works+0x90/0x90* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>] > xlog_cil_force_lsn+0x8a/0x210 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167335b>] ? > getxattr+0x11b/0x180* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>] > _xfs_log_force_lsn+0x74/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ? > down_read+0x12/0x40* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr008:153777 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr008 D ffff9b9b7d61ac80 0 > 153777 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>] > _xfs_log_force_lsn+0x2d1/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr009:153778 blocked > for more than 120 seconds.* > *Mar 24 06:24:06 storage2 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.* > *Mar 24 06:24:06 storage2 kernel: glfs_iotwr009 D ffff9b9958c920e0 0 > 153778 1 0x00000080* > *Mar 24 06:24:06 storage2 kernel: Call Trace:* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>] > _xfs_log_force_lsn+0x2d1/0x310 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? > wake_up_state+0x20/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] > xfs_file_fsync+0xfd/0x1c0 [xfs]* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] > SyS_fdatasync+0x13/0x20* > *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] > system_call_fastpath+0x25/0x2a* > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > https://lists.gluster.org/mailman/listinfo/gluster-users > > --0000000000006d508e05db44ff40 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi Peter,<div><br></div><div>I think Staril means, running= the command;=C2=A0hosted-engine --set-maintenance --mode=3Dlocal, this is = also possible from the ovirt ui, via the ribbon on the hosts section;</div>= <div><img src=3D"cid:ii_l1akxwfw0" alt=3D"image.png" width=3D"383" height= =3D"297"><br></div><div><br></div><div>From the log's it seems gluster = has difficulty find the shared's, e.g.;</div><div>.shard/e5f699e2-de11-= 41be-bd24-e29876928f0f.1279 (be318638-e8a0-4c6d-977d-7a937aa84806/e5f699e2-= de11-41be-bd24-e29876928f0f.1279<br></div><div>Do these files exist within = your brick's directories?=C2=A0</div><div>Did you try to repair the fil= esystem=C2=A0using xfs_repair?</div><div><br></div><div>Best regards,</div>= <div><br></div><div>Olaf</div></div><br><div class=3D"gmail_quote"><div dir= =3D"ltr" class=3D"gmail_attr">Op ma 28 mrt. 2022 om 10:01 schreef Peter Sch= midt <<a href=3D"mailto:peterschmidt18351@xxxxxxxxxx";>peterschmidt18351@= yandex.com</a>>:<br></div><blockquote class=3D"gmail_quote" style=3D"mar= gin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1= ex"><div>=C2=A0</div><div>Hi Olaf,</div><div>=C2=A0</div><div>I tried runni= ng "gluster volume start hdd force" but sadly it did not change a= nything.</div><div>=C2=A0</div><div>the raid rebuild has finished now and e= verything seems to be fine:</div><div><div>md6 : active raid6 sdu1[2] sdx1[= 5] sds1[0] sdt1[1] sdz1[7] sdv1[3] sdw1[4] sdaa1[8] sdy1[6]</div><div>=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 68364119040 blocks super 1.2 level 6, 512k chun= k, algorithm 2 [9/9] [UUUUUUUUU]</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 b= itmap: 0/73 pages [0KB], 65536KB chunk</div></div><div>=C2=A0</div><div>Bes= t regards</div><div>Peter</div><div>=C2=A0</div><div>25.03.2022, 12:36, &qu= ot;Olaf Buitelaar" <<a href=3D"mailto:olaf.buitelaar@xxxxxxxxx"; tar= get=3D"_blank">olaf.buitelaar@xxxxxxxxx</a>>:</div><blockquote><div>Hi P= eter,<div>=C2=A0</div><div>I see your raid array is rebuilding, could it be= your xfs needs a repair, using xfs_repair?</div><div>did you try running g= luster v hdd start force?=C2=A0</div><div>=C2=A0</div><div>Kind regards,=C2= =A0</div><div>=C2=A0</div><div>Olaf</div><div>=C2=A0</div></div>=C2=A0<div>= <div>Op do 24 mrt. 2022 om 15:54 schreef Peter Schmidt <<a href=3D"mailt= o:peterschmidt18351@xxxxxxxxxx" rel=3D"noopener noreferrer" target=3D"_blan= k">peterschmidt18351@xxxxxxxxxx</a>>:</div><blockquote style=3D"border-l= eft:1px solid rgb(204,204,204);margin:0px 0px 0px 0.8ex;padding-left:1ex"><= div><div><div><div><div>Hello everyone,</div><div>=C2=A0</div><div>I'm = running an oVirt cluster on top of a distributed-replicate gluster volume a= nd one of the bricks cannot be mounted anymore from my oVirt hosts. This mo= rning I also noticed a stack trace and a spike in TCP connections on one of= the three gluster nodes (storage2), which I have attached at the end of th= is mail. Only this particular brick on storage2 seems to be causing trouble= :</div><div><em>Brick storage2:/data/glusterfs/hdd/brick3/brick</em></div><= div><em>Status: Transport endpoint is not connected</em></div><div>=C2=A0</= div><div>I don't know what's causing this or how to resolve this is= sue. I would appreciate it if someone could take a look at my logs and poin= t me in the right direction. If any additional logs are required, please le= t me know. Thank you in advance!</div><div>=C2=A0</div><div>Operating syste= m on all hosts: Centos 7.9.2009</div><div>oVirt version: 4.3.10.4-1</div><d= iv>Gluster versions:</div><div>- storage1: 6.10-1</div><div>- storage2: 6.7= -1</div><div>- storage3: 6.7-1</div><div>=C2=A0</div><div>#################= ###################</div><div># brick is not connected/mounted on the oVirt= hosts</div><div>=C2=A0</div><div><em>[xlator.protocol.client.hdd-client-7.= priv]</em></div><div><em>fd.0.remote_fd =3D -1</em></div><div><em>------ = =3D ------</em></div><div><em>granted-posix-lock[0] =3D owner =3D 9d673ffe3= 23e25cd, cmd =3D F_SETLK fl_type =3D F_RDLCK, fl_start =3D 100, fl_end =3D = 100, user_flock: l_type =3D F_RDLCK, l_start =3D 100, l_len =3D 1</em></div= ><div><em>granted-posix-lock[1] =3D owner =3D 9d673ffe323e25cd, cmd =3D F_S= ETLK fl_type =3D F_RDLCK, fl_start =3D 101, fl_end =3D 101, user_flock: l_t= ype =3D F_RDLCK, l_start =3D 101, l_len =3D 1</em></div><div><em>------ =3D= ------</em></div><div><em>connected =3D 0</em></div><div><em>total_bytes_r= ead =3D <span>11383136800</span></em></div><div><em>ping_timeout =3D 10</em= ></div><div><em>total_bytes_written =3D <span>16699851552</span></em></div>= <div><em>ping_msgs_sent =3D 1</em></div><div><em>msgs_sent =3D 2</em></div>= <div>=C2=A0</div><div>####################################</div><div># moun= t log from one of the oVirt hosts</div><div># the IP 172.22.102.142 corresp= onds to my gluster node "storage2"</div><div># the port 49154 cor= responds to the brick storage2:/data/glusterfs/hdd/brick3/brick=C2=A0=C2=A0= =C2=A0=C2=A0 =C2=A0</div><div>=C2=A0</div><div><em>[<span>2022-03-24 10</sp= an>:59:28.138178] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] 0-hdd-client-7:= socket disconnected</em></div><div><em>[<span>2022-03-24 10</span>:59:38.1= 42698] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-hdd-client-7: changing port = to 49154 (from 0)</em></div><div><em>The message "I [MSGID: 114018] [c= lient.c:2331:client_rpc_notify] 0-hdd-client-7: disconnected from hdd-clien= t-7. Client process will keep trying to connect to glusterd until brick'= ;s port is available" repeated 4 times between [<span>2022-03-24 10</s= pan>:58:04.114741] and [<span>2022-03-24 10</span>:59:28.137380]</em></div>= <div><em>The message "W [MSGID: 114032] [client-handshake.c:1546:clien= t_dump_version_cbk] 0-hdd-client-7: received RPC status error [Transport en= dpoint is not connected]" repeated 4 times between [<span>2022-03-24 1= 0</span>:58:04.115169] and [<span>2022-03-24 10</span>:59:28.138052]</em></= div><div><em>[<span>2022-03-24 10</span>:59:49.143217] C [rpc-clnt-ping.c:1= 55:rpc_clnt_ping_timer_expired] 0-hdd-client-7: server <a href=3D"http://17= 2.22.102.142:49154/" target=3D"_blank">172.22.102.142:49154</a> has not res= ponded in the last 10 seconds, disconnecting.</em></div><div><em>[<span>202= 2-03-24 10</span>:59:49.143838] I [MSGID: 114018] [client.c:2331:client_rpc= _notify] 0-hdd-client-7: disconnected from hdd-client-7. Client process wil= l keep trying to connect to glusterd until brick's port is available</e= m></div><div><em>[<span>2022-03-24 10</span>:59:49.144540] E [rpc-clnt.c:34= 6:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0= x13b)[0x7f<span>6724643</span>adb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x= 7f67243ea7e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f67243ea8fe] (-->= /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f67243eb987] (--= > /lib64/libgfrpc.so.0(+0xf518)[0x7f67243ec518] ))))) 0-hdd-client-7: fo= rced unwinding frame type(GF-DUMP) op(DUMP(1)) called at <span>2022-03-24 1= 0</span>:59:38.145208 (xid=3D0x861)</em></div><div><em>[<span>2022-03-24 10= </span>:59:49.144557] W [MSGID: 114032] [client-handshake.c:1546:client_dum= p_version_cbk] 0-hdd-client-7: received RPC status error [Transport endpoin= t is not connected]</em></div><div><em>[<span>2022-03-24 10</span>:59:49.14= 4653] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so= .0(_gf_log_callingfn+0x13b)[0x7f<span>6724643</span>adb] (--> /lib64/lib= gfrpc.so.0(+0xd7e4)[0x7f67243ea7e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0= x7f67243ea8fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9= 7)[0x7f67243eb987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f67243ec518] ))= ))) 0-hdd-client-7: forced unwinding frame type(GF-DUMP) op(NULL(2)) called= at <span>2022-03-24 10</span>:59:38.145218 (xid=3D0x862)</em></div><div><e= m>[<span>2022-03-24 10</span>:59:49.144665] W [rpc-clnt-ping.c:210:rpc_clnt= _ping_cbk] 0-hdd-client-7: socket disconnected</em></div><div>=C2=A0</div><= div>####################################</div><div># netcat/telnet to the b= rick's port of storage2 are working</div><div>=C2=A0</div><div><em>[<a = href=3D"mailto:root@storage1"; rel=3D"noopener noreferrer" target=3D"_blank"= >root@storage1</a> ~]#=C2=A0 netcat -z -v 172.22.102.<span>142 49154</span>= </em></div><div><em>Connection to 172.22.102.<span>142 49154</span> port [t= cp/*] succeeded!</em></div><div>=C2=A0</div><div><em>[<a href=3D"mailto:roo= t@storage3" rel=3D"noopener noreferrer" target=3D"_blank">root@storage3</a>= ~]# netcat -z -v 172.22.102.<span>142 49154</span></em></div><div><em>Conn= ection to 172.22.102.<span>142 49154</span> port [tcp/*] succeeded!</em></d= iv><div>=C2=A0</div><div><em>[<a href=3D"mailto:root@ovirthost1"; rel=3D"noo= pener noreferrer" target=3D"_blank">root@ovirthost1</a> /var/log/glusterfs]= #=C2=A0 netcat -z -v 172.22.102.<span>142 49154</span></em></div><div><em>C= onnection to 172.22.102.<span>142 49154</span> port [tcp/*] succeeded!</em>= </div><div>=C2=A0</div><div>####################################</div><div>= # gluster peer status - all gluster peers are connected</div><div><em>[<a h= ref=3D"mailto:root@storage3"; rel=3D"noopener noreferrer" target=3D"_blank">= root@storage3</a> ~]#=C2=A0 gluster peer status</em></div><div><em>Number o= f Peers: 2</em></div><div>=C2=A0</div><div><em>Hostname: storage1</em></div= ><div><em>Uuid: 055e79c2-b1ff-4a<span>82-9296-205</span>d<span>6877904</spa= n>e</em></div><div><em>State: Peer in Cluster (Connected)</em></div><div>= =C2=A0</div><div><em>Hostname: storage2</em></div><div><em>Uuid: d7adcb92-2= e71-41a9-80d4-13180ee673cf</em></div><div><em>State: Peer in Cluster (Conne= cted)</em></div><div>=C2=A0</div><div>####################################<= /div><div># Configuration of the volume</div><div><em>Volume Name: hdd</em>= </div><div><em>Type: Distributed-Replicate</em></div><div><em>Volume ID: 1b= 47c2f8-5024-4b85-aa7f-a3f767bb076c</em></div><div><em>Status: Started</em><= /div><div><em>Snapshot Count: 0</em></div><div><em>Number of Bricks: 4 x 3 = =3D 12</em></div><div><em>Transport-type: tcp</em></div><div><em>Bricks:</e= m></div><div><em>Brick1: storage1:/data/glusterfs/hdd/brick1/brick</em></di= v><div><em>Brick2: storage2:/data/glusterfs/hdd/brick1/brick</em></div><div= ><em>Brick3: storage3:/data/glusterfs/hdd/brick1/brick</em></div><div><em>B= rick4: storage1:/data/glusterfs/hdd/brick2/brick</em></div><div><em>Brick5:= storage2:/data/glusterfs/hdd/brick2/brick</em></div><div><em>Brick6: stora= ge3:/data/glusterfs/hdd/brick2/brick</em></div><div><em>Brick7: storage1:/d= ata/glusterfs/hdd/brick3/brick</em></div><div><em>Brick8: storage2:/data/gl= usterfs/hdd/brick3/brick</em></div><div><em>Brick9: storage3:/data/glusterf= s/hdd/brick3/brick</em></div><div><em>Brick10: storage1:/data/glusterfs/hdd= /brick4/brick</em></div><div><em>Brick11: storage2:/data/glusterfs/hdd/bric= k4/brick</em></div><div><em>Brick12: storage3:/data/glusterfs/hdd/brick4/br= ick</em></div><div><em>Options Reconfigured:</em></div><div><em>storage.own= er-gid: 36</em></div><div><em>storage.owner-uid: 36</em></div><div><em>serv= er.event-threads: 4</em></div><div><em>client.event-threads: 4</em></div><d= iv><em>cluster.choose-local: off</em></div><div><em>user.cifs: off</em></di= v><div><em>features.shard: on</em></div><div><em>cluster.shd-wait-qlength: = 10000</em></div><div><em>cluster.shd-max-threads: 8</em></div><div><em>clus= ter.locking-scheme: granular</em></div><div><em>cluster.data-self-heal-algo= rithm: full</em></div><div><em>cluster.server-quorum-type: server</em></div= ><div><em>cluster.eager-lock: enable</em></div><div><em>network.remote-dio:= enable</em></div><div><em>performance.low-prio-threads: 32</em></div><div>= <em>performance.io-cache: off</em></div><div><em>performance.read-ahead: of= f</em></div><div><em>performance.quick-read: off</em></div><div><em>auth.al= low: *</em></div><div><em>network.ping-timeout: 10</em></div><div><em>clust= er.quorum-type: auto</em></div><div><em>transport.address-family: inet</em>= </div><div><em>nfs.disable: on</em></div><div><em>performance.client-io-thr= eads: on</em></div><div>=C2=A0</div><div>##################################= ##</div><div># gluster volume status. The brick running on port 49154 is su= pposedly online</div><div>=C2=A0</div><div><em>Status of volume: hdd</em></= div><div><em>Gluster process=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 TCP Port=C2=A0 RDMA Port= =C2=A0 Online=C2=A0 Pid</em></div><div><em>--------------------------------= ----------------------------------------------</em></div><div><em>Brick sto= rage1:/data/gluste</em></div><div><em>rfs/hdd/brick1/brick=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49158=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 9142</em></div><div><em>Brick storage2:/data/glust= e</em></div><div><em>rfs/hdd/brick1/brick=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49152=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 115896</em></div><div><em>Brick storage3:/data/gluste</em></div><div= ><em>rfs/hdd/brick1/brick=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 49158=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 131775</e= m></div><div><em>Brick storage1:/data/gluste</em></div><div><em>rfs/hdd/bri= ck2/brick=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49= 159=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 9151</em></div><div><em>Bri= ck storage2:/data/gluste</em></div><div><em>rfs/hdd/brick2/brick=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49153=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 115904</em></div><div><em>Brick storage3:/da= ta/gluste</em></div><div><em>rfs/hdd/brick2/brick=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49159=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 131783</em></div><div><em>Brick storage1:/data/gluste</em><= /div><div><em>rfs/hdd/brick3/brick=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 49160=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 9163</em></div><div><em>Brick storage2:/data/gluste</em></div><div><em>rfs/= hdd/brick3/brick=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 49154=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 115913</em></div><= div><em>Brick storage3:/data/gluste</em></div><div><em>rfs/hdd/brick3/brick= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49160=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 131792</em></div><div><em>Brick stora= ge1:/data/gluste</em></div><div><em>rfs/hdd/brick4/brick=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49161=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 9170</em></div><div><em>Brick storage2:/data/gluste</= em></div><div><em>rfs/hdd/brick4/brick=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 49155=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 115923</em></div><div><em>Brick storage3:/data/gluste</em></div><div><e= m>rfs/hdd/brick4/brick=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 49161=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 131800</em><= /div><div><em>Self-heal Daemon on localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 170468</em></div><div><em>Self-heal Daemon o= n storage3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 132= 263</em></div><div><em>Self-heal Daemon on storage1=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 9512</em></div><div>=C2=A0</div><div><= em>Task Status of Volume hdd</em></div><div><em>---------------------------= ---------------------------------------------------</em></div><div><em>Ther= e are no active volume tasks</em></div><div>=C2=A0</div><div>##############= ######################</div><div># gluster volume heal hdd info split-brain= . All bricks are connected and showing no entries (0), except for brick3 on= storage2</div><div><em>Brick storage2:/data/glusterfs/hdd/brick3/brick</em= ></div><div><em>Status: Transport endpoint is not connected</em></div><div>= <em>Number of entries in split-brain: -</em></div><div>=C2=A0</div><div>###= #################################</div><div># gluster volume heal hdd info.= Only brick3 seems to be affected and it has lots of entries. brick3 on sto= rage2 is not connected</div><div>=C2=A0</div><div><em>Brick storage1:/data/= glusterfs/hdd/brick3/brick</em></div><div><em>/538befbf-ffa7-4a8c-8827-cee6= 79d589f4/images/615fa<span>020-9737-4</span>b83-a3c1-a61e32400d59/f<span>49= 17758</span>-deae-4a62-bf4d-5b9a95a7db5b</em></div><div><em><gfid:f3d0b1= 9a-2544-48c5-90b7-addd561113bc></em></div><div><em>/.shard/753a8a81-bd06= -4c8c-9515-d54123f6fe4d.1</em></div><div><em>/.shard/c7f5f88f-dc<span>85-46= 45-9178</span>-c7df8e46a99d.83</em></div><div><em>/538befbf-ffa7-4a8c-8827-= cee679d589f4/images/bc4362e6-cd43-4ab8-b8fa-0ea72405b7da/ea9c0e7c-d2c7-43c8= -b19f-7a3076cc6743</em></div><div><em>/.shard/dc46e963-2b<span>68-4802-9537= -42</span>f25ea97ae2.10872</em></div><div><em>/.shard/dc46e963-2b<span>68-4= 802-9537-42</span>f25ea97ae2.1901</em></div><div><em>/538befbf-ffa7-4a8c-88= 27-cee679d589f4/images/e48e80fb-d42f-47a4-9a56-07fd7ad868b3/31fd839f-85bf-4= c42-ac0e-7055d903df40</em></div><div><em>/.shard/82700f9b-c7e0-4568-a565-64= c9a770449f.223</em></div><div><em>/.shard/82700f9b-c7e0-4568-a565-64c9a7704= 49f.243</em></div><div><em>/.shard/dc46e963-2b<span>68-4802-9537-42</span>f= 25ea97ae2.10696</em></div><div><em>/.shard/dc46e963-2b<span>68-4802-9537-42= </span>f25ea97ae2.10902</em></div><div><em>..</em></div><div><em>Status: Co= nnected</em></div><div><em>Number of entries: 664</em></div><div>=C2=A0</di= v><div><em>Brick storage2:/data/glusterfs/hdd/brick3/brick</em></div><div><= em>Status: Transport endpoint is not connected</em></div><div><em>Number of= entries: -</em></div><div>=C2=A0</div><div><em>Brick storage3:/data/gluste= rfs/hdd/brick3/brick</em></div><div><em>/538befbf-ffa7-4a8c-8827-cee679d589= f4/images/615fa<span>020-9737-4</span>b83-a3c1-a61e32400d59/f<span>4917758<= /span>-deae-4a62-bf4d-5b9a95a7db5b</em></div><div><em><gfid:f3d0b19a-254= 4-48c5-90b7-addd561113bc></em></div><div><em>/.shard/753a8a81-bd06-4c8c-= 9515-d54123f6fe4d.1</em></div><div><em>..</em></div><div><em>Status: Connec= ted</em></div><div><em>Number of entries: 664</em></div><div>=C2=A0</div><d= iv>####################################</div><div># /data/glusterfs/hdd/bri= ck3 on storage2 is running inside of a software RAID</div><div>=C2=A0</div>= <div><em>md6 : active raid6 sdac1[6] sdz1[3] sdx1[1] sdad1[7] sdaa1[4] sdy1= [2] sdw1[0] sdab1[5] sdae1[8]</em></div><div><em>=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 <span>68364119040</span> blocks super 1.2 level 6, 512k chunk, algor= ithm 2 [9/9] [UUUUUUUUU]</em></div><div><em>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = [=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>........]=C2=A0 check =3D 64.4% (6= 290736128/<span>9766302720</span>) finish=3D3220.5min speed=3D17985K/sec</e= m></div><div><em>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bitmap: 10/73 pages [40KB],= 65536KB chunk</em></div><div>=C2=A0</div><div>############################= ########</div><div># glfsheal-hdd.log on storage2</div><div>=C2=A0</div><di= v><em>[<span>2022-03-24 10</span>:15:33.238884] I [MSGID: 114046] [client-h= andshake.c:1106:client_setvolume_cbk] 0-hdd-client-10: Connected to hdd-cli= ent-10, attached to remote volume '/data/glusterfs/hdd/brick4/brick'= ;.</em></div><div><em>[<span>2022-03-24 10</span>:15:33.238931] I [MSGID: 1= 08002] [afr-common.c:5607:afr_notify] 0-hdd-replicate-3: Client-quorum is m= et</em></div><div><em>[<span>2022-03-24 10</span>:15:33.241616] I [MSGID: 1= 14046] [client-handshake.c:1106:client_setvolume_cbk] 0-hdd-client-11: Conn= ected to hdd-client-11, attached to remote volume '/data/glusterfs/hdd/= brick4/brick'.</em></div><div><em>[<span>2022-03-24 10</span>:15:44.078= 651] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-hdd-client-7: se= rver <a href=3D"http://172.22.102.142:49154/"; target=3D"_blank">172.22.102.= 142:49154</a> has not responded in the last 10 seconds, disconnecting.</em>= </div><div><em>[<span>2022-03-24 10</span>:15:44.078891] I [MSGID: 114018] = [client.c:2331:client_rpc_notify] 0-hdd-client-7: disconnected from hdd-cli= ent-7. Client process will keep trying to connect to glusterd until brick&#= 39;s port is available</em></div><div><em>[<span>2022-03-24 10</span>:15:44= .079954] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs= .so.0(_gf_log_callingfn+0x13b)[0x7fc6c0cadadb] (--> /lib64/libgfrpc.so.0= (+0xd7e4)[0x7fc6c019f7e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7fc6c019f= 8fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7fc6c= 01a0987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7fc6c01a1518] ))))) 0-hdd-= client-7: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at <span>= 2022-03-24 10</span>:15:33.209640 (xid=3D0x5)</em></div><div><em>[<span>202= 2-03-24 10</span>:15:44.080008] W [MSGID: 114032] [client-handshake.c:1547:= client_dump_version_cbk] 0-hdd-client-7: received RPC status error [Transpo= rt endpoint is not connected]</em></div><div><em>[<span>2022-03-24 10</span= >:15:44.080526] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libgl= usterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc6c0cadadb] (--> /lib64/libgfr= pc.so.0(+0xd7e4)[0x7fc6c019f7e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f= c6c019f8fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[= 0x7fc6c01a0987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7fc6c01a1518] )))))= 0-hdd-client-7: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at= <span>2022-03-24 10</span>:15:33.209655 (xid=3D0x6)</em></div><div><em>[<s= pan>2022-03-24 10</span>:15:44.080574] W [rpc-clnt-ping.c:210:rpc_clnt_ping= _cbk] 0-hdd-client-7: socket disconnected</em></div><div>=C2=A0</div><div>#= ###################################</div><div># stack trace on storage2 tha= t happened this morning</div><div>=C2=A0</div><div><em>Mar 24 06:24:06 stor= age2 kernel: INFO: task glfs_iotwr000:115974 blocked for more than 120 seco= nds.</em></div><div><em>Mar 24 06:24:06 storage2 kernel: "echo 0 > = /proc/sys/kernel/hung_task_timeout_secs" disables this message.</em></= div><div><em>Mar 24 06:24:06 storage2 kernel: glfs_iotwr000=C2=A0=C2=A0 D f= fff9b91b<span>8951070</span>=C2=A0=C2=A0=C2=A0=C2=A0 <span>0 115974</span>= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1 0x<span>00000080</span></em></div><div><em= >Mar 24 06:24:06 storage2 kernel: Call Trace:</em></div><div><em>Mar 24 06:= 24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70</em></= div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>] _x= fs_log_force_lsn+0x2d1/0x310 [xfs]</em></div><div><em>Mar 24 06:24:06 stora= ge2 kernel: [<ffffffffa14db4d0>] ? wake_up_state+0x20/0x20</em></div>= <div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] xfs_fi= le_fsync+0xfd/0x1c0 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 kerne= l: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0</em></div><div><em>Mar 24 = 06:24:06 storage2 kernel: [<ffffffffa167ff03>] SyS_fdatasync+0x13/0x2= 0</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede= >] system_call_fastpath+0x25/0x2a</em></div><div><em>Mar 24 06:24:06 sto= rage2 kernel: INFO: task glfs_iotwr001:121353 blocked for more than 120 sec= onds.</em></div><div><em>Mar 24 06:24:06 storage2 kernel: "echo 0 >= /proc/sys/kernel/hung_task_timeout_secs" disables this message.</em><= /div><div><em>Mar 24 06:24:06 storage2 kernel: glfs_iotwr001=C2=A0=C2=A0 D = ffff9b9b7d4dac80=C2=A0=C2=A0=C2=A0=C2=A0 <span>0 121353</span>=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 1 0x<span>00000080</span></em></div><div><em>Mar 24 06:2= 4:06 storage2 kernel: Call Trace:</em></div><div><em>Mar 24 06:24:06 storag= e2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70</em></div><div><em= >Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>] _xfs_log_force= _lsn+0x2d1/0x310 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 kernel: = [<ffffffffa14db4d0>] ? wake_up_state+0x20/0x20</em></div><div><em>Mar= 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] xfs_file_fsync+0xf= d/0x1c0 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffff= ffffa167fbf7>] do_fsync+0x67/0xb0</em></div><div><em>Mar 24 06:24:06 sto= rage2 kernel: [<ffffffffa167ff03>] SyS_fdatasync+0x13/0x20</em></div>= <div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] system= _call_fastpath+0x25/0x2a</em></div><div><em>Mar 24 06:24:06 storage2 kernel= : INFO: task glfs_iotwr002:121354 blocked for more than 120 seconds.</em></= div><div><em>Mar 24 06:24:06 storage2 kernel: "echo 0 > /proc/sys/k= ernel/hung_task_timeout_secs" disables this message.</em></div><div><e= m>Mar 24 06:24:06 storage2 kernel: glfs_iotwr002=C2=A0=C2=A0 D ffff9b9b7d75= ac80=C2=A0=C2=A0=C2=A0=C2=A0 <span>0 121354</span>=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 1 0x<span>00000080</span></em></div><div><em>Mar 24 06:24:06 storage= 2 kernel: Call Trace:</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [= <ffffffffa1b80a29>] schedule+0x29/0x70</em></div><div><em>Mar 24 06:2= 4:06 storage2 kernel: [<ffffffffc05056e1>] _xfs_log_force_lsn+0x2d1/0= x310 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<fffffff= fa14db4d0>] ? wake_up_state+0x20/0x20</em></div><div><em>Mar 24 06:24:06= storage2 kernel: [<ffffffffc04e5a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs= ]</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7= >] do_fsync+0x67/0xb0</em></div><div><em>Mar 24 06:24:06 storage2 kernel= : [<ffffffffa167ff03>] SyS_fdatasync+0x13/0x20</em></div><div><em>Mar= 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] system_call_fastpa= th+0x25/0x2a</em></div><div><em>Mar 24 06:24:06 storage2 kernel: INFO: task= glfs_iotwr003:121355 blocked for more than 120 seconds.</em></div><div><em= >Mar 24 06:24:06 storage2 kernel: "echo 0 > /proc/sys/kernel/hung_t= ask_timeout_secs" disables this message.</em></div><div><em>Mar 24 06:= 24:06 storage2 kernel: glfs_iotwr003=C2=A0=C2=A0 D ffff9b9b7d51ac80=C2=A0= =C2=A0=C2=A0=C2=A0 <span>0 121355</span>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1 0x= <span>00000080</span></em></div><div><em>Mar 24 06:24:06 storage2 kernel: C= all Trace:</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<fffffff= fa1b80a29>] schedule+0x29/0x70</em></div><div><em>Mar 24 06:24:06 storag= e2 kernel: [<ffffffffa1b7e531>] schedule_timeout+0x221/0x2d0</em></di= v><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ? tt= wu_do_wakeup+0x19/0xe0</em></div><div><em>Mar 24 06:24:06 storage2 kernel: = [<ffffffffa14d78df>] ? ttwu_do_activate+0x6f/0x80</em></div><div><em>= Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ? try_to_wake_u= p+0x190/0x390</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffff= ffffa1b80ddd>] wait_for_completion+0xfd/0x140</em></div><div><em>Mar 24 = 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? wake_up_state+0x20/0= x20</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9= aa>] flush_work+0x10a/0x1b0</em></div><div><em>Mar 24 06:24:06 storage2 = kernel: [<ffffffffa14bb6c0>] ? move_linked_works+0x90/0x90</em></div>= <div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>] xlog_c= il_force_lsn+0x8a/0x210 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 k= ernel: [<ffffffffc<span>0505484</span>>] _xfs_log_force_lsn+0x74/0x31= 0 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1= 5bcb1f>] ? filemap_fdatawait_range+0x1f/0x30</em></div><div><em>Mar 24 0= 6:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ? down_read+0x12/0x40</= em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>= ;] xfs_file_fsync+0xfd/0x1c0 [xfs]</em></div><div><em>Mar 24 06:24:06 stora= ge2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0</em></div><div><e= m>Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] SyS_fdatasync= +0x13/0x20</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<fffffff= fa1b8dede>] system_call_fastpath+0x25/0x2a</em></div><div><em>Mar 24 06:= 24:06 storage2 kernel: INFO: task glfs_iotwr004:121356 blocked for more tha= n 120 seconds.</em></div><div><em>Mar 24 06:24:06 storage2 kernel: "ec= ho 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this messa= ge.</em></div><div><em>Mar 24 06:24:06 storage2 kernel: glfs_iotwr004=C2=A0= =C2=A0 D ffff9b9b7d75ac80=C2=A0=C2=A0=C2=A0=C2=A0 <span>0 121356</span>=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 1 0x<span>00000080</span></em></div><div><em>Ma= r 24 06:24:06 storage2 kernel: Call Trace:</em></div><div><em>Mar 24 06:24:= 06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70</em></div= ><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>] sched= ule_timeout+0x221/0x2d0</em></div><div><em>Mar 24 06:24:06 storage2 kernel:= [<ffffffffa14d77a9>] ? ttwu_do_wakeup+0x19/0xe0</em></div><div><em>M= ar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ? ttwu_do_activa= te+0x6f/0x80</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<fffff= fffa14db210>] ? try_to_wake_up+0x190/0x390</em></div><div><em>Mar 24 06:= 24:06 storage2 kernel: [<ffffffffa1b80ddd>] wait_for_completion+0xfd/= 0x140</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d= b4d0>] ? wake_up_state+0x20/0x20</em></div><div><em>Mar 24 06:24:06 stor= age2 kernel: [<ffffffffa14be9aa>] flush_work+0x10a/0x1b0</em></div><d= iv><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ? move_l= inked_works+0x90/0x90</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [= <ffffffffc05070ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]</em></div><di= v><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc<span>0505484</span>&g= t;] _xfs_log_force_lsn+0x74/0x310 [xfs]</em></div><div><em>Mar 24 06:24:06 = storage2 kernel: [<ffffffffa15bcb1f>] ? filemap_fdatawait_range+0x1f/= 0x30</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7f= d22>] ? down_read+0x12/0x40</em></div><div><em>Mar 24 06:24:06 storage2 = kernel: [<ffffffffc04e5a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]</em></di= v><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_f= sync+0x67/0xb0</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<fff= fffffa167ff03>] SyS_fdatasync+0x13/0x20</em></div><div><em>Mar 24 06:24:= 06 storage2 kernel: [<ffffffffa1b8dede>] system_call_fastpath+0x25/0x= 2a</em></div><div><em>Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotw= r005:153774 blocked for more than 120 seconds.</em></div><div><em>Mar 24 06= :24:06 storage2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeou= t_secs" disables this message.</em></div><div><em>Mar 24 06:24:06 stor= age2 kernel: glfs_iotwr005=C2=A0=C2=A0 D ffff9b9b7d61ac80=C2=A0=C2=A0=C2=A0= =C2=A0 <span>0 153774</span>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1 0x<span>000000= 80</span></em></div><div><em>Mar 24 06:24:06 storage2 kernel: Call Trace:</= em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>= ;] schedule+0x29/0x70</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [= <ffffffffa1b7e531>] schedule_timeout+0x221/0x2d0</em></div><div><em>M= ar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ? ttwu_do_wakeup= +0x19/0xe0</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<fffffff= fa14d78df>] ? ttwu_do_activate+0x6f/0x80</em></div><div><em>Mar 24 06:24= :06 storage2 kernel: [<ffffffffa14db210>] ? try_to_wake_up+0x190/0x39= 0</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd= >] wait_for_completion+0xfd/0x140</em></div><div><em>Mar 24 06:24:06 sto= rage2 kernel: [<ffffffffa14db4d0>] ? wake_up_state+0x20/0x20</em></di= v><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>] flus= h_work+0x10a/0x1b0</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<= ;ffffffffa14bb6c0>] ? move_linked_works+0x90/0x90</em></div><div><em>Mar= 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>] xlog_cil_force_lsn= +0x8a/0x210 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<= ffffffffa167335b>] ? getxattr+0x11b/0x180</em></div><div><em>Mar 24 06:2= 4:06 storage2 kernel: [<ffffffffc<span>0505484</span>>] _xfs_log_forc= e_lsn+0x74/0x310 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 kernel: = [<ffffffffa1b7fd22>] ? down_read+0x12/0x40</em></div><div><em>Mar 24 = 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] xfs_file_fsync+0xfd/0x= 1c0 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffff= a167fbf7>] do_fsync+0x67/0xb0</em></div><div><em>Mar 24 06:24:06 storage= 2 kernel: [<ffffffffa167ff03>] SyS_fdatasync+0x13/0x20</em></div><div= ><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>] system_cal= l_fastpath+0x25/0x2a</em></div><div><em>Mar 24 06:24:06 storage2 kernel: IN= FO: task glfs_iotwr006:153775 blocked for more than 120 seconds.</em></div>= <div><em>Mar 24 06:24:06 storage2 kernel: "echo 0 > /proc/sys/kerne= l/hung_task_timeout_secs" disables this message.</em></div><div><em>Ma= r 24 06:24:06 storage2 kernel: glfs_iotwr006=C2=A0=C2=A0 D ffff9b9b7d49ac80= =C2=A0=C2=A0=C2=A0=C2=A0 <span>0 153775</span>=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 1 0x<span>00000080</span></em></div><div><em>Mar 24 06:24:06 storage2 k= ernel: Call Trace:</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<= ;ffffffffa1b80a29>] schedule+0x29/0x70</em></div><div><em>Mar 24 06:24:0= 6 storage2 kernel: [<ffffffffa1b7e531>] schedule_timeout+0x221/0x2d0<= /em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9&g= t;] ? ttwu_do_wakeup+0x19/0xe0</em></div><div><em>Mar 24 06:24:06 storage2 = kernel: [<ffffffffa14d78df>] ? ttwu_do_activate+0x6f/0x80</em></div><= div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ? try_t= o_wake_up+0x190/0x390</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [= <ffffffffa1b80ddd>] wait_for_completion+0xfd/0x140</em></div><div><em= >Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? wake_up_stat= e+0x20/0x20</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffff= ffa14be9aa>] flush_work+0x10a/0x1b0</em></div><div><em>Mar 24 06:24:06 s= torage2 kernel: [<ffffffffa14bb6c0>] ? move_linked_works+0x90/0x90</e= m></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>= ] xlog_cil_force_lsn+0x8a/0x210 [xfs]</em></div><div><em>Mar 24 06:24:06 st= orage2 kernel: [<ffffffffa167335b>] ? getxattr+0x11b/0x180</em></div>= <div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc<span>0505484</span= >>] _xfs_log_force_lsn+0x74/0x310 [xfs]</em></div><div><em>Mar 24 06:24:= 06 storage2 kernel: [<ffffffffa1b7fd22>] ? down_read+0x12/0x40</em></= div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] xf= s_file_fsync+0xfd/0x1c0 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 k= ernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0</em></div><div><em>Mar= 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] SyS_fdatasync+0x13= /0x20</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8= dede>] system_call_fastpath+0x25/0x2a</em></div><div><em>Mar 24 06:24:06= storage2 kernel: INFO: task glfs_iotwr007:153776 blocked for more than 120= seconds.</em></div><div><em>Mar 24 06:24:06 storage2 kernel: "echo 0 = > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</= em></div><div><em>Mar 24 06:24:06 storage2 kernel: glfs_iotwr007=C2=A0=C2= =A0 D ffff9b9958c962a0=C2=A0=C2=A0=C2=A0=C2=A0 <span>0 153776</span>=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 1 0x<span>00000080</span></em></div><div><em>Mar 2= 4 06:24:06 storage2 kernel: Call Trace:</em></div><div><em>Mar 24 06:24:06 = storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70</em></div><d= iv><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>] schedule= _timeout+0x221/0x2d0</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [&= lt;ffffffffa14d7782>] ? check_preempt_curr+0x92/0xa0</em></div><div><em>= Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ? ttwu_do_wakeu= p+0x19/0xe0</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffff= ffa14db210>] ? try_to_wake_up+0x190/0x390</em></div><div><em>Mar 24 06:2= 4:06 storage2 kernel: [<ffffffffa1b80ddd>] wait_for_completion+0xfd/0= x140</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db= 4d0>] ? wake_up_state+0x20/0x20</em></div><div><em>Mar 24 06:24:06 stora= ge2 kernel: [<ffffffffa14be9aa>] flush_work+0x10a/0x1b0</em></div><di= v><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ? move_li= nked_works+0x90/0x90</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [&= lt;ffffffffc05070ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]</em></div><div= ><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa167335b>] ? getxattr= +0x11b/0x180</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<fffff= fffc<span>0505484</span>>] _xfs_log_force_lsn+0x74/0x310 [xfs]</em></div= ><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ? dow= n_read+0x12/0x40</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<f= fffffffc04e5a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]</em></div><div><em>Mar= 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0= </em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03&= gt;] SyS_fdatasync+0x13/0x20</em></div><div><em>Mar 24 06:24:06 storage2 ke= rnel: [<ffffffffa1b8dede>] system_call_fastpath+0x25/0x2a</em></div><= div><em>Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr008:153777 bl= ocked for more than 120 seconds.</em></div><div><em>Mar 24 06:24:06 storage= 2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" d= isables this message.</em></div><div><em>Mar 24 06:24:06 storage2 kernel: g= lfs_iotwr008=C2=A0=C2=A0 D ffff9b9b7d61ac80=C2=A0=C2=A0=C2=A0=C2=A0 <span>0= 153777</span>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1 0x<span>00000080</span></em>= </div><div><em>Mar 24 06:24:06 storage2 kernel: Call Trace:</em></div><div>= <em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x= 29/0x70</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc0= 5056e1>] _xfs_log_force_lsn+0x2d1/0x310 [xfs]</em></div><div><em>Mar 24 = 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ? wake_up_state+0x20/0= x20</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a= 3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]</em></div><div><em>Mar 24 06:24:06 = storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0</em></div><d= iv><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>] SyS_fdat= async+0x13/0x20</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ff= ffffffa1b8dede>] system_call_fastpath+0x25/0x2a</em></div><div><em>Mar 2= 4 06:24:06 storage2 kernel: INFO: task glfs_iotwr009:153778 blocked for mor= e than 120 seconds.</em></div><div><em>Mar 24 06:24:06 storage2 kernel: &qu= ot;echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this = message.</em></div><div><em>Mar 24 06:24:06 storage2 kernel: glfs_iotwr009= =C2=A0=C2=A0 D ffff9b9958c920e0=C2=A0=C2=A0=C2=A0=C2=A0 <span>0 153778</spa= n>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1 0x<span>00000080</span></em></div><div><= em>Mar 24 06:24:06 storage2 kernel: Call Trace:</em></div><div><em>Mar 24 0= 6:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70</em>= </div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>] = _xfs_log_force_lsn+0x2d1/0x310 [xfs]</em></div><div><em>Mar 24 06:24:06 sto= rage2 kernel: [<ffffffffa14db4d0>] ? wake_up_state+0x20/0x20</em></di= v><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>] xfs_= file_fsync+0xfd/0x1c0 [xfs]</em></div><div><em>Mar 24 06:24:06 storage2 ker= nel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0</em></div><div><em>Mar 2= 4 06:24:06 storage2 kernel: [<ffffffffa167ff03>] SyS_fdatasync+0x13/0= x20</em></div><div><em>Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8de= de>] system_call_fastpath+0x25/0x2a</em></div></div></div></div></div>__= ______<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href=3D"https://meet.google.com/cpu-eiue-hvk"; rel=3D"noreferrer"= target=3D"_blank">https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href=3D"mailto:Gluster-users@xxxxxxxxxxx"; target=3D"_blank">Gluster-user= s@xxxxxxxxxxx</a><br> <a href=3D"https://lists.gluster.org/mailman/listinfo/gluster-users"; rel=3D= "noreferrer" target=3D"_blank">https://lists.gluster.org/mailman/listinfo/g= luster-users</a><br> </blockquote></div></blockquote></blockquote></div> --0000000000006d508e05db44ff40-- --0000000000006d508e05db44ff41 Content-Type: image/png; name="image.png" Content-Disposition: inline; filename="image.png" Content-Transfer-Encoding: base64 Content-ID: <ii_l1akxwfw0> X-Attachment-Id: ii_l1akxwfw0 iVBORw0KGgoAAAANSUhEUgAAAX8AAAEpCAYAAAB7kyKNAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAFhFSURBVHhe7Z0HXBRH+8d/UqIoIiogICoidsVY sCuxtxg06muNKdYk/o0mr8aWRFM00SQmvInGlhh7jIliorEQu1hQVEQFREBBQEBFpNf/zN0uLMve cQccIvd8P5/HnZl9dndmduY3s7N7WCWPAYIgCMKoMBG2BEEQhBFB4k8QBGGEkPgTBEEYIST+BEEQ RgiJP0EQhBFC4k8QBGGEkPgTBEEYIST+BEEQRgiJP0EQhBFC4k8QBGGEkPgTBEEYIST+BEEQRgiJ P0EQhBFC4k8QBGGEkPgTBEEYIST+BEEQRgiJP0EQhBFC4k8QBGGEkPgTBEEYIST+BEEQRgiJP0EQ hBFC4k8QBGGEkPgTBEEYIST+BEEQRkiV3NzcPCFcplSpUkUIKZOXZ5DLEpUYbW2K2hOhRHE6ZMxU OXr0aJn2GgsLC9SpUwctWrRQxeWVL3bSoKAg3L9/XxUmCG1oa1PUngglitMhgtVJZmZmmU+ZTp48 ib59++ZXuLgVOyrfHjt2DB4eHqo4QRSHUpui9kRoQ5MOEWpMeMcpa+Pk5ubmh6XwNL5PDJOR6WIc pTbF49SeyJSMo0mHCAO+8JVWuvRm8K3YWQlCH+RtStxSeyI0odRmCDUGF39phYtx6qxESZC2KalR eyI0oaRDhBqDLftIw1KUfMjIijNpe5GitJ+MjJu0TRBFMdjMX1r5IkppBKEr8vZD7YkoDmojmjHo j7yUKpxuAlEa5O2H2hNRHNRGlDHYso8uKB1LRqZkuqB0HJnxGqEd+vMOBEEQRgiJP0EQhAYe3wpD a68A1BNtbQh+eijsfM6hZR+y58J0Qek4MuO1suDAtSe4mZaDONGSUrDK/4mw9/mGZv5lTPzZbVi1 +n/YdFYyPUg4j00sbdXm84gXkgiC0NBfiHLh+Zj5n/oWYye9hbFzdyFcaX8FsuSIAFy6fAVXI5IL 0pMjcJWlXboWgWSJb+ktBWe+m40Jk6bjs2MpCvsrj+mC0nHKdhYreHuaNB9bIkqy/1laBu6e2YUv 5v8fJr/O88js9ZmYMv8bbDnD2leW0jEV1xT7SxkZoR2a+RPE80J2DA6umIeFPx1BQHQKMnKE9JxM JEffwIFdpxBpJqQRZYKLlbxCq6BhLQsh/HxD4i/hyfGv8fprU/DlaSGhgqA5X9XR473vsX3rOizp U11II54X9Gtv2bixeRW23ExGDizgMuRdfLtuHXZt3YRt677GpzMHYuiYgVD/AWOirOg+tCXipjdD hGgzW+Fw5xeEvc83VZKTk8v8+ejcuXPo3r07zMzMYGpqChMT9RjD/85GTk4OsrOz4evri27duqnS i+XMdxi/PhCwHYgV3/wHzqpEX3w5+WcEtp+Gza9bYNeaLfC5k8hmQ2awcGiOIZOmYExbK5UnJ/tR APPZhmNhj5CWDZi+UAP1mvfAiHEj0MshGf67f8Lao2FIFmdTKtrg7S1z0JsHs1MRfvIP/HrkEiLi 1LMucwsHtBw4Du+Mao1aKn8gYscCLDyUALvBH+P7CQ3ViXd3472PjiCuUP6BjDh/7N/5N07djEaC OlOo7dQBo995A30d2Iwj+1Gx+Tr19VSsDQDaTd+IBT3VezkZkSewcf3f8IvidcJPbY0GnV7B/03t DUfJZEZ1/I12mLVhIix2b8DGY2F4nJmjWLZnibxNiX/SuUTtSWg712CDYZ99iUmNhOR8NO/XtV5L 3d7kpBzHp7O241aOKRqN+hRfetYTdmjGMG0gE/f+3Yjv9wTiQUomG4hYf6vriI4DxmHcwGaoK5yX l3/vpt/xz60YVflhZgHHlv3w+vQRcJM0KMX+wtA179pQ0iH6s84FmGh6b5396BS+/XANTlXwF9s5 4QewcJEXDoSoGwnLOdJibuDPb3/EP2Les69ijcpH3RE5OZkpiL7+L44Hs4T7Pvj1kLwjykg4hnXb TiI4puBxOystBgHeXljh/UCdoBfZuP77L/jz8j218HPY4/vju+exYdkmXOJJuuRLgeyw3Vjw8Tac uSvWCT91IiJ8t2D+ot24I1wun5wI7PvoE6w6dFvV6Tli2b49nKSKF0tKEDZ//hX2hslPLpKJ4B2f YdHucFby5xOd67Us2puMlDPnmfCzQFU2OdBB+A3VBmK9V2DRr/6IVgk/h/W3h/dw5tBFJIii/MQX X7Py/3ldEH5Odhor/99Y8eF3xWqK1rwv/Rux6qRyIe7KbdisuoIqXwn29U0siaocf0jQZNGPlyG/ F1z4v160Bf55NWAtpFVYEqMRY9UL73+9Bju3bMTmL0aiWVWWnnMHJ08IXxAEXIB/KtvavoSlGzeq /X5YikVvTMHEl6oDjf6D77d8jGG2anc+i+Y+O6WzMPvB+PibL/HjTz8J+37Cpve6wIp1gbt+fkgQ 3HTHDJ2mLYfXt6uxWXU+ZhtXYLQL25UaAF82m9cpX0V4gL0bjiCWdZrqzUdjuSq/LK+fjkZLVtSc 2CP4qchg9QRRsVbo894KdV42LsXYpqpKRMjpU7qVrWpVVE0Jx+6VXyoMAEz4ty3DZ4fiUK1CrZcm 4MBHUzF+stz4rF+OHvVaFu1NRmz8Y3XAtSVeVIe0YKg28BDn/CJZSlW0eX0ltnK/n73w48fvYsq0 AWiu8snGxS3bcY2V39Re0i+/fgd97E1Z2w7Ez1sua5kAiHm3QMtxC7FOqL+tP32kyk/OvX+w219w LQeOBCfjoVTrczKw89ZTIfJ8Y5J5aT34AJAovCHPesiEf+EWBNbsiXmLJqOtVdG36MWZrigdq2zC ASicrqYmer8+GZ1szVVpLzgNxYBWrJEx0lKELwhqVIdKcpLu4HxAHNK5X836aNPHHS6mknOqjmKI 8UKWgshz+7Bq8Qd4400uEDMx5fsLUM2J0lLwVPRTnYAjPVZMLZyWlXALh37+CrNnzFSLztSF2BPG /TKQmiLxVR3LkByrcV/Uafiq/jfDJhj13iA4W5iydFNYNBqE90c0UblGnT+NSNnxVj0nYVoHG7zA 08zrw7NfC6hqUVo2bWbqjHGfzsfw2lHY/dWX+PNOlrAvA8FbmfAfSULTCYuwcGA9mMqP1cF0Qek4 TaYr+cfoU69l0t4KW1TsI7WvqWnx9WewNlAN1dm5ePuMunIN91LYPTa1QO0mL6JfGzvh+pdw0j+D +cj6pW17THu9G5sssaP9fXFe5VtwbVYJ6nhWMEJUeU/DrV0rMGOqekB+beZn+O02P28G7gTdFa6l 3QjtmKx6qw34ALB4jT8ePjqNb9iMP9CKCf/i1wutzVVcnNCkmRAUqF+vthASaPYKZvZ1QtXsSBzy WoQ3p3yAResOI+iR5vlHYZJw6pslWPrbeUQ8lHxlUQr4o+38xetxMDAWSfnPxmXAvRj1bwlsXdCy hiolnxotXGDHAwkxuKtKKaChi6sQEnC0Q10hqDNmjTGeDwB1ovD7Sr4ElIqQbZ/is6Nq4V80yJ49 71QkbDD00w3Y8avc3kQ7wSMffeq11O2tKPXr1RFCOmCwNlAdfd8czXTBDIkB27H47Vl4c/532HY2 ikmywP0oRKv6RyM0b6VKKaBVUzTm25xY3NP03y3fj0GcECQMi0mtnrMgDgBz/vu8CT/HFGbFKooV 3F7/BBt+WIK3B7eHc41URPjuwaf/nYevz+iwph2yH7sC0ljAAi1GvI9v1q5Vi8SnA9QdSW9ScWzX MdVjuWm9znj7o2/xiyA8M90El9LCX64LQV0wLb4SdUM6AHz+fgUW/hKiU72Wsr0pUMtKUPHw27ih DhWPAdqAmUN/LPD6Ht/NH41+TdnQkHADB9cvw/RFf8jeI/AX8UJQH2zqCEvNTfDaGumAXGDfjS94 MWxo2tq+oH76yccUXZ1qCuHnG9WPvKx6vIuVb7aBZb2e+O/iki31SE1XlI5VNuEAhlIajxTyF5PF R0nBTC0aoue4t/HF999hxfAGMM1Jhv/hU4gX9ucjOUZlDx4ikafb9sRrI1rAXvUYnYesuIf570vy fcV4oWvzuJCuiscj7pG6W7YZPhU9m1iqH7XzHoBdSk3+sVryxfcVnFid1tQZ9Xn8YQRCkgv7JgeF qWdVNg5oKKQVOT7fpLvk+4oxvgS0bB6G16vBhH9hiZd6pKYLSscpm3AAp8g+Dfv1rFduJW5vCla3 sxt7xmU89cO+E08UffLN4G3AHLatBuKtxZ9j/app6MzUOuv+cRy4xvY5usCZvypAFEJDpMcwu3kb 4XyXqT0aOgppPM7gIVW8en04qrQ1CsE3xaXDkllZ0K5fayS9W/CpZ9TsNtjesnJ8IZ9filo938Wa 5ZPhVvB1ZOUh5Di2H7+FyKRMdZxNSfJnRJZWwkzDDKZCbUSHhULlmZmKVD57MTNVj/5JrEHG8z3Z SI04ie+3Xy143NUL6bWC1dfIjMelXRtwuMjjsJZ8KVHXHR1VPf8O/vA6jHCVI8/vYXy79w7fAacu PdVCYkjYE8C45V/jo4GVZMavT72Wtr0pUW8AXm7LVTUDN7YsxfJ9txAjOmen4lHoVRy4fE8dN1gb eIiz+//CpdDH+fnMzRFLVgO1VAVrgy6qfD7F6c1bcEnVX3jzvoINm31V78iqtu+GzqpUJVpgQA8b ts3AxU1fYfvleHXd8Pw/CsOZXccRrIoTpeX5+PMO0vlBoXSFNJWpdvBd6nhcAA7+uhoLZs/CxDem Y+LUOVjyF/9qwQIdenUQZqUOeLGVeq0r/thKvMn9pn+OP6PYvo494M7f4GXcwpZ5/BzvYNrS7Qhy 6oaOqlmKeF1uPK5KkqQVJKrjDvDo4aQaUOKPfYNpU/m1FmP10TR07qH+FQDzyvfVmC++X7VH6m8L z7f6Q/VhRfAfWPLOO0J+/0BQGl9m6o9pw20FX6XjCyx/jyz9WZguKB2n0Yo7psh+Peq1tO0t/5pS s0DPqTPwUj02YrAniBv7VuO/qjyoz/9/n6/BjhNRgq+h2kAKwk//hdWff6hus+zab877GRfZY7Fp PXf0asx9TOH+2ni4sf6S8+AMVqv6C/dbixMPcgGL1njjtfaSJ8H8SwjxPDQaNQ1DeDnT7uLg/xar 64bn//0vsfZQAB7kH6vdyoJr/96A1Y8hcF6vNievQEy8VUk+9RS2lZu2/TCijT0sXxCLy3+Y0gB9 Z36C2V2rC2lA8wnvYHRTa7ygmuab4AVrezhYsqBZO0yfOwzNrYX1P7NqcOw4CUtnT8CLDXiC/tgP mYWZ3exhoZoW82u5YsiceZja16XIi1aN+dKAmcsYfPnZVPRrWkc4P+ucL1jDudsEfPbpGDSpFFPx 8kfnei1te9OEVWtM/Xw5PnjlRTjXrQZzIZm3x5oOTdC3i/pLHo5h2oAD+gzrUuja/Mdrjm1exeJP X83/8SKsumH+l/MxvqOkDnifaTMMH3z+DnoVt7pg5oyJn3+Cd/u4oq6YeaEOO77SD2X1WkwXrrMn l8LvTXJwPqpyfOpZ5cmTJ2UzREq4ePGiTr/w7dxZ88MfQUiRtymlX/hSeyKkKOmQvr/w3bbrCl6T fRbl0sEFdwY8N1/EaIT+nj/Zc2G6oHQcmfEaoR3jWPYhCIIoAQObW6KuVCVNq2J8y0ryqaewJQiC IGTYtW+KhHntkfehYP9thc+dKods0rIP2XNhuqB0HJnxGqEdmvkTBEEYIST+BEEQRggt+5A9F6YL SseRGa8R2qGZP0EQhBFCM3+y58J0Qek4MuM1QjtlPvPPyMhAtWrVhJhmdPEhCI4ubYraEyFFVx0y ZqocPXq0TIdICwsL1K5dG87OzjA3N1f9pFr65x24ZWVlISIiAtHR0ap0gtCGUpsS/7wDtSdCCU06 RP+BewFVYmNjy0T8eaVy439Dg/8tDV7h0r/DwuGdVfxbLLzD8i2P02MaoYS2NiVC7YmQUpwOcSPU VImPjy9z8Rcrnpt0tOUdks/UeCcVOyp1VkIT2tqUCLUnQkpxOsSNUFMlMTGxzHoJr1heyWLFc5N2 VA7vrGIn5cbj1FEJTRTXpqg9EXK0tZnSiP+dO+r/CKdJk4I/nf08UyUlJaVMxV+seNHENI44I+Md VDQxjSCU0NamqD0RSmhrMyWFC794PG9flWEAqJKRkVEmPUWsGLGSRRPTOGKnFDuoaGJaSSnNsUTF Rdp+pCamSduO1MQ0wviQtg+piWklISwsTHWsi4tLfpy3LzGuCyW9tiGpkp2dXWa9RKmS5YWWdkp9 Oip1ZuOkuDZV0vZEVF6KazP6IBd+kZIMAJooad5KSxX2qGyQnlJcgXTpoNSJCSna2hS1FUKJ0gir JuEXKcsBQKQ8B4IqLPMVqtdQJyYI4llTnPCLGGIAEDH0QFAhxJ8EnyCIioKuwi9iyAFAxBADwTMT /9JelgYMgiDKGi7k/OsgfYWcH8e/NtPluNIKeVkNBM9E/EtySRJ7giAMSUmFX0SfAUBKScS8LAaA chd/XS9X1n4EQRCaCA8PL5Xwi4gDQOPGjYWUAnQV7LL200S5ir8ul9LkU5pjCYIgNMGFn/8KuLTC L8IHAP5rcz4A6CLQmnxKc6wulJv4F3cZTfuV0sspywRBVHLKWvhFpAOAEkqiXdJBoKQDQLmIv7ZL 6CLu+h5PEARRHPzPgBtC+EXEAYD/WWmONpGW71Py1ed4XTC4+Osj3PrGRbRdgyAIQg4Xfv7XPg0l /CJ8AOB/cVbXJwB941L0HQCqXLt2zWDKqU2Udd0n9ysurgslOYYgiMpBeno66tevb3DhF+EDAP+P hvh/MCNFk7C3bdu20D5Nfkpo2yenTP+qpxRtAivdpykcFRVF/zMTQRBlDhf98hJ+ET4AcNMGHxy4 eHfu3FlIKSzmmsJydB0ADCL+pRV+Hvbz80P//v2FFIIgiMqPj48PPDw8dBL90g4Ahf+nFQNTnPDz rWgEQRDGiFwHpXqoKVwSylz8NWWouAJIt9L9BEEQxoRUA+VbjqawFE3pUsp15i9HXjC+1SXTBEEQ lRmpFsq3ZUWZir+mzEnTNRWIb+VpBEEQxoY2TZRvOdKwFE3pIs9kzV9pKzeCIAhjRJMeatqWFIOL f3EZ5PulPvI4QRCEMVESTSyJZpbbzF/MnHQrmjwuphEEQRgbSlqoFJduS4JBxV9TxuTpPC43giAI Y6Q4PZTHRTSla6LMxF+XC0t9eFibEQRBGCNKeig1EWlYE9p8ymXZR58M61IggiCIygrXQH10sqSa WW5r/lLEwimZuJ8gCMIYkWuikpUFZSL+SpmRpxWXYWmhivMlCIKorEh1sDgtlO9X8td0DoPP/KUX 1pSx4nwIgiCMCakuyjVSRJqmtL84yn3ZRyyIaCJiWJ5OEARhTEg1UKqFYrp0f2kotfjrkgltPtKC yLcEQRDGhpIeatNEXfRSyccgM39dMsNR8uNpuh5PEARR2ZBqoFQLddVFXf3KfdlHCaWCEgRBGDOG 1kWDir8009KCiKaEtn3PL+kI2DADwzy6oUuXLujZdzb2hB/FgoEeeGNblOBD6E2IF0ay+px7UIg/ Mw5iLsvHSK8QIU4cnNsFXUZ6oXLWiGHvty76KO6X+mk6RhPlOvNXypy0EPpmXmey4nB1+1JM9uyL nuymcQHu0rMvBo+djpW/BeGp4GYoso4vw+yNoWj4mhd2eu/E6mXT0M8mFSmpmcjJFpyeBQfnsrp4 CUtOC3FGcZ02xGskO2Yua/5EZUF1z8V+obJu8BjgiclL9+JOuuBEaCEEXiNZvb22GbFCSlkg1UQl bSytXj7TZR9NBSptoQrx+CDmDfPEjB+OI6XxCLyzcCEWcntjMJpVjcTNJEvUFFwNxfl/zyHN/mXM nuoOF3sXuPdqi9o1PfG/M+ew9Q0nwesZ4NoYjqgNu3pCvILgv2IQzaLLG+sumCL2jYUf4rUelog9 /CUmsQnLrSzB5znj6W/T0aVcHgubwdWZbWzsYK9OKBVyDRTDZaqLDIOLv64ZLuuCqQnBmumf4dRT J4xbdxh/fDcbE0aMwAhuU+fDa8s/2DzN8OL7NCWN1bQpTIU4oQ1/HD6RKISJcqO6K14S+wazqUu3 47c57YB7f2L7ecHnueIpDv57TQg/vxhSP5/5C1/DiL6ap3+sxtZ7QLs56zHXrZqQqo3HuLBhDsb2 7al+/O3mgWGTl2L/3cJTH9XSx5g1CInzhdeMYfDoJjwqD5uMlUfjIHo/Pb0G08f2xXJfFonejtdU j9QjoZ7UFl03VD1+85nK4wvsvEPUS1RDVyGA71Stb4/BmvAs3No1B6NU7w/YNUctEPKXhbv7Fwjp /L3CWCzYfzc/L4o41IMtnODcWIiXgqy7+7F0slgX6uvP2XCB1Whh5H58+c1z8ofYw6vh2vcs/+9i H9P+6O2vqfczK3bylh2Hoysn579T6eYxDJNXHkWcrPCPr29n1/ZE354F1x67YD8K315ZG1AtgYzF 9DWndVgeVB8r3gNN+eAUrgd+H+dgwwV5benmx9tNt3lHkX5nLxaw9qZe2uyJvmPnYHtAyddtardv w2ayaUiRFVzXvHMeX9iAOaM80I3nSdWfVuKoQoXo2n44xfrG7sP7Q4bgW679vstUPtwKPU2yPrZh ztj8ttBzyGQsVegv/FqF65T3t7qwsxYcBOxZgn3DMuhIGjCETj4z8VcqDE8ru0Jm4fSZq8i1eAnj X60tpGnjKY7OG43ZP4ei4YQV2OTtjZ1e0+GecRpfjJsAL/mzb9Q+zJq0FNdcp+Hr7d7w3vQJXra9 jz+WzMBaoY1Ve3EMPv1+B97pyCL2I7CandPbez3e1NZGsoPhNX0+/rUYol6i+mA4CtyTELXnI3x+ uQ3m/LhXdc2+5mfwxdtfw2f7bMz0rovXvt4O750/4I3mCTj+xdv4TjVyaKBmTdSAGczMhXhJCVmD CeO+wGmTlzDXaycr4yYsGWGL0J9nYzRT7vxO+fQoFr2l9lu4gdfFTvzw8QT0qGYFcwe2v/lk/Pjb /0FdXavZfu7jjY/7qo7WSMD3k7AquC3+j9cJu/YnL9vi/h9LMEO8EQIpl8/htkM/zF7N88iu/XZH pJz8Am9LKilkzXTM4W1gymrs5NfftBIzhjujulmtYpcHH3svwkdn66vvATu/1/RWeLR3CcYvOlp4 4IjajLeE+lLVw04vTHcJxc9zphduZ7r6MXL9V+GVtzYDA5bhJ952f3gbHVPYJGL2Qhws4UutrPsx 7N5ZoIa04HrkiVUIFn10FvVf+xrbeZ5Yf2r1aC+WjF+Eo9I86dp+OLr41u2Hhb+sxAi+BtPxPeaj bkfrxY6XdRpLRs/Gz6ENMUXVFtg5+prg9BdvYf5BydUeH8R81l5PpnTE2z9wv5+wbEAO1k9cBJ8M wUfAskZ19nBfds/2mrSw7PSRkZKSklcaS05OLmJPnz5VWVJSUt6TJ09UlpiYmPf48eO8R48e5T18 +DAvPj4+Ly4uLi82NlZl0dHReVFRUXn37t3Li4iIyDt69CgrZ2kIzvt+ROe8zpN+yYsRUrRybWXe kM5d86bteiQkCGTezPuWnafrzD15SUJS8Pcj8jp37pw3bn2kkCKQeSBvTtfOeSO+DxYS1ByYw/Ix 4nuWIynMl51D6qvy42nf3szLFNLyCf4+bwTb13XarrxCOYxcnzeO5bvraNn5M4/lLfDonPcKO5c+ KOe1AHXZ57DciyTl7ZnZNa/zkOV5l2WZfuQ9K8+j85C8ldeEBFUZRuTJqkdG0XrRiFAnncetzyt8 JzJZOVietJRDTVE/VfnnFJRON9R57jxwWd55eR3smpbXtVCZhfoa8W3ezUK+kXnrx0nbma5+Yrvx yFtwTHZxoX6KK07Re56W9+j6tryZfdl1Rv8oSdc3TwPzlhWtkLxphfqIHu1HL1+h/ysU/trKIXmd PWbleRfqSJl5xxZ4sHOvzBNP4ftJ37zOXaflySVBfU91bKN6wnXvwYMHKn3kOsk1k+sn11KurVxz 09LS8tLT0/MyMjLyMjMz87KyslSWnZ2dl5OTo7Lc3NwiJueZL/sYHB1fwoSc8MVDdMLL8qcE85YY 1seRzawOy2ZQzTBouOx9gbkrGrOLRYeHCgklwKQ7ps1qCU2TcdfuHiiUQ6f6rHy5sOs+gOVIgnl9 ODDH2HvhQoIe5C9RFbXXtkcLTiKnccY/F/YDRqCDLNO12aN3e5OH8Dngr05wYHk1iYb3ci/4Kq2F lJBmg4aj8J0wh6v6RkD7nRD8JDg51gXOr8G8vXeg74KJhXsfdJHXwdB+aIto+PmK34Gcx/mruWjI 6qtlIV8ndHSzRu5Vtl8V19VPwKInhvaRXbyZK/h7yIjQwk9AihS65x4YPO1XJPVajF073pG0K33z 5I4+RSsE/dqyy/n5Cl/G6NF+9PLVRDjOXHwIE/dX8EqhjmSOTi+ykj68iDOqLnML568kA51ehlwS 1Pf0+adCiD8bhISQAUiI0+nzq9BwJmqOTdFC1qg4zVSv8uPxIEYdV2MDO11GFX2xbwxXhTyI2Gi4 aFk+chb+8qOwTekiW+wMCUUE27g0b6mOSxEGw8Q44Q7UHIXV66bC9cFuzB3eB0MmfwivQ/qLrBxN dSIn/c4heM0Zi8EDhDVohcHMbc5mfD6oKi6unIA+HqMwfeV2XNVxoKqt9NmUamkNSE5KUsdDbiE0 F7j367j8AVW0t/nLjtxs9bqzrn4ite1Qqo+28pclvbHxjWYwyTVDm1eGoJG0LZZJnmqiprpCoKoR fdqPPr4auYXb91hWTy0pUob+qpcE2chWFSIc99ipHJu2KDoRE+5peWEofazEM38H1LNlm8hQ3FYn ELoi+/JDai+5VhecSkY1t2lY948P/vzhbXRBAHZ/MgH9PZfieKGF3bLn8cG5GDZhGY6YD8Di1b9h 798+OHPhArZOdBQ8BMztMGDpH/A5sgWfjGqAhwd+wAzPYZiy+VaRl4GloeN7aqEtah9D+opDV79S Y1IDNvb2sGfWdupcvFL3Ifav3g6lnyCWW54MiPSdUmEr5p1cJeKZiL9BZ/r51IRHd/YYl3YGf+rw xqtl04bsWfQ2ghR6eEgon2/Yoh5/KUkUpllLuLJWFBZ8S0iQkBWKcDZ7si4yM6+G+u4TsXTLP/DZ MA4Ocf9g1UZtb6ZLSyy8d/oi2eUt/LRqKnq1YSJnW1M1o7sbqTxTNK/ZHINnf4c/fLyxuGsOAtd5 YX8xzSg28q4QkhAbhwS2sbSyUscbu8KZ1df9B49VQlvUarPa0cPPEJh3wKxZ3VE1ZDO+kfYdffMU G4miNRKLOHWFQFUj+rSfErU1OewJgY33sdEJqKtYBlvUVE31G6MhO5XiPc1iTwdC8FlQVvrJqrLy Yj9xOgZYpsH3m/n4Tfa5ppzG/XvBEZfw95+yKWiWP/44FA2TDoMw1NC/Bnsu6Yo+XS0Qe3Qf/GVV /PjPfTifWxf9h3UQUopSzW0YerFO9jjugZCiJjcnRwiVBUlISmabmlaopU5Qk3Ucx/xyhYgG2JPA KwPcWIYS8ICLlhZy/Y7huKwOov46jBC4oHd/QZTMe6HniyaI/Xs9pB+WFEFXPwNRc+jbeNWR9Z0f fii4r/rmKdcPx4pWCA6HAC69+wvv4vRpPyVoa9mypSg0w0vd6wKXtmKt1l+vtUSnthasCAeLfC31 9OhxXBfCzzOVWvx5Y/3kh9fROMMf344bhokfemHHvn3Yp7KNWDl/Or4/Jvg2exvLJzrh+ndjMWXV IQTGxiLMbx9WTZuHfY8bYvzsVwz+S+Dnk5oY+vF8dM/Yh9mjF2C7XxhiYwNxeuM8vPbdNVTvPgtT mXaqCFmPGZOXYuMRP4Sx+o0N88O+VcvhHW2JngO7Ck4d0LElFxgvbA9kPuxcd0r954+EDn9tPeZv VF878PRGzBv5ES6aWgg+ao4uGYU5XjtwWnVtwW/NeZg07IP+xSwH1LW8jqWyOpj5cxgse7+Fcflv pGti1OIP0B6++GzsFKzacQR+YcyXtcmNK6dj1JKjevoZimZ4+/0BsHy4H6u3izdAzzzVtcT1paOx YLukzmf+jDDL3ngrv0L0aD96+TaDeztLlch/fZL7hSH4rlrF3eZ8hlF1o7B9Gsvbxn3sXofB78g+ 7PD6EJNnrM//0ya9Zr6BZhm+WD5Jeq0FmPRVAGTN5rmk3MW/fJZ8CjBv+Q62/rkOswfZI9lvN75f sQIruH29A8fYE11tpglqzNFy9lZsm98Byf98jimenkzwV+NEbi8s3rUDswt/3kBIqT0UK3euxn/s r2P97PHw9JyChTvuwfUtL+xZPbTg6ySHlqxLXsGOT2dhPKtfz/Fz4XUmF4M+34nlA8Sh1R5jly1E H+ub+GEK8xn5Ln7R4WOV4nCbsxaL+9ggeLP62jM/OQrTKb/iwAL3Qp2g0YsNEPPXOszn12Z+05b8 hnuus7Buq/SrFyXsMWDVVqx8OR37P5ysqoP5W2+i/vjV2Ll8AJMtCU6j8T9WX2+1SsY/az7CrPFT 8N+vVuO3y0CX7pKXmbr6GQjzXh9gejs2Zm/+CvvFmb4eebIfsApbV76M9P1MVFldTpm/FTfrj8fq ncuRf7s5urYfjh6+3d5bjoluyTgwn/tNxmfHhEc3846Yv2cbFvSvhetbv8J/p4zHrI++xsZ/Y+DQ rRPyV3ed3sDPuxajZ7WLWDuLn2MmWLPBlC17MF17YzA4ZaGjVfh3o0K4RChlQkzjW3k4NzdXMSzG Rbt9+zb69++vOpYgCMIY8PHxgZubG0xMTPKtSpUqRbbyMEceliNPqzDLPuIAQBAEQRheEyv3mj9B EAShCIk/QRCEEULiTxAEYYSQ+BMEQRghJP4EQRBGCIk/QRCEEULiTxAEYYSQ+BMEQRghJP4EQRBG CIk/QRCEEULiTxAEYYSQ+BMEQRghJP4EQRBGCIk/QRCEEULiTxAEYYSQ+BMEQRghJP4EQRBGCIk/ QRCEEULiTxAEYYSQ+Jcp5/DFQA+MWx8ixAmCIComVa5du1aq/yFY6T8YlqaJYfE/I87Nzc2P87A0 XdzHt9nZ2ejfv78qXiZEbcD4URvxZPQGHJznJiSWNeewtN/78H95HfbPNdQ1ZByciy7LfIWIZrp/ cgGrhwoRgiAqJD4+PnBzc4OJiQmqVKmi2oomxvlWHubIw3LkaVUyMzMNJv6isEvDcsEXt6KJ8Vu3 bpWp+AesGoppex4C1iOw9vBCdBDSn3uiLmLfpWghwgjcjRV/PUSXKW+jr52QxnDsNAKdnYQIQRAV knIV/4yMjAon/tzKVvwDsGroNPh2GADLo2fg9OVRrOhjLuyrZKieBCIwcetezG4mpBEE8VxQnuJv FGv+Wcd34sDDuuj+n5no7ZKGE7v346mwr4DHuLBhDsb27YkuXbow6waPAWMxfc3pfN+su/uxdPIw eHTj+5n17AvPyR9iT/4S/0HMZekjvaRr/vLzFjbR9+DcLug27yjS7+zFgrF90VO1vyf6jp2D7QHp Kp/S8RjXty/FZE/x3F3Qs+9YLNh/F1mChxrd8ovHF7Bhzlj07Sns6+aBAWOnY83pojVLEETFwwjE /yn27z6BtLp9MMjNCUP6uiDX/zAOyjQqZM10zPk5FA2nrMZOb294b1qJGcOdUd2sFmpyh6dHseit L3Da5CUs3MD2e+/EDx9PQI9qVjB3UJ1CEfV57+PFT3bA58wZ+Pz9P0xtaQnUHIzVZy5gr2R6nuu/ Cq+8tRkYsAw/sTzs/OFtdEy5AK/ZC4vkV39ScPncbTj0m43VO1n+d/6Atzum4OQXb+O7AMGFoVt+ Q7Bm+hz8HNoQU1bvZHXhjU0rZ2C4c3WY1VLVFkEQFZzKL/5Pj8Dnai7q9hkE/grWafggNMM17NsV pd4vEHr7HnK7voNVE93hYm8P+za9MGH2V/huuvDiNuYWQpMd4bloHga3YfvtXeA+cCrmr18MT416 F4JDR++hKhPchR4NUdPcHDVtO2Pahy/D/ulJHDwvuIkkZ6Lj0j34cmovtGF5cHGfiFXfjodjmi+O nhZ8SowT3vhhO76aPQLuLiz/Lu6YuPwddDV5CN8T4pOKrvkNxe17uej6zipMdHdhdWGPNr0mYPZX 30GsLoIgKjYm4hp7RTARabi0RO3aA/9cRwz2FFTJfjD6sclr2N6dkEx44eRYFzi/BvP23oHiIotD fdibRMN7uRd84wovlBRH1Ro1hFAxWPTEUPm7iGaucGabiFDpUlIZYe6KxvZCWELx+XWCurrmYe+d sliSIgiivKnkM/8A7NwbBrgMxav5qyv2GPVyO+ChDw74C0kMtzmb8fmgqri4cgL6eIzC9JXbcVUq 8jVHYfW6qXB9sBtzh/fBkMkfwuuQhoEin2Z4qXtdJO5fgRUn76t8s54GY7vX34i17IlBXdVe+dS2 Qz0hWPak484hL8wZOxgDPLoJa/ivYbvkQyHd8+uGOZs/x6CqF7FyQh94jJqOlduvQs8xkSCIZ0jl Fn//A/B5yLZhGzFK8tKy/7fXWGIiDv9+vOBlp7kdBiz9Az5HtuCTUQ3w8MAPmOE5DFM238r3qeY2 Dev+8cGfP7yNLmxg2f3JBPT3XIrjjwUHBfigsnRABg7OfxUe7No9B07HjlxPrN75CXqV2wdHj3Fw 7jBMWHYE5gMWY/Vve/G3zxlcuLAVEx0FFwFd82tuNwBL//DBkS2fYFSDhzjwwwx4DpuCzbdoBCCI 54EKs+zDkcdLRxaO/34YiSbNMWLhQiyU2YjmJkg7sRv7ZS9SzWs2x+DZ3+EPH28s7pqDwHVeMp9q qO8+EUu3/AOfDePgEPcPVm2ULiDJeHAEvx0zxSsbTjKxvYAL507iwLrZ6G5Xjp+axnpjp28yXN76 CauE9wm2Nfn17yIyVu2Sj175NUfN5oMx+zs2aHovRtecQKzzUvqSiiCIikblnfk/3Y/dJ9Jg8dKb +O+IERghs/+++RIscq+ymb4GqWJPAq8McANyE/AgQUiTUc1tGHrZs3l13AMhRYGAy7iVVRt29qZC wjMgKQnJbFPTqpY6LpB1/Bj81D+qLqCE+TW3ewXq6noADdVFEEQFotKK/9ODh+Gf64KJ7/Zh89Oi mPd5FxNdcuG/Zxf4dz9Hl4zCHK8dOB0Yi9jYWASe3oh5a87DpGEf9G/MHELWY8bkpdh4xA9hbH9s mB/2rVoO72hL9BwoX7yX0GsMBtvcwJrh0u/mu8Fj2Ax4XdCyXlSWNHsJ3esC19bPx0a/MFa+QJze OA8jP7oIUwvBR0Sn/B7FklFz4LXjNAJ5XQjnW3PeBA379AevLoIgKjYVYtmHo5RWcqKwa981mHQY jXEa/6SBE8aN7gCTsL3YGQA0erEBYv5ah/lTPOHp6YlpS37DPddZWLf1HajeFTu0ZNsr2PHpLIxn +z3Hz4XXmVwM+nwnlg/Q8K3n4+NYOmkhztmPxnuFlp3+Dy/bhmL7nLnYJl92MQhumLN2MfrYBGPz rPGsfDPxyVFTTPn1ABa4S8Z/nfPbCC82iMFf6+ZjCq8Lz2lY8ts9uM5ah63vFPxugSCIikuV1NTU UimtklCLaVIhF8NKf95BDItxvg0JCSnbP+z2DFD9PSGfXsp/Syh2M17zXAubCvQH1563/BJEZYP+ vEMl4UHcY6B6DVgKcSlZd+8hDtawU/jO/lnxvOWXIIiSU6F+5CW1ykDXgT1hGe2N5V4n1e8JVBYG v+0LMPqDA8ju/h5mVqA/L/q85ZcgiJJTJSUlpUIu+9y+ffu5X/bhn5ve3f8FPtp0iolpivr3AibV YGXXCD3+8z7e+c+LKM8vPovnecsvQVQuyvVPOpP4EwRBVAzKdc1fFN9nZRxN6QRBEIRhoBe+BEEQ RgiJP0EQhBHyTJd9OErp3AiCIAjDQTN/giAII4TEnyAIwgihH3kRBEEYITTzJwiCMEJI/AmCIIyQ Z7bsw1FKF40gCIIwHDTzJwiCMEJI/AmCIIyQZ7Lsw1FKlxpBEARhOGjmTxAEYYSQ+BMEQRghJP4E QRBGSLmv+XOU0uVGEARBGA7jmfknXsQfmzZhx6loIYEgCMJ4MRrxjw4IRSLbpkXeRqQ6iSAIwmgx kj/sFo3QqDRYNmqE2ukRCArNEdIJgiCMkyqPHj0qldIqCbWYJhVyMSw16X/YLv8P3O/du1dm/4F7 TuhRbDsZj8ZDhqLauT9w44XOGD+8LaoJ+1XE+eK3v+6j0fDBqBPkA787j5Cey0ZH81qo17Y7BrR3 hLngqr9vJBoMH40W8f/C59J9PM3OhV3n1zC87QvMIRV3L56AX9ADPMniJzFDdesGcPPwQOs6pjzz OLrtJOKcB2KiRwP1OfOJxMntRxBh54FJA1zBvJHz6AZOngxAZGIqsnNNYFazHlp2eQmdG1VXH0IQ RIWlXP8Dd2FbiUnHzVtRyLZwgqujNZo1sEZuXDiC04XdhUhF+L/euJzmhK6Dx2CM50C8WC8LD/wP Y9/5OMFHRB9fICP0CA5eeQqbVu7o0qUL2jXgwp+OkMN74HPjCWq27I2hY8bAs78b6uVE4bz3n/CN ZU8opo3R2NEM6RHBCFOfqoCwYESkm8GxcWOV8CPxEry9zyOqSn10UeWpP9xqPcGNYwfU5yIIghCo kMs+ZUr6bUQk5MLCyRWOLGrdohFqIx6hAfwNgJxspNdoC89B7mjiYAUrmwZoP2gkujmaIOnWZQQV 0k99fJMRdjsX7V8dhb7ubdCmTRs0tGbJ0RdxKSoHth08Mci9CRysrGDToD36jhyIFtWTEOx3kw0P pnBtUh9m2fcRJlP/sLD7yDarjyauXPrTcf30dTyu3gIDh/dGi/w8DUFrK/FcBEEQakyioqJgKLt/ /34RU0qPjo7Ot5iYGNW2rEgMCEFcriWcW3DpZ1g1Q8PaLD00AEpXqeXYGIUXSKqhRXMmvrlxiLor JAno42vZ1B2tZSsvcRExSIMdmrSR7TC1R9MGlgVPKC6t0NgiG/fvhCJ/TMkJxZ372bBo3AouqoR7 iGaDnFWjFrBXPQaIWMPBphpyE6KZB0EQhJoqbKZdrmv+4rq+NCyu90vjAQEBZbDmH41TO/7B7apt MWpUZyaDatKv7cf2S0/hOnAi8pfRVWvzt2DVYwqGtBDSROT7ysg36J9NOJvUEsPHdmdDgIygf7Dp bBJaDh+L7mxn9Kkd+CfcFh6TBoBP9AveY0xAbz6uCddJVh+tgCN6TBkCeXYJgqg40Jp/WcE/60xj 28Trqm/8Nwm2/VI8S0xHRJBkJl3BcXRzhXV2NMLDeY5z2DYa2daucBMeaETqdRyDMWOUzANNBB+C IIhKLP45CA2KQLpJbbh26aJ6ySo119omyI66hZuyhfDkxIdCSEJyKhsqzGDO39FK0MdXCdvaVsz/ MeIVRqC4R0nsXwvUYC4qrJuhgXU2om7fRk5OOMKjs2HdoFn+0wzq1oEVu5vJKamwsrJSsOoFXyAR BGH0VF7xT7+JW1HZMHNyQ8826pesUuvp5gSz3ARE3C6s/kn376h+DFZAKm7cuo9sEzs4NRKSBPTx VaKuS31YIg53AlOFFIGcSNyKSIaJXWM0z/8e1RotGtVGblwk7gTdRlR2bTRqkS/9gKkznGxMkHLn CoJkpyMIgpBTacU/PTgccbnWaOmu/v5djqmrO1pa5yIuJKCQgJunhOBv71MIiklCUkIkrhz2xsXY XFi17IgWshPp46uIXRd4tLTEQ/8/4X0qCDFJSUiIDMSpv44jNMMKzd1bFfotglWbprDLjcala3Hs 2KZoIz4VqKiGtr06wQ7ROPenN05dDUZkQgwbWALhd2o/dvuECH4EQRCVVvwTERAaz2bOzeAmmRwX xhpuzexgkhiKAMlnP7Ydh6J9zYe4cuQP/O59BFcfmKN+l+EY0bXIK1m9fJUxhX33ERjuboesiHM4 +Pvv8Pa5jPt5Tujq+Sq6F/5sh+l7Uzjb5CItLRc2zk0L/0iNY90WQ0f2Res6WYi4cgZHvA/ihN9l BLOxwt5J1zwRBGEMVPKvffRA2xc8cvTxJQiC0BH62ocgCIIwKCT+BEEQRgiJP0EQhBFCa/4EQRAV BFrzJwiCIAwKiT9BEIQRQuJPEARhhJD4EwRBGCEk/gRBEEYIiT9BEIQRUqE/9TRt1k11LEEQhDGQ E3Ku3D71rNDi36QrfedPEITxcOc8fedPEARBGBASf4IgCCOExJ8gCMIIIfEnCIIwQkj8CYIgjBAS f4IgCCOExJ8gCMIIIfEnCIIwQkj8DUTUzvfw3utf4qIQ14Y+vobhFH56fTw+3RkhxPXj4jfj8d5/ tyFKiJcFhjjn88bNjdMwb9Fuo64DwnBUbvG/uw2fMlF7T9GepdiWD5mRp7HPayGWzpycX+6506Zh 8YKl2H04DKmC33OLcH9/OiPEi1DcoBaLg4tYvby3CWFCii6oBiZel8uPaK3DR/sXlK6t5eQiOy0V mUKUIMoSo5j5O/Saiv+8IbdhcBH2Vz4yELZzHhYuWYPToUADj//kl3t43xdhlxsB/5AEVBW8jZaQ Azh3n20TL+LCTXWSrlhaWwPBx3AmQUgoQizOn78LC4vqQlx/Ws3YhNWr3yhxOw35eUaJn+aIyo9R iL91i37o0UdubWEj7K9sxO//BP87FIPaHmzW77UCU8a/nF/uvuPfxXsrN+PL/+sMU8HfWAk7dwlJ tt3QvmES/P+9iBwhXRdS69VHY9O78D8VK6TIUA0sdmjSwkpIKG9u4LJ/khAmiKKYCVujhq+5r7rS FfMWNMeltRtx9s5jZOaY4gXrBnAf/1+M6lpXIpRPEH5wG/b4XEbswzRksxQzC0e0GvcB3njJsYig 5sQcw9a1u3H17hPma45qDi0xcMr76Ne0uHl3Nh6cWIcte/0QnZiBXNOqsG05DKOnj0GLWoKLEtkX 8dffd4Gmk/HeW26oKSSXiCcBOLj+V5wMikY6K6jJC7Xh2Gk0Jk/pi3oKLSf5+u9Yu/UAQh6o81vL qRc857yOjnUkztnR8P/9Vxw6HYz4FObHauwFa1f0mDoXI9pqK1gZw+rpxNlEWPUYhGE1I3Hl70Pw TemMXjWE/cWQW7U7PNqH4ueTBxD26pQis/OQMxeRVL8/Otc7g0AhLR8d64AvL22PGYZ5X0+CE0/g y1wfX0b7T5fAxfd/+O1YKJ5k5qjvS49JmDapO6x5VQdvwedfH0E8Xy86tBDvHeIHAy2n7cTMnjyk W9vi199pOgtfjUrF1h934mYMb+8a2rAeZdL5nAzef3as3YOAKN4neRu0hG23d/D+W+1RTeVRwn5C 0AvffBKOYc1HaxHm9CqmfuqFTz6aAffacTi79lP8xbS0gDTcvn4XddwnYPoXzO+LRRjaIh2Bv3yG P0IEl3xuw/vzP5Defir+7xt+zmnoWC0U+1cskJ2zKPH7l+DLX/xh0mEK3mPHfvjBKNS77411X2zD XT7iaCDn/DFcz6iKtoMHlE74U85iw4crcPS+PV5651N88s1KzBzZGllXNuDLRQp5SDqOzT9dRd3B H2CekN8GLG3LotW4nCL4qIjCjcAMuA6YpSqXup4jcXz19zhdyM+wpJ44xOrJGm27NYdtj85wyLmF y2f0yYAZ2vXogBcSL+GS/L6zgeXM+RQ06TsUdYWkwpSmDuJxbuUCbL/dAIPmrGD35VNM6FkbD4// D16/C0s8jUfgnRXj4MqCtT3mMx92DWaTOqt369O2coM2Y+mn+4Eu76ja8Ifzx8A1/Qb2r1qNiyW8 rzqf8+5OrFi8AVdTGqDHmKl4bea7GD30RTRo6CAIf8n7CUHiX0BOMqz6fYb3X++P5k62qOPaC/9Z 8jpamsYhwFe6bmqPgR+uxJTxgp9TW/SbNRbNTRMRdFm+vpoK22HLMW1kJzjbiOd8H11rxuLYnlOa lxlSjmDnvruoPWAR5rzeS3WsY+vhmLbQE/UeHIL3Cc0KERMZw2ZdjeDSsnQPdWF7tiEwsxk8l83D kI5NUcemAZoPfZd11MGorZSHDDO0mbkM/+nbFo5ifpdNRONMfxzYL6kXs8547Yul+I+0Tt7sg7pM fG9cEXwMTgouXwxBrnUndGrGovX6oENDIPzkQSatetBhENytE+F3uPCSkWpgyW6G9t00PEaUqg5y kFJrIOZ+PAU9Wjdg96Up3F9fhlfdTPHw8hn1l0EvWLF0KzafZh3cojYLs2sws3yBJejbttKy4Dpj NaYIeVX5zh2MuhnX4C/Nqz5l0umcKTi9/QDia/bBjK8WYsSQfujUrSd6jHwXr/W3F1xK3k8IIxH/ WxvUX2dIreiLsEbo0FtoVCJmDWFfB3gYfU9I0IDgV5RGaNtZ9uxp1hod21khN/gqNL5jDLiG8Bx7 tO/TpPAyUr1WaFQzB+GBV4UETVigulx3inz59B5Yv9FABAKuJwLNPdBLln1TFw+42ebgzvlThb90 qdoa7drKBpxavfGiC6u/m1fwSEhSpFFD2AnBkqB0f9W2FrcEn0I8OIizwTmo27WfsFxjg07ujYD7 x3C8yNObNpqje1c7ZF4/i8D8WaZ6YDFrPxjddVxCUqFHHdTv2ge2QliNGRwd2TNGfBSihRSN6Nu2 qnaAeyfZfRXyGhcp70MyNJVJp3NeRVBoDmq790MzTfOYUvcT48Zov/YZ3rVw9wGsYa3TG+AMRJ/d hrULZmDhtEmCyCzEccUpo/I5q1uwh9aMVKQJcTlREVFs9h4LH/4ZYiEx+xznn7LH5pziXk2mIVU+ 6bHpjuFi+Xs1EBI1cQ+xrDx1G7oU7lQqnOHowDaJDwsLulUd1BaCBdRgZWWbtJSCgSL7Ie4c/BGr 3p+BD6eI5dIg0jqi/DUXNw/wrMoJO3QMMXBCp77OQgpQp58HGiMR187cEFJ0w6lvTzhkXMZJcZap GljM0LqHlhfqpawDqzol/1RB77aleF8V0KdMupzzbjhiWFbsnZsICUUpfT8xboz2a5/2jfWZlok8 wcVv3sFXm87BrMs0zFixGsvWbsX3v65AH/lYUmpawZOvnSrZtC6CT1EcHPj86S4igtXxfGq4sBmS UP4Wio8phif7DvYteg9e+6PR6OVZeO8rL3yxYServ7fRUnApCcpfc3FrxYZfOTdwwY891SAKh+dL BOOdLQhnqcnnj+CaPmvF9Yaia9OCJ6H4sxcRY90LvTuodxfBQHWgHyVrWxp5pmUq47IYEbTmrw8J x3AiIBUOL3+Uv45vXZ0/k95HvOK6xgM84N+Ry3j8+Al79K0OPilWwqGBA7sxcUhMUq/VFjErzV8K mbq7o4lpBm4cly3L6IUL6tcDHt4LU3gvEYHoGLaxrotCQ8ijGFZaOQlI5DprwZ4AeDTgAM4+MEO7 qQXvBlTr0PfvM8/yIefSEfg/NUV9D4UnBY9GMJHO4nWiBjp3bQmT26xtPAjG8ZNRcPDQ8huSZ1wH pWlbGjFEmeo7ql6Wx0bcUccVMEhZjAgT/n/lGsquX7+uaIGBgYXsxo0b+Xbz5k2VVUhSUpDONtVq FP6OJufSRYQoPmHG4sYF2Xfg2RfhdzUDL7TtyuYsypi274DGpgm4uPcU2NOrftToi8E9rZEZ8CvW /hlcwl+HOqF9e/YEEXwSp9k4JSXn5hFcjjdFk6691YIuknMD1y7JpswPjsP/Huuk7bupB4pU/mvV qrBQDZgFxF/w1+9Fa4lJge+Ry8is2hGDJis8KUx+FW2r5iD84lm9Bs7qLw1mx0Xh3Dc/wS/RCW49 ZO+OpJRzHciXPkrVtjRhiDKZdYFbc1M89vsXIRqexAxSFiPCxNXVFYayJk2aKJqLi4tGa9y4scrK ksSgf3H2uNyu6z8raeSOFtZA+N5v8M+NSDxKuI3re1dh2dpAmCpNMqrWQeq/H+HbX08jIiEe0Td8 sPvzdbiWbY/er2hZE64xEOMntGAzqvVY8ekmHDt3HdGhl1ie97Dj52LzOcFPETM0m/wRRjYH7nkv xcIPlmL3Xmm52TmOFvOdKcNpzGz0sbkD70UfYffZ26yskQg+tgnfeR1Hcr3B8HxJtmxWqwYi1s3F poMsr6ysEZf3YMMKb8RYdMCAQYIYss7axDQJFzf/D36h8XgUdR1nf12Mb4480vgUVKaknMLl4Bw4 DB6PdkovEc06Y/hgJ+QGH2WzeCFNF8w6oGPbqkh6EIvs5gPwEntq0ki51UEruDZm4nlmO/7l12Ft NZqXqVRtSwMGKVMNdB83GLZPj2Pdh9/iHzGf/2zD2h3nBRcDlMWIMIpln5jTG7F7s9wO6PX3XNQ0 x6gF0+BmHQGflfOx7IPPsO2CKQYuXYMxrRSkvMkoLFkyHrZ3duLH+bPx1cotuJzeGmOWfYnhjQQf Ddj2X4wFcz3RIPUsDvy0HF999g327DiMULRBq+J+72/miJcWrcGH04fB1TwGl/8uKDc/x7WMeug0 eAQ6aHtvaNYEIz5fjlHNU3H5549ZWefjp52Xkdt+GhYsn4RGMvGs3fl9zJvdG5mnVuObD2Zj9Y9/ I9J2MKYvn4uO4jjBOuubc4fBOeMydnw2G8s+/gaHo1riteUfoYc2wSwj4g8fQ7hpS/QQByMFbAcN YELGZvGH5C9NtGGGdv26w5IN54079yj8RCSn3OrABn1mvAU3qzD8za8z/wscEcb8UrUtJQxUJlOX SZi/7C28WCMUxzYI+fzTF8lVC97klHlZjIgqKSkpeUK4ROTlFT1cTONbeTg3N1djWIzz7e3bt9Gk a3/VsQRBEMbAnfM+cHNzg4mJCapUqaLaiibG+VYe5sjDcuRp9MKXIAjCCCHxJwiCMEJI/AmCIIwQ En+CIAgjhMSfIAjCCCHxJwiCMEJI/AmCIIwQEn+CIAgjhMSfIAjCCKnQv/A1bdZNdSxBEIQxkBNy rtx+4Vuhxb9/f/rzDgRBGA8+PvTnHQiCIAgDQuJPEARhhJD4EwRBGCEk/gRBEEYIiT9BEIQRQuJP EARhhJD4EwRBGCEk/gRBEEYIiT9BEIQRQuIvI+mqN37ddgQh6UJCWRLni982bcI/QUJcG/r4GoQg /MOu/5tvnBDXjzjf37Bp0z/sLGWHIc75vBFxYht+3XMRJbsrBFFA5RZ/QUA3FbJfmLjvwUHfICRk CX5ScvOQm5UFpV2VgZyn93Hl1H7s2fYrfhHr5Jct2L5rL4743kGi4Pf8Egff31iZtIyaQf+w/b/5 ahDQHIQe/ZXVyV+4rscEQHVOVpe//HUd2g5LuvSn0A5LOIjxP4OSlYlsIUoQJcUoZv71Oo7BmDFj 4DmwJ7p0aY+mdqZ4HHwW3mwQOB+bI3ipseowAm++OQytqwkJlYYcPL1xGDv3HMLV8HS80LAV3Lt0 YfXBrI0zaldNxf2Yh4KvEZN+E7eimLTmxiE8WL/Hv2oWFmzsuY3AJCGhCIkIuvsY5ubmQlx/nPtO xpvje8JRiOtL+rX9WgdGwngwCvE3rWYFKysr2DRojjZtXkT3gSMxcWx/uFo+xY0jRxCUKjhWYnLC jmPv+Sjk2nSC56T/4JXe7qwu2qjNvTeGjpyIN0d1hrXgb6yk345AApPWRg3NEB8aoNeTUGbNWrAx eYy7QRqOig5AaKIl6tVjg8QzIR3Bd+OFMGHsmC5evHipEC4X+F/tFJGHxTjfPnr0CC4uLqp4iUmJ xI2QBFRt2AFNbYQ0EXNrONum4fatcERn2KNto5qqZL6uvOtIJKp3aAr1ITl4dOMEDv17CucvXob/ lSu4FhiCiAd5qNukHmqofIDU6Cs47XMSp89fwCX/K7hyLRBh8Wawd7aDhTjEivmp74zMwEM4esIX fsz3WmAYYnJZfhxqwlRw1ZT3nEc3cOLQvzh1/iIu+1/D9dAYZFrWR31rbbPJRFw6dg4xVVzRf2QX 2OdfRBsJCL0SiQxbNmA2EEvJScXdi0fh8+8ZnLvkz8p5HcERj2BSryHs8gvKs38DIQlVUd85E4GH juKErx+ru2sIDItBrrUzHGpKM5GK6Cun4XPyNM5fuCTUcRjizezhbGeRP0MRz9kw/94okYLIGyFI qNoQHYrcdDUJoVcQmWGL5m0a5N8/NYnwP3MJ8TVbYkjTKggKjkKmfVsITUMr/Jz3TNqgk3UsgqMy Yd+2EeSHRfqfQXBuE7SvywYIeTlyHiHk/HEcP+mLC368nbF7y66fatUADST3li8vefunwyk/7/zd zF8Is2wD65h/cfDQKeG+sPYXkwtrZweoqjopEIf2HEZwEutjTyNxhdUxt9vpTvn3V5e2peof55Ph 1DAL144cwjFfwTc4AklVneBUt6pkRqnbfdXvnIzUu7h41Af/njmX39dC7ufBvpk9qqscZH2WtdHQ mExY1q8Prd2kAhAWFsYmB/Xy/0KnrsaRh+XI0wrVqdFh1wZNagNpMREaX6Clh/jgbzZjrlK/CwaP GYMxngPRvaUjTF8wgZXgw8mMjsbjGkxw+ntizBhPDHyxHrLun8ehs9GCRwHxlw8iIMsJXQfz5aih 6OZshgT/w9h3vpjXeImX4O19HlFV6qMLP9azP9xqPcGNYwfgK1u+KkTcTYSzyai1azs00En4NZGO kMN74HPjCWq2ZE8LfCmtvxvq5UThvPefCnmIx+WDAchy6qquu6Hd4GyWAP/D+1C4qJmIjn6MGmyg 6++pruMX62Xh/vlDUKg+w6GamZvAxrkpqjduDEezNETejhR26oIpXJvUh1laFELl+c4JRVBEJuya uRVqNwU8QlRMNuya98yvq8YWj3Hr2L86vHvIxYOLv+HADaBRj8EF7e+BP476BKnfQdRoip7D+8CV 63y9jqplUG7DOtble/VrW8mhOOJ9FvG12qLvcDGvT3H79EFcKOl91fWc6SE4vMcHNx5VYQNgF/Ts 2RNd2ruijo0VWFdWkXjJG96SPsvbaK0nN3DsgC+0dRNjw7jFn3VDqxqsCpKfsK6nTFJCIrIsXdG9 dws4WDF/mwZo7t4br7zUGtLXAtadhmHUwO5sFmUDKysbNGjfH+3tTRQHliyr1vAc5I4mDnw5ygEt eg9HH9eqSLp1GUEaG2c6rp++jsfVW2Dg8N5owY9leWk/aAhaWyUh2O+m5heNj54gmeXWxqGUizrR F3EpKge2HTwxyL2Jqj5sGrRH35ED0aK6Uh6yYNW6wNfKoQV6cwGqmoRbl4PY/EzEGp2GjcLA7m3Q gHViVbn6t4e9SRpiIooZEMuQ6NAopJnYwLkpu7OmrmhS3wzpEUEI1UcwXFox0UpD+I1QSfnY3bt5 C1G5wrmVYNfrO+oV9JbWVXdXWObGISpC8NFCVk499Bg9EO5NHIT2Nwj9mlsiOzocqsNNq8LSqjrM +eTPtJpqGZSbZVU+G9CzbeVmwKLZy/DsLdwv1X3tDEeTZESGSu+XHvdVx3NGX7yEqBxbdHh1FPq6 t0Hz5uzJ9MXuGNjVRf3UnH4dp68/RvUWAzFc6LO8jQ4a0hpWScHwu1nsSGo0GLn4s+ZZU/2gqAkr tt+EzUp8fcPxVK9Zgynq1FI+d22nJsLjqYgpGrjURzXe0e8KSUW4h+iEXFg1aiFbtrGGg0015CZE Mw9tmMG8qhDMR/05Z8GXUNo/LY2LiEEa7NCkjaxcpvZo2sASuXHhKPyOtDacmsh9G8ClPstvXBQ0 FpVjWgcaqk83os8WKpfUFJ8m2Mz8RngaTOybopWgzy7snphlR+GWXoLhiBbOatENz28v6bgdkQAT p5b559YJuzoanhKKYla/CVxlT3V2dfjRSXhU7Pipb9uqjUYtZBMJ4X4lP9E0jRLQeF91OedD3HuQ xgrWBPImmM+9aCTkWrFz2RcsoXKsHWBTLRcJ0dp7iTFh9OL/6EmyEFKmWtvBGNS2NlKCj2H3lu3Y e8QXQQrfiGYlBMH34B5sl3xC+dct5XNbWCp06armTJ6zkZUpxOXEJSAxl3XlG/uKiNmRUCZO/BNA wVUZdu4MIZiPPVqLX/y0dSj0JKOEqq4sa8NWJjIctdCkIaXQly4WbLYpBCVUNTdj2clCQVGzkBDk i4N7tmPbr78I5foLGqpPN6ybqMulYE0UHoBUM/NsMzg1bVogGi7N4cwEIy48WPNTlQJ2rRrDWjpo JAYgJM4E9Zu4FhYkKTlPcf/KMezdtR1bfhHv7VnouupVrbqlECoBerct5ftaFH3uqy7njMdj1r4s a9tqrMc49qSeywa8G/sKl2PTpiNQF0V7LzEmjFz84/CIi5UVEzR1ggLmcOw8HBMnscfXFx1Y+wvG We9t+O1YKMSPhFKD/sFO73O4a9oIXfuNxKixk/DGlCkY3rIUHVID4merRc0DTQSfIlhboQaTr4cP 5N8gWqOh+MWPszUbfJ4FqQj6Zye8z92FaaOu6DdyFMZOegNTpgxHqaqvun3B10wysy8ya0xHcHgc E41s3Du5uYhgID4UAfp89mPtBlfbgkEjMSQSiRaN0UrT9ws5sfD9cw8OXU9C3Ta9MXTUGIyfPIXV QY8Sf9JZEkrUtjRioPuqE/XQUbEczDz0L0llxajFPyfyFiLYLMSaPe4Kr700Y26NBu37YuTESRjS whKp4RdxSTUtS8LNm9HIsm6NIeI6vmVV1cwk8anyN6TJiQrf0yenMqEwg/kLQlxO3Tqw4q8nUlLz 12sLW3U2TGnAvjEcLYDHdwI1vtjWBdvabGqW/BjxCstfcapR1AI1Cs3ekqFcVCaJZuZQFTXpJm5G Z8G69ZD8dwPqdehEaKi+sofNzEPjWe4d3Yo+Kbg5slIlIjRA1zk4pxqaN7aDiWrQiEZAaCKsXd00 C/ndQNxOMkHD7sPRu00D2LA6qM5v5sNEVoPlQGnaliYMcl+tYVWDN8H4Qu9TpNRlT6AmrNZSUpXK wUxVsQTHaMU/59E1HDweijRzJ7R10+dFKHsSaFof1ZGBNFXPTEcmXwUyr1p42SQnFHdjlR8xk+7L f0mbg9A795Ft5gjnRkKSHFNnONmYIOXOlRL8LsERnVrbwiQ5GP/6hOj57qKAui71YcmGjzuBsgzk ROIWG0VN7BqjeaFKSML9O7IpM6uXO/ezYeboDFVR0zNVv6Y2r1p40Skn9C40VF+ZEx0Qyu6HNVy7 SX77IJp7N7iy5pEWFarzEgynWquWcDJjg8bRswhPYxOHZlraGGtArEbwgkocC0hkbUL+rFYm5OYU Fs9StS0NGOS+2qMxn8XE3YG8CYqYOjvBxiQFd64E5T+ZE8oYhfjnpCchKSkJCZHBCAz0w6n9v2HL 3ktIMLFBu4H90UzLYnfcxb+w95gfgiMThHME4pRvKJLNHdG4Ifewg7MDa5Dx13DULxIJzCfmjh8O 7z6NmCpKCylmqJ4Zgr+9TyEohuUrIRKBp/6C771cWDVvV+SlXQHV0LZXJ3a1aJz70xunrgYjMiGG CXEg/E7tx24m6tqo3q4/+rpaIv3uaezZ+QeO+F5ldREo2FX4XolSLVFoxa4LPNgz+0P/P+F9Kggx Yn38dRyhGVZo7t6q8ABoVh2ZIX/LfH1xL5f5thPWv+2coa6+o/BT1TErk99h7D4dA8XqK3MicTsy DWYN26Gjoj5bo2O7hjBLC8cNfT77MW3MhMoMaazcuXbNoHV+wQTLziQdYef/lbSJfdgflKL/jFsr dnC0ZWeMu4kzd3hdJ+DBI37XS9e2FDHQfXXs1AGOpvHwV+XzDmJYXQWzPn3M5ypUD5nV2qJXJzs2 op/Dn6yPXQ1mfTKGDRaqfr8bJSlKZcUoxP/B5d/x+++/w/vIGVzwu4l7qS+gUYfBGD3eE52K+cWT lQ3rtXG34HvEW30OH39E5zmj18iCQcOxx2B0bVAdjwOPwJv5HDp7F1XaDsf4rvYKFWwDt9FD0b7m Q1w58gd+9z4Cv4hs2HcZjhFdWaPVhnVbDB3ZF63rZCHiyhkc8T6IE36XERzH5kROxRzLnlUaebyK 0YM7oFGNLMTdvoILFy6ozS8A4Y/NYdvUDa3tBXdFTGHffQSGu9shK+IcDqrq4zLu5zmhq+er6C6v Sxs3jB7aHjUfXsGRP3j9+yEi2x5dho9AQVEd0WNwVzSo/hiBqjo+hLN3q6Dt8PHoam/45pkTGoSI dAs0bq35Zaypa2s0tshG1C0tn9MWwRSuLZyZrKp/N6BlfqESrH59W8ImOwrnDrK2+tcxXH9ix9rY YDTT6cWq7jh39UBLm0yEneB1/RfOhqWod5SqbSlhoPvKP0fNz+cJHGT954zfLTyCRf5Aad12KEb2 bY06WRG4cob1yYMn4Hc5mD2z2qNERamkVElJSSn4mW0JkP5KV0T6S115mL9t1xQW43x7+/Zt9O/f X3UsQRCEMeDj4wM3NzeYmJiA/yKXb0UT43wrD3PkYTnytCrXrl0zmPhzxDDfisIuxpXEn8O32dnZ JP4EQRgV5Sr+THgNJv58Kw/LBV/ciibGAwICSPwJgjAqylP8K8yavzTjBEEQxo6hNfGZij+JPUEQ hG6UtV6Wu/iT4BMEQZSOstDRCrPsQxAEQZQfJP4EQRBGyDMRf1r6IQiCKBllpZ808ycIgjBCKoT4 05MAQRCEMobSR5r5EwRBGCEVUvzpSYAgCGOlvPTvmYm/UgF5Ggk/QRDGjiYtLEt9fOYzfxJ7giAI 7RhCJw0u/rpmmgYBgiCIwhhSP5/pzF/MsDTjPFySghAEQVQG5BoohstaF8tV/JUyLy2YdH+NGjWE EEEQROUnNTUVpqYF/5+cVBOl2iiilKYPBv17/hweFuNiWDTp3++Xb+Pj4xEVFaU6jiAIorLDJ7z8 P7Fq1apVkb/fr/T3/EXjSMMcaVhEnlYhxF8a5ltpWGmrZOL5pVuONEwQBGEolMRXulUyuagHBQWh devWqriS4EvDoknPLyINi8jTDCL+HDGdb+VhqUkFXRfRl4ZFk55bRBrmyOMEQRBlQRFRlYmwGBfD oimJ+q1bt9CmTRudhZ+b9NxiWAl5eqnFn6N0CjFNutVmugi/Ulx6bjEsIg1rQ1c/giCME02CKkfq JxVjeZibXMx5/ObNm2jbtm2h/ZpEX2riuaVbKYppTPhKrXxKpxDTpFt5mJtUzJXCcrEXjafLzyUi DXPkcYIgiLJELq7SOA+LxtEk5nLxF9OUwtw48rB0K0UxjQljqZVR6RTSNB4W42JYbsUNAsWZiBiW pmlCFx+CIAgRJRGVoyTCPFycieIvXfYR06V+UpOeW0QaFlFMYwJYJgqodBoxTbotzuRiX5z4S88r Ig1L0ZROEARREpRElSMXYzEuhpVMKv5iXO6jZOJ5pVspSmmcZyL+0rDclMReniY/XoyLSMMceZwg CKIskQusNC4VZtHkcVHodRV/6fFiWLqVopTGKRfx5/CwGBfDpTXxXEpbEXmcIAjCEMhFVpMw860m k4p/cSY9l4g0LKKUxikz8edoE14xzLfysL4mPVYMS7ci8jhBEIQhkQutGJdu5WGp3bhxo9ALXyWT HiuGRaRhjjwuxaDizxHTlLaiyePaTOorhqVbEXmcIAiiPNAkwNKtPCyaJvGX+4pxpa0UpTSRchN/ jhiWbkWTx+Um3S+GlbYcaZggCKK8kYquJpHmW3lYSfyl+6Vx6ZYjDYsopYkYXPw5Yrq2rWi6xpW2 IvK4FG37CIIgdEWrsMr2iXGlrWgcqfhz5PuL28rRlM4pV/HniGGlrTysaZ90y1FKU6K4/QRBEPqg TVw5SuIsT+NbMaxJ/MWw0pYjDUvRlM4pU/HnaDqdNF0MK201hbVtOdKwFE3pBEEQZYkuAiyGNW0D AwPh5uamCvO04vw50rAUTekiz1T8OWJYulVKU9pypGGOPE4QBPEskIuvkmDLt1z8pWv+HPmWoyks RVO6SJmLP0fTKaXpSuHithylNI48rg19fAmCIOQUJ6xS5L5iXJouhqXiz9Hmy5GGpWhKl1Ku4s+R 7isuLN9ypGGOPM5RSiMIgigvlMRXnqYk4nLx5+gSlqNtn4hBxJ+j7bTSfaUJc+RxTejqRxAEoQ+6 CC1H7qck5NevX89f8+co+XDk55KibZ8Ug4k/R9up5fukcU1hTnFxXSjJMQRBECK6CqwU+TFKcS7+ 2mb+HHlcirZ9cgwq/hxtp5fv0zcuou0aBEEQzxpNoixPF5d9RAwl/ByDiz9H2yWU9snT9D2eIAii oqGLcCvN/EXKUvg5Va5du1Yu6lmcSOsj4iT4BEE8z2gTa+maP6c4YS+J8HOqpKSklJuS6iLamnxK cyxBEMSzQBdhlvqEhoYW+pFXcZRU+DnlKv4cXQW6rP0IgiCeJboINRf/du3aCTHtlEb4AeD/ATCr jnT2JSGYAAAAAElFTkSuQmCC --0000000000006d508e05db44ff41-- --===============1701454019224294576== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users --===============1701454019224294576==--
