We are also facing following issue which is mentioned in case on openshift origin while we are creating pvc for pods. (Please provide workaround to move further (pod restart doesn't workout)
https://bugzilla.redhat.com/show_bug.cgi?id=1630117
https://bugzilla.redhat.com/show_bug.cgi?id=1636912
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume Server busy. Retry
operation later..
at a time to create only one volume when in-flight operations are zero. Once volume requested it reaches to 8.
Now single volume not able
to create and we are till now mostly 10 volumes are created.
Please find heketidb dump and log
[negroni] Completed 200 OK in 98.699µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 106.654µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 185.406µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 102.664µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 192.658µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 198.611µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 124.254µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 101.491µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 116.997µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 100.171µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 109.238µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/28 06:50:57 operations in-flight (8) exceeds limit
(8)
[negroni] Completed 429 Too Many Requests in 191.118µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 188.791µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 94.436µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 110.893µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 112.132µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 96.15µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 112.682µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 140.543µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 182.066µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 151.572µs
BR
Salam
From:
Shaik Salam/HYD/TCS
To:
rtalur@xxxxxxxxxx
Cc:
"gluster-users@xxxxxxxxxxx
List" <gluster-users@xxxxxxxxxxx>, "John Mulligan"
<jmulliga@xxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date:
02/08/2019 12:05 PM
Subject:
Failed to provision
volume with StorageClass "glusterfs-storage": glusterfs: server
busy
Hi Raghavendra,
We are also facing following issue which is mentioned in case on openshift origin while we are creating pvc for pods. (Please provide workaround to move further (pod restart doesn't workout)
https://bugzilla.redhat.com/show_bug.cgi?id=1630117
https://bugzilla.redhat.com/show_bug.cgi?id=1636912
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume Server busy. Retry
operation later..
at a time to create only one volume when in-flight operations are zero. Once volume requested it reaches to 8.
Now single volume not able
to create and we are till now mostly 10 volumes are created.
Please find heketidb dump and log
[negroni] Completed 200 OK in 98.699µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 106.654µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 185.406µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 102.664µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 192.658µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 198.611µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 124.254µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 101.491µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 116.997µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 100.171µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 109.238µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/28 06:50:57 operations in-flight (8) exceeds limit
(8)
[negroni] Completed 429 Too Many Requests in 191.118µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 188.791µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 94.436µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 110.893µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 112.132µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 96.15µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 112.682µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 140.543µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 182.066µs
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6
[negroni] Completed 200 OK in 151.572µs
BR
Salam
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
[root@master ~]# oc logs heketi-storage-6-8dcdn
Setting up heketi database
File: /var/lib/heketi/heketi.db
Size: 262144 Blocks: 512 IO Block: 131072 regular file
Device: 200054h/2097236d Inode: 13717556549865401166 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-01-21 14:17:53.065291018 +0000
Modify: 2019-02-14 06:20:40.547041647 +0000
Change: 2019-02-14 06:20:40.558041722 +0000
Birth: -
Heketi v8.0.0-1-g082b556-release-8
[heketi] INFO 2019/02/14 07:12:08 Loaded kubernetes executor
[heketi] ERROR 2019/02/14 07:12:08 /src/github.com/heketi/heketi/apps/glusterfs/app.go:177: Heketi was terminated while performing one or more operations. Server may refuse to start as long as pending operations are present in the db.
[heketi] INFO 2019/02/14 07:12:08 Please refer to the Heketi troubleshooting documentation for more information on how to resolve this issue.
[heketi] WARNING 2019/02/14 07:12:08 Ignoring stale pending operations.Server will be running with incomplete/inconsistent state in DB.
[heketi] INFO 2019/02/14 07:12:08 Block: Auto Create Block Hosting Volume set to true
[heketi] INFO 2019/02/14 07:12:08 Block: New Block Hosting Volume size 100 GB
[heketi] INFO 2019/02/14 07:12:08 GlusterFS Application Loaded
[heketi] INFO 2019/02/14 07:12:08 Started Node Health Cache Monitor
Authorization loaded
Listening on port 8080
[heketi] INFO 2019/02/14 07:12:18 Starting Node Health Status refresh
[cmdexec] INFO 2019/02/14 07:12:18 Check Glusterd service status in node storage.matrix.nokia.com
[kubeexec] DEBUG 2019/02/14 07:12:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: storage.matrix.nokia.com Pod: glusterfs-storage-9h5cf Command: systemctl status glusterd
Result: ? glusterd.service - GlusterFS, a clustered file-system server
Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-01-18 13:27:57 UTC; 3 weeks 5 days ago
Process: 23951 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 23952 (glusterd)
CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b11242_cd44_11e8_a361_5254007b74fe.slice/docker-ce7cfe74053c991b94bda7e05b33316da1af1ce519ce170748f1ece89f266143.scope/system.slice/glusterd.service
+- 558 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id heketidbstorage.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_2b6ed29607d007a75f7a7c1bf0da7bae-brick -p /var/run/gluster/vols/heketidbstorage/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_2b6ed29607d007a75f7a7c1bf0da7bae-brick.pid -S /var/run/gluster/88c0da28ee877a32.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_2b6ed29607d007a75f7a7c1bf0da7bae/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_2b6ed29607d007a75f7a7c1bf0da7bae-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154
+- 901 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id glusterfs-registry-volume.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_b22c3ab4d7bbc420e05340e8e187bcf5-brick -p /var/run/gluster/vols/glusterfs-registry-volume/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_b22c3ab4d7bbc420e05340e8e187bcf5-brick.pid -S /var/run/gluster/b64892014146a0d5.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_b22c3ab4d7bbc420e05340e8e187bcf5/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_b22c3ab4d7bbc420e05340e8e187bcf5-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49155 --xlator-option glusterfs-registry-volume-server.listen-port=49155
+- 2130 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_2e2e6b9cf174901b370ea79a266c651b.192.168.89.220.var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_62c4432fc7a6e4aeb11573e22eaf53d8-brick -p /var/run/gluster/vols/vol_2e2e6b9cf174901b370ea79a266c651b/192.168.89.220-var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_62c4432fc7a6e4aeb11573e22eaf53d8-brick.pid -S /var/run/gluster/5ab438223cc6750c.socket --brick-name /var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_62c4432fc7a6e4aeb11573e22eaf53d8/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_62c4432fc7a6e4aeb11573e22eaf53d8-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49161 --xlator-option vol_2e2e6b9cf174901b370ea79a266c651b-server.listen-port=49161
+- 2713 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_c5fa076446479cf414397591c0af1c7f.192.168.89.220.var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_ac7b22198b6424785edd32d6f0a078d2-brick -p /var/run/gluster/vols/vol_c5fa076446479cf414397591c0af1c7f/192.168.89.220-var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_ac7b22198b6424785edd32d6f0a078d2-brick.pid -S /var/run/gluster/b586da88dc5b2118.socket --brick-name /var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_ac7b22198b6424785edd32d6f0a078d2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_ac7b22198b6424785edd32d6f0a078d2-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49162 --xlator-option vol_c5fa076446479cf414397591c0af1c7f-server.listen-port=49162
+- 9064 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_bc5012fd39c5b3ab958b9da4b0256d3a.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_fded2411a02e03c7e0219eb9a2b08c9a-brick -p /var/run/gluster/vols/vol_bc5012fd39c5b3ab958b9da4b0256d3a/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_fded2411a02e03c7e0219eb9a2b08c9a-brick.pid -S /var/run/gluster/8b6fba20e4ec888e.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_fded2411a02e03c7e0219eb9a2b08c9a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_fded2411a02e03c7e0219eb9a2b08c9a-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49166 --xlator-option vol_bc5012fd39c5b3ab958b9da4b0256d3a-server.listen-port=49166
+- 23952 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
+- 32058 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/06a8fe3d13500a44.socket --xlator-option *replicate*.node-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name glustershd
+- 71770 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_88fbf36bf87106e549dca765f171cf69.192.168.89.220.var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_22982c432dd089512ab2bde1e0663f9a-brick -p /var/run/gluster/vols/vol_88fbf36bf87106e549dca765f171cf69/192.168.89.220-var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_22982c432dd089512ab2bde1e0663f9a-brick.pid -S /var/run/gluster/adbe241d384df1c2.socket --brick-name /var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_22982c432dd089512ab2bde1e0663f9a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_22982c432dd089512ab2bde1e0663f9a-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49167 --xlator-option vol_88fbf36bf87106e549dca765f171cf69-server.listen-port=49167
+- 72776 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_47c2af6849ac9ad7d7fbc897ae3ae80c.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_025b1bf4bbabea839ef139c4453f9bcd-brick -p /var/run/gluster/vols/vol_47c2af6849ac9ad7d7fbc897ae3ae80c/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_025b1bf4bbabea839ef139c4453f9bcd-brick.pid -S /var/run/gluster/e6f521dfc39ee69b.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_025b1bf4bbabea839ef139c4453f9bcd/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_025b1bf4bbabea839ef139c4453f9bcd-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49158 --xlator-option vol_47c2af6849ac9ad7d7fbc897ae3ae80c-server.listen-port=49158
+- 82920 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_63428c902b5d577a7739eb66a050a420.192.168.89.220.var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_85c1670a17f956abd0f3dd99cad53a7e-brick -p /var/run/gluster/vols/vol_63428c902b5d577a7739eb66a050a420/192.168.89.220-var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_85c1670a17f956abd0f3dd99cad53a7e-brick.pid -S /var/run/gluster/c82b614596a19c6c.socket --brick-name /var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_85c1670a17f956abd0f3dd99cad53a7e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_85c1670a17f956abd0f3dd99cad53a7e-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49171 --xlator-option vol_63428c902b5d577a7739eb66a050a420-server.listen-port=49171
+- 86318 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_937aa673e0d60126082bbd8c1589e383.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_275d68b56cfc42c8b46aa90eb227eb25-brick -p /var/run/gluster/vols/vol_937aa673e0d60126082bbd8c1589e383/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_275d68b56cfc42c8b46aa90eb227eb25-brick.pid -S /var/run/gluster/5ae55f9a21c8e488.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_275d68b56cfc42c8b46aa90eb227eb25/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_275d68b56cfc42c8b46aa90eb227eb25-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49172 --xlator-option vol_937aa673e0d60126082bbd8c1589e383-server.listen-port=49172
+- 87342 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_62905ab0e959e7663f4576501c0c9b69.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_9684d9e56109e011585af33ecd8039cc-brick -p /var/run/gluster/vols/vol_62905ab0e959e7663f4576501c0c9b69/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_9684d9e56109e011585af33ecd8039cc-brick.pid -S /var/run/gluster/e63ba38f2720a00f.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_9684d9e56109e011585af33ecd8039cc/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_9684d9e56109e011585af33ecd8039cc-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49157 --xlator-option vol_62905ab0e959e7663f4576501c0c9b69-server.listen-port=49157
+- 97945 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_dcee4f7f020d47e58fb48612bbba19d1.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_16de636f8337f47a87996fca50313260-brick -p /var/run/gluster/vols/vol_dcee4f7f020d47e58fb48612bbba19d1/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_16de636f8337f47a87996fca50313260-brick.pid -S /var/run/gluster/5bbb028f813f880f.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_16de636f8337f47a87996fca50313260/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_16de636f8337f47a87996fca50313260-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49168 --xlator-option vol_dcee4f7f020d47e58fb48612bbba19d1-server.listen-port=49168
+-103617 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_5101331e9d33a5d04adab92837b9d5ad.192.168.89.220.var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_079acb5b4a0b0f3c45ed87f9c7def14b-brick -p /var/run/gluster/vols/vol_5101331e9d33a5d04adab92837b9d5ad/192.168.89.220-var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_079acb5b4a0b0f3c45ed87f9c7def14b-brick.pid -S /var/run/gluster/20aa83a13cec2563.socket --brick-name /var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_079acb5b4a0b0f3c45ed87f9c7def14b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_079acb5b4a0b0f3c45ed87f9c7def14b-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49159 --xlator-option vol_5101331e9d33a5d04adab92837b9d5ad-server.listen-port=49159
+-103808 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_2e6ad6aa03f7fe219807cb135ca1c766.192.168.89.220.var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_2fd555bc1cbd1b49ef04e7b88d34297b-brick -p /var/run/gluster/vols/vol_2e6ad6aa03f7fe219807cb135ca1c766/192.168.89.220-var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_2fd555bc1cbd1b49ef04e7b88d34297b-brick.pid -S /var/run/gluster/a375f4b3a8cba7c0.socket --brick-name /var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_2fd555bc1cbd1b49ef04e7b88d34297b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_2fd555bc1cbd1b49ef04e7b88d34297b-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49163 --xlator-option vol_2e6ad6aa03f7fe219807cb135ca1c766-server.listen-port=49163
+-103990 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_78d426889b7bd661f3d6c6f7814b6d4f.192.168.89.220.var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_8dba95329e797beb06eca079b9ae6a4b-brick -p /var/run/gluster/vols/vol_78d426889b7bd661f3d6c6f7814b6d4f/192.168.89.220-var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_8dba95329e797beb06eca079b9ae6a4b-brick.pid -S /var/run/gluster/8d3f3f80d9e62092.socket --brick-name /var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_8dba95329e797beb06eca079b9ae6a4b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_8dba95329e797beb06eca079b9ae6a4b-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49164 --xlator-option vol_78d426889b7bd661f3d6c6f7814b6d4f-server.listen-port=49164
+-105270 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_9f04d82be19e2ea8ee80deb9098cd390.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_0bd819f4969e7adbba32e2c77afc202c-brick -p /var/run/gluster/vols/vol_9f04d82be19e2ea8ee80deb9098cd390/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_0bd819f4969e7adbba32e2c77afc202c-brick.pid -S /var/run/gluster/0882e30a8d5b1272.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_0bd819f4969e7adbba32e2c77afc202c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_0bd819f4969e7adbba32e2c77afc202c-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49165 --xlator-option vol_9f04d82be19e2ea8ee80deb9098cd390-server.listen-port=49165
+-121771 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_7e73769a754d7f9f6213ef3b4551af0e.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_c2fe3a63636192e85d8adeefff18c92b-brick -p /var/run/gluster/vols/vol_7e73769a754d7f9f6213ef3b4551af0e/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_c2fe3a63636192e85d8adeefff18c92b-brick.pid -S /var/run/gluster/130cc033b68aedd6.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_c2fe3a63636192e85d8adeefff18c92b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_c2fe3a63636192e85d8adeefff18c92b-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49156 --xlator-option vol_7e73769a754d7f9f6213ef3b4551af0e-server.listen-port=49156
+-130924 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_dbc601f7162df326784aea34c4ebe8f2.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_4fa8a73223d69de183d8e58618722084-brick -p /var/run/gluster/vols/vol_dbc601f7162df326784aea34c4ebe8f2/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_4fa8a73223d69de183d8e58618722084-brick.pid -S /var/run/gluster/88109cbb53cf4280.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_4fa8a73223d69de183d8e58618722084/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_4fa8a73223d69de183d8e58618722084-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49160 --xlator-option vol_dbc601f7162df326784aea34c4ebe8f2-server.listen-port=49160
Jan 18 13:27:56 storage.matrix.nokia.com systemd[1]: Starting GlusterFS, a clustered file-system server...
Jan 18 13:27:57 storage.matrix.nokia.com systemd[1]: Started GlusterFS, a clustered file-system server.
[heketi] INFO 2019/02/14 07:12:18 Periodic health check status: node 5f090e0aa757345d22ebdb15897bd68a up=true
[cmdexec] INFO 2019/02/14 07:12:18 Check Glusterd service status in node app1.matrix.nokia.com
[kubeexec] DEBUG 2019/02/14 07:12:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: app1.matrix.nokia.com Pod: glusterfs-storage-31fjz Command: systemctl status glusterd
Result: ? glusterd.service - GlusterFS, a clustered file-system server
Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2019-02-03 10:05:10 UTC; 1 weeks 3 days ago
Process: 198 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 199 (glusterd)
CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b10041_cd44_11e8_a361_5254007b74fe.slice/docker-b96fad2fc728c53d50ae7c4bc5a83b155d21566fd3b3877d4bd364513804907d.scope/system.slice/glusterd.service
+-199 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
+-238 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/46eb4bfe4652baa1.socket --xlator-option *replicate*.node-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name glustershd
+-282 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id glusterfs-registry-volume.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_7ef53adf2bce0e240229941ac7f43661-brick -p /var/run/gluster/vols/glusterfs-registry-volume/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_7ef53adf2bce0e240229941ac7f43661-brick.pid -S /var/run/gluster/f223a3f7986d97b5.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_7ef53adf2bce0e240229941ac7f43661/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_7ef53adf2bce0e240229941ac7f43661-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49152 --xlator-option glusterfs-registry-volume-server.listen-port=49152
+-291 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id heketidbstorage.192.168.89.218.var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_79e211c7bee7d61bfdabb0add8e42b33-brick -p /var/run/gluster/vols/heketidbstorage/192.168.89.218-var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_79e211c7bee7d61bfdabb0add8e42b33-brick.pid -S /var/run/gluster/f905b930c13d9816.socket --brick-name /var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_79e211c7bee7d61bfdabb0add8e42b33/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_79e211c7bee7d61bfdabb0add8e42b33-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153
+-306 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_2e2e6b9cf174901b370ea79a266c651b.192.168.89.218.var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_6f738c7cc82f66dbda8bc0b6d74883c0-brick -p /var/run/gluster/vols/vol_2e2e6b9cf174901b370ea79a266c651b/192.168.89.218-var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_6f738c7cc82f66dbda8bc0b6d74883c0-brick.pid -S /var/run/gluster/ac938e354448789e.socket --brick-name /var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_6f738c7cc82f66dbda8bc0b6d74883c0/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_6f738c7cc82f66dbda8bc0b6d74883c0-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49154 --xlator-option vol_2e2e6b9cf174901b370ea79a266c651b-server.listen-port=49154
+-321 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_2e6ad6aa03f7fe219807cb135ca1c766.192.168.89.218.var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_980bd302213bbdd2d9aa09dbe8abf62b-brick -p /var/run/gluster/vols/vol_2e6ad6aa03f7fe219807cb135ca1c766/192.168.89.218-var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_980bd302213bbdd2d9aa09dbe8abf62b-brick.pid -S /var/run/gluster/db1300311bc7697d.socket --brick-name /var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_980bd302213bbdd2d9aa09dbe8abf62b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_980bd302213bbdd2d9aa09dbe8abf62b-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49155 --xlator-option vol_2e6ad6aa03f7fe219807cb135ca1c766-server.listen-port=49155
+-330 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_47c2af6849ac9ad7d7fbc897ae3ae80c.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_1f52101af46329a1d5e758156c8267e0-brick -p /var/run/gluster/vols/vol_47c2af6849ac9ad7d7fbc897ae3ae80c/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_1f52101af46329a1d5e758156c8267e0-brick.pid -S /var/run/gluster/17890cacc5ba5074.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_1f52101af46329a1d5e758156c8267e0/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_1f52101af46329a1d5e758156c8267e0-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49156 --xlator-option vol_47c2af6849ac9ad7d7fbc897ae3ae80c-server.listen-port=49156
+-339 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_5101331e9d33a5d04adab92837b9d5ad.192.168.89.218.var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_17f8162940314babf2b7d75976890130-brick -p /var/run/gluster/vols/vol_5101331e9d33a5d04adab92837b9d5ad/192.168.89.218-var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_17f8162940314babf2b7d75976890130-brick.pid -S /var/run/gluster/6d420a71217c335e.socket --brick-name /var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_17f8162940314babf2b7d75976890130/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_17f8162940314babf2b7d75976890130-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49157 --xlator-option vol_5101331e9d33a5d04adab92837b9d5ad-server.listen-port=49157
+-348 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_62905ab0e959e7663f4576501c0c9b69.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_31ed93bf445cf1fd00d34660a80e1078-brick -p /var/run/gluster/vols/vol_62905ab0e959e7663f4576501c0c9b69/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_31ed93bf445cf1fd00d34660a80e1078-brick.pid -S /var/run/gluster/73b2ef5a8bf5baa1.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_31ed93bf445cf1fd00d34660a80e1078/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_31ed93bf445cf1fd00d34660a80e1078-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49158 --xlator-option vol_62905ab0e959e7663f4576501c0c9b69-server.listen-port=49158
+-357 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_63428c902b5d577a7739eb66a050a420.192.168.89.218.var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_586c05c5ce5bd7474cfb9a9cd190225c-brick -p /var/run/gluster/vols/vol_63428c902b5d577a7739eb66a050a420/192.168.89.218-var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_586c05c5ce5bd7474cfb9a9cd190225c-brick.pid -S /var/run/gluster/27400e8d3608aa83.socket --brick-name /var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_586c05c5ce5bd7474cfb9a9cd190225c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_586c05c5ce5bd7474cfb9a9cd190225c-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49159 --xlator-option vol_63428c902b5d577a7739eb66a050a420-server.listen-port=49159
+-366 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_78d426889b7bd661f3d6c6f7814b6d4f.192.168.89.218.var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_d3230ee5de8fe8300f47eb4bcea6f2e2-brick -p /var/run/gluster/vols/vol_78d426889b7bd661f3d6c6f7814b6d4f/192.168.89.218-var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_d3230ee5de8fe8300f47eb4bcea6f2e2-brick.pid -S /var/run/gluster/87c81a7cb1ad328b.socket --brick-name /var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_d3230ee5de8fe8300f47eb4bcea6f2e2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_d3230ee5de8fe8300f47eb4bcea6f2e2-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49160 --xlator-option vol_78d426889b7bd661f3d6c6f7814b6d4f-server.listen-port=49160
+-375 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_7e73769a754d7f9f6213ef3b4551af0e.192.168.89.218.var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_84e395f5464cd25dce66b8e60f395982-brick -p /var/run/gluster/vols/vol_7e73769a754d7f9f6213ef3b4551af0e/192.168.89.218-var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_84e395f5464cd25dce66b8e60f395982-brick.pid -S /var/run/gluster/c7917e048093731a.socket --brick-name /var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_84e395f5464cd25dce66b8e60f395982/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_84e395f5464cd25dce66b8e60f395982-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49161 --xlator-option vol_7e73769a754d7f9f6213ef3b4551af0e-server.listen-port=49161
+-384 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_937aa673e0d60126082bbd8c1589e383.192.168.89.218.var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_5f22e9906d3566116ec474bb618213e5-brick -p /var/run/gluster/vols/vol_937aa673e0d60126082bbd8c1589e383/192.168.89.218-var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_5f22e9906d3566116ec474bb618213e5-brick.pid -S /var/run/gluster/926dadf68659a463.socket --brick-name /var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_5f22e9906d3566116ec474bb618213e5/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_5f22e9906d3566116ec474bb618213e5-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49162 --xlator-option vol_937aa673e0d60126082bbd8c1589e383-server.listen-port=49162
+-393 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_9f04d82be19e2ea8ee80deb9098cd390.192.168.89.218.var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_281606b3024ac98416fedee94528e3c7-brick -p /var/run/gluster/vols/vol_9f04d82be19e2ea8ee80deb9098cd390/192.168.89.218-var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_281606b3024ac98416fedee94528e3c7-brick.pid -S /var/run/gluster/a3fd18961e1f360b.socket --brick-name /var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_281606b3024ac98416fedee94528e3c7/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_281606b3024ac98416fedee94528e3c7-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49163 --xlator-option vol_9f04d82be19e2ea8ee80deb9098cd390-server.listen-port=49163
+-402 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_bc5012fd39c5b3ab958b9da4b0256d3a.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_08599e162e105063a8e2efec868d2a76-brick -p /var/run/gluster/vols/vol_bc5012fd39c5b3ab958b9da4b0256d3a/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_08599e162e105063a8e2efec868d2a76-brick.pid -S /var/run/gluster/42ae4fefd1cb6554.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_08599e162e105063a8e2efec868d2a76/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_08599e162e105063a8e2efec868d2a76-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49164 --xlator-option vol_bc5012fd39c5b3ab958b9da4b0256d3a-server.listen-port=49164
+-411 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_c5fa076446479cf414397591c0af1c7f.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_ae2fa0410160c9fb489af8e5ad063ac3-brick -p /var/run/gluster/vols/vol_c5fa076446479cf414397591c0af1c7f/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_ae2fa0410160c9fb489af8e5ad063ac3-brick.pid -S /var/run/gluster/8bb099144ac69c55.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_ae2fa0410160c9fb489af8e5ad063ac3/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_ae2fa0410160c9fb489af8e5ad063ac3-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49165 --xlator-option vol_c5fa076446479cf414397591c0af1c7f-server.listen-port=49165
+-420 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_dbc601f7162df326784aea34c4ebe8f2.192.168.89.218.var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_ff62a0d3e0fe5c53c3725235d2832893-brick -p /var/run/gluster/vols/vol_dbc601f7162df326784aea34c4ebe8f2/192.168.89.218-var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_ff62a0d3e0fe5c53c3725235d2832893-brick.pid -S /var/run/gluster/53bb40a037c87c29.socket --brick-name /var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_ff62a0d3e0fe5c53c3725235d2832893/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_ff62a0d3e0fe5c53c3725235d2832893-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49166 --xlator-option vol_dbc601f7162df326784aea34c4ebe8f2-server.listen-port=49166
+-429 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_dcee4f7f020d47e58fb48612bbba19d1.192.168.89.218.var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_800e2a99110af93ca69bd950c9ea7ef8-brick -p /var/run/gluster/vols/vol_dcee4f7f020d47e58fb48612bbba19d1/192.168.89.218-var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_800e2a99110af93ca69bd950c9ea7ef8-brick.pid -S /var/run/gluster/fbf6e334c35a83bb.socket --brick-name /var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_800e2a99110af93ca69bd950c9ea7ef8/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_800e2a99110af93ca69bd950c9ea7ef8-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49167 --xlator-option vol_dcee4f7f020d47e58fb48612bbba19d1-server.listen-port=49167
Feb 03 10:05:08 app1.matrix.nokia.com systemd[1]: Starting GlusterFS, a clustered file-system server...
Feb 03 10:05:10 app1.matrix.nokia.com systemd[1]: Started GlusterFS, a clustered file-system server.
[heketi] INFO 2019/02/14 07:12:18 Periodic health check status: node b72660bc8e28725f612d33bd675f8341 up=true
[cmdexec] INFO 2019/02/14 07:12:18 Check Glusterd service status in node app2.matrix.nokia.com
[negroni] Started POST /volumes
[heketi] INFO 2019/02/14 07:12:40 Allocating brick set #0
[negroni] Started POST /volumes
[negroni] Started GET /clusters
[negroni] Completed 401 Unauthorized in 119.867µs
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 449.406µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 322.589µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started GET /db/dump
[negroni] Completed 200 OK in 7.607085ms
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[heketi] WARNING 2019/02/14 07:17:41 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 250.4µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/02/14 07:17:56 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 191.165µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/02/14 07:18:11 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 339.575µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/02/14 07:18:26 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 200.437µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/02/14 07:18:41 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 456.234µs{"clusterentries":{"e66ce2684350cb023c36d63b3d2bbf6f":{"Info":{"id":"e66ce2684350cb023c36d63b3d2bbf6f","nodes":["5f090e0aa757345d22ebdb15897bd68a","b72660bc8e28725f612d33bd675f8341","f829fc134bc9f983a2865cc0f46a31a6"],"volumes":["2e2e6b9cf174901b370ea79a266c651b","2e6ad6aa03f7fe219807cb135ca1c766","47c2af6849ac9ad7d7fbc897ae3ae80c","5101331e9d33a5d04adab92837b9d5ad","62905ab0e959e7663f4576501c0c9b69","63428c902b5d577a7739eb66a050a420","78d426889b7bd661f3d6c6f7814b6d4f","7e73769a754d7f9f6213ef3b4551af0e","83dcb62de26517790f155b51526cace5","88fbf36bf87106e549dca765f171cf69","937aa673e0d60126082bbd8c1589e383","9f04d82be19e2ea8ee80deb9098cd390","bc5012fd39c5b3ab958b9da4b0256d3a","c3b5f711128418ce9a7c68b514e34618","c5fa076446479cf414397591c0af1c7f","d7a1642b3bb2f1b98c7eee66c4fc393b","dbc601f7162df326784aea34c4ebe8f2","dcee4f7f020d47e58fb48612bbba19d1"],"block":true,"file":true,"blockvolumes":[]}}},"volumeentries":{"2e2e6b9cf174901b370ea79a266c651b":{"Info":{"size":10,"name":"vol_2e2e6b9cf174901b370ea79a266c651b","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2004,"snapshot":{"enable":false,"factor":1},"id":"2e2e6b9cf174901b370ea79a266c651b","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_2e2e6b9cf174901b370ea79a266c651b","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["62c4432fc7a6e4aeb11573e22eaf53d8","6f738c7cc82f66dbda8bc0b6d74883c0","c607b4aed1593917939ba85df5eefaae"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"2e6ad6aa03f7fe219807cb135ca1c766":{"Info":{"size":70,"name":"vol_2e6ad6aa03f7fe219807cb135ca1c766","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2007,"snapshot":{"enable":false,"factor":1},"id":"2e6ad6aa03f7fe219807cb135ca1c766","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_2e6ad6aa03f7fe219807cb135ca1c766","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["2fd555bc1cbd1b49ef04e7b88d34297b","980bd302213bbdd2d9aa09dbe8abf62b","fb2e1c7f95f9926166c585ca16ac2402"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"47c2af6849ac9ad7d7fbc897ae3ae80c":{"Info":{"size":50,"name":"vol_47c2af6849ac9ad7d7fbc897ae3ae80c","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2001,"snapshot":{"enable":false,"factor":1},"id":"47c2af6849ac9ad7d7fbc897ae3ae80c","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_47c2af6849ac9ad7d7fbc897ae3ae80c","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["025b1bf4bbabea839ef139c4453f9bcd","1f52101af46329a1d5e758156c8267e0","1f69ed2975c20811515f054a74669be6"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"5101331e9d33a5d04adab92837b9d5ad":{"Info":{"size":70,"name":"vol_5101331e9d33a5d04adab92837b9d5ad","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2006,"snapshot":{"enable":false,"factor":1},"id":"5101331e9d33a5d04adab92837b9d5ad","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_5101331e9d33a5d04adab92837b9d5ad","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["079acb5b4a0b0f3c45ed87f9c7def14b","17f8162940314babf2b7d75976890130","413e08918a40ce6806dd6c2711d923f3"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"62905ab0e959e7663f4576501c0c9b69":{"Info":{"size":1,"name":"vol_62905ab0e959e7663f4576501c0c9b69","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2002,"snapshot":{"enable":false,"factor":1},"id":"62905ab0e959e7663f4576501c0c9b69","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_62905ab0e959e7663f4576501c0c9b69","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["31ed93bf445cf1fd00d34660a80e1078","9684d9e56109e011585af33ecd8039cc","d343f2a8b3c19bda521f93d6327451f2"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"63428c902b5d577a7739eb66a050a420":{"Info":{"size":25,"name":"vol_63428c902b5d577a7739eb66a050a420","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2013,"snapshot":{"enable":false,"factor":1},"id":"63428c902b5d577a7739eb66a050a420","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_63428c902b5d577a7739eb66a050a420","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["3c16802735df66a463468ce8262ef0a6","586c05c5ce5bd7474cfb9a9cd190225c","85c1670a17f956abd0f3dd99cad53a7e"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"78d426889b7bd661f3d6c6f7814b6d4f":{"Info":{"size":70,"name":"vol_78d426889b7bd661f3d6c6f7814b6d4f","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2008,"snapshot":{"enable":false,"factor":1},"id":"78d426889b7bd661f3d6c6f7814b6d4f","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_78d426889b7bd661f3d6c6f7814b6d4f","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["28b3599a579db6e987f46ab0120f9d8c","8dba95329e797beb06eca079b9ae6a4b","d3230ee5de8fe8300f47eb4bcea6f2e2"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"7e73769a754d7f9f6213ef3b4551af0e":{"Info":{"size":5,"name":"vol_7e73769a754d7f9f6213ef3b4551af0e","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{"data":4,"redundancy":2}},"snapshot":{"enable":false,"factor":1},"id":"7e73769a754d7f9f6213ef3b4551af0e","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_7e73769a754d7f9f6213ef3b4551af0e","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["774b52d13a2cf996784181cf6d7db93c","84e395f5464cd25dce66b8e60f395982","c2fe3a63636192e85d8adeefff18c92b"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"83dcb62de26517790f155b51526cace5":{"Info":{"size":200,"name":"glusterfs-registry-volume","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{"data":4,"redundancy":2}},"snapshot":{"enable":false,"factor":1},"id":"83dcb62de26517790f155b51526cace5","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:glusterfs-registry-volume","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["7ef53adf2bce0e240229941ac7f43661","b22c3ab4d7bbc420e05340e8e187bcf5","b85460c3ba53bb7db5ab712f7b684c51"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"88fbf36bf87106e549dca765f171cf69":{"Info":{"size":5,"name":"vol_88fbf36bf87106e549dca765f171cf69","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{"data":4,"redundancy":2}},"snapshot":{"enable":false,"factor":1},"id":"88fbf36bf87106e549dca765f171cf69","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_88fbf36bf87106e549dca765f171cf69","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["22982c432dd089512ab2bde1e0663f9a","e26c7a93f2a7b6d6455ab5fa7615dfb1","ef253aa4b2756fc293aa101c2eb743cb"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"937aa673e0d60126082bbd8c1589e383":{"Info":{"size":20,"name":"vol_937aa673e0d60126082bbd8c1589e383","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2003,"snapshot":{"enable":false,"factor":1},"id":"937aa673e0d60126082bbd8c1589e383","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_937aa673e0d60126082bbd8c1589e383","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["275d68b56cfc42c8b46aa90eb227eb25","5f22e9906d3566116ec474bb618213e5","bb29a705b3f39522b131277d64761033"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"9f04d82be19e2ea8ee80deb9098cd390":{"Info":{"size":20,"name":"vol_9f04d82be19e2ea8ee80deb9098cd390","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2014,"snapshot":{"enable":false,"factor":1},"id":"9f04d82be19e2ea8ee80deb9098cd390","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_9f04d82be19e2ea8ee80deb9098cd390","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["0bd819f4969e7adbba32e2c77afc202c","281606b3024ac98416fedee94528e3c7","9603a0a95036d6362ae2b38bdbb40428"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"bc5012fd39c5b3ab958b9da4b0256d3a":{"Info":{"size":4,"name":"vol_bc5012fd39c5b3ab958b9da4b0256d3a","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2009,"snapshot":{"enable":false,"factor":1},"id":"bc5012fd39c5b3ab958b9da4b0256d3a","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_bc5012fd39c5b3ab958b9da4b0256d3a","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["08599e162e105063a8e2efec868d2a76","c4b582ef0712e8c2d750c97ff443e47c","fded2411a02e03c7e0219eb9a2b08c9a"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"c3b5f711128418ce9a7c68b514e34618":{"Info":{"size":2,"name":"heketidbstorage","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"snapshot":{"enable":false,"factor":1},"id":"c3b5f711128418ce9a7c68b514e34618","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:heketidbstorage","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["2b6ed29607d007a75f7a7c1bf0da7bae","79e211c7bee7d61bfdabb0add8e42b33","ac26b8e8306405623200599d05e87fe2"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"c5fa076446479cf414397591c0af1c7f":{"Info":{"size":10,"name":"vol_c5fa076446479cf414397591c0af1c7f","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2005,"snapshot":{"enable":false,"factor":1},"id":"c5fa076446479cf414397591c0af1c7f","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_c5fa076446479cf414397591c0af1c7f","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["93eb3caff9f179d522b5498b515bf991","ac7b22198b6424785edd32d6f0a078d2","ae2fa0410160c9fb489af8e5ad063ac3"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"d7a1642b3bb2f1b98c7eee66c4fc393b":{"Info":{"size":1,"name":"vol_d7a1642b3bb2f1b98c7eee66c4fc393b","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2011,"snapshot":{"enable":false,"factor":1},"id":"d7a1642b3bb2f1b98c7eee66c4fc393b","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_d7a1642b3bb2f1b98c7eee66c4fc393b","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["11fe695a405dcd4854f088ddc073b4a5","27b1f0d7139161a7b618a0c0e50eed24","4220211c169c178ed693131a02ae7fb0"],"GlusterVolumeOptions":null,"Pending":{"Id":"8192ada38b173ad13b4d5f4b74f09fae"}},"dbc601f7162df326784aea34c4ebe8f2":{"Info":{"size":20,"name":"vol_dbc601f7162df326784aea34c4ebe8f2","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2000,"snapshot":{"enable":false,"factor":1},"id":"dbc601f7162df326784aea34c4ebe8f2","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_dbc601f7162df326784aea34c4ebe8f2","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["4fa8a73223d69de183d8e58618722084","c5d298199cfbc91ebf013707709bdbb6","ff62a0d3e0fe5c53c3725235d2832893"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"dcee4f7f020d47e58fb48612bbba19d1":{"Info":{"size":25,"name":"vol_dcee4f7f020d47e58fb48612bbba19d1","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2010,"snapshot":{"enable":false,"factor":1},"id":"dcee4f7f020d47e58fb48612bbba19d1","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_dcee4f7f020d47e58fb48612bbba19d1","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["16de636f8337f47a87996fca50313260","800e2a99110af93ca69bd950c9ea7ef8","fd6a1a83004a11c0007e9392913aea8f"],"GlusterVolumeOptions":null,"Pending":{"Id":""}}},"brickentries":{"025b1bf4bbabea839ef139c4453f9bcd":{"Info":{"id":"025b1bf4bbabea839ef139c4453f9bcd","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_025b1bf4bbabea839ef139c4453f9bcd/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"47c2af6849ac9ad7d7fbc897ae3ae80c","size":52428800},"TpSize":52428800,"PoolMetadataSize":262144,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"079acb5b4a0b0f3c45ed87f9c7def14b":{"Info":{"id":"079acb5b4a0b0f3c45ed87f9c7def14b","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_079acb5b4a0b0f3c45ed87f9c7def14b/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"5101331e9d33a5d04adab92837b9d5ad","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"08599e162e105063a8e2efec868d2a76":{"Info":{"id":"08599e162e105063a8e2efec868d2a76","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_08599e162e105063a8e2efec868d2a76/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"bc5012fd39c5b3ab958b9da4b0256d3a","size":4194304},"TpSize":4194304,"PoolMetadataSize":24576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"0bd819f4969e7adbba32e2c77afc202c":{"Info":{"id":"0bd819f4969e7adbba32e2c77afc202c","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_0bd819f4969e7adbba32e2c77afc202c/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"9f04d82be19e2ea8ee80deb9098cd390","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"11fe695a405dcd4854f088ddc073b4a5":{"Info":{"id":"11fe695a405dcd4854f088ddc073b4a5","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_11fe695a405dcd4854f088ddc073b4a5/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"d7a1642b3bb2f1b98c7eee66c4fc393b","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":"8192ada38b173ad13b4d5f4b74f09fae"},"LvmThinPool":"tp_11fe695a405dcd4854f088ddc073b4a5","LvmLv":"","SubType":1},"16de636f8337f47a87996fca50313260":{"Info":{"id":"16de636f8337f47a87996fca50313260","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_16de636f8337f47a87996fca50313260/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"dcee4f7f020d47e58fb48612bbba19d1","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"17f8162940314babf2b7d75976890130":{"Info":{"id":"17f8162940314babf2b7d75976890130","path":"/var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_17f8162940314babf2b7d75976890130/brick","device":"127a3d697690febddfa2a41b3a820bde","node":"b72660bc8e28725f612d33bd675f8341","volume":"5101331e9d33a5d04adab92837b9d5ad","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"1f52101af46329a1d5e758156c8267e0":{"Info":{"id":"1f52101af46329a1d5e758156c8267e0","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_1f52101af46329a1d5e758156c8267e0/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"47c2af6849ac9ad7d7fbc897ae3ae80c","size":52428800},"TpSize":52428800,"PoolMetadataSize":262144,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"1f69ed2975c20811515f054a74669be6":{"Info":{"id":"1f69ed2975c20811515f054a74669be6","path":"/var/lib/heketi/mounts/vg_339b50a1c21d42f39d5568dcfbbf8844/brick_1f69ed2975c20811515f054a74669be6/brick","device":"339b50a1c21d42f39d5568dcfbbf8844","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"47c2af6849ac9ad7d7fbc897ae3ae80c","size":52428800},"TpSize":52428800,"PoolMetadataSize":262144,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"22982c432dd089512ab2bde1e0663f9a":{"Info":{"id":"22982c432dd089512ab2bde1e0663f9a","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_22982c432dd089512ab2bde1e0663f9a/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"88fbf36bf87106e549dca765f171cf69","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"275d68b56cfc42c8b46aa90eb227eb25":{"Info":{"id":"275d68b56cfc42c8b46aa90eb227eb25","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_275d68b56cfc42c8b46aa90eb227eb25/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"937aa673e0d60126082bbd8c1589e383","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"27b1f0d7139161a7b618a0c0e50eed24":{"Info":{"id":"27b1f0d7139161a7b618a0c0e50eed24","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_27b1f0d7139161a7b618a0c0e50eed24/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"d7a1642b3bb2f1b98c7eee66c4fc393b","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":"8192ada38b173ad13b4d5f4b74f09fae"},"LvmThinPool":"tp_2a41705745608f49c0510f9910cd49db","LvmLv":"","SubType":1},"281606b3024ac98416fedee94528e3c7":{"Info":{"id":"281606b3024ac98416fedee94528e3c7","path":"/var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_281606b3024ac98416fedee94528e3c7/brick","device":"88c314d3366471a0dd89ca4d5a3c8ef8","node":"b72660bc8e28725f612d33bd675f8341","volume":"9f04d82be19e2ea8ee80deb9098cd390","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"28b3599a579db6e987f46ab0120f9d8c":{"Info":{"id":"28b3599a579db6e987f46ab0120f9d8c","path":"/var/lib/heketi/mounts/vg_06f12f2c23a257117083a6d86d6f4087/brick_28b3599a579db6e987f46ab0120f9d8c/brick","device":"06f12f2c23a257117083a6d86d6f4087","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"78d426889b7bd661f3d6c6f7814b6d4f","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"2b6ed29607d007a75f7a7c1bf0da7bae":{"Info":{"id":"2b6ed29607d007a75f7a7c1bf0da7bae","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_2b6ed29607d007a75f7a7c1bf0da7bae/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"c3b5f711128418ce9a7c68b514e34618","size":2097152},"TpSize":2097152,"PoolMetadataSize":12288,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"2fd555bc1cbd1b49ef04e7b88d34297b":{"Info":{"id":"2fd555bc1cbd1b49ef04e7b88d34297b","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_2fd555bc1cbd1b49ef04e7b88d34297b/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"2e6ad6aa03f7fe219807cb135ca1c766","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"31ed93bf445cf1fd00d34660a80e1078":{"Info":{"id":"31ed93bf445cf1fd00d34660a80e1078","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_31ed93bf445cf1fd00d34660a80e1078/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"62905ab0e959e7663f4576501c0c9b69","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"3c16802735df66a463468ce8262ef0a6":{"Info":{"id":"3c16802735df66a463468ce8262ef0a6","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_3c16802735df66a463468ce8262ef0a6/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"63428c902b5d577a7739eb66a050a420","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"413e08918a40ce6806dd6c2711d923f3":{"Info":{"id":"413e08918a40ce6806dd6c2711d923f3","path":"/var/lib/heketi/mounts/vg_06f12f2c23a257117083a6d86d6f4087/brick_413e08918a40ce6806dd6c2711d923f3/brick","device":"06f12f2c23a257117083a6d86d6f4087","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"5101331e9d33a5d04adab92837b9d5ad","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"4220211c169c178ed693131a02ae7fb0":{"Info":{"id":"4220211c169c178ed693131a02ae7fb0","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_4220211c169c178ed693131a02ae7fb0/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"d7a1642b3bb2f1b98c7eee66c4fc393b","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":"8192ada38b173ad13b4d5f4b74f09fae"},"LvmThinPool":"tp_4220211c169c178ed693131a02ae7fb0","LvmLv":"","SubType":1},"4fa8a73223d69de183d8e58618722084":{"Info":{"id":"4fa8a73223d69de183d8e58618722084","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_4fa8a73223d69de183d8e58618722084/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"dbc601f7162df326784aea34c4ebe8f2","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"586c05c5ce5bd7474cfb9a9cd190225c":{"Info":{"id":"586c05c5ce5bd7474cfb9a9cd190225c","path":"/var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_586c05c5ce5bd7474cfb9a9cd190225c/brick","device":"88c314d3366471a0dd89ca4d5a3c8ef8","node":"b72660bc8e28725f612d33bd675f8341","volume":"63428c902b5d577a7739eb66a050a420","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"5f22e9906d3566116ec474bb618213e5":{"Info":{"id":"5f22e9906d3566116ec474bb618213e5","path":"/var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_5f22e9906d3566116ec474bb618213e5/brick","device":"127a3d697690febddfa2a41b3a820bde","node":"b72660bc8e28725f612d33bd675f8341","volume":"937aa673e0d60126082bbd8c1589e383","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"62c4432fc7a6e4aeb11573e22eaf53d8":{"Info":{"id":"62c4432fc7a6e4aeb11573e22eaf53d8","path":"/var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_62c4432fc7a6e4aeb11573e22eaf53d8/brick","device":"25eb42cb007aab028afd0ee42b72bae9","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"2e2e6b9cf174901b370ea79a266c651b","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"6f738c7cc82f66dbda8bc0b6d74883c0":{"Info":{"id":"6f738c7cc82f66dbda8bc0b6d74883c0","path":"/var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_6f738c7cc82f66dbda8bc0b6d74883c0/brick","device":"88c314d3366471a0dd89ca4d5a3c8ef8","node":"b72660bc8e28725f612d33bd675f8341","volume":"2e2e6b9cf174901b370ea79a266c651b","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"774b52d13a2cf996784181cf6d7db93c":{"Info":{"id":"774b52d13a2cf996784181cf6d7db93c","path":"/var/lib/heketi/mounts/vg_3343a86e75865dd02b054fb268781815/brick_774b52d13a2cf996784181cf6d7db93c/brick","device":"3343a86e75865dd02b054fb268781815","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"7e73769a754d7f9f6213ef3b4551af0e","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"79e211c7bee7d61bfdabb0add8e42b33":{"Info":{"id":"79e211c7bee7d61bfdabb0add8e42b33","path":"/var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_79e211c7bee7d61bfdabb0add8e42b33/brick","device":"127a3d697690febddfa2a41b3a820bde","node":"b72660bc8e28725f612d33bd675f8341","volume":"c3b5f711128418ce9a7c68b514e34618","size":2097152},"TpSize":2097152,"PoolMetadataSize":12288,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"7ef53adf2bce0e240229941ac7f43661":{"Info":{"id":"7ef53adf2bce0e240229941ac7f43661","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_7ef53adf2bce0e240229941ac7f43661/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"83dcb62de26517790f155b51526cace5","size":209715200},"TpSize":209715200,"PoolMetadataSize":1048576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"800e2a99110af93ca69bd950c9ea7ef8":{"Info":{"id":"800e2a99110af93ca69bd950c9ea7ef8","path":"/var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_800e2a99110af93ca69bd950c9ea7ef8/brick","device":"e7e08adf3c8cd3dee08dd006b1596074","node":"b72660bc8e28725f612d33bd675f8341","volume":"dcee4f7f020d47e58fb48612bbba19d1","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"84e395f5464cd25dce66b8e60f395982":{"Info":{"id":"84e395f5464cd25dce66b8e60f395982","path":"/var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_84e395f5464cd25dce66b8e60f395982/brick","device":"88c314d3366471a0dd89ca4d5a3c8ef8","node":"b72660bc8e28725f612d33bd675f8341","volume":"7e73769a754d7f9f6213ef3b4551af0e","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"85c1670a17f956abd0f3dd99cad53a7e":{"Info":{"id":"85c1670a17f956abd0f3dd99cad53a7e","path":"/var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_85c1670a17f956abd0f3dd99cad53a7e/brick","device":"25eb42cb007aab028afd0ee42b72bae9","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"63428c902b5d577a7739eb66a050a420","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"8dba95329e797beb06eca079b9ae6a4b":{"Info":{"id":"8dba95329e797beb06eca079b9ae6a4b","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_8dba95329e797beb06eca079b9ae6a4b/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"78d426889b7bd661f3d6c6f7814b6d4f","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"93eb3caff9f179d522b5498b515bf991":{"Info":{"id":"93eb3caff9f179d522b5498b515bf991","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_93eb3caff9f179d522b5498b515bf991/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"c5fa076446479cf414397591c0af1c7f","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"9603a0a95036d6362ae2b38bdbb40428":{"Info":{"id":"9603a0a95036d6362ae2b38bdbb40428","path":"/var/lib/heketi/mounts/vg_3343a86e75865dd02b054fb268781815/brick_9603a0a95036d6362ae2b38bdbb40428/brick","device":"3343a86e75865dd02b054fb268781815","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"9f04d82be19e2ea8ee80deb9098cd390","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"9684d9e56109e011585af33ecd8039cc":{"Info":{"id":"9684d9e56109e011585af33ecd8039cc","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_9684d9e56109e011585af33ecd8039cc/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"62905ab0e959e7663f4576501c0c9b69","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"980bd302213bbdd2d9aa09dbe8abf62b":{"Info":{"id":"980bd302213bbdd2d9aa09dbe8abf62b","path":"/var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_980bd302213bbdd2d9aa09dbe8abf62b/brick","device":"e7e08adf3c8cd3dee08dd006b1596074","node":"b72660bc8e28725f612d33bd675f8341","volume":"2e6ad6aa03f7fe219807cb135ca1c766","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ac26b8e8306405623200599d05e87fe2":{"Info":{"id":"ac26b8e8306405623200599d05e87fe2","path":"/var/lib/heketi/mounts/vg_06f12f2c23a257117083a6d86d6f4087/brick_ac26b8e8306405623200599d05e87fe2/brick","device":"06f12f2c23a257117083a6d86d6f4087","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"c3b5f711128418ce9a7c68b514e34618","size":2097152},"TpSize":2097152,"PoolMetadataSize":12288,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ac7b22198b6424785edd32d6f0a078d2":{"Info":{"id":"ac7b22198b6424785edd32d6f0a078d2","path":"/var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_ac7b22198b6424785edd32d6f0a078d2/brick","device":"25eb42cb007aab028afd0ee42b72bae9","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"c5fa076446479cf414397591c0af1c7f","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ae2fa0410160c9fb489af8e5ad063ac3":{"Info":{"id":"ae2fa0410160c9fb489af8e5ad063ac3","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_ae2fa0410160c9fb489af8e5ad063ac3/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"c5fa076446479cf414397591c0af1c7f","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"b22c3ab4d7bbc420e05340e8e187bcf5":{"Info":{"id":"b22c3ab4d7bbc420e05340e8e187bcf5","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_b22c3ab4d7bbc420e05340e8e187bcf5/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"83dcb62de26517790f155b51526cace5","size":209715200},"TpSize":209715200,"PoolMetadataSize":1048576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"b85460c3ba53bb7db5ab712f7b684c51":{"Info":{"id":"b85460c3ba53bb7db5ab712f7b684c51","path":"/var/lib/heketi/mounts/vg_339b50a1c21d42f39d5568dcfbbf8844/brick_b85460c3ba53bb7db5ab712f7b684c51/brick","device":"339b50a1c21d42f39d5568dcfbbf8844","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"83dcb62de26517790f155b51526cace5","size":209715200},"TpSize":209715200,"PoolMetadataSize":1048576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"bb29a705b3f39522b131277d64761033":{"Info":{"id":"bb29a705b3f39522b131277d64761033","path":"/var/lib/heketi/mounts/vg_3343a86e75865dd02b054fb268781815/brick_bb29a705b3f39522b131277d64761033/brick","device":"3343a86e75865dd02b054fb268781815","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"937aa673e0d60126082bbd8c1589e383","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"c2fe3a63636192e85d8adeefff18c92b":{"Info":{"id":"c2fe3a63636192e85d8adeefff18c92b","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_c2fe3a63636192e85d8adeefff18c92b/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"7e73769a754d7f9f6213ef3b4551af0e","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"c4b582ef0712e8c2d750c97ff443e47c":{"Info":{"id":"c4b582ef0712e8c2d750c97ff443e47c","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_c4b582ef0712e8c2d750c97ff443e47c/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"bc5012fd39c5b3ab958b9da4b0256d3a","size":4194304},"TpSize":4194304,"PoolMetadataSize":24576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"c5d298199cfbc91ebf013707709bdbb6":{"Info":{"id":"c5d298199cfbc91ebf013707709bdbb6","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_c5d298199cfbc91ebf013707709bdbb6/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"dbc601f7162df326784aea34c4ebe8f2","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"c607b4aed1593917939ba85df5eefaae":{"Info":{"id":"c607b4aed1593917939ba85df5eefaae","path":"/var/lib/heketi/mounts/vg_3343a86e75865dd02b054fb268781815/brick_c607b4aed1593917939ba85df5eefaae/brick","device":"3343a86e75865dd02b054fb268781815","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"2e2e6b9cf174901b370ea79a266c651b","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"d3230ee5de8fe8300f47eb4bcea6f2e2":{"Info":{"id":"d3230ee5de8fe8300f47eb4bcea6f2e2","path":"/var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_d3230ee5de8fe8300f47eb4bcea6f2e2/brick","device":"127a3d697690febddfa2a41b3a820bde","node":"b72660bc8e28725f612d33bd675f8341","volume":"78d426889b7bd661f3d6c6f7814b6d4f","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"d343f2a8b3c19bda521f93d6327451f2":{"Info":{"id":"d343f2a8b3c19bda521f93d6327451f2","path":"/var/lib/heketi/mounts/vg_339b50a1c21d42f39d5568dcfbbf8844/brick_d343f2a8b3c19bda521f93d6327451f2/brick","device":"339b50a1c21d42f39d5568dcfbbf8844","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"62905ab0e959e7663f4576501c0c9b69","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"e26c7a93f2a7b6d6455ab5fa7615dfb1":{"Info":{"id":"e26c7a93f2a7b6d6455ab5fa7615dfb1","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_e26c7a93f2a7b6d6455ab5fa7615dfb1/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"88fbf36bf87106e549dca765f171cf69","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ef253aa4b2756fc293aa101c2eb743cb":{"Info":{"id":"ef253aa4b2756fc293aa101c2eb743cb","path":"/var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_ef253aa4b2756fc293aa101c2eb743cb/brick","device":"e7e08adf3c8cd3dee08dd006b1596074","node":"b72660bc8e28725f612d33bd675f8341","volume":"88fbf36bf87106e549dca765f171cf69","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"fb2e1c7f95f9926166c585ca16ac2402":{"Info":{"id":"fb2e1c7f95f9926166c585ca16ac2402","path":"/var/lib/heketi/mounts/vg_06f12f2c23a257117083a6d86d6f4087/brick_fb2e1c7f95f9926166c585ca16ac2402/brick","device":"06f12f2c23a257117083a6d86d6f4087","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"2e6ad6aa03f7fe219807cb135ca1c766","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"fd6a1a83004a11c0007e9392913aea8f":{"Info":{"id":"fd6a1a83004a11c0007e9392913aea8f","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_fd6a1a83004a11c0007e9392913aea8f/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"dcee4f7f020d47e58fb48612bbba19d1","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"fded2411a02e03c7e0219eb9a2b08c9a":{"Info":{"id":"fded2411a02e03c7e0219eb9a2b08c9a","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_fded2411a02e03c7e0219eb9a2b08c9a/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"bc5012fd39c5b3ab958b9da4b0256d3a","size":4194304},"TpSize":4194304,"PoolMetadataSize":24576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ff62a0d3e0fe5c53c3725235d2832893":{"Info":{"id":"ff62a0d3e0fe5c53c3725235d2832893","path":"/var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_ff62a0d3e0fe5c53c3725235d2832893/brick","device":"e7e08adf3c8cd3dee08dd006b1596074","node":"b72660bc8e28725f612d33bd675f8341","volume":"dbc601f7162df326784aea34c4ebe8f2","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0}},"nodeentries":{"5f090e0aa757345d22ebdb15897bd68a":{"State":"online","Info":{"zone":1,"hostnames":{"manage":["storage.matrix.nokia.com"],"storage":["192.168.89.220"]},"cluster":"e66ce2684350cb023c36d63b3d2bbf6f","id":"5f090e0aa757345d22ebdb15897bd68a"},"Devices":["018e77166eb3b38a0c250eb8faede2f7","25eb42cb007aab028afd0ee42b72bae9","5054cf122b4f54021cb9d67752d23c1d","9c007bfbe66a2522ad5dd851bb18d03a"]},"b72660bc8e28725f612d33bd675f8341":{"State":"online","Info":{"zone":1,"hostnames":{"manage":["app1.matrix.nokia.com"],"storage":["192.168.89.218"]},"cluster":"e66ce2684350cb023c36d63b3d2bbf6f","id":"b72660bc8e28725f612d33bd675f8341"},"Devices":["127a3d697690febddfa2a41b3a820bde","88c314d3366471a0dd89ca4d5a3c8ef8","cea64ccf8a96a13dacc1cb3ac3ea13de","e7e08adf3c8cd3dee08dd006b1596074"]},"f829fc134bc9f983a2865cc0f46a31a6":{"State":"online","Info":{"zone":1,"hostnames":{"manage":["app2.matrix.nokia.com"],"storage":["192.168.89.219"]},"cluster":"e66ce2684350cb023c36d63b3d2bbf6f","id":"f829fc134bc9f983a2865cc0f46a31a6"},"Devices":["06f12f2c23a257117083a6d86d6f4087","3343a86e75865dd02b054fb268781815","339b50a1c21d42f39d5568dcfbbf8844","ae7c5467f294a953bfb274c1e6afc26d"]}},"deviceentries":{"018e77166eb3b38a0c250eb8faede2f7":{"State":"online","Info":{"name":"/dev/sda","storage":{"total":292835328,"free":199589888,"used":93245440},"id":"018e77166eb3b38a0c250eb8faede2f7"},"Bricks":["0bd819f4969e7adbba32e2c77afc202c","16de636f8337f47a87996fca50313260","275d68b56cfc42c8b46aa90eb227eb25","2b6ed29607d007a75f7a7c1bf0da7bae","c2fe3a63636192e85d8adeefff18c92b"],"NodeId":"5f090e0aa757345d22ebdb15897bd68a","ExtentSize":4096},"06f12f2c23a257117083a6d86d6f4087":{"State":"online","Info":{"name":"/dev/sdf","storage":{"total":292835328,"free":48340992,"used":244494336},"id":"06f12f2c23a257117083a6d86d6f4087"},"Bricks":["28b3599a579db6e987f46ab0120f9d8c","413e08918a40ce6806dd6c2711d923f3","ac26b8e8306405623200599d05e87fe2","fb2e1c7f95f9926166c585ca16ac2402"],"NodeId":"f829fc134bc9f983a2865cc0f46a31a6","ExtentSize":4096},"127a3d697690febddfa2a41b3a820bde":{"State":"online","Info":{"name":"/dev/sde","storage":{"total":292835328,"free":120684544,"used":172150784},"id":"127a3d697690febddfa2a41b3a820bde"},"Bricks":["17f8162940314babf2b7d75976890130","5f22e9906d3566116ec474bb618213e5","79e211c7bee7d61bfdabb0add8e42b33","d3230ee5de8fe8300f47eb4bcea6f2e2"],"NodeId":"b72660bc8e28725f612d33bd675f8341","ExtentSize":4096},"25eb42cb007aab028afd0ee42b72bae9":{"State":"online","Info":{"name":"/dev/sde","storage":{"total":292835328,"free":244355072,"used":48480256},"id":"25eb42cb007aab028afd0ee42b72bae9"},"Bricks":["62c4432fc7a6e4aeb11573e22eaf53d8","85c1670a17f956abd0f3dd99cad53a7e","ac7b22198b6424785edd32d6f0a078d2"],"NodeId":"5f090e0aa757345d22ebdb15897bd68a","ExtentSize":4096},"3343a86e75865dd02b054fb268781815":{"State":"online","Info":{"name":"/dev/sde","storage":{"total":292835328,"free":218001408,"used":74833920},"id":"3343a86e75865dd02b054fb268781815"},"Bricks":["774b52d13a2cf996784181cf6d7db93c","9603a0a95036d6362ae2b38bdbb40428","bb29a705b3f39522b131277d64761033","c607b4aed1593917939ba85df5eefaae"],"NodeId":"f829fc134bc9f983a2865cc0f46a31a6","ExtentSize":4096},"339b50a1c21d42f39d5568dcfbbf8844":{"State":"online","Info":{"name":"/dev/sdb","storage":{"total":292835328,"free":27267072,"used":265568256},"id":"339b50a1c21d42f39d5568dcfbbf8844"},"Bricks":["1f69ed2975c20811515f054a74669be6","b85460c3ba53bb7db5ab712f7b684c51","d343f2a8b3c19bda521f93d6327451f2"],"NodeId":"f829fc134bc9f983a2865cc0f46a31a6","ExtentSize":4096},"5054cf122b4f54021cb9d67752d23c1d":{"State":"online","Info":{"name":"/dev/sdd","storage":{"total":292835328,"free":1970176,"used":290865152},"id":"5054cf122b4f54021cb9d67752d23c1d"},"Bricks":["025b1bf4bbabea839ef139c4453f9bcd","4fa8a73223d69de183d8e58618722084","9684d9e56109e011585af33ecd8039cc","b22c3ab4d7bbc420e05340e8e187bcf5","fded2411a02e03c7e0219eb9a2b08c9a"],"NodeId":"5f090e0aa757345d22ebdb15897bd68a","ExtentSize":4096},"88c314d3366471a0dd89ca4d5a3c8ef8":{"State":"online","Info":{"name":"/dev/sdf","storage":{"total":292835328,"free":212733952,"used":80101376},"id":"88c314d3366471a0dd89ca4d5a3c8ef8"},"Bricks":["281606b3024ac98416fedee94528e3c7","586c05c5ce5bd7474cfb9a9cd190225c","6f738c7cc82f66dbda8bc0b6d74883c0","84e395f5464cd25dce66b8e60f395982"],"NodeId":"b72660bc8e28725f612d33bd675f8341","ExtentSize":4096},"9c007bfbe66a2522ad5dd851bb18d03a":{"State":"online","Info":{"name":"/dev/sdf","storage":{"total":292835328,"free":44122112,"used":248713216},"id":"9c007bfbe66a2522ad5dd851bb18d03a"},"Bricks":["079acb5b4a0b0f3c45ed87f9c7def14b","11fe695a405dcd4854f088ddc073b4a5","22982c432dd089512ab2bde1e0663f9a","2fd555bc1cbd1b49ef04e7b88d34297b","8dba95329e797beb06eca079b9ae6a4b"],"NodeId":"5f090e0aa757345d22ebdb15897bd68a","ExtentSize":4096},"ae7c5467f294a953bfb274c1e6afc26d":{"State":"online","Info":{"name":"/dev/sdd","storage":{"total":292835328,"free":196923392,"used":95911936},"id":"ae7c5467f294a953bfb274c1e6afc26d"},"Bricks":["27b1f0d7139161a7b618a0c0e50eed24","3c16802735df66a463468ce8262ef0a6","93eb3caff9f179d522b5498b515bf991","c4b582ef0712e8c2d750c97ff443e47c","c5d298199cfbc91ebf013707709bdbb6","e26c7a93f2a7b6d6455ab5fa7615dfb1","fd6a1a83004a11c0007e9392913aea8f"],"NodeId":"f829fc134bc9f983a2865cc0f46a31a6","ExtentSize":4096},"cea64ccf8a96a13dacc1cb3ac3ea13de":{"State":"online","Info":{"name":"/dev/sdd","storage":{"total":292835328,"free":11452416,"used":281382912},"id":"cea64ccf8a96a13dacc1cb3ac3ea13de"},"Bricks":["08599e162e105063a8e2efec868d2a76","1f52101af46329a1d5e758156c8267e0","31ed93bf445cf1fd00d34660a80e1078","4220211c169c178ed693131a02ae7fb0","7ef53adf2bce0e240229941ac7f43661","ae2fa0410160c9fb489af8e5ad063ac3"],"NodeId":"b72660bc8e28725f612d33bd675f8341","ExtentSize":4096},"e7e08adf3c8cd3dee08dd006b1596074":{"State":"online","Info":{"name":"/dev/sdc","storage":{"total":292835328,"free":145293312,"used":147542016},"id":"e7e08adf3c8cd3dee08dd006b1596074"},"Bricks":["800e2a99110af93ca69bd950c9ea7ef8","980bd302213bbdd2d9aa09dbe8abf62b","ef253aa4b2756fc293aa101c2eb743cb","ff62a0d3e0fe5c53c3725235d2832893"],"NodeId":"b72660bc8e28725f612d33bd675f8341","ExtentSize":4096}},"blockvolumeentries":{},"dbattributeentries":{"DB_BRICK_HAS_SUBTYPE_FIELD":{"Key":"DB_BRICK_HAS_SUBTYPE_FIELD","Value":"yes"},"DB_CLUSTER_HAS_FILE_BLOCK_FLAG":{"Key":"DB_CLUSTER_HAS_FILE_BLOCK_FLAG","Value":"yes"},"DB_GENERATION_ID":{"Key":"DB_GENERATION_ID","Value":"d90958f6a026bc80efe34bea84fa074f"},"DB_HAS_PENDING_OPS_BUCKET":{"Key":"DB_HAS_PENDING_OPS_BUCKET","Value":"yes"}},"pendingoperations":{"8192ada38b173ad13b4d5f4b74f09fae":{"Id":"8192ada38b173ad13b4d5f4b74f09fae","Timestamp":1548654582,"Type":1,"Actions":[{"Change":1,"Id":"27b1f0d7139161a7b618a0c0e50eed24","Delta":null},{"Change":1,"Id":"11fe695a405dcd4854f088ddc073b4a5","Delta":null},{"Change":1,"Id":"4220211c169c178ed693131a02ae7fb0","Delta":null},{"Change":2,"Id":"d7a1642b3bb2f1b98c7eee66c4fc393b","Delta":null}],"Status":"stale"}}}_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
