Could you please have look my issue If you have time (atleast workaround).
I am unable to send mail to "John Mulligan" <John_Mulligan@xxxxxxxxxx>" who is currently handling issue
https://bugzilla.redhat.com/show_bug.cgi?id=1636912
BR
Salam
From: Shaik Salam/HYD/TCS
To: "John Mulligan" <John_Mulligan@xxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>, "Madhu Rajanna" <mrajanna@xxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
Date: 01/24/2019 12:21 PM
Subject: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
Hi All,
We are facing also following issue on openshift origin while we are creating pvc for pods. (atlease provide workaround to move further)
Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume
Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later..
Please find heketidb dump and log
[negroni] Completed 429 Too Many Requests in 250.763µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 169.08µs
[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c
[negroni] Completed 404 Not Found in 148.125µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 496.624µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 101.673µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 209.681µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 103.595µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 297.594µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 96.75µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 477.007µs
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 165.38µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 488.253µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 171.836µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 208.59µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 125.141µs
[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606
[negroni] Completed 404 Not Found in 138.687µs
[negroni] Started POST /volumes
BR
Salam
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
[root@master ~]# oc rsh heketi-storage-6-2bgwr
//localhost:8080 ; export HEKETI_CLI_USER=admin ; export HEKETI_CLI_SERVER=http:/
sh-4.4# heketi-cli db dump
{"clusterentries":{"e66ce2684350cb023c36d63b3d2bbf6f":{"Info":{"id":"e66ce2684350cb023c36d63b3d2bbf6f","nodes":["5f090e0aa757345d22ebdb15897bd68a","b72660bc8e28725f612d33bd675f8341","f829fc134bc9f983a2865cc0f46a31a6"],"volumes":["2e2e6b9cf174901b370ea79a266c651b","2e6ad6aa03f7fe219807cb135ca1c766","47c2af6849ac9ad7d7fbc897ae3ae80c","5101331e9d33a5d04adab92837b9d5ad","62905ab0e959e7663f4576501c0c9b69","63428c902b5d577a7739eb66a050a420","78d426889b7bd661f3d6c6f7814b6d4f","7e73769a754d7f9f6213ef3b4551af0e","83dcb62de26517790f155b51526cace5","88fbf36bf87106e549dca765f171cf69","937aa673e0d60126082bbd8c1589e383","9f04d82be19e2ea8ee80deb9098cd390","bc5012fd39c5b3ab958b9da4b0256d3a","c3b5f711128418ce9a7c68b514e34618","c5fa076446479cf414397591c0af1c7f","dbc601f7162df326784aea34c4ebe8f2","dcee4f7f020d47e58fb48612bbba19d1"],"block":true,"file":true,"blockvolumes":[]}}},"volumeentries":{"2e2e6b9cf174901b370ea79a266c651b":{"Info":{"size":10,"name":"vol_2e2e6b9cf174901b370ea79a266c651b","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2004,"snapshot":{"enable":false,"factor":1},"id":"2e2e6b9cf174901b370ea79a266c651b","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_2e2e6b9cf174901b370ea79a266c651b","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["62c4432fc7a6e4aeb11573e22eaf53d8","6f738c7cc82f66dbda8bc0b6d74883c0","c607b4aed1593917939ba85df5eefaae"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"2e6ad6aa03f7fe219807cb135ca1c766":{"Info":{"size":70,"name":"vol_2e6ad6aa03f7fe219807cb135ca1c766","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2007,"snapshot":{"enable":false,"factor":1},"id":"2e6ad6aa03f7fe219807cb135ca1c766","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_2e6ad6aa03f7fe219807cb135ca1c766","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["2fd555bc1cbd1b49ef04e7b88d34297b","980bd302213bbdd2d9aa09dbe8abf62b","fb2e1c7f95f9926166c585ca16ac2402"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"47c2af6849ac9ad7d7fbc897ae3ae80c":{"Info":{"size":50,"name":"vol_47c2af6849ac9ad7d7fbc897ae3ae80c","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2001,"snapshot":{"enable":false,"factor":1},"id":"47c2af6849ac9ad7d7fbc897ae3ae80c","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_47c2af6849ac9ad7d7fbc897ae3ae80c","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["025b1bf4bbabea839ef139c4453f9bcd","1f52101af46329a1d5e758156c8267e0","1f69ed2975c20811515f054a74669be6"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"5101331e9d33a5d04adab92837b9d5ad":{"Info":{"size":70,"name":"vol_5101331e9d33a5d04adab92837b9d5ad","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2006,"snapshot":{"enable":false,"factor":1},"id":"5101331e9d33a5d04adab92837b9d5ad","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_5101331e9d33a5d04adab92837b9d5ad","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["079acb5b4a0b0f3c45ed87f9c7def14b","17f8162940314babf2b7d75976890130","413e08918a40ce6806dd6c2711d923f3"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"62905ab0e959e7663f4576501c0c9b69":{"Info":{"size":1,"name":"vol_62905ab0e959e7663f4576501c0c9b69","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2002,"snapshot":{"enable":false,"factor":1},"id":"62905ab0e959e7663f4576501c0c9b69","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_62905ab0e959e7663f4576501c0c9b69","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["31ed93bf445cf1fd00d34660a80e1078","9684d9e56109e011585af33ecd8039cc","d343f2a8b3c19bda521f93d6327451f2"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"63428c902b5d577a7739eb66a050a420":{"Info":{"size":25,"name":"vol_63428c902b5d577a7739eb66a050a420","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2013,"snapshot":{"enable":false,"factor":1},"id":"63428c902b5d577a7739eb66a050a420","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_63428c902b5d577a7739eb66a050a420","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["3c16802735df66a463468ce8262ef0a6","586c05c5ce5bd7474cfb9a9cd190225c","85c1670a17f956abd0f3dd99cad53a7e"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"78d426889b7bd661f3d6c6f7814b6d4f":{"Info":{"size":70,"name":"vol_78d426889b7bd661f3d6c6f7814b6d4f","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2008,"snapshot":{"enable":false,"factor":1},"id":"78d426889b7bd661f3d6c6f7814b6d4f","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_78d426889b7bd661f3d6c6f7814b6d4f","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["28b3599a579db6e987f46ab0120f9d8c","8dba95329e797beb06eca079b9ae6a4b","d3230ee5de8fe8300f47eb4bcea6f2e2"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"7e73769a754d7f9f6213ef3b4551af0e":{"Info":{"size":5,"name":"vol_7e73769a754d7f9f6213ef3b4551af0e","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{"data":4,"redundancy":2}},"snapshot":{"enable":false,"factor":1},"id":"7e73769a754d7f9f6213ef3b4551af0e","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_7e73769a754d7f9f6213ef3b4551af0e","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["774b52d13a2cf996784181cf6d7db93c","84e395f5464cd25dce66b8e60f395982","c2fe3a63636192e85d8adeefff18c92b"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"83dcb62de26517790f155b51526cace5":{"Info":{"size":200,"name":"glusterfs-registry-volume","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{"data":4,"redundancy":2}},"snapshot":{"enable":false,"factor":1},"id":"83dcb62de26517790f155b51526cace5","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:glusterfs-registry-volume","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["7ef53adf2bce0e240229941ac7f43661","b22c3ab4d7bbc420e05340e8e187bcf5","b85460c3ba53bb7db5ab712f7b684c51"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"88fbf36bf87106e549dca765f171cf69":{"Info":{"size":5,"name":"vol_88fbf36bf87106e549dca765f171cf69","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{"data":4,"redundancy":2}},"snapshot":{"enable":false,"factor":1},"id":"88fbf36bf87106e549dca765f171cf69","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_88fbf36bf87106e549dca765f171cf69","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["22982c432dd089512ab2bde1e0663f9a","e26c7a93f2a7b6d6455ab5fa7615dfb1","ef253aa4b2756fc293aa101c2eb743cb"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"937aa673e0d60126082bbd8c1589e383":{"Info":{"size":20,"name":"vol_937aa673e0d60126082bbd8c1589e383","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2003,"snapshot":{"enable":false,"factor":1},"id":"937aa673e0d60126082bbd8c1589e383","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_937aa673e0d60126082bbd8c1589e383","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["275d68b56cfc42c8b46aa90eb227eb25","5f22e9906d3566116ec474bb618213e5","bb29a705b3f39522b131277d64761033"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"9f04d82be19e2ea8ee80deb9098cd390":{"Info":{"size":20,"name":"vol_9f04d82be19e2ea8ee80deb9098cd390","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2014,"snapshot":{"enable":false,"factor":1},"id":"9f04d82be19e2ea8ee80deb9098cd390","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_9f04d82be19e2ea8ee80deb9098cd390","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["0bd819f4969e7adbba32e2c77afc202c","281606b3024ac98416fedee94528e3c7","9603a0a95036d6362ae2b38bdbb40428"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"bc5012fd39c5b3ab958b9da4b0256d3a":{"Info":{"size":4,"name":"vol_bc5012fd39c5b3ab958b9da4b0256d3a","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2009,"snapshot":{"enable":false,"factor":1},"id":"bc5012fd39c5b3ab958b9da4b0256d3a","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_bc5012fd39c5b3ab958b9da4b0256d3a","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["08599e162e105063a8e2efec868d2a76","c4b582ef0712e8c2d750c97ff443e47c","fded2411a02e03c7e0219eb9a2b08c9a"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"c3b5f711128418ce9a7c68b514e34618":{"Info":{"size":2,"name":"heketidbstorage","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"snapshot":{"enable":false,"factor":1},"id":"c3b5f711128418ce9a7c68b514e34618","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:heketidbstorage","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["2b6ed29607d007a75f7a7c1bf0da7bae","79e211c7bee7d61bfdabb0add8e42b33","ac26b8e8306405623200599d05e87fe2"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"c5fa076446479cf414397591c0af1c7f":{"Info":{"size":10,"name":"vol_c5fa076446479cf414397591c0af1c7f","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2005,"snapshot":{"enable":false,"factor":1},"id":"c5fa076446479cf414397591c0af1c7f","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_c5fa076446479cf414397591c0af1c7f","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["93eb3caff9f179d522b5498b515bf991","ac7b22198b6424785edd32d6f0a078d2","ae2fa0410160c9fb489af8e5ad063ac3"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"dbc601f7162df326784aea34c4ebe8f2":{"Info":{"size":20,"name":"vol_dbc601f7162df326784aea34c4ebe8f2","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2000,"snapshot":{"enable":false,"factor":1},"id":"dbc601f7162df326784aea34c4ebe8f2","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_dbc601f7162df326784aea34c4ebe8f2","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["4fa8a73223d69de183d8e58618722084","c5d298199cfbc91ebf013707709bdbb6","ff62a0d3e0fe5c53c3725235d2832893"],"GlusterVolumeOptions":null,"Pending":{"Id":""}},"dcee4f7f020d47e58fb48612bbba19d1":{"Info":{"size":25,"name":"vol_dcee4f7f020d47e58fb48612bbba19d1","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{}},"gid":2010,"snapshot":{"enable":false,"factor":1},"id":"dcee4f7f020d47e58fb48612bbba19d1","cluster":"e66ce2684350cb023c36d63b3d2bbf6f","mount":{"glusterfs":{"hosts":["192.168.89.220","192.168.89.218","192.168.89.219"],"device":"192.168.89.220:vol_dcee4f7f020d47e58fb48612bbba19d1","options":{"backup-volfile-servers":"192.168.89.218,192.168.89.219"}}},"blockinfo":{}},"Bricks":["16de636f8337f47a87996fca50313260","800e2a99110af93ca69bd950c9ea7ef8","fd6a1a83004a11c0007e9392913aea8f"],"GlusterVolumeOptions":null,"Pending":{"Id":""}}},"brickentries":{"025b1bf4bbabea839ef139c4453f9bcd":{"Info":{"id":"025b1bf4bbabea839ef139c4453f9bcd","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_025b1bf4bbabea839ef139c4453f9bcd/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"47c2af6849ac9ad7d7fbc897ae3ae80c","size":52428800},"TpSize":52428800,"PoolMetadataSize":262144,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"079acb5b4a0b0f3c45ed87f9c7def14b":{"Info":{"id":"079acb5b4a0b0f3c45ed87f9c7def14b","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_079acb5b4a0b0f3c45ed87f9c7def14b/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"5101331e9d33a5d04adab92837b9d5ad","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"08599e162e105063a8e2efec868d2a76":{"Info":{"id":"08599e162e105063a8e2efec868d2a76","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_08599e162e105063a8e2efec868d2a76/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"bc5012fd39c5b3ab958b9da4b0256d3a","size":4194304},"TpSize":4194304,"PoolMetadataSize":24576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"0bd819f4969e7adbba32e2c77afc202c":{"Info":{"id":"0bd819f4969e7adbba32e2c77afc202c","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_0bd819f4969e7adbba32e2c77afc202c/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"9f04d82be19e2ea8ee80deb9098cd390","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"16de636f8337f47a87996fca50313260":{"Info":{"id":"16de636f8337f47a87996fca50313260","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_16de636f8337f47a87996fca50313260/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"dcee4f7f020d47e58fb48612bbba19d1","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"17f8162940314babf2b7d75976890130":{"Info":{"id":"17f8162940314babf2b7d75976890130","path":"/var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_17f8162940314babf2b7d75976890130/brick","device":"127a3d697690febddfa2a41b3a820bde","node":"b72660bc8e28725f612d33bd675f8341","volume":"5101331e9d33a5d04adab92837b9d5ad","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"1f52101af46329a1d5e758156c8267e0":{"Info":{"id":"1f52101af46329a1d5e758156c8267e0","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_1f52101af46329a1d5e758156c8267e0/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"47c2af6849ac9ad7d7fbc897ae3ae80c","size":52428800},"TpSize":52428800,"PoolMetadataSize":262144,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"1f69ed2975c20811515f054a74669be6":{"Info":{"id":"1f69ed2975c20811515f054a74669be6","path":"/var/lib/heketi/mounts/vg_339b50a1c21d42f39d5568dcfbbf8844/brick_1f69ed2975c20811515f054a74669be6/brick","device":"339b50a1c21d42f39d5568dcfbbf8844","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"47c2af6849ac9ad7d7fbc897ae3ae80c","size":52428800},"TpSize":52428800,"PoolMetadataSize":262144,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"22982c432dd089512ab2bde1e0663f9a":{"Info":{"id":"22982c432dd089512ab2bde1e0663f9a","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_22982c432dd089512ab2bde1e0663f9a/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"88fbf36bf87106e549dca765f171cf69","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"275d68b56cfc42c8b46aa90eb227eb25":{"Info":{"id":"275d68b56cfc42c8b46aa90eb227eb25","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_275d68b56cfc42c8b46aa90eb227eb25/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"937aa673e0d60126082bbd8c1589e383","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"281606b3024ac98416fedee94528e3c7":{"Info":{"id":"281606b3024ac98416fedee94528e3c7","path":"/var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_281606b3024ac98416fedee94528e3c7/brick","device":"88c314d3366471a0dd89ca4d5a3c8ef8","node":"b72660bc8e28725f612d33bd675f8341","volume":"9f04d82be19e2ea8ee80deb9098cd390","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"28b3599a579db6e987f46ab0120f9d8c":{"Info":{"id":"28b3599a579db6e987f46ab0120f9d8c","path":"/var/lib/heketi/mounts/vg_06f12f2c23a257117083a6d86d6f4087/brick_28b3599a579db6e987f46ab0120f9d8c/brick","device":"06f12f2c23a257117083a6d86d6f4087","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"78d426889b7bd661f3d6c6f7814b6d4f","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"2b6ed29607d007a75f7a7c1bf0da7bae":{"Info":{"id":"2b6ed29607d007a75f7a7c1bf0da7bae","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_2b6ed29607d007a75f7a7c1bf0da7bae/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"c3b5f711128418ce9a7c68b514e34618","size":2097152},"TpSize":2097152,"PoolMetadataSize":12288,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"2fd555bc1cbd1b49ef04e7b88d34297b":{"Info":{"id":"2fd555bc1cbd1b49ef04e7b88d34297b","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_2fd555bc1cbd1b49ef04e7b88d34297b/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"2e6ad6aa03f7fe219807cb135ca1c766","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"31ed93bf445cf1fd00d34660a80e1078":{"Info":{"id":"31ed93bf445cf1fd00d34660a80e1078","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_31ed93bf445cf1fd00d34660a80e1078/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"62905ab0e959e7663f4576501c0c9b69","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"3c16802735df66a463468ce8262ef0a6":{"Info":{"id":"3c16802735df66a463468ce8262ef0a6","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_3c16802735df66a463468ce8262ef0a6/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"63428c902b5d577a7739eb66a050a420","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"413e08918a40ce6806dd6c2711d923f3":{"Info":{"id":"413e08918a40ce6806dd6c2711d923f3","path":"/var/lib/heketi/mounts/vg_06f12f2c23a257117083a6d86d6f4087/brick_413e08918a40ce6806dd6c2711d923f3/brick","device":"06f12f2c23a257117083a6d86d6f4087","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"5101331e9d33a5d04adab92837b9d5ad","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"4fa8a73223d69de183d8e58618722084":{"Info":{"id":"4fa8a73223d69de183d8e58618722084","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_4fa8a73223d69de183d8e58618722084/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"dbc601f7162df326784aea34c4ebe8f2","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"586c05c5ce5bd7474cfb9a9cd190225c":{"Info":{"id":"586c05c5ce5bd7474cfb9a9cd190225c","path":"/var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_586c05c5ce5bd7474cfb9a9cd190225c/brick","device":"88c314d3366471a0dd89ca4d5a3c8ef8","node":"b72660bc8e28725f612d33bd675f8341","volume":"63428c902b5d577a7739eb66a050a420","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"5f22e9906d3566116ec474bb618213e5":{"Info":{"id":"5f22e9906d3566116ec474bb618213e5","path":"/var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_5f22e9906d3566116ec474bb618213e5/brick","device":"127a3d697690febddfa2a41b3a820bde","node":"b72660bc8e28725f612d33bd675f8341","volume":"937aa673e0d60126082bbd8c1589e383","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"62c4432fc7a6e4aeb11573e22eaf53d8":{"Info":{"id":"62c4432fc7a6e4aeb11573e22eaf53d8","path":"/var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_62c4432fc7a6e4aeb11573e22eaf53d8/brick","device":"25eb42cb007aab028afd0ee42b72bae9","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"2e2e6b9cf174901b370ea79a266c651b","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"6f738c7cc82f66dbda8bc0b6d74883c0":{"Info":{"id":"6f738c7cc82f66dbda8bc0b6d74883c0","path":"/var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_6f738c7cc82f66dbda8bc0b6d74883c0/brick","device":"88c314d3366471a0dd89ca4d5a3c8ef8","node":"b72660bc8e28725f612d33bd675f8341","volume":"2e2e6b9cf174901b370ea79a266c651b","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"774b52d13a2cf996784181cf6d7db93c":{"Info":{"id":"774b52d13a2cf996784181cf6d7db93c","path":"/var/lib/heketi/mounts/vg_3343a86e75865dd02b054fb268781815/brick_774b52d13a2cf996784181cf6d7db93c/brick","device":"3343a86e75865dd02b054fb268781815","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"7e73769a754d7f9f6213ef3b4551af0e","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"79e211c7bee7d61bfdabb0add8e42b33":{"Info":{"id":"79e211c7bee7d61bfdabb0add8e42b33","path":"/var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_79e211c7bee7d61bfdabb0add8e42b33/brick","device":"127a3d697690febddfa2a41b3a820bde","node":"b72660bc8e28725f612d33bd675f8341","volume":"c3b5f711128418ce9a7c68b514e34618","size":2097152},"TpSize":2097152,"PoolMetadataSize":12288,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"7ef53adf2bce0e240229941ac7f43661":{"Info":{"id":"7ef53adf2bce0e240229941ac7f43661","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_7ef53adf2bce0e240229941ac7f43661/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"83dcb62de26517790f155b51526cace5","size":209715200},"TpSize":209715200,"PoolMetadataSize":1048576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"800e2a99110af93ca69bd950c9ea7ef8":{"Info":{"id":"800e2a99110af93ca69bd950c9ea7ef8","path":"/var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_800e2a99110af93ca69bd950c9ea7ef8/brick","device":"e7e08adf3c8cd3dee08dd006b1596074","node":"b72660bc8e28725f612d33bd675f8341","volume":"dcee4f7f020d47e58fb48612bbba19d1","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"84e395f5464cd25dce66b8e60f395982":{"Info":{"id":"84e395f5464cd25dce66b8e60f395982","path":"/var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_84e395f5464cd25dce66b8e60f395982/brick","device":"88c314d3366471a0dd89ca4d5a3c8ef8","node":"b72660bc8e28725f612d33bd675f8341","volume":"7e73769a754d7f9f6213ef3b4551af0e","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"85c1670a17f956abd0f3dd99cad53a7e":{"Info":{"id":"85c1670a17f956abd0f3dd99cad53a7e","path":"/var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_85c1670a17f956abd0f3dd99cad53a7e/brick","device":"25eb42cb007aab028afd0ee42b72bae9","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"63428c902b5d577a7739eb66a050a420","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"8dba95329e797beb06eca079b9ae6a4b":{"Info":{"id":"8dba95329e797beb06eca079b9ae6a4b","path":"/var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_8dba95329e797beb06eca079b9ae6a4b/brick","device":"9c007bfbe66a2522ad5dd851bb18d03a","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"78d426889b7bd661f3d6c6f7814b6d4f","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"93eb3caff9f179d522b5498b515bf991":{"Info":{"id":"93eb3caff9f179d522b5498b515bf991","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_93eb3caff9f179d522b5498b515bf991/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"c5fa076446479cf414397591c0af1c7f","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"9603a0a95036d6362ae2b38bdbb40428":{"Info":{"id":"9603a0a95036d6362ae2b38bdbb40428","path":"/var/lib/heketi/mounts/vg_3343a86e75865dd02b054fb268781815/brick_9603a0a95036d6362ae2b38bdbb40428/brick","device":"3343a86e75865dd02b054fb268781815","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"9f04d82be19e2ea8ee80deb9098cd390","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"9684d9e56109e011585af33ecd8039cc":{"Info":{"id":"9684d9e56109e011585af33ecd8039cc","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_9684d9e56109e011585af33ecd8039cc/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"62905ab0e959e7663f4576501c0c9b69","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"980bd302213bbdd2d9aa09dbe8abf62b":{"Info":{"id":"980bd302213bbdd2d9aa09dbe8abf62b","path":"/var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_980bd302213bbdd2d9aa09dbe8abf62b/brick","device":"e7e08adf3c8cd3dee08dd006b1596074","node":"b72660bc8e28725f612d33bd675f8341","volume":"2e6ad6aa03f7fe219807cb135ca1c766","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ac26b8e8306405623200599d05e87fe2":{"Info":{"id":"ac26b8e8306405623200599d05e87fe2","path":"/var/lib/heketi/mounts/vg_06f12f2c23a257117083a6d86d6f4087/brick_ac26b8e8306405623200599d05e87fe2/brick","device":"06f12f2c23a257117083a6d86d6f4087","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"c3b5f711128418ce9a7c68b514e34618","size":2097152},"TpSize":2097152,"PoolMetadataSize":12288,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ac7b22198b6424785edd32d6f0a078d2":{"Info":{"id":"ac7b22198b6424785edd32d6f0a078d2","path":"/var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_ac7b22198b6424785edd32d6f0a078d2/brick","device":"25eb42cb007aab028afd0ee42b72bae9","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"c5fa076446479cf414397591c0af1c7f","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ae2fa0410160c9fb489af8e5ad063ac3":{"Info":{"id":"ae2fa0410160c9fb489af8e5ad063ac3","path":"/var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_ae2fa0410160c9fb489af8e5ad063ac3/brick","device":"cea64ccf8a96a13dacc1cb3ac3ea13de","node":"b72660bc8e28725f612d33bd675f8341","volume":"c5fa076446479cf414397591c0af1c7f","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"b22c3ab4d7bbc420e05340e8e187bcf5":{"Info":{"id":"b22c3ab4d7bbc420e05340e8e187bcf5","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_b22c3ab4d7bbc420e05340e8e187bcf5/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"83dcb62de26517790f155b51526cace5","size":209715200},"TpSize":209715200,"PoolMetadataSize":1048576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"b85460c3ba53bb7db5ab712f7b684c51":{"Info":{"id":"b85460c3ba53bb7db5ab712f7b684c51","path":"/var/lib/heketi/mounts/vg_339b50a1c21d42f39d5568dcfbbf8844/brick_b85460c3ba53bb7db5ab712f7b684c51/brick","device":"339b50a1c21d42f39d5568dcfbbf8844","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"83dcb62de26517790f155b51526cace5","size":209715200},"TpSize":209715200,"PoolMetadataSize":1048576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"bb29a705b3f39522b131277d64761033":{"Info":{"id":"bb29a705b3f39522b131277d64761033","path":"/var/lib/heketi/mounts/vg_3343a86e75865dd02b054fb268781815/brick_bb29a705b3f39522b131277d64761033/brick","device":"3343a86e75865dd02b054fb268781815","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"937aa673e0d60126082bbd8c1589e383","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"c2fe3a63636192e85d8adeefff18c92b":{"Info":{"id":"c2fe3a63636192e85d8adeefff18c92b","path":"/var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_c2fe3a63636192e85d8adeefff18c92b/brick","device":"018e77166eb3b38a0c250eb8faede2f7","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"7e73769a754d7f9f6213ef3b4551af0e","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"c4b582ef0712e8c2d750c97ff443e47c":{"Info":{"id":"c4b582ef0712e8c2d750c97ff443e47c","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_c4b582ef0712e8c2d750c97ff443e47c/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"bc5012fd39c5b3ab958b9da4b0256d3a","size":4194304},"TpSize":4194304,"PoolMetadataSize":24576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"c5d298199cfbc91ebf013707709bdbb6":{"Info":{"id":"c5d298199cfbc91ebf013707709bdbb6","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_c5d298199cfbc91ebf013707709bdbb6/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"dbc601f7162df326784aea34c4ebe8f2","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"c607b4aed1593917939ba85df5eefaae":{"Info":{"id":"c607b4aed1593917939ba85df5eefaae","path":"/var/lib/heketi/mounts/vg_3343a86e75865dd02b054fb268781815/brick_c607b4aed1593917939ba85df5eefaae/brick","device":"3343a86e75865dd02b054fb268781815","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"2e2e6b9cf174901b370ea79a266c651b","size":10485760},"TpSize":10485760,"PoolMetadataSize":53248,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"d3230ee5de8fe8300f47eb4bcea6f2e2":{"Info":{"id":"d3230ee5de8fe8300f47eb4bcea6f2e2","path":"/var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_d3230ee5de8fe8300f47eb4bcea6f2e2/brick","device":"127a3d697690febddfa2a41b3a820bde","node":"b72660bc8e28725f612d33bd675f8341","volume":"78d426889b7bd661f3d6c6f7814b6d4f","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"d343f2a8b3c19bda521f93d6327451f2":{"Info":{"id":"d343f2a8b3c19bda521f93d6327451f2","path":"/var/lib/heketi/mounts/vg_339b50a1c21d42f39d5568dcfbbf8844/brick_d343f2a8b3c19bda521f93d6327451f2/brick","device":"339b50a1c21d42f39d5568dcfbbf8844","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"62905ab0e959e7663f4576501c0c9b69","size":1048576},"TpSize":1048576,"PoolMetadataSize":8192,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"e26c7a93f2a7b6d6455ab5fa7615dfb1":{"Info":{"id":"e26c7a93f2a7b6d6455ab5fa7615dfb1","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_e26c7a93f2a7b6d6455ab5fa7615dfb1/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"88fbf36bf87106e549dca765f171cf69","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ef253aa4b2756fc293aa101c2eb743cb":{"Info":{"id":"ef253aa4b2756fc293aa101c2eb743cb","path":"/var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_ef253aa4b2756fc293aa101c2eb743cb/brick","device":"e7e08adf3c8cd3dee08dd006b1596074","node":"b72660bc8e28725f612d33bd675f8341","volume":"88fbf36bf87106e549dca765f171cf69","size":5242880},"TpSize":5242880,"PoolMetadataSize":28672,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"fb2e1c7f95f9926166c585ca16ac2402":{"Info":{"id":"fb2e1c7f95f9926166c585ca16ac2402","path":"/var/lib/heketi/mounts/vg_06f12f2c23a257117083a6d86d6f4087/brick_fb2e1c7f95f9926166c585ca16ac2402/brick","device":"06f12f2c23a257117083a6d86d6f4087","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"2e6ad6aa03f7fe219807cb135ca1c766","size":73400320},"TpSize":73400320,"PoolMetadataSize":368640,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"fd6a1a83004a11c0007e9392913aea8f":{"Info":{"id":"fd6a1a83004a11c0007e9392913aea8f","path":"/var/lib/heketi/mounts/vg_ae7c5467f294a953bfb274c1e6afc26d/brick_fd6a1a83004a11c0007e9392913aea8f/brick","device":"ae7c5467f294a953bfb274c1e6afc26d","node":"f829fc134bc9f983a2865cc0f46a31a6","volume":"dcee4f7f020d47e58fb48612bbba19d1","size":26214400},"TpSize":26214400,"PoolMetadataSize":131072,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"fded2411a02e03c7e0219eb9a2b08c9a":{"Info":{"id":"fded2411a02e03c7e0219eb9a2b08c9a","path":"/var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_fded2411a02e03c7e0219eb9a2b08c9a/brick","device":"5054cf122b4f54021cb9d67752d23c1d","node":"5f090e0aa757345d22ebdb15897bd68a","volume":"bc5012fd39c5b3ab958b9da4b0256d3a","size":4194304},"TpSize":4194304,"PoolMetadataSize":24576,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0},"ff62a0d3e0fe5c53c3725235d2832893":{"Info":{"id":"ff62a0d3e0fe5c53c3725235d2832893","path":"/var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_ff62a0d3e0fe5c53c3725235d2832893/brick","device":"e7e08adf3c8cd3dee08dd006b1596074","node":"b72660bc8e28725f612d33bd675f8341","volume":"dbc601f7162df326784aea34c4ebe8f2","size":20971520},"TpSize":20971520,"PoolMetadataSize":106496,"Pending":{"Id":""},"LvmThinPool":"","LvmLv":"","SubType":0}},"nodeentries":{"5f090e0aa757345d22ebdb15897bd68a":{"State":"online","Info":{"zone":1,"hostnames":{"manage":["storage.matrix.nokia.com"],"storage":["192.168.89.220"]},"cluster":"e66ce2684350cb023c36d63b3d2bbf6f","id":"5f090e0aa757345d22ebdb15897bd68a"},"Devices":["018e77166eb3b38a0c250eb8faede2f7","25eb42cb007aab028afd0ee42b72bae9","5054cf122b4f54021cb9d67752d23c1d","9c007bfbe66a2522ad5dd851bb18d03a"]},"b72660bc8e28725f612d33bd675f8341":{"State":"online","Info":{"zone":1,"hostnames":{"manage":["app1.matrix.nokia.com"],"storage":["192.168.89.218"]},"cluster":"e66ce2684350cb023c36d63b3d2bbf6f","id":"b72660bc8e28725f612d33bd675f8341"},"Devices":["127a3d697690febddfa2a41b3a820bde","88c314d3366471a0dd89ca4d5a3c8ef8","cea64ccf8a96a13dacc1cb3ac3ea13de","e7e08adf3c8cd3dee08dd006b1596074"]},"f829fc134bc9f983a2865cc0f46a31a6":{"State":"online","Info":{"zone":1,"hostnames":{"manage":["app2.matrix.nokia.com"],"storage":["192.168.89.219"]},"cluster":"e66ce2684350cb023c36d63b3d2bbf6f","id":"f829fc134bc9f983a2865cc0f46a31a6"},"Devices":["06f12f2c23a257117083a6d86d6f4087","3343a86e75865dd02b054fb268781815","339b50a1c21d42f39d5568dcfbbf8844","ae7c5467f294a953bfb274c1e6afc26d"]}},"deviceentries":{"018e77166eb3b38a0c250eb8faede2f7":{"State":"online","Info":{"name":"/dev/sda","storage":{"total":292835328,"free":199589888,"used":93245440},"id":"018e77166eb3b38a0c250eb8faede2f7"},"Bricks":["0bd819f4969e7adbba32e2c77afc202c","16de636f8337f47a87996fca50313260","275d68b56cfc42c8b46aa90eb227eb25","2b6ed29607d007a75f7a7c1bf0da7bae","c2fe3a63636192e85d8adeefff18c92b"],"NodeId":"5f090e0aa757345d22ebdb15897bd68a","ExtentSize":4096},"06f12f2c23a257117083a6d86d6f4087":{"State":"online","Info":{"name":"/dev/sdf","storage":{"total":292835328,"free":48340992,"used":244494336},"id":"06f12f2c23a257117083a6d86d6f4087"},"Bricks":["28b3599a579db6e987f46ab0120f9d8c","413e08918a40ce6806dd6c2711d923f3","ac26b8e8306405623200599d05e87fe2","fb2e1c7f95f9926166c585ca16ac2402"],"NodeId":"f829fc134bc9f983a2865cc0f46a31a6","ExtentSize":4096},"127a3d697690febddfa2a41b3a820bde":{"State":"online","Info":{"name":"/dev/sde","storage":{"total":292835328,"free":120684544,"used":172150784},"id":"127a3d697690febddfa2a41b3a820bde"},"Bricks":["17f8162940314babf2b7d75976890130","5f22e9906d3566116ec474bb618213e5","79e211c7bee7d61bfdabb0add8e42b33","d3230ee5de8fe8300f47eb4bcea6f2e2"],"NodeId":"b72660bc8e28725f612d33bd675f8341","ExtentSize":4096},"25eb42cb007aab028afd0ee42b72bae9":{"State":"online","Info":{"name":"/dev/sde","storage":{"total":292835328,"free":244355072,"used":48480256},"id":"25eb42cb007aab028afd0ee42b72bae9"},"Bricks":["62c4432fc7a6e4aeb11573e22eaf53d8","85c1670a17f956abd0f3dd99cad53a7e","ac7b22198b6424785edd32d6f0a078d2"],"NodeId":"5f090e0aa757345d22ebdb15897bd68a","ExtentSize":4096},"3343a86e75865dd02b054fb268781815":{"State":"online","Info":{"name":"/dev/sde","storage":{"total":292835328,"free":218001408,"used":74833920},"id":"3343a86e75865dd02b054fb268781815"},"Bricks":["774b52d13a2cf996784181cf6d7db93c","9603a0a95036d6362ae2b38bdbb40428","bb29a705b3f39522b131277d64761033","c607b4aed1593917939ba85df5eefaae"],"NodeId":"f829fc134bc9f983a2865cc0f46a31a6","ExtentSize":4096},"339b50a1c21d42f39d5568dcfbbf8844":{"State":"online","Info":{"name":"/dev/sdb","storage":{"total":292835328,"free":27267072,"used":265568256},"id":"339b50a1c21d42f39d5568dcfbbf8844"},"Bricks":["1f69ed2975c20811515f054a74669be6","b85460c3ba53bb7db5ab712f7b684c51","d343f2a8b3c19bda521f93d6327451f2"],"NodeId":"f829fc134bc9f983a2865cc0f46a31a6","ExtentSize":4096},"5054cf122b4f54021cb9d67752d23c1d":{"State":"online","Info":{"name":"/dev/sdd","storage":{"total":292835328,"free":1970176,"used":290865152},"id":"5054cf122b4f54021cb9d67752d23c1d"},"Bricks":["025b1bf4bbabea839ef139c4453f9bcd","4fa8a73223d69de183d8e58618722084","9684d9e56109e011585af33ecd8039cc","b22c3ab4d7bbc420e05340e8e187bcf5","fded2411a02e03c7e0219eb9a2b08c9a"],"NodeId":"5f090e0aa757345d22ebdb15897bd68a","ExtentSize":4096},"88c314d3366471a0dd89ca4d5a3c8ef8":{"State":"online","Info":{"name":"/dev/sdf","storage":{"total":292835328,"free":212733952,"used":80101376},"id":"88c314d3366471a0dd89ca4d5a3c8ef8"},"Bricks":["281606b3024ac98416fedee94528e3c7","586c05c5ce5bd7474cfb9a9cd190225c","6f738c7cc82f66dbda8bc0b6d74883c0","84e395f5464cd25dce66b8e60f395982"],"NodeId":"b72660bc8e28725f612d33bd675f8341","ExtentSize":4096},"9c007bfbe66a2522ad5dd851bb18d03a":{"State":"online","Info":{"name":"/dev/sdf","storage":{"total":292835328,"free":45178880,"used":247656448},"id":"9c007bfbe66a2522ad5dd851bb18d03a"},"Bricks":["079acb5b4a0b0f3c45ed87f9c7def14b","22982c432dd089512ab2bde1e0663f9a","2fd555bc1cbd1b49ef04e7b88d34297b","8dba95329e797beb06eca079b9ae6a4b"],"NodeId":"5f090e0aa757345d22ebdb15897bd68a","ExtentSize":4096},"ae7c5467f294a953bfb274c1e6afc26d":{"State":"online","Info":{"name":"/dev/sdd","storage":{"total":292835328,"free":197980160,"used":94855168},"id":"ae7c5467f294a953bfb274c1e6afc26d"},"Bricks":["3c16802735df66a463468ce8262ef0a6","93eb3caff9f179d522b5498b515bf991","c4b582ef0712e8c2d750c97ff443e47c","c5d298199cfbc91ebf013707709bdbb6","e26c7a93f2a7b6d6455ab5fa7615dfb1","fd6a1a83004a11c0007e9392913aea8f"],"NodeId":"f829fc134bc9f983a2865cc0f46a31a6","ExtentSize":4096},"cea64ccf8a96a13dacc1cb3ac3ea13de":{"State":"online","Info":{"name":"/dev/sdd","storage":{"total":292835328,"free":12509184,"used":280326144},"id":"cea64ccf8a96a13dacc1cb3ac3ea13de"},"Bricks":["08599e162e105063a8e2efec868d2a76","1f52101af46329a1d5e758156c8267e0","31ed93bf445cf1fd00d34660a80e1078","7ef53adf2bce0e240229941ac7f43661","ae2fa0410160c9fb489af8e5ad063ac3"],"NodeId":"b72660bc8e28725f612d33bd675f8341","ExtentSize":4096},"e7e08adf3c8cd3dee08dd006b1596074":{"State":"online","Info":{"name":"/dev/sdc","storage":{"total":292835328,"free":145293312,"used":147542016},"id":"e7e08adf3c8cd3dee08dd006b1596074"},"Bricks":["800e2a99110af93ca69bd950c9ea7ef8","980bd302213bbdd2d9aa09dbe8abf62b","ef253aa4b2756fc293aa101c2eb743cb","ff62a0d3e0fe5c53c3725235d2832893"],"NodeId":"b72660bc8e28725f612d33bd675f8341","ExtentSize":4096}},"blockvolumeentries":{},"dbattributeentries":{"DB_BRICK_HAS_SUBTYPE_FIELD":{"Key":"DB_BRICK_HAS_SUBTYPE_FIELD","Value":"yes"},"DB_CLUSTER_HAS_FILE_BLOCK_FLAG":{"Key":"DB_CLUSTER_HAS_FILE_BLOCK_FLAG","Value":"yes"},"DB_GENERATION_ID":{"Key":"DB_GENERATION_ID","Value":"d90958f6a026bc80efe34bea84fa074f"},"DB_HAS_PENDING_OPS_BUCKET":{"Key":"DB_HAS_PENDING_OPS_BUCKET","Value":"yes"}},"pendingoperations":{}}heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 250.763µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 169.08µs [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 148.125µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 496.624µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 101.673µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 209.681µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 103.595µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 297.594µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 96.75µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 477.007µs [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 165.38µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 488.253µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 171.836µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 208.59µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 125.141µs [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.687µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 392.814µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 208.28µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 360.289µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 169.465µs [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 197.691µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:10:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 349.65µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:10:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 80.175µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:10:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 191.618µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:10:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 104.703µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:10:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 139.424µs [heketi] WARNING 2019/01/23 12:10:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 356.587µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:11:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 603.166µs [heketi] WARNING 2019/01/23 12:11:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 409.484µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:11:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 224.426µs [heketi] WARNING 2019/01/23 12:11:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 244.469µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:11:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 198.034µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:11:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 91.59µs [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 128.194µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:11:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 227.904µs [heketi] WARNING 2019/01/23 12:11:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 129.223µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:12:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 203.491µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:12:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 109.563µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:12:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 262.199µs [heketi] WARNING 2019/01/23 12:12:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 168.901µs [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 111.988µs [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:12:34 operations in-flight (8) exceeds limit (8) [negroni] Started POST /volumes [negroni] Completed 429 Too Many Requests in 339.567µs [heketi] WARNING 2019/01/23 12:12:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 304.878µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:12:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 357.406µs [heketi] WARNING 2019/01/23 12:12:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 400.669µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:13:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 240.076µs [heketi] WARNING 2019/01/23 12:13:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 118.418µs [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:13:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 230.416µs [heketi] WARNING 2019/01/23 12:13:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 178.939µs
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
