Hi Igor,
Just to update you on our progress.
- We've not had another repro of this since switching to bitmap allocator / upgrading to the latest octopus release. I'll try to gather the full set of diags if we do see this again.
- I think my issues with an empty /var/lib/ceph/osd/ceph-N/ folder are because we're running ceph in container which is using a mounted filesystem. As soon as I stop the OSD, the container terminated and the filesystem disappears. There's probably a way to decouple the ceph process from the lifetime of the container, but I've not figured it out yet.
Cheers again for all your help,
Dave
-----Original Message-----
From: Igor Fedotov <ifedotov@xxxxxxx>
Sent: 26 July 2021 13:30
To: Dave Piper <david.piper@xxxxxxxxxxxxx>; ceph-users@xxxxxxx
Subject: Re: [EXTERNAL] Re: OSDs flapping with "_open_alloc loaded 132 GiB in 2930776 extents available 113 GiB"
Dave,
please see inline
On 7/26/2021 1:57 PM, Dave Piper wrote:
Hi Igor,
So to get more verbose but less log one can set both debug-bluestore and debug-bluefs to 1/20. ...
More verbose logging attached. I've trimmed the file to a single restart attempt to keep the filesize down; let me know if there's not enough here.
Jul 26 10:25:07 condor_sc0 docker[19100]: -9628>
2021-07-26T10:25:05.512+0000 7f9b3ed48f40 20 bluefs _read got 32768
Jul 26 10:25:07 condor_sc0 docker[19100]: -9627>
2021-07-26T10:25:05.512+0000 7f9b3ed48f40 10 bluefs _read h
0x563e8bd3ff80 0xb2d0000~8000 from file(ino 316842 size 0xe6a476e
mtime
2021-07-14T15:54:21.751044+0000 allocated e6b0000 extents
[1:0x1128740000~10000,1:0x1128780000~10000,1:0x112ad60000~10000,1:0x1
165fa0000~10000,1:0x11662a0000~10000,1:0x1167410000~10000,1:0x116be10
000~10000,1:0x112a330000~20000,1:0x1170890000~10000,1:0x11751e0000~10
000,1:0x11770c0000~10000,1:0x1177760000~10000,1:0x1178050000~10000,1:
0x117b2d0000~10000,1:0x117b580000~10000,1:0x12a9770000~10000,1:0x12a9
820000~10000,1:0x12a98d0000~10000,1:0x12a98f0000~10000,1:0x12a9930000
~10000,1:0x12a9a00000~10000,1:0x12a9b30000~10000,1:0x12a9bd0000~10000
,1:0x12a9c30000~10000,1:0x12a9c70000~10000,1:0x12a9ca0000~10000,1:0x1
2a9cc0000~10000,1:0x12a9df0000~10000,1:0x12a9e90000~10000,1:0x12a9ec0
000~10000,1:0x12a9f30000~10000,1:0x12a9f90000~10000,1:0x12a9ff0000~10
000,1:0x12aa020000~10000,1:0x12aa050000~10000,1:0x12aa090000~10000,1:
0x12aa0b0000~10000,1:0x12aa150000~10000,1:0x12aa220000~10000,1:0x12aa
370000~10000,1:0x12aa410000~10000,1:0x12aa440000~10000,1:0x12aa490000
~10000,1:0x12aa4c0000~10000,1:0x12aa510000~10000,1:0x12aa5a0000~10000
,1:0x12aa7b0000~10000,1:0x12aa910000~10000,1:0x12aa950000~10000,1:0x1
2aa970000~10000,1:0x12aa9a0000~10000,1:0x12aa9d0000~10000,1:0x12aab60
000~10000,1:0x12aac10000~10000,1:0x12ad420000~10000,1:0x12ad4d0000~10
000,1:0x12ad7a0000~10000,1:0x12ad810000~10000,1:0x12e4450000~10000,1:
0x12e7fe0000~10000,1:0x12ecc40000~10000,1:0x12f2510000~10000,1:0x12f2
bd0000~10000,1:0x12f3430000~10000,1:0x12f5210000~10000,1:0x132a050000
~10000,1:0x133f8e0000~10000,1:0x13433d0000~10000,1:0x1344ef0000~10000
,1:0x13478d0000~10000,1:0x134b7f0000~10000,1:0x134d450000~10000,1:0x1
34dc60000~10000,1:0x134dc80000~10000,1:0x13511c0000~10000,1:0x1351230
000~10000,1:0x1352af0000~10000,1:0x1355cc0000~10000,1:0x135ec80000~10
000,1:0x136cfb0000~10000,1:0x13737a0000~10000,1:0x1374750000~10000,1:
0x137fe70000~10000,1:0x1392590000~10000,1:0x1393a20000~10000,1:0x1396
670000~10000,1:0x139d560000~10000,1:0x13a17d0000~10000,1:0x13a20e0000
~10000,1:0x13a2ea0000~10000,1:0x13a5420000~10000,1:0x13a6d70000~10000
,1:0x13abef0000~10000,1:0x13bc840000~10000,1:0x13caa00000~10000,1:0x1
3cedf0000~10000,1:0x2a00540000~10000,1:0x2a02630000~10000,1:0x2a04100
000~10000,1:0x2a08550000~10000,1:0x2a091e0000~10000,1:0x2a09410000~10
000,1:0x2a17c70000~10000,1:0x2a1a860000~10000,1:0x2a1a930000~10000,1:
0x2a1fa00000~10000,1:0x2a21bc0000~10000,1:0x2a24d20000~10000,1:0x2a2b
5a0000~10000,1:0x2a2c9b0000~10000,1:0x2a36c30000~10000,1:0x2a37c40000
~10000,1:0x2a39c60000~10000,1:0x2a43480000~10000,1:0x2a43b80000~10000
,1:0x2a46250000~10000,1:0x2a4ba60000~10000,1:0x2a526b0000~10000,1:0x2
a55b00000~10000,1:0x2a59920000~10000,1:0x2a5fee0000~10000,1:0x2a686c0
000~10000,1:0x2a76280000~10000,1:0x2a8b7d0000~10000,1:0x2a91f00000~10
000,1:0x2a98930000~10000,1:0x2a9b4f0000~10000,1:0x2a9e720000~10000,1:
0x2aa5920000~10000,1:0x2aac110000~10000,1:0x2ab57f0000~10000,1:0x2ac2
690000~10000,1:0x2ac57e0000~10000,1:0x2ad6710000~10000,1:0x2ad9040000
~10000,1:0x2afa900000~10000,1:0x2afb110000~10000,1:0x2afd070000~10000
,1:0x2b15220000~10000,1:0x2b189b0000~10000,1:0x2b330f0000~10000,1:0x2
b33810000~10000,1:0x2b365a0000~10000,1:0x2b36aa0000~10000,1:0x2b36d60
000~10000,1:0x2b397b0000~10000,1:0x2b3f510000~10000,1:0x2b42960000~10
000,1:0x2b43e60000~10000,1:0x2b5e900000~10000,1:0x2b62060000~10000,1:
0x2b63060000~10000,1:0x2b652f0000~10000,1:0x2b7bf60000~10000,1:0x2b87
6d0000~10000,1:0x2b8a3f0000~10000,1:0x2b94810000~10000,1:0x2b951e0000
~10000,1:0x2b957e0000~10000,1:0x2b98a80000~10000,1:0x2b99480000~10000
,1:0x2b9b880000~10000,1:0x2ba1030000~10000,1:0x2ba2170000~10000,1:0x2
bb6670000~10000,1:0x2bd2870000~10000,1:0x2bd2a80000~10000,1:0x2bed7a0
000~10000,1:0x2bf3800000~10000,1:0x2bf44f0000~10000,1:0x2bf8d40000~10
000,1:0x2c129b0000~10000,1:0x2c142a0000~10000,1:0x2c171b0000~10000,1:
0x2c1abd0000~10000,1:0x2c1d810000~10000,1:0x2c23ea0000~10000,1:0x2c25
9b0000~10000,1:0x2c25f20000~10000,1:0x2c28330000~10000,1:0x2c320f0000
~10000,1:0x2c48430000~10000,1:0x2c56500000~10000,1:0x2c60fa0000~10000
,1:0x2c65a40000~10000,1:0x2c68f40000~10000,1:0x2c9c130000~10000,1:0x2
ca1350000~10000,1:0x2ca5880000~10000,1:0x2cd4380000~10000,1:0x2ce2b40
000~10000,1:0x2cf0e00000~10000,1:0x2d005a0000~10000,1:0x2d0f700000~10
000,1:0x2d0fc00000~10000,1:0x2d19ed0000~10000,1:0x2d3ddc0000~10000,1:
0x2d3fbc0000~10000,1:0x2d43040000~10000,1:0x2d6de50000~10000,1:0x13bc
d60000~20000,1:0x13bf180000~20000,1:0x13c3ec0000~20000,1:0x13c5a00000
~20000,1:0x13c6070000~20000,1:0x13ce790000~20000,1:0x13cfc70000~20000
,1:0x13d0970000~20000,1:0x2a08d60000~20000,1:0x2a09170000~20000,1:0x2
a09b30000~20000,1:0x2a11950000~20000,1:0x2a17890000~20000,1:0x2a28550
000~20000,1:0x2a34d20000~20000,1:0x2a57410000~20000,1:0x2a58c00000~20
000,1:0x2a5c7c0000~20000,1:0x2a5cf40000~20000,1:0x2a663b0000~20000,1:
0x2a68660000~20000,1:0x2a687e0000~20000,1:0x2a73d50000~20000,1:0x2a9f
ca0000~20000,1:0x2aa5300000~20000,1:0x2aaaaf0000~20000,1:0x2ab7700000
~20000,1:0x2ac1d70000~20000,1:0x2ac6f30000~20000,1:0x2ad93a0000~20000
,1:0x2ad9ad0000~20000,1:0x2ada2f0000~20000,1:0x2ae5ca0000~20000,1:0x2
ae82c0000~20000,1:0x2aebe10000~20000,1:0x13bcd20000~20000,1:0x2b0ff10
000~20000,1:0x2b11530000~20000,1:0x2b18790000~20000,1:0x2b35c80000~20
000,1:0x2b4d290000~20000,1:0x2b54d70000~20000,1:0x2b5cf40000~20000,1:
0x2b5d990000~20000,1:0x2b5e3e0000~20000,1:0x2b5ee10000~20000,1:0x2b5f
860000~20000,1:0x2b5fd80000~20000,1:0x2b8fc60000~20000,1:0x2b91fd0000
~20000,1:0x2b934f0000~20000,1:0x2ba72b0000~20000,1:0x2baaef0000~20000
,1:0x2bb7b80000~20000,1:0x2bbe290000~20000,1:0x2bc7460000~20000,1:0x2
bf7200000~20000,1:0x2c09340000~20000,1:0x2c217e0000~20000,1:0x2c231e0
000~20000,1:0x2c45f30000~20000,1:0x2c4c4f0000~20000,1:0x2c4c810000~20
000,1:0x2c4efa0000~20000,1:0x2c95750000~20000,1:0x2c99b30000~20000,1:
0x2c9b130000~20000,1:0x2ca5360000~20000,1:0x2cb9540000~20000,1:0x2cb9
c40000~20000,1:0x2cd8e90000~20000,1:0x2d399b0000~20000,1:0x2b05b20000
~20000,1:0x2d3d140000~20000,1:0x2d3dca0000~20000,1:0x11754f0000~30000
,1:0x12e67e0000~30000,1:0x1304830000~30000,1:0x139eba0000~30000,1:0x1
3a32e0000~30000,1:0x13b1680000~30000,1:0x13b3c00000~30000,1:0x13b95a0
000~30000,1:0x13bd6d0000~30000,1:0x13c3790000~30000,1:0x2a0bea0000~30
000,1:0x2a193b0000~30000,1:0x2a19830000~30000,1:0x2a1ab80000~30000,1:
0x2a1b570000~30000,1:0x2a29740000~30000,1:0x2a2a170000~30000,1:0x2a2b
c00000~30000,1:0x2a2ed80000~30000,1:0x2a36800000~30000,1:0x2a38750000
~30000,1:0x2a3e360000~10000,1:0x2d3a330000~20000,1:0x2a43150000~30000
,1:0x2a456f0000~30000,1:0x2a54c40000~30000,1:0x2a63f60000~30000,1:0x2
a665d0000~30000,1:0x2a6b350000~30000,1:0x2a791a0000~30000,1:0x2a7a5d0
000~30000,1:0x2a7f2e0000~30000,1:0x2a846c0000~30000,1:0x2a84df0000~30
000,1:0x2aa91f0000~30000,1:0x2aaf450000~30000,1:0x2ab3130000~30000,1:
0x2ab4ac0000~30000,1:0x2aca290000~30000,1:0x2addc80000~30000,1:0x2ae9
de0000~30000,1:0x2af3770000~30000,1:0x2b04740000~30000,1:0x2b06190000
~30000,1:0x2b07620000~30000,1:0x2b09880000~30000,1:0x2b294d0000~10000
,1:0x2a3e340000~20000,1:0x2b38870000~30000,1:0x2b5d460000~30000,1:0x2
b5deb0000~30000,1:0x2b5f330000~30000,1:0x2b602a0000~30000,1:0x2b64580
000~30000,1:0x2b81730000~30000,1:0x2b8b710000~30000,1:0x2b9b990000~30
000,1:0x2b9ed00000~30000,1:0x2ba1540000~30000,1:0x2ba5080000~30000,1:
0x2bafe50000~30000,1:0x2bd3ab0000~10000,1:0x2b294e0000~20000,1:0x2bd5
890000~30000,1:0x2bdc5d0000~30000,1:0x2bdc900000~30000,1:0x2bf19b0000
~30000,1:0x2c08e10000~30000,1:0x2c25bc0000~30000,1:0x2c33c00000~30000
,1:0x2c509d0000~30000,1:0x2c67770000~30000,1:0x2c6f5a0000~30000,1:0x2
c784e0000~30000,1:0x2c8c030000~10000,1:0x2bd3a90000~20000,1:0x2c8f2e0
000~30000,1:0x2c97b30000~30000,1:0x2cbb770000~30000,1:0x2cc0290000~30
000,1:0x2cc7110000~30000,1:0x2cd3ef0000~30000,1:0x2cde190000~30000,1:
0x2cf6f10000~30000,1:0x2d3ce10000~30000,1:0x2d63960000~30000,1:0x1125
ef0000~40000,1:0x112ac00000~40000,1:0x112ac80000~40000,1:0x1162280000
~40000,1:0x11686e0000~20000,1:0x2c8c010000~20000,1:0x116adf0000~40000
,1:0x116dd40000~40000,1:0x1170380000~40000,1:0x1175ac0000~40000,1:0x1
176e80000~40000,1:0x11777c0000~40000,1:0x1177820000~40000,1:0x1178270
000~40000,1:0x11787f0000~40000,1:0x11686c0000~20000,1:0x117fc30000~20
000,1:0x11807e0000~40000,1:0x1183a90000~40000,1:0x11845d0000~40000,1:
0x118aad0000~40000,1:0x118d230000~40000,1:0x118d4a0000~40000,1:0x1196
d80000~40000,1:0x12a84c0000~40000,1:0x12ae7c0000~20000,1:0x117fc10000
~20000,1:0x12b01a0000~40000,1:0x12b2710000~40000,1:0x12c70b0000~40000
,1:0x12cd280000~40000,1:0x12cee60000~40000,1:0x12d4690000~40000,1:0x1
2d8a90000~40000,1:0x12dd5e0000~40000,1:0x12df360000~40000,1:0x12e5e70
000~10000,1:0x12e5ea0000~10000,1:0x12ae7e0000~20000,1:0x12e6b10000~40
000,1:0x12e9730000~40000,1:0x12e9870000~40000,1:0x12ea390000~40000,1:
0x12eae80000~40000,1:0x12ec6b0000~40000,1:0x1312b90000~40000,1:0x1321
7e0000~40000,1:0x1327060000~40000,1:0x12e5e80000~20000,1:0x1333780000
~20000,1:0x1338b00000~40000,1:0x1338ef0000~40000,1:0x1339290000~40000
,1:0x133e320000~40000,1:0x133ebe0000~40000,1:0x1341650000~40000,1:0x1
3467e0000~40000,1:0x13469e0000~40000,1:0x1333760000~20000,1:0x1347f10
000~20000,1:0x1348c70000~40000,1:0x134c250000~40000,1:0x13502b0000~40
000,1:0x13685c0000~40000,1:0x136b9f0000~40000,1:0x136cb60000~40000,1:
0x136ce00000~40000,1:0x13731d0000~40000,1:0x13796c0000~20000,1:0x1347
ef0000~20000,1:0x137b700000~40000,1:0x1384a40000~40000,1:0x138ee10000
~40000,1:0x1399730000~40000,1:0x13b8130000~40000,1:0x13bb250000~40000
,1:0x13c3ad0000~40000,1:0x13c5c20000~40000,1:0x13c7320000~10000,1:0x1
3c7350000~10000,1:0x13796e0000~20000,1:0x13c74e0000~40000,1:0x13c8180
000~40000,1:0x13cb700000~40000,1:0x13cbc40000~40000,1:0x2a06450000~40
000,1:0x2a0d440000~40000,1:0x2a17450000~40000,1:0x2a1c4d0000~40000,1:
0x2a300b0000~10000,1:0x2a300e0000~10000,1:0x13c7330000~20000,1:0x2a37
f50000~40000,1:0x2a39320000~40000,1:0x2a3e690000~40000,1:0x2a47d50000
~40000,1:0x2a4d5b0000~40000,1:0x2a555d0000~40000,1:0x2a77a60000~40000
,1:0x2a7de60000~40000,1:0x2a300c0000~20000,1:0x2a888a0000~20000,1:0x2
a8a670000~40000,1:0x2a97750000~40000,1:0x2a9cec0000~40000,1:0x2aa8970
000~40000,1:0x2aa8bb0000~40000,1:0x2ac61e0000~40000,1:0x2ad5cc0000~40
000,1:0x2adc150000~40000,1:0x2a88880000~20000,1:0x2afd290000~20000,1:
0x2b06880000~40000,1:0x2b09db0000~40000,1:0x2b16790000~40000,1:0x2b26
a90000~40000,1:0x2b2bf90000~40000,1:0x2b2f9d0000~40000,1:0x2b48570000
~40000,1:0x2b4f890000~40000,1:0x2b5ca00000~10000,1:0x2b5ca30000~10000
,1:0x2afd270000~20000,1:0x2b604d0000~40000,1:0x2b61620000~40000,1:0x2
b69e50000~40000,1:0x2b6fb00000~40000,1:0x2b7a820000~40000,1:0x2b91080
000~40000,1:0x2b964f0000~40000,1:0x2b97f70000~40000,1:0x2b98f80000~20
000,1:0x2b5ca10000~20000,1:0x2bb3590000~40000,1:0x2bcdca0000~40000,1:
0x2bda690000~40000,1:0x2be93c0000~40000,1:0x2bfd6b0000~40000,1:0x2c35
b40000~40000,1:0x2c49df0000~40000,1:0x2c5f560000~40000,1:0x2c9d240000
~20000,1:0x2b98fa0000~20000,1:0x2cc4a50000~40000,1:0x2ccb1c0000~40000
,1:0x2cda090000~40000,1:0x2ced0b0000~40000,1:0x2d3d260000~40000,1:0x2
d43a40000~40000,1:0x2d7f160000~40000,1:0x2d7fdb0000~40000,1:0x11259b0
000~50000,1:0x112a9b0000~30000,1:0x2c9d260000~20000,1:0x112aab0000~50
000,1:0x116a9a0000~50000,1:0x1174ed0000~50000,1:0x1178990000~50000,1:
0x117f3a0000~50000,1:0x1183c60000~50000,1:0x118a020000~50000,1:0x118b
320000~50000,1:0x112a9e0000~20000,1:0x118c9f0000~30000,1:0x12964f0000
~50000,1:0x12acdd0000~50000,1:0x12b16b0000~50000,1:0x12d8cd0000~50000
,1:0x12e78d0000~50000,1:0x12ee0c0000~50000,1:0x12ff530000~50000,1:0x1
2ff780000~10000,1:0x118c9d0000~20000,1:0x12ff7b0000~20000,1:0x1306460
000~50000,1:0x13073b0000~50000,1:0x130d8e0000~50000,1:0x1315b40000~50
000,1:0x131da80000~50000,1:0x1324a10000~50000,1:0x1325360000~20000,1:
0x13253a0000~10000,1:0x12ff790000~20000,1:0x1331830000~50000,1:0x133d
7f0000~50000,1:0x1343ca0000~50000,1:0x1350a80000~50000,1:0x13570a0000
~50000,1:0x1358080000~50000,1:0x135ae50000~50000,1:0x135bb30000~20000
,1:0x135bb70000~10000,1:0x1325380000~20000,1:0x1360ca0000~50000,1:0x1
368510000~50000,1:0x1368f00000~50000,1:0x136b5a0000~50000,1:0x1377320
000~50000,1:0x1377470000~50000,1:0x1377640000~50000,1:0x135bb50000~20
000,1:0x137b010000~30000,1:0x13896b0000~50000,1:0x13981a0000~50000,1:
0x13a0980000~50000,1:0x13bc9d0000~50000,1:0x13be130000~50000,1:0x13c0
be0000~50000,1:0x137aff0000~20000,1:0x13c3160000~30000,1:0x13c3c10000
~50000,1:0x13c49a0000~50000,1:0x13c6280000~50000,1:0x13c6f90000~50000
,1:0x13cc5f0000~50000,1:0x13cf5e0000~50000,1:0x2a038b0000~50000,1:0x1
3c3140000~20000,1:0x2a065c0000~30000,1:0x2a140b0000~50000,1:0x2a14e00
000~50000,1:0x2a17d80000~50000,1:0x2a1ec10000~50000,1:0x2a318d0000~50
000,1:0x2a38ae0000~50000,1:0x2a38d50000~30000,1:0x2a065a0000~20000,1:
0x2a40ab0000~50000,1:0x2a43dd0000~50000,1:0x2a4c060000~50000,1:0x2a4e
e30000~50000,1:0x2a597d0000~50000,1:0x2a655f0000~50000,1:0x2a67700000
~50000,1:0x2a38d80000~20000,1:0x2a7c9f0000~30000,1:0x2a84070000~50000
,1:0x2ab4760000~50000,1:0x2abed50000~50000,1:0x2ac7d50000~50000,1:0x2
ac8560000~50000,1:0x2acaf20000~50000,1:0x2ad42f0000~20000,1:0x2ad4330
000~10000,1:0x2a7c9d0000~20000,1:0x2af58b0000~50000,1:0x2afb420000~50
000,1:0x2affa60000~50000,1:0x2b03920000~50000,1:0x2b05d40000~50000,1:
0x2b06fc0000~50000,1:0x2b12ce0000~50000,1:0x2b17860000~20000,1:0x2b17
8a0000~10000,1:0x2ad4310000~20000,1:0x2b4fd10000~50000,1:0x2b97520000
~50000,1:0x2b9e3b0000~50000,1:0x2be5ec0000~50000,1:0x2c11280000~50000
,1:0x2c29b60000~50000,1:0x2c2a7b0000~50000,1:0x2c2de50000~30000,1:0x2
b17880000~20000,1:0x2c6ba50000~50000,1:0x2c9d680000~50000,1:0x11622d0
000~60000,1:0x1166c20000~60000,1:0x1171050000~60000,1:0x1172020000~60
000,1:0x11771d0000~30000,1:0x1177220000~10000,1:0x2c2de80000~20000,1:
0x1177870000~60000,1:0x117abd0000~60000,1:0x117b840000~60000,1:0x117f
f90000~60000,1:0x1183a20000~60000,1:0x118b110000~40000,1:0x1177200000
~20000,1:0x118b180000~60000,1:0x118e1d0000~60000,1:0x128cdf0000~60000
,1:0x128ce60000~60000,1:0x12a82a0000~10000,1:0x118b150000~20000,1:0x1
2a82d0000~30000,1:0x12b39e0000~60000,1:0x12c03a0000~60000,1:0x12c9310
000~60000,1:0x12cbc50000~60000,1:0x12a82b0000~20000,1:0x12da380000~40
000,1:0x12e7490000~60000,1:0x12ec7c0000~60000,1:0x12ede10000~60000,1:
0x12ee660000~40000,1:0x12da360000~20000,1:0x12ef4c0000~60000,1:0x12f1
490000~60000,1:0x130e480000~60000,1:0x130f3a0000~60000,1:0x1311430000
~10000,1:0x12ee6a0000~20000,1:0x1311460000~30000,1:0x1311a90000~60000
,1:0x1311d00000~60000,1:0x1312260000~60000,1:0x131e4f0000~40000,1:0x1
311440000~20000,1:0x131f360000~60000,1:0x1322ab0000~60000,1:0x1347d50
000~60000,1:0x134ef60000~60000,1:0x135b0f0000~20000,1:0x131e530000~20
000,1:0x135b130000~20000,1:0x1369950000~60000,1:0x136a480000~60000,1:
0x137e590000~60000,1:0x137eb30000~40000,1:0x135b110000~20000,1:0x1382
d30000~60000,1:0x138e070000~60000,1:0x1395e40000~60000,1:0x1397330000
~60000,1:0x13a0cd0000~10000,1:0x137eb70000~20000,1:0x13a0d00000~30000
,1:0x13c9ff0000~60000,1:0x13cc9f0000~60000,1:0x2a08090000~60000,1:0x2
a252c0000~30000,1:0x2a25310000~10000,1:0x13a0ce0000~20000,1:0x2a33bc0
000~60000,1:0x2a38250000~60000,1:0x2a41100000~60000,1:0x2a436e0000~60
000,1:0x2a252f0000~20000,1:0x2a79bb0000~40000,1:0x2a80a70000~60000,1:
0x2a8aac0000~60000,1:0x2a9d9c0000~60000,1:0x2ab09f0000~30000,1:0x2ab0
a40000~10000,1:0x2a79b90000~20000,1:0x2abd150000~60000,1:0x2ac0950000
~60000,1:0x2ac10f0000~60000,1:0x2ada7e0000~60000,1:0x2ab0a20000~20000
,1:0x2af8660000~40000,1:0x2b02800000~60000,1:0x2b02960000~60000,1:0x2
b031c0000~60000,1:0x2af8640000~20000,1:0x2b15350000~40000,1:0x2b344c0
000~60000,1:0x2b35520000~60000,1:0x2b3a5d0000~60000,1:0x2b97710000~20
000,1:0x2b15330000~20000,1:0x2b97750000~20000,1:0x2bc22f0000~60000,1:
0x2bdf940000~60000,1:0x2bea590000~60000,1:0x2bf59f0000~40000,1:0x2b97
730000~20000,1:0x2bf9450000~60000,1:0x2c00080000~60000,1:0x2c75870000
~60000,1:0x2c93de0000~60000,1:0x2bf5a30000~20000,1:0x2c9dc00000~40000
,1:0x2cc47f0000~60000,1:0x2cd4220000~60000,1:0x2cec450000~60000,1:0x2
d3a0d0000~20000,1:0x2d3a110000~20000,1:0x2c9dbe0000~20000,1:0x2d3c940
000~60000,1:0x11663b0000~70000,1:0x1169f00000~70000,1:0x116a370000~30
000,1:0x116a3c0000~20000,1:0x2d3a0f0000~20000,1:0x116d510000~70000,1:
0x116f290000~70000,1:0x117f2e0000~70000,1:0x1182780000~20000,1:0x116a
3a0000~20000,1:0x11827c0000~30000,1:0x1188270000~70000,1:0x1189c90000
~70000,1:0x118ac20000~70000,1:0x118d0b0000~20000,1:0x11827a0000~20000
,1:0x118d0f0000~30000,1:0x118d6c0000~70000,1:0x118e440000~70000,1:0x1
28ccd0000~70000,1:0x128ced0000~20000,1:0x118d0d0000~20000,1:0x128cf10
000~30000,1:0x128dbb0000~70000,1:0x1297320000~70000,1:0x12a5500000~70
000,1:0x12b9b40000~20000,1:0x128cef0000~20000,1:0x12b9b80000~30000,1:
0x12e03b0000~70000,1:0x12e5d00000~70000,1:0x12e6670000~70000,1:0x12f2
9c0000~10000,1:0x12b9b60000~20000,1:0x12f29f0000~40000,1:0x1300220000
~70000,1:0x1300480000~70000,1:0x131c0c0000~70000,1:0x131c230000~10000
,1:0x12f29d0000~20000,1:0x131c260000~40000,1:0x1323350000~70000,1:0x1
32afe0000~70000,1:0x132c340000~70000,1:0x131c240000~20000,1:0x1331ba0
000~50000,1:0x1331df0000~70000,1:0x1335830000~70000,1:0x1336e50000~50
000,1:0x1331b80000~20000,1:0x1340220000~70000,1:0x134da70000~70000,1:
0x1350580000~70000,1:0x1353cb0000~40000,1:0x1353d10000~10000,1:0x1336
ea0000~20000,1:0x1360630000~70000,1:0x13648d0000~70000,1:0x1369190000
~70000,1:0x1371770000~50000,1:0x1353cf0000~20000,1:0x1373c10000~70000
,1:0x137fdf0000~70000,1:0x1385a80000~70000,1:0x138a460000~30000,1:0x1
3717c0000~20000,1:0x138a4b0000~20000,1:0x138ad60000~70000,1:0x138ef70
000~70000,1:0x13986f0000~70000,1:0x139e190000~70000,1:0x13afbc0000~40
000,1:0x13afc20000~10000,1:0x138a490000~20000,1:0x13bc490000~70000,1:
0x13bd080000~70000,1:0x13c3e10000~70000,1:0x13cb8d0000~70000,1:0x13cc
340000~70000,1:0x13afc00000~20000,1:0x29ffe30000~50000,1:0x2a09670000
~70000,1:0x2a10b00000~70000,1:0x2a20210000~70000,1:0x2a32650000~70000
,1:0x2a3a230000~30000,1:0x29ffe10000~20000,1:0x2a3a280000~20000,1:0x2
a4bdf0000~70000,1:0x2a54e90000~70000,1:0x2a56aa0000~70000,1:0x2a79cf0
000~70000,1:0x2a7ac00000~50000,1:0x2a3a260000~20000,1:0x2a7c060000~70
000,1:0x2a85e10000~70000,1:0x2a93f70000~70000,1:0x2aa12c0000~70000,1:
0x2ac3160000~70000,1:0x2a7ac50000~20000,1:0x2ad15d0000~50000,1:0x2b01
8b0000~70000,1:0x2b0bf80000~70000,1:0x2b419f0000~70000,1:0x2b5c490000
~70000,1:0x2c04d10000~40000,1:0x2c04d70000~10000,1:0x2ad15b0000~20000
,1:0x2c0d910000~70000,1:0x2c0fa80000~70000,1:0x2c150c0000~70000,1:0x2
c192b0000~70000,1:0x2c383a0000~70000,1:0x2c4eb30000~10000,1:0x2c04d50
000~20000,1:0x2c4eb60000~40000,1:0x2cc7340000~70000,1:0x2ce1dd0000~70
000,1:0x2d00bb0000~70000,1:0x2d083c0000~70000,1:0x2d414d0000~50000,1:
0x2c4eb40000~20000,1:0x1125570000~80000,1:0x1125de0000~80000,1:0x112a
cd0000~80000,1:0x116c310000~80000,1:0x11708b0000~30000,1:0x2d41520000
~20000,1:0x1170900000~30000,1:0x1172cb0000~80000,1:0x1175720000~80000
,1:0x11792f0000~80000,1:0x117baa0000~80000,1:0x117e810000~30000,1:0x1
1708e0000~20000,1:0x117e860000~30000,1:0x117ef60000~80000,1:0x1183350
000~80000,1:0x1185d90000~80000,1:0x1188c80000~80000,1:0x117e840000~20
000,1:0x118d760000~60000,1:0x118d890000~80000,1:0x128d8b0000~80000,1:
0x12981c0000~80000,1:0x12a7ed0000~60000,1:0x118d740000~20000,1:0x12ad
1b0000~80000,1:0x12ad860000~80000,1:0x12cb1a0000~80000,1:0x12cece0000
~80000,1:0x12da0e0000~30000,1:0x12a7f30000~20000,1:0x12da130000~30000
,1:0x12e6c50000~80000,1:0x12eac00000~80000,1:0x12eb0c0000~80000,1:0x1
2eb840000~80000,1:0x12f1c90000~20000,1:0x12da110000~20000,1:0x12f1cd0
000~40000,1:0x12ffea0000~80000,1:0x130ce30000~80000,1:0x130e6a0000~80
000,1:0x130fa60000~80000,1:0x12f1cb0000~20000,1:0x13237a0000~60000,1:
0x132ac60000~80000,1:0x132b230000~80000,1:0x132c0c0000~80000,1:0x1338
7e0000~80000,1:0x1323780000~20000,1:0x133c6a0000~60000,1:0x13414d0000
~80000,1:0x13485b0000~80000,1:0x134eaa0000~80000,1:0x1353ab0000~60000
,1:0x133c680000~20000,1:0x13560a0000~80000,1:0x1366b40000~80000,1:0x1
367960000~80000,1:0x13683d0000~80000,1:0x1370260000~50000,1:0x13702d0
000~10000,1:0x1353b10000~20000,1:0x13743d0000~80000,1:0x1383b20000~80
000,1:0x138a9b0000~80000,1:0x138c6e0000~80000,1:0x1390ee0000~30000,1:
0x13702b0000~20000,1:0x1390f30000~30000,1:0x13928b0000~80000,1:0x1398
d80000~80000,1:0x139f970000~80000,1:0x13b1ab0000~80000,1:0x1390f10000
~20000,1:0x13ba5f0000~60000,1:0x13ca490000~80000,1:0x13d10f0000~80000
,1:0x2a05d50000~80000,1:0x2a075d0000~60000,1:0x13ba5d0000~20000,1:0x2
a224d0000~80000,1:0x2a2e600000~80000,1:0x2a390a0000~80000,1:0x2a481f0
000~80000,1:0x2a4bc70000~30000,1:0x2a07630000~20000,1:0x2a4bcc0000~30
000,1:0x2a5a6c0000~80000,1:0x2a5d160000~80000,1:0x2a75700000~80000,1:
0x2a7dce0000~80000,1:0x2a86e90000~20000,1:0x2a4bca0000~20000,1:0x2a86
ed0000~40000,1:0x2a8d920000~80000,1:0x2a98f40000~80000,1:0x2abae10000
~60000,1:0x2a86eb0000~20000,1:0x2ad2320000~80000,1:0x2b00d30000~80000
,1:0x2b07c10000~80000,1:0x2b66200000~40000,1:0x2abae70000~20000,1:0x2
b66260000~20000,1:0x2b6ba90000~80000,1:0x2b7d880000~80000,1:0x2be9700
000~80000,1:0x2c15230000~10000,1:0x2b66240000~20000,1:0x2c15260000~50
000,1:0x2c2f160000~80000,1:0x2cad5a0000~80000,1:0x2ce3050000~50000,1:
0x2ce30c0000~10000,1:0x2c15240000~20000,1:0x2ce3cd0000~80000,1:0x2d0f
980000~80000,1:0x2d59e90000~80000,1:0x2d5a210000~10000,1:0x2ce30a0000
~20000,1:0x2d5a240000~50000,1:0x11279a0000~90000,1:0x1171430000~90000
,1:0x1175f00000~30000,1:0x2d5a220000~20000,1:0x1175f50000~40000,1:0x1
177b00000~90000,1:0x11781d0000~90000,1:0x11782f0000~70000,1:0x1175f30
000~20000,1:0x117b3e0000~90000,1:0x11803d0000~90000,1:0x1180650000~90
000,1:0x1178360000~20000,1:0x1187320000~70000,1:0x11884f0000~90000,1:
0x118c930000~90000,1:0x118d610000~10000,1:0x1187300000~20000,1:0x118d
640000~60000,1:0x118d7f0000~90000,1:0x1284d20000~90000,1:0x128cd50000
~20000,1:0x118d620000~20000,1:0x128cd90000~50000,1:0x1297030000~90000
,1:0x12aec10000~90000,1:0x12b2f50000~40000,1:0x128cd70000~20000,1:0x1
2b2fb0000~30000,1:0x12b4530000~90000,1:0x12ba3b0000~90000,1:0x12d58e0
000~60000,1:0x12d5960000~10000,1:0x12b2f90000~20000,1:0x12dd820000~80
000]) Jul 26 10:25:07 condor_sc0 docker[19100]: -9626>
2021-07-26T10:25:05.512+0000 7f9b3ed48f40 20 bluefs _read fetching
0x0~80000 of 1:0x1185d90000~80000
Jul 26 10:25:07 condor_sc0 docker[19100]: -9625>
2021-07-26T10:25:05.513+0000 7f9b3ed48f40 20 bluefs _read left
0x80000 len 0x8000
^^^ I expected this stuff to count up to -1 where the most interesting stuff should be...
Jul 26 10:25:09 condor_sc0 docker[19550]: Error response from daemon:
No such container: ceph-osd-1
Jul 26 10:25:19 condor_sc0 systemd[1]: ceph-osd@1.service holdoff
time over, scheduling restart.
Jul 26 10:25:19 condor_sc0 systemd[1]: Stopped Ceph OSD.
Jul 26 10:25:19 condor_sc0 systemd[1]: Starting Ceph OSD...
It would be also great to collect the output for the following commands: ...
I've tried running ceph-bluestore-tool previously on this system but both commands fails with the following error:
[qs-admin@condor_sc0 ~]$ sudo docker exec 419d997e5a05
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-1 --command
bluefs-stats error from cold_open: (11) Resource temporarily
unavailable
2021-07-26T10:50:15.032+0000 7f9a9bf68240 -1
bluestore(/var/lib/ceph/osd/ceph-1) _lock_fsid failed to lock
/var/lib/ceph/osd/ceph-1/fsid (is another ceph-osd still
running?)(11) Resource temporarily unavailable
[qs-admin@condor_sc0 ~]$
There's only one OSD running on this server; should I be stopping it / the other OSDs in the cluster before running the `ceph-bluestore-tool` command? Previously when the OSDs were failing to start, /var/lib/ceph/osd/ceph-1/ was empty but it now contains the following:
Each folder under /var/lib/ceph/osd/ceph-N/ should be used
exclusively by a single OSD. ceph-bluestore-tool to be run when the
respective OSD is shutdown, other OSDs referring to different folders shouldn't matter.
Is the above valid for your case?
[qs-admin@condor_sc0 ~]$ sudo docker exec 419d997e5a05 ls
/var/lib/ceph/osd/ceph-1 block ceph_fsid fsid keyring ready
require_osd_release type whoami
[qs-admin@condor_sc0 ~]$
And finally you can try to switch to bitmap allocator as a workaround ...
Switching to the bitmap allocator as you suggested has led to both failing OSDs starting up successfully. I've now got 3/3 OSDs up and in! The cluster still has MDS issues that were blocked behind getting the OSDs running as mentioned in my original post, but I think these are unrelated to the OSD problem as it's an issue we've seen in isolation elsewhere.
So - that's a big step forward! Should I retry with my original config on the latest octopus release and see if this is now fixed?
yeah, looks like this is a hybrid allocator bug - hence you can
upgrade and check if this is fixed.
Cheers again,
Dave
-----Original Message-----
From: Igor Fedotov <ifedotov@xxxxxxx>
Sent: 26 July 2021 11:14
To: Dave Piper <david.piper@xxxxxxxxxxxxx>; ceph-users@xxxxxxx
Subject: Re: [EXTERNAL] Re: OSDs flapping with "_open_alloc loaded 132 GiB in 2930776 extents available 113 GiB"
Hi Dave,
Some notes first:
1) The following behavior is fine, BlueStore mounts in two stages - the first one is read-only and among other things it loads allocation map from DB. And that's exactly the case here.
Jul 26 08:55:31 condor_sc0 docker[15282]:
2021-07-26T08:55:31.703+0000
7f0e15b3df40 1 bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc
loaded
132 GiB in 2930776 extents available 113 GiB Jul 26 08:55:31
condor_sc0 docker[15282]: 2021-07-26T08:55:31.703+0000
7f0e15b3df40 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all
background work Jul 26 08:55:31 condor_sc0 docker[15282]:
2021-07-26T08:55:31.704+0000
7f0e15b3df40 4 rocksdb: [db/db_impl.cc:563] Shutdown complete
2) What's really broken is the following allocation attempt:
Jul 26 08:55:34 condor_sc0 docker[15282]:
2021-07-26T08:55:34.767+0000
7f0e15b3df40 1 bluefs _allocate failed to allocate 0x100716 on bdev
1, free 0xd0000; fallback to bdev 2 Jul 26 08:55:34 condor_sc0
docker[15282]: 2021-07-26T08:55:34.767+0000
7f0e15b3df40 1 bluefs _allocate unable to allocate 0x100716 on bdev
2, free 0xffffffffffffffff; fallback to slow device expander Jul 26
08:55:35 condor_sc0 docker[15282]: 2021-07-26T08:55:35.042+0000
7f0e15b3df40 -1 bluestore(/var/lib/ceph/osd/ceph-1)
allocate_bluefs_freespace failed to allocate on 0x40000000 min_size
0x110000 > allocated total 0x0 bluefs_shared_alloc_size 0x10000
allocated 0x0 available 0x 1c09738000 Jul 26 08:55:35 condor_sc0
docker[15282]: 2021-07-26T08:55:35.044+0000
7f0e15b3df40 -1 bluefs _allocate failed to expand slow device to fit
+0x100716
Jul 26 08:55:35 condor_sc0 docker[15282]:
2021-07-26T08:55:35.044+0000
7f0e15b3df40 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
0x100716
This occurs during BlueFS recovery and that's an attempt to get more space to write out the bluefs log. This shouldn't fail given the plenty of free space:
... available 0x 1c09738000 ...
So to get more verbose but less log one can set both debug-bluestore and debug-bluefs to 1/20. This way just last 10000 lines of the log preceeding the crash would be at level 20. Which seems sufficient for the troubleshooting.
It would be also great to collect the output for the following commands:
ceph-bluestore-tool --path <osd-dir> --command bluefs-bdev-sizes
ceph-bluestore-tool --path <osd-dir> --command bluefs-stats
And finally you can try to switch to bitmap allocator as a workaround - we've fixed a couple of issues in Hybrid one which prevented from proper allcoations under some circumstances. The fixes were made after v15.2.11 release hence this might be the case. So please try setting:
bluestore_allocator = bitmap
bluefs_allocator = bitmap
Thanks,
Igor
On 7/26/2021 12:14 PM, Dave Piper wrote:
Hi Igor,
Thanks for your time looking into this.
I've attached a 5 minute window of OSD logs, which includes several restart attempt (each one takes ~25 seconds).
When I said it looked like we were starting up in a different state, I'm referring to how "Recovered from manifest file" log appears twice, with different logs afterwards. This behaviour seems to repeat reliably on each restart of the OSD. My interpretation of this was that when the initial recovery attempt leads to the rocksdb shutdown, ceph is automatically trying to start the OSD in some alternative state but that this is also failing (with the bdev errors I copied in). Possibly I'm inferring too much.
I tried turning up the logging levels for rocksdb and bluestore but they're both very spammy so I've not included this in the attached logs. Let me know if you think that would be helpful.
My ceph version is 15.2.11. We're running a containerized deployment using docker image ceph-daemon:v5.0.10-stable-5.0-octopus-centos-8 .
[qs-admin@condor_sc0 metaswitch]$ sudo docker exec b732f9135b42
ceph version ceph version 15.2.11
(e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)
Cheers,
Dave
-----Original Message-----
From: Igor Fedotov <ifedotov@xxxxxxx>
Sent: 23 July 2021 20:45
To: Dave Piper <david.piper@xxxxxxxxxxxxx>; ceph-users@xxxxxxx
Subject: [EXTERNAL] Re: OSDs flapping with "_open_alloc loaded 132 GiB in 2930776 extents available 113 GiB"
Hi Dave,
The follow log line indicates that allocator has just completed loading information about free disk blocks into memory. And it looks perfectly fine.
_open_alloc loaded 132 GiB in 2930776 extents available 113 GiB
Subsequent rocksdb shutdown looks weird without any other log output indicating the issue.
Curious what do you mean under "
After that we seem to try starting up in a slightly different state and get a different set of errors:
"
The resulted errors show lack of disk space at some point but I'd definitely like to get the full startup log.
Please also specify which Octopus version do you have?
Thanks,
Igor
On 7/23/2021 6:48 PM, Dave Piper wrote:
Hi all,
We've got a containerized test cluster with 3 OSDs and ~ 220GiB of data. Shortly after upgrading from nautilus -> octopus, 2 of the 3 OSDs have started flapping. I've also got alarms about the MDS being damaged, which we've seen elsewhere and have a recovery process for, but I'm unable to run this (I suspect because I've only got 1 functioning OSD). My RGWs are also failing to start, again I suspect because of the bad state of OSDs. I've tried restarting all OSDs, rebooting all servers, checked auth (all looks fine) - but I'm still in the same state.
My OSDs seem to be failing at the "_open_alloc opening allocation metadata" step; looking at logs for each OSD restart, the OSD writes this log, then no logs for a few minutes and then logs:
bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 132 GiB in 2930776 extents available 113 GiB
rocksdb: [db/db_impl.cc:390] Shutdown: canceling all
background work
After that we seem to try starting up in a slightly different state and get a different set of errors:
bluefs _allocate failed to allocate 0x100716 on bdev 1, free 0xd0000; fallback to bdev 2
bluefs _allocate unable to allocate 0x100716 on bdev 2,
free 0xffffffffffffffff; fallback to slow device expander
and eventually crash and log a heap of stack dumps.
I don't know what extents are but I seem to have a lot of them, and more than I've got capacity for? Maybe I'm running out of RAM or disk space somewhere, but I've got 21GB of free RAM on the server, and each OSD has a 350GiB device attached to it.
I'm wondering if anyone has seen anything like this before or can suggest next debug steps to take?
Cheers,
Dave
Full OSD logs surrounding the "_open_alloc opening allocation metadata" step:
Jul 23 00:07:13 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:13.818+0000 7f3de111bf40 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1626998833819439, "job": 1, "event":
"recovery_started", "log_files": [392088, 392132]}
Jul 23 00:07:13 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:13.818+0000 7f3de111bf40 4 rocksdb:
[db/db_impl_open.cc:583] Recovering log #392088 mode 0
Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:17.240+0000 7f3de111bf40 4 rocksdb:
[db/db_impl_open.cc:583] Recovering log #392132 mode 0
Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:17.486+0000 7f3de111bf40 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1626998837486404, "job": 1, "event":
"recovery_finished"}
Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:17.486+0000 7f3de111bf40 1
bluestore(/var/lib/ceph/osd/ceph-1) _open_db opened rocksdb path
db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buf
fer
_
number_to_merge=1,recycle_log_file_num=4,write_buffer_size=2684354
56,
w
ritable_file_max_buffer_size=0,compaction_readahead_size=2097152,m
ax_
b
ackground_compactions=2
Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:17.524+0000 7f3de111bf40 1 freelist init
Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:17.524+0000 7f3de111bf40 1 freelist
_init_from_label
Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:17.529+0000 7f3de111bf40 1
bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc opening allocation
metadata
Jul 23 00:07:18 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:18.238+0000 7f3de111bf40 1 HybridAllocator
_spillover_range constructing fallback allocator
Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:20.563+0000 7f3de111bf40 1
bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 132 GiB in
2930776 extents available 113 GiB
Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:20.563+0000 7f3de111bf40 4 rocksdb:
[db/db_impl.cc:390] Shutdown: canceling all background work
Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
2021-07-23T00:07:20.565+0000 7f3de111bf40 4 rocksdb:
[db/db_impl.cc:563] Shutdown complete
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send
an email to ceph-users-leave@xxxxxxx
