I have just checked we do not have pglog_hardlimt set, it could be worth a
try, the initial start for this was down to a PGNUM increase in one of the
pools. Commiting this took the cluster offline initially..
Its almost as if this is being processed when booting the OSD.
On Thu, 6 Jan 2022 at 13:50, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote:
> Hi,
>
> I'm not sure if it's 100% related, but in addition to the large
> buffer_anon, there is also far too much memory used by the osd_pglog:
>
> "osd_pglog": {
> "items": 45797772,
> "bytes": 4854818176
> },
>
> Do the related PGs have a very large number of pglog entries? (See the
> LOG and DISK_LOG columns in `ceph pg dump`)
> Do you have pglog_hardlimit enabled? (`ceph osd dump | grep pglog`)
>
> Normally a pglog can grow to ~10k entries -- if you find some with
> many times more, you may find it useful to trim the pglog. There is a
> way to do this with ceph-objectstore-tool -- but it should only be
> used under expert advice, because it may cause data loss.
>
> If your pglogs are under 10k entries long, then they normally
> shouldn't consume so many gigabytes. So there is likely some unknown
> issue causing even just a few thousand entries to consume gigs of
> memory. Have you shared the entire debug_osd=20 logs somewhere?
>
> BTW there are several related trackers:
>
> https://tracker.ceph.com/issues/53729
> https://tracker.ceph.com/issues/51609
> https://tracker.ceph.com/issues/47775
>
> (And a related thread from our less serious instance of this type of
> bug:
> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7IMIWCKIHXNULEBHVUIXQQGYUDJAO2SF/
> )
>
> Cheers, Dan
>
> On Thu, Jan 6, 2022 at 12:24 PM 胡 玮文 <huww98@xxxxxxxxxxx> wrote:
> >
> > Hi Lee,
> >
> > I assume the huge memory consumption is temporary. Once the OSD is up
> and stable, it would release the memory.
> >
> > So how about allocate a large swap temporarily just to let the OSD up. I
> remember that someone else on the list have resolved a similar issue with
> swap.
> >
> > Weiwen Hu
> >
> > > 在 2022年1月6日，15:21，Lee <lquince@xxxxxxxxx> 写道：
> > >
> > > Hi,
> > >
> > > Is there any sort of work around for this? I have 4 OSD's still
> off-line
> > > spread across three nodes. I cannot for love nor money get them to
> start,
> > > if I am correct I actually only need one more to boot and I should be
> home
> > > and dryish..
> > >
> > > Its the same issue every time the daemon starts then logs about shards
> and
> > > dies.. I cannot physically get more than 196gb of Ram in the servers
> due to
> > > DIMM's and slots. I've tried add a swap and that fails also.
> > >
> > > What are basically my next steps? As far as I know, im yet to lose
> data,
> > > its just stuck in the OSD's that will not start.
> > >
> > > So far its been a long night.
> > >
> > > Kind Regards
> > >
> > > Lee
> > >
> > >> On Thu, 6 Jan 2022 at 02:37, Lee <lquince@xxxxxxxxx> wrote:
> > >>
> > >> Hi Igor,
> > >>
> > >> As it starts to eat memory. It's basically when it is resizing
> shards..
> > >> I've so far managed only to recover one OSD.. Just about to add
> additional
> > >> memory to another node.
> > >>
> > >> 2022-01-06 02:22:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:13.575+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:13.627+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:13.675+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:13.727+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:13.775+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:13.827+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:14 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:14.027+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:14 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:14.079+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:14 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:14.127+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:14 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:14.179+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:14 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:14.227+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:14 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:14.279+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:14 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:14.327+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.031+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.079+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.131+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.175+0000 7f22de971700 5 osd.51 24448261 heartbeat
> > >> osd_stat(store_statfs(0x2231238000/0x67710000/0x3a38800000, data
> > >> 0x17919fd8c4/0x179feb4000, compress 0x0/0x0/0x0, omap 0x45519, meta
> > >> 0x676caae7), peers [] op hist [])
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.175+0000 7f22de971700 20 osd.51 24448261
> > >> check_full_status cur ratio 0.412718, physical ratio 0.412718, new
> state
> > >> none
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.179+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.231+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.283+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.331+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.383+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:15 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:15.431+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.015+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.063+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.115+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.163+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.215+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.615+0000 7f22fcc26700 10 osd.51 24448261
> do_waiters --
> > >> start
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.615+0000 7f22fcc26700 10 osd.51 24448261
> do_waiters --
> > >> finish
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.619+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.615+0000 7f22fcc26700 20 osd.51 24448261 tick
> > >> last_purged_snaps_scrub 2022-01-06T00:25:27.654967+0000 next
> > >> 2022-01-07T00:25:27.654967+0000
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.667+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.719+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.771+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.819+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.923+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.975+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.615+0000 7f22fcc26700 10 osd.51 24448261
> do_waiters --
> > >> start
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.615+0000 7f22fcc26700 10 osd.51 24448261
> do_waiters --
> > >> finish
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.619+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.615+0000 7f22fcc26700 20 osd.51 24448261 tick
> > >> last_purged_snaps_scrub 2022-01-06T00:25:27.654967+0000 next
> > >> 2022-01-07T00:25:27.654967+0000
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.667+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.719+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.771+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.819+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.923+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >> 2022-01-06T02:22:16.975+0000 7f22f519e700 20
> > >> bluestore.MempoolThread(0x55bed6dcaa98) _resize_shards cache_size:
> > >> 134217728 kv_alloc: 67108864 kv_used: 65059840 meta_alloc: 67108864
> > >> meta_used: 79017 data_alloc: 67108864 data_used: 0
> > >> 2022-01-06 02:22:25 bb-ceph-enc-rm63-osd03-31 init.scope
> > >> ceph-osd@51.service: Main process exited, code=killed, status=9/KILL
> > >> 2022-01-06 02:22:25 bb-ceph-enc-rm63-osd03-31 init.scope
> > >> ceph-osd@51.service: Failed with result 'signal'.
> > >> 2022-01-06 02:22:35 bb-ceph-enc-rm63-osd03-31 init.scope
> > >> ceph-osd@51.service: Scheduled restart job, restart counter is at 1.
> > >>
> > >>
> > >> OSD Config > Deployed by Croit.
> > >> "name": "osd.51",
> > >> "cluster": "ceph",
> > >> "admin_socket": "/var/run/ceph/ceph-osd.51.asok",
> > >> "admin_socket_mode": "",
> > >> "allow_ansi": "Terminal",
> > >> "auth_allow_insecure_global_id_reclaim": "true",
> > >> "auth_client_required": "cephx",
> > >> "auth_cluster_required": "cephx",
> > >> "auth_debug": "false",
> > >> "auth_expose_insecure_global_id_reclaim": "true",
> > >> "auth_mon_ticket_ttl": "259200.000000",
> > >> "auth_service_required": "cephx",
> > >> "auth_service_ticket_ttl": "3600.000000",
> > >> "auth_supported": "cephx",
> > >> "bdev_aio": "true",
> > >> "bdev_aio_max_queue_depth": "1024",
> > >> "bdev_aio_poll_ms": "250",
> > >> "bdev_aio_reap_max": "16",
> > >> "bdev_async_discard": "false",
> > >> "bdev_block_size": "4096",
> > >> "bdev_debug_aio": "false",
> > >> "bdev_debug_aio_log_age": "5.000000",
> > >> "bdev_debug_aio_suicide_timeout": "60.000000",
> > >> "bdev_debug_inflight_ios": "false",
> > >> "bdev_enable_discard": "false",
> > >> "bdev_flock_retry": "3",
> > >> "bdev_flock_retry_interval": "0.100000",
> > >> "bdev_inject_crash": "0",
> > >> "bdev_inject_crash_flush_delay": "2",
> > >> "bdev_ioring_hipri": "false",
> > >> "bdev_ioring_sqthread_poll": "false",
> > >> "bdev_nvme_retry_count": "-1",
> > >> "bdev_nvme_unbind_from_kernel": "false",
> > >> "bluefs_alloc_size": "1048576",
> > >> "bluefs_allocator": "hybrid",
> > >> "bluefs_buffered_io": "true",
> > >> "bluefs_check_for_zeros": "false",
> > >> "bluefs_compact_log_sync": "false",
> > >> "bluefs_log_compact_min_ratio": "5.000000",
> > >> "bluefs_log_compact_min_size": "16777216",
> > >> "bluefs_log_replay_check_allocations": "true",
> > >> "bluefs_max_log_runway": "4194304",
> > >> "bluefs_max_prefetch": "1048576",
> > >> "bluefs_min_flush_size": "524288",
> > >> "bluefs_min_log_runway": "1048576",
> > >> "bluefs_replay_recovery": "false",
> > >> "bluefs_replay_recovery_disable_compact": "false",
> > >> "bluefs_shared_alloc_size": "65536",
> > >> "bluefs_sync_write": "false",
> > >> "bluestore_2q_cache_kin_ratio": "0.500000",
> > >> "bluestore_2q_cache_kout_ratio": "0.500000",
> > >> "bluestore_alloc_stats_dump_interval": "86400.000000",
> > >> "bluestore_allocator": "hybrid",
> > >> "bluestore_avl_alloc_bf_free_pct": "4",
> > >> "bluestore_avl_alloc_bf_threshold": "131072",
> > >> "bluestore_bitmapallocator_blocks_per_zone": "1024",
> > >> "bluestore_bitmapallocator_span_size": "1024",
> > >> "bluestore_blobid_prealloc": "10240",
> > >> "bluestore_block_create": "true",
> > >> "bluestore_block_db_create": "false",
> > >> "bluestore_block_db_path": "",
> > >> "bluestore_block_db_size": "0",
> > >> "bluestore_block_path": "",
> > >> "bluestore_block_preallocate_file": "false",
> > >> "bluestore_block_size": "107374182400",
> > >> "bluestore_block_wal_create": "false",
> > >> "bluestore_block_wal_path": "",
> > >> "bluestore_block_wal_size": "100663296",
> > >> "bluestore_bluefs": "true",
> > >> "bluestore_bluefs_alloc_failure_dump_interval": "0.000000",
> > >> "bluestore_bluefs_balance_interval": "1.000000",
> > >> "bluestore_bluefs_db_compatibility": "true",
> > >> "bluestore_bluefs_env_mirror": "false",
> > >> "bluestore_bluefs_gift_ratio": "0.020000",
> > >> "bluestore_bluefs_max_free": "10737418240",
> > >> "bluestore_bluefs_max_ratio": "0.900000",
> > >> "bluestore_bluefs_min": "1073741824",
> > >> "bluestore_bluefs_min_free": "1073741824",
> > >> "bluestore_bluefs_min_ratio": "0.020000",
> > >> "bluestore_bluefs_reclaim_ratio": "0.200000",
> > >> "bluestore_cache_autotune": "true",
> > >> "bluestore_cache_autotune_interval": "5.000000",
> > >> "bluestore_cache_kv_ratio": "0.400000",
> > >> "bluestore_cache_meta_ratio": "0.400000",
> > >> "bluestore_cache_size": "0",
> > >> "bluestore_cache_size_hdd": "1073741824",
> > >> "bluestore_cache_size_ssd": "1073741824",
> > >> "bluestore_cache_trim_interval": "0.050000",
> > >> "bluestore_cache_trim_max_skip_pinned": "1000",
> > >> "bluestore_cache_type": "2q",
> > >> "bluestore_clone_cow": "true",
> > >> "bluestore_compression_algorithm": "snappy",
> > >> "bluestore_compression_max_blob_size": "0",
> > >> "bluestore_compression_max_blob_size_hdd": "524288",
> > >> "bluestore_compression_max_blob_size_ssd": "65536",
> > >> "bluestore_compression_min_blob_size": "0",
> > >> "bluestore_compression_min_blob_size_hdd": "131072",
> > >> "bluestore_compression_min_blob_size_ssd": "8192",
> > >> "bluestore_compression_mode": "none",
> > >> "bluestore_compression_required_ratio": "0.875000",
> > >> "bluestore_csum_type": "crc32c",
> > >> "bluestore_debug_enforce_settings": "default",
> > >> "bluestore_debug_freelist": "false",
> > >> "bluestore_debug_fsck_abort": "false",
> > >> "bluestore_debug_inject_bug21040": "false",
> > >> "bluestore_debug_inject_csum_err_probability": "0.000000",
> > >> "bluestore_debug_inject_read_err": "false",
> > >> "bluestore_debug_misc": "false",
> > >> "bluestore_debug_no_reuse_blocks": "false",
> > >> "bluestore_debug_omit_block_device_write": "false",
> > >> "bluestore_debug_omit_kv_commit": "false",
> > >> "bluestore_debug_permit_any_bdev_label": "false",
> > >> "bluestore_debug_prefill": "0.000000",
> > >> "bluestore_debug_prefragment_max": "1048576",
> > >> "bluestore_debug_random_read_err": "0.000000",
> > >> "bluestore_debug_randomize_serial_transaction": "0",
> > >> "bluestore_debug_small_allocations": "0",
> > >> "bluestore_debug_too_many_blobs_threshold": "24576",
> > >> "bluestore_default_buffered_read": "true",
> > >> "bluestore_default_buffered_write": "false",
> > >> "bluestore_deferred_batch_ops": "0",
> > >> "bluestore_deferred_batch_ops_hdd": "64",
> > >> "bluestore_deferred_batch_ops_ssd": "16",
> > >> "bluestore_extent_map_inline_shard_prealloc_size": "256",
> > >> "bluestore_extent_map_shard_max_size": "1200",
> > >> "bluestore_extent_map_shard_min_size": "150",
> > >> "bluestore_extent_map_shard_target_size": "500",
> > >> "bluestore_extent_map_shard_target_size_slop": "0.200000",
> > >> "bluestore_freelist_blocks_per_key": "128",
> > >> "bluestore_fsck_error_on_no_per_pool_omap": "false",
> > >> "bluestore_fsck_error_on_no_per_pool_stats": "false",
> > >> "bluestore_fsck_on_mkfs": "true",
> > >> "bluestore_fsck_on_mkfs_deep": "false",
> > >> "bluestore_fsck_on_mount": "false",
> > >> "bluestore_fsck_on_mount_deep": "false",
> > >> "bluestore_fsck_on_umount": "false",
> > >> "bluestore_fsck_on_umount_deep": "false",
> > >> "bluestore_fsck_quick_fix_on_mount": "false",
> > >> "bluestore_fsck_quick_fix_threads": "2",
> > >> "bluestore_fsck_read_bytes_cap": "67108864",
> > >> "bluestore_gc_enable_blob_threshold": "0",
> > >> "bluestore_gc_enable_total_threshold": "0",
> > >> "bluestore_hybrid_alloc_mem_cap": "67108864",
> > >> "bluestore_ignore_data_csum": "false",
> > >> "bluestore_ioring": "false",
> > >> "bluestore_kv_sync_util_logging_s": "10.000000",
> > >> "bluestore_kvbackend": "rocksdb",
> > >> "bluestore_log_collection_list_age": "60.000000",
> > >> "bluestore_log_omap_iterator_age": "5.000000",
> > >> "bluestore_log_op_age": "5.000000",
> > >> "bluestore_max_alloc_size": "0",
> > >> "bluestore_max_blob_size": "0",
> > >> "bluestore_max_blob_size_hdd": "524288",
> > >> "bluestore_max_blob_size_ssd": "65536",
> > >> "bluestore_max_defer_interval": "3.000000",
> > >> "bluestore_max_deferred_txc": "32",
> > >> "bluestore_min_alloc_size": "0",
> > >> "bluestore_min_alloc_size_hdd": "65536",
> > >> "bluestore_min_alloc_size_ssd": "4096",
> > >> "bluestore_nid_prealloc": "1024",
> > >> "bluestore_prefer_deferred_size": "0",
> > >> "bluestore_prefer_deferred_size_hdd": "32768",
> > >> "bluestore_prefer_deferred_size_ssd": "0",
> > >> "bluestore_retry_disk_reads": "3",
> > >> "bluestore_rocksdb_cf": "false",
> > >> "bluestore_rocksdb_cfs": "M= P= L=",
> > >> "bluestore_rocksdb_options":
> > >>
> "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2",
> > >> "bluestore_rocksdb_options_annex": "",
> > >> "bluestore_spdk_coremask": "0x1",
> > >> "bluestore_spdk_io_sleep": "5",
> > >> "bluestore_spdk_max_io_completion": "0",
> > >> "bluestore_spdk_mem": "512",
> > >> "bluestore_sync_submit_transaction": "false",
> > >> "bluestore_throttle_bytes": "67108864",
> > >> "bluestore_throttle_cost_per_io": "0",
> > >> "bluestore_throttle_cost_per_io_hdd": "670000",
> > >> "bluestore_throttle_cost_per_io_ssd": "4000",
> > >> "bluestore_throttle_deferred_bytes": "134217728",
> > >> "bluestore_throttle_trace_rate": "0.000000",
> > >> "bluestore_tracing": "false",
> > >> "bluestore_volume_selection_policy": "use_some_extra",
> > >> "bluestore_volume_selection_reserved": "0",
> > >> "bluestore_volume_selection_reserved_factor": "2.000000",
> > >> "bluestore_warn_on_bluefs_spillover": "false",
> > >> "bluestore_warn_on_legacy_statfs": "false",
> > >> "bluestore_warn_on_no_per_pool_omap": "true",
> > >> "cephadm_path": "/usr/sbin/cephadm",
> > >> "cephx_cluster_require_signatures": "false",
> > >> "cephx_cluster_require_version": "1",
> > >> "cephx_require_signatures": "false",
> > >> "cephx_require_version": "1",
> > >> "cephx_service_require_signatures": "false",
> > >> "cephx_service_require_version": "1",
> > >> "cephx_sign_messages": "true",
> > >> "chdir": "",
> > >> "client_acl_type": "",
> > >> "client_cache_mid": "0.750000",
> > >> "client_cache_size": "16384",
> > >> "client_caps_release_delay": "5",
> > >> "client_check_pool_perm": "true",
> > >> "client_debug_force_sync_read": "false",
> > >> "client_debug_getattr_caps": "false",
> > >> "client_debug_inject_tick_delay": "0",
> > >> "client_die_on_failed_dentry_invalidate": "true",
> > >> "client_die_on_failed_remount": "false",
> > >> "client_dirsize_rbytes": "true",
> > >> "client_force_lazyio": "false",
> > >> "client_fs": "",
> > >> "client_inject_fixed_oldest_tid": "false",
> > >> "client_inject_release_failure": "false",
> > >> "client_max_inline_size": "4096",
> > >> "client_mds_namespace": "",
> > >> "client_metadata": "",
> > >> "client_mount_gid": "-1",
> > >> "client_mount_timeout": "300.000000",
> > >> "client_mount_uid": "-1",
> > >> "client_mountpoint": "/",
> > >> "client_notify_timeout": "10",
> > >> "client_oc": "true",
> > >> "client_oc_max_dirty": "104857600",
> > >> "client_oc_max_dirty_age": "5.000000",
> > >> "client_oc_max_objects": "1000",
> > >> "client_oc_size": "209715200",
> > >> "client_oc_target_dirty": "8388608",
> > >> "client_permissions": "true",
> > >> "client_quota_df": "true",
> > >> "client_readahead_max_bytes": "0",
> > >> "client_readahead_max_periods": "4",
> > >> "client_readahead_min": "4194304",
> > >> "client_reconnect_stale": "false",
> > >> "client_shutdown_timeout": "30",
> > >> "client_snapdir": ".snap",
> > >> "client_tick_interval": "1.000000",
> > >> "client_trace": "",
> > >> "client_try_dentry_invalidate": "false",
> > >> "client_use_faked_inos": "false",
> > >> "client_use_random_mds": "false",
> > >> "clog_to_graylog": "false",
> > >> "clog_to_graylog_host": "127.0.0.1",
> > >> "clog_to_graylog_port": "12201",
> > >> "clog_to_monitors": "default=true",
> > >> "clog_to_syslog": "false",
> > >> "clog_to_syslog_facility": "default=daemon audit=local0",
> > >> "clog_to_syslog_level": "info",
> > >> "cluster_addr": "-",
> > >> "cluster_network": "172.30.254.253/16",
> > >> "cluster_network_interface": "",
> > >> "colors": "Terminal",
> > >> "compressor_zlib_isal": "false",
> > >> "compressor_zlib_level": "5",
> > >> "compressor_zstd_level": "1",
> > >> "container_image": "docker.io/ceph/ceph:v15",
> > >> "continuation_prompt": ">",
> > >> "crash_dir": "/var/lib/ceph/crash",
> > >> "crimson_osd_obc_lru_size": "10",
> > >> "crush_location": "host=bb-ceph-enc-rm63-osd03-31",
> > >> "crush_location_hook": "",
> > >> "crush_location_hook_timeout": "10",
> > >> "daemonize": "false",
> > >> "debug_allow_any_pool_priority": "false",
> > >> "debug_asok": "1/5",
> > >> "debug_asok_assert_abort": "false",
> > >> "debug_asserts_on_shutdown": "false",
> > >> "debug_auth": "1/5",
> > >> "debug_bdev": "1/3",
> > >> "debug_bluefs": "1/5",
> > >> "debug_bluestore": "20/0",
> > >> "debug_buffer": "0/1",
> > >> "debug_civetweb": "1/10",
> > >> "debug_client": "0/0",
> > >> "debug_compressor": "1/5",
> > >> "debug_context": "0/1",
> > >> "debug_crush": "1/1",
> > >> "debug_crypto": "1/5",
> > >> "debug_deliberately_leak_memory": "false",
> > >> "debug_disable_randomized_ping": "false",
> > >> "debug_dpdk": "1/5",
> > >> "debug_eventtrace": "1/5",
> > >> "debug_filer": "0/1",
> > >> "debug_filestore": "1/3",
> > >> "debug_finisher": "1/1",
> > >> "debug_fuse": "1/5",
> > >> "debug_heartbeat_testing_span": "0",
> > >> "debug_heartbeatmap": "1/5",
> > >> "debug_immutable_obj_cache": "0/5",
> > >> "debug_javaclient": "1/5",
> > >> "debug_journal": "1/3",
> > >> "debug_journaler": "0/5",
> > >> "debug_kstore": "1/5",
> > >> "debug_leveldb": "4/5",
> > >> "debug_lockdep": "0/1",
> > >> "debug_mds": "1/5",
> > >> "debug_mds_balancer": "1/5",
> > >> "debug_mds_locker": "1/5",
> > >> "debug_mds_log": "1/5",
> > >> "debug_mds_log_expire": "1/5",
> > >> "debug_mds_migrator": "1/5",
> > >> "debug_memdb": "4/5",
> > >> "debug_mgr": "1/5",
> > >> "debug_mgrc": "1/5",
> > >> "debug_mon": "1/5",
> > >> "debug_monc": "0/10",
> > >> "debug_ms": "0/0",
> > >> "debug_none": "0/5",
> > >> "debug_objclass": "0/5",
> > >> "debug_objectcacher": "0/5",
> > >> "debug_objecter": "0/1",
> > >> "debug_optracker": "0/5",
> > >> "debug_osd": "20/0",
> > >> "debug_paxos": "1/5",
> > >> "debug_perfcounter": "1/5",
> > >> "debug_prioritycache": "1/5",
> > >> "debug_rados": "0/5",
> > >> "debug_rbd": "0/5",
> > >> "debug_rbd_mirror": "0/5",
> > >> "debug_rbd_replay": "0/5",
> > >> "debug_rbd_rwl": "0/5",
> > >> "debug_refs": "0/0",
> > >> "debug_reserver": "1/1",
> > >> "debug_rgw": "1/5",
> > >> "debug_rgw_sync": "1/5",
> > >> "debug_rocksdb": "3/5",
> > >> "debug_shell": "false",
> > >> "debug_striper": "0/1",
> > >> "debug_test": "0/5",
> > >> "debug_throttle": "1/1",
> > >> "debug_timer": "0/1",
> > >> "debug_tp": "0/5",
> > >> "device_failure_prediction_mode": "none",
> > >> "echo": "false",
> > >> "editor": "vim",
> > >> "enable_experimental_unrecoverable_data_corrupting_features": "",
> > >> "erasure_code_dir": "/usr/lib/ceph/erasure-code",
> > >> "err_to_graylog": "false",
> > >> "err_to_stderr": "true",
> > >> "err_to_syslog": "true",
> > >> "event_tracing": "false",
> > >> "fake_statfs_for_testing": "0",
> > >> "fatal_signal_handlers": "true",
> > >> "feedback_to_output": "false",
> > >> "filer_max_purge_ops": "10",
> > >> "filer_max_truncate_ops": "128",
> > >> "filestore_apply_finisher_threads": "1",
> > >> "filestore_blackhole": "false",
> > >> "filestore_btrfs_clone_range": "true",
> > >> "filestore_btrfs_snap": "true",
> > >> "filestore_caller_concurrency": "10",
> > >> "filestore_collect_device_partition_information": "true",
> > >> "filestore_commit_timeout": "600.000000",
> > >> "filestore_debug_inject_read_err": "false",
> > >> "filestore_debug_omap_check": "false",
> > >> "filestore_debug_random_read_err": "0.000000",
> > >> "filestore_debug_verify_split": "false",
> > >> "filestore_dump_file": "",
> > >> "filestore_expected_throughput_bytes": "209715200.000000",
> > >> "filestore_expected_throughput_ops": "200.000000",
> > >> "filestore_fadvise": "true",
> > >> "filestore_fail_eio": "true",
> > >> "filestore_fd_cache_shards": "16",
> > >> "filestore_fd_cache_size": "128",
> > >> "filestore_fiemap": "false",
> > >> "filestore_fiemap_threshold": "4096",
> > >> "filestore_fsync_flushes_journal_data": "false",
> > >> "filestore_index_retry_probability": "0.000000",
> > >> "filestore_inject_stall": "0",
> > >> "filestore_journal_parallel": "false",
> > >> "filestore_journal_trailing": "false",
> > >> "filestore_journal_writeahead": "false",
> > >> "filestore_kill_at": "0",
> > >> "filestore_max_alloc_hint_size": "1048576",
> > >> "filestore_max_inline_xattr_size": "0",
> > >> "filestore_max_inline_xattr_size_btrfs": "2048",
> > >> "filestore_max_inline_xattr_size_other": "512",
> > >> "filestore_max_inline_xattr_size_xfs": "65536",
> > >> "filestore_max_inline_xattrs": "0",
> > >> "filestore_max_inline_xattrs_btrfs": "10",
> > >> "filestore_max_inline_xattrs_other": "2",
> > >> "filestore_max_inline_xattrs_xfs": "10",
> > >> "filestore_max_sync_interval": "5.000000",
> > >> "filestore_max_xattr_value_size": "0",
> > >> "filestore_max_xattr_value_size_btrfs": "65536",
> > >> "filestore_max_xattr_value_size_other": "1024",
> > >> "filestore_max_xattr_value_size_xfs": "65536",
> > >> "filestore_merge_threshold": "-10",
> > >> "filestore_min_sync_interval": "0.010000",
> > >> "filestore_odsync_write": "false",
> > >> "filestore_omap_backend": "rocksdb",
> > >> "filestore_omap_backend_path": "",
> > >> "filestore_omap_header_cache_size": "1024",
> > >> "filestore_ondisk_finisher_threads": "1",
> > >> "filestore_op_thread_suicide_timeout": "180",
> > >> "filestore_op_thread_timeout": "60",
> > >> "filestore_op_threads": "2",
> > >> "filestore_punch_hole": "false",
> > >> "filestore_queue_high_delay_multiple": "0.000000",
> > >> "filestore_queue_high_delay_multiple_bytes": "0.000000",
> > >> "filestore_queue_high_delay_multiple_ops": "0.000000",
> > >> "filestore_queue_high_threshhold": "0.900000",
> > >> "filestore_queue_low_threshhold": "0.300000",
> > >> "filestore_queue_max_bytes": "104857600",
> > >> "filestore_queue_max_delay_multiple": "0.000000",
> > >> "filestore_queue_max_delay_multiple_bytes": "0.000000",
> > >> "filestore_queue_max_delay_multiple_ops": "0.000000",
> > >> "filestore_queue_max_ops": "50",
> > >> "filestore_rocksdb_options":
> > >>
> "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression",
> > >> "filestore_seek_data_hole": "false",
> > >> "filestore_sloppy_crc": "false",
> > >> "filestore_sloppy_crc_block_size": "65536",
> > >> "filestore_splice": "false",
> > >> "filestore_split_multiple": "2",
> > >> "filestore_split_rand_factor": "20",
> > >> "filestore_update_to": "1000",
> > >> "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400",
> > >> "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040",
> > >> "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000",
> > >> "filestore_wbthrottle_btrfs_inodes_start_flusher": "500",
> > >> "filestore_wbthrottle_btrfs_ios_hard_limit": "5000",
> > >> "filestore_wbthrottle_btrfs_ios_start_flusher": "500",
> > >> "filestore_wbthrottle_enable": "true",
> > >> "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400",
> > >> "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040",
> > >> "filestore_wbthrottle_xfs_inodes_hard_limit": "5000",
> > >> "filestore_wbthrottle_xfs_inodes_start_flusher": "500",
> > >> "filestore_wbthrottle_xfs_ios_hard_limit": "5000",
> > >> "filestore_wbthrottle_xfs_ios_start_flusher": "500",
> > >> "filestore_xfs_extsize": "false",
> > >> "filestore_zfs_snap": "false",
> > >> "fio_dir": "/tmp/fio",
> > >> "fsid": "e682fa26-0026-4a40-871e-cc6112b1bd11",
> > >> "fuse_allow_other": "true",
> > >> "fuse_atomic_o_trunc": "true",
> > >> "fuse_big_writes": "true",
> > >> "fuse_debug": "false",
> > >> "fuse_default_permissions": "false",
> > >> "fuse_disable_pagecache": "false",
> > >> "fuse_max_write": "0",
> > >> "fuse_multithreaded": "true",
> > >> "fuse_require_active_mds": "true",
> > >> "fuse_set_user_groups": "true",
> > >> "fuse_syncfs_on_mksnap": "true",
> > >> "fuse_use_invalidate_cb": "true",
> > >> "gss_ktab_client_file":
> "/var/lib/ceph/osd.51/gss_client_osd.51.ktab",
> > >> "gss_target_name": "ceph",
> > >> "heartbeat_file": "",
> > >> "heartbeat_inject_failure": "0",
> > >> "heartbeat_interval": "5",
> > >> "host": "",
> > >> "immutable_object_cache_client_dedicated_thread_num": "2",
> > >> "immutable_object_cache_max_inflight_ops": "128",
> > >> "immutable_object_cache_max_size": "1073741824",
> > >> "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache",
> > >> "immutable_object_cache_sock":
> > >> "/var/run/ceph/immutable_object_cache_sock",
> > >> "immutable_object_cache_watermark": "0.100000",
> > >> "inject_early_sigterm": "false",
> > >> "journal_aio": "true",
> > >> "journal_align_min_size": "65536",
> > >> "journal_block_align": "true",
> > >> "journal_block_size": "4096",
> > >> "journal_dio": "true",
> > >> "journal_discard": "false",
> > >> "journal_force_aio": "false",
> > >> "journal_ignore_corruption": "false",
> > >> "journal_max_write_bytes": "10485760",
> > >> "journal_max_write_entries": "100",
> > >> "journal_replay_from": "0",
> > >> "journal_throttle_high_multiple": "0.000000",
> > >> "journal_throttle_high_threshhold": "0.900000",
> > >> "journal_throttle_low_threshhold": "0.600000",
> > >> "journal_throttle_max_multiple": "0.000000",
> > >> "journal_write_header_frequency": "0",
> > >> "journal_zero_on_create": "false",
> > >> "journaler_prefetch_periods": "10",
> > >> "journaler_prezero_periods": "5",
> > >> "journaler_write_head_interval": "15",
> > >> "key": "",
> > >> "keyfile": "",
> > >> "keyring": "/var/lib/ceph/osd/ceph-51/keyring",
> > >> "kstore_backend": "rocksdb",
> > >> "kstore_default_stripe_size": "65536",
> > >> "kstore_fsck_on_mount": "false",
> > >> "kstore_fsck_on_mount_deep": "true",
> > >> "kstore_max_bytes": "67108864",
> > >> "kstore_max_ops": "512",
> > >> "kstore_nid_prealloc": "1024",
> > >> "kstore_onode_map_size": "1024",
> > >> "kstore_rocksdb_options": "compression=kNoCompression",
> > >> "kstore_sync_submit_transaction": "false",
> > >> "kstore_sync_transaction": "false",
> > >> "leveldb_block_size": "0",
> > >> "leveldb_bloom_size": "0",
> > >> "leveldb_cache_size": "134217728",
> > >> "leveldb_compact_on_mount": "false",
> > >> "leveldb_compression": "true",
> > >> "leveldb_log": "",
> > >> "leveldb_log_to_ceph_log": "true",
> > >> "leveldb_max_open_files": "0",
> > >> "leveldb_paranoid": "false",
> > >> "leveldb_write_buffer_size": "8388608",
> > >> "lockdep": "false",
> > >> "lockdep_force_backtrace": "false",
> > >> "log_coarse_timestamps": "true",
> > >> "log_file": "/dev/null",
> > >> "log_flush_on_exit": "false",
> > >> "log_graylog_host": "127.0.0.1",
> > >> "log_graylog_port": "12201",
> > >> "log_max_new": "1000",
> > >> "log_max_recent": "10000",
> > >> "log_stderr_prefix": "",
> > >> "log_stop_at_utilization": "0.970000",
> > >> "log_to_file": "true",
> > >> "log_to_graylog": "false",
> > >> "log_to_stderr": "false",
> > >> "log_to_syslog": "true",
> > >> "max_completion_items": "50",
> > >> "max_rotating_auth_attempts": "10",
> > >> "mds_action_on_write_error": "1",
> > >> "mds_bal_export_pin": "true",
> > >> "mds_bal_fragment_dirs": "true",
> > >> "mds_bal_fragment_fast_factor": "1.500000",
> > >> "mds_bal_fragment_interval": "5",
> > >> "mds_bal_fragment_size_max": "100000",
> > >> "mds_bal_idle_threshold": "0.000000",
> > >> "mds_bal_interval": "10",
> > >> "mds_bal_max": "-1",
> > >> "mds_bal_max_until": "-1",
> > >> "mds_bal_merge_size": "50",
> > >> "mds_bal_midchunk": "0.300000",
> > >> "mds_bal_min_rebalance": "0.100000",
> > >> "mds_bal_min_start": "0.200000",
> > >> "mds_bal_minchunk": "0.001000",
> > >> "mds_bal_mode": "0",
> > >> "mds_bal_need_max": "1.200000",
> > >> "mds_bal_need_min": "0.800000",
> > >> "mds_bal_replicate_threshold": "8000.000000",
> > >> "mds_bal_sample_interval": "3.000000",
> > >> "mds_bal_split_bits": "3",
> > >> "mds_bal_split_rd": "25000.000000",
> > >> "mds_bal_split_size": "10000",
> > >> "mds_bal_split_wr": "10000.000000",
> > >> "mds_bal_target_decay": "10.000000",
> > >> "mds_bal_unreplicate_threshold": "0.000000",
> > >> "mds_beacon_grace": "15.000000",
> > >> "mds_beacon_interval": "4.000000",
> > >> "mds_cache_memory_limit": "4294967296",
> > >> "mds_cache_mid": "0.700000",
> > >> "mds_cache_release_free_interval": "10",
> > >> "mds_cache_reservation": "0.050000",
> > >> "mds_cache_trim_decay_rate": "1.000000",
> > >> "mds_cache_trim_interval": "1",
> > >> "mds_cache_trim_threshold": "262144",
> > >> "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000",
> > >> "mds_cap_revoke_eviction_timeout": "0.000000",
> > >> "mds_client_delegate_inos_pct": "50",
> > >> "mds_client_prealloc_inos": "1000",
> > >> "mds_client_writeable_range_max_inc_objs": "1024",
> > >> "mds_damage_table_max_entries": "10000",
> > >> "mds_data": "/var/lib/ceph/mds/ceph-51",
> > >> "mds_debug_auth_pins": "false",
> > >> "mds_debug_frag": "false",
> > >> "mds_debug_scatterstat": "false",
> > >> "mds_debug_subtrees": "false",
> > >> "mds_decay_halflife": "5.000000",
> > >> "mds_default_dir_hash": "2",
> > >> "mds_defer_session_stale": "true",
> > >> "mds_dir_keys_per_op": "16384",
> > >> "mds_dir_max_commit_size": "10",
> > >> "mds_dirstat_min_interval": "1.000000",
> > >> "mds_dump_cache_after_rejoin": "false",
> > >> "mds_dump_cache_on_map": "false",
> > >> "mds_dump_cache_threshold_file": "0",
> > >> "mds_dump_cache_threshold_formatter": "1073741824",
> > >> "mds_early_reply": "true",
> > >> "mds_enable_op_tracker": "true",
> > >> "mds_enforce_unique_name": "true",
> > >> "mds_export_ephemeral_distributed": "false",
> > >> "mds_export_ephemeral_random": "false",
> > >> "mds_export_ephemeral_random_max": "0.010000",
> > >> "mds_forward_all_requests_to_auth": "false",
> > >> "mds_freeze_tree_timeout": "30.000000",
> > >> "mds_hack_allow_loading_invalid_metadata": "false",
> > >> "mds_health_cache_threshold": "1.500000",
> > >> "mds_health_summarize_threshold": "10",
> > >> "mds_heartbeat_grace": "15.000000",
> > >> "mds_inject_migrator_session_race": "false",
> > >> "mds_inject_traceless_reply_probability": "0.000000",
> > >> "mds_join_fs": "",
> > >> "mds_journal_format": "1",
> > >> "mds_kill_create_at": "0",
> > >> "mds_kill_export_at": "0",
> > >> "mds_kill_import_at": "0",
> > >> "mds_kill_journal_at": "0",
> > >> "mds_kill_journal_expire_at": "0",
> > >> "mds_kill_journal_replay_at": "0",
> > >> "mds_kill_link_at": "0",
> > >> "mds_kill_mdstable_at": "0",
> > >> "mds_kill_openc_at": "0",
> > >> "mds_kill_rename_at": "0",
> > >> "mds_log_events_per_segment": "1024",
> > >> "mds_log_max_events": "-1",
> > >> "mds_log_max_segments": "128",
> > >> "mds_log_pause": "false",
> > >> "mds_log_segment_size": "0",
> > >> "mds_log_skip_corrupt_events": "false",
> > >> "mds_log_warn_factor": "2.000000",
> > >> "mds_max_caps_per_client": "1048576",
> > >> "mds_max_completed_flushes": "100000",
> > >> "mds_max_completed_requests": "100000",
> > >> "mds_max_export_size": "20971520",
> > >> "mds_max_file_recover": "32",
> > >> "mds_max_purge_files": "64",
> > >> "mds_max_purge_ops": "8192",
> > >> "mds_max_purge_ops_per_pg": "0.500000",
> > >> "mds_max_retries_on_remount_failure": "5",
> > >> "mds_max_scrub_ops_in_progress": "5",
> > >> "mds_max_snaps_per_dir": "100",
> > >> "mds_max_xattr_pairs_size": "65536",
> > >> "mds_min_caps_per_client": "100",
> > >> "mds_min_caps_working_set": "10000",
> > >> "mds_mon_shutdown_timeout": "5.000000",
> > >> "mds_numa_node": "-1",
> > >> "mds_oft_prefetch_dirfrags": "true",
> > >> "mds_op_complaint_time": "30.000000",
> > >> "mds_op_history_duration": "600",
> > >> "mds_op_history_size": "20",
> > >> "mds_op_log_threshold": "5",
> > >> "mds_purge_queue_busy_flush_period": "1.000000",
> > >> "mds_recall_global_max_decay_threshold": "131072",
> > >> "mds_recall_max_caps": "30000",
> > >> "mds_recall_max_decay_rate": "1.500000",
> > >> "mds_recall_max_decay_threshold": "131072",
> > >> "mds_recall_warning_decay_rate": "60.000000",
> > >> "mds_recall_warning_threshold": "262144",
> > >> "mds_reconnect_timeout": "45.000000",
> > >> "mds_replay_interval": "1.000000",
> > >> "mds_replay_unsafe_with_closed_session": "false",
> > >> "mds_request_load_average_decay_rate": "60.000000",
> > >> "mds_root_ino_gid": "0",
> > >> "mds_root_ino_uid": "0",
> > >> "mds_scatter_nudge_interval": "5.000000",
> > >> "mds_session_blacklist_on_evict": "true",
> > >> "mds_session_blacklist_on_timeout": "true",
> > >> "mds_session_cache_liveness_decay_rate": "300.000000",
> > >> "mds_session_cache_liveness_magnitude": "10",
> > >> "mds_session_cap_acquisition_decay_rate": "10.000000",
> > >> "mds_session_cap_acquisition_throttle": "500000",
> > >> "mds_session_max_caps_throttle_ratio": "1.100000",
> > >> "mds_sessionmap_keys_per_op": "1024",
> > >> "mds_shutdown_check": "0",
> > >> "mds_skip_ino": "0",
> > >> "mds_snap_max_uid": "4294967294",
> > >> "mds_snap_min_uid": "0",
> > >> "mds_snap_rstat": "false",
> > >> "mds_task_status_update_interval": "2.000000",
> > >> "mds_thrash_exports": "0",
> > >> "mds_thrash_fragments": "0",
> > >> "mds_tick_interval": "5.000000",
> > >> "mds_verify_backtrace": "1",
> > >> "mds_verify_scatter": "false",
> > >> "mds_wipe_ino_prealloc": "false",
> > >> "mds_wipe_sessions": "false",
> > >> "mempool_debug": "false",
> > >> "memstore_debug_omit_block_device_write": "false",
> > >> "memstore_device_bytes": "1073741824",
> > >> "memstore_page_set": "false",
> > >> "memstore_page_size": "65536",
> > >> "mgr_client_bytes": "134217728",
> > >> "mgr_client_messages": "512",
> > >> "mgr_client_service_daemon_unregister_timeout": "1.000000",
> > >> "mgr_connect_retry_interval": "1.000000",
> > >> "mgr_data": "/var/lib/ceph/mgr/ceph-51",
> > >> "mgr_debug_aggressive_pg_num_changes": "false",
> > >> "mgr_disabled_modules": "",
> > >> "mgr_initial_modules": "restful iostat",
> > >> "mgr_mds_bytes": "134217728",
> > >> "mgr_mds_messages": "128",
> > >> "mgr_module_path": "/usr/share/ceph/mgr",
> > >> "mgr_mon_bytes": "134217728",
> > >> "mgr_mon_messages": "128",
> > >> "mgr_osd_bytes": "536870912",
> > >> "mgr_osd_messages": "8192",
> > >> "mgr_service_beacon_grace": "60.000000",
> > >> "mgr_stats_period": "5",
> > >> "mgr_stats_threshold": "5",
> > >> "mgr_tick_period": "2",
> > >> "mon_accept_timeout_factor": "2.000000",
> > >> "mon_allow_pool_delete": "true",
> > >> "mon_cache_target_full_warn_ratio": "0.660000",
> > >> "mon_clean_pg_upmaps_per_chunk": "256",
> > >> "mon_client_bytes": "104857600",
> > >> "mon_client_directed_command_retry": "2",
> > >> "mon_client_hunt_interval": "1.500000",
> > >> "mon_client_hunt_interval_backoff": "1.500000",
> > >> "mon_client_hunt_interval_max_multiple": "10.000000",
> > >> "mon_client_hunt_interval_min_multiple": "1.000000",
> > >> "mon_client_hunt_parallel": "3",
> > >> "mon_client_log_interval": "1.000000",
> > >> "mon_client_max_log_entries_per_message": "1000",
> > >> "mon_client_ping_interval": "10.000000",
> > >> "mon_client_ping_timeout": "30.000000",
> > >> "mon_clock_drift_allowed": "0.100000",
> > >> "mon_clock_drift_warn_backoff": "5.000000",
> > >> "mon_cluster_log_file": "/dev/null",
> > >> "mon_cluster_log_file_level": "debug",
> > >> "mon_cluster_log_to_file": "true",
> > >> "mon_cluster_log_to_graylog": "false",
> > >> "mon_cluster_log_to_graylog_host": "127.0.0.1",
> > >> "mon_cluster_log_to_graylog_port": "12201",
> > >> "mon_cluster_log_to_stderr": "false",
> > >> "mon_cluster_log_to_syslog": "true",
> > >> "mon_cluster_log_to_syslog_facility": "daemon",
> > >> "mon_cluster_log_to_syslog_level": "info",
> > >> "mon_compact_on_bootstrap": "false",
> > >> "mon_compact_on_start": "false",
> > >> "mon_compact_on_trim": "true",
> > >> "mon_config_key_max_entry_size": "65536",
> > >> "mon_cpu_threads": "4",
> > >> "mon_crush_min_required_version": "hammer",
> > >> "mon_daemon_bytes": "419430400",
> > >> "mon_data": "/var/lib/ceph/mon/ceph-51",
> > >> "mon_data_avail_crit": "5",
> > >> "mon_data_avail_warn": "30",
> > >> "mon_data_size_warn": "16106127360",
> > >> "mon_debug_block_osdmap_trim": "false",
> > >> "mon_debug_deprecated_as_obsolete": "false",
> > >> "mon_debug_dump_json": "false",
> > >> "mon_debug_dump_location": "/var/log/ceph/ceph-osd.51.tdump",
> > >> "mon_debug_dump_transactions": "false",
> > >> "mon_debug_extra_checks": "false",
> > >> "mon_debug_no_initial_persistent_features": "false",
> > >> "mon_debug_no_require_bluestore_for_ec_overwrites": "false",
> > >> "mon_debug_no_require_nautilus": "false",
> > >> "mon_debug_no_require_octopus": "false",
> > >> "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false",
> > >> "mon_delta_reset_interval": "10.000000",
> > >> "mon_dns_srv_name": "ceph-mon",
> > >> "mon_election_timeout": "5.000000",
> > >> "mon_enable_op_tracker": "true",
> > >> "mon_fake_pool_delete": "false",
> > >> "mon_force_quorum_join": "false",
> > >> "mon_globalid_prealloc": "10000",
> > >> "mon_health_detail_to_clog": "true",
> > >> "mon_health_log_update_period": "5",
> > >> "mon_health_max_detail": "50",
> > >> "mon_health_to_clog": "true",
> > >> "mon_health_to_clog_interval": "600",
> > >> "mon_health_to_clog_tick_interval": "60.000000",
> > >> "mon_host": "10.28.0.4,10.28.0.2,10.28.0.3",
> > >> "mon_host_override": "",
> > >> "mon_initial_members": "",
> > >> "mon_inject_pg_merge_bounce_probability": "0.000000",
> > >> "mon_inject_sync_get_chunk_delay": "0.000000",
> > >> "mon_inject_transaction_delay_max": "10.000000",
> > >> "mon_inject_transaction_delay_probability": "0.000000",
> > >> "mon_keyvaluedb": "rocksdb",
> > >> "mon_lease": "5.000000",
> > >> "mon_lease_ack_timeout_factor": "2.000000",
> > >> "mon_lease_renew_interval_factor": "0.600000",
> > >> "mon_log_max_summary": "50",
> > >> "mon_max_log_entries_per_event": "4096",
> > >> "mon_max_log_epochs": "500",
> > >> "mon_max_mdsmap_epochs": "500",
> > >> "mon_max_mgrmap_epochs": "500",
> > >> "mon_max_osd": "10000",
> > >> "mon_max_pg_per_osd": "200",
> > >> "mon_max_pool_pg_num": "65536",
> > >> "mon_max_snap_prune_per_epoch": "100",
> > >> "mon_mds_blacklist_interval": "86400.000000",
> > >> "mon_mds_force_trim_to": "0",
> > >> "mon_mds_skip_sanity": "false",
> > >> "mon_memory_autotune": "true",
> > >> "mon_memory_target": "2147483648",
> > >> "mon_mgr_beacon_grace": "30",
> > >> "mon_mgr_blacklist_interval": "86400.000000",
> > >> "mon_mgr_digest_period": "5",
> > >> "mon_mgr_inactive_grace": "60",
> > >> "mon_mgr_mkfs_grace": "120",
> > >> "mon_mgr_proxy_client_bytes_ratio": "0.300000",
> > >> "mon_min_osdmap_epochs": "500",
> > >> "mon_op_complaint_time": "30",
> > >> "mon_op_history_duration": "600",
> > >> "mon_op_history_size": "20",
> > >> "mon_op_history_slow_op_size": "20",
> > >> "mon_op_history_slow_op_threshold": "10",
> > >> "mon_op_log_threshold": "5",
> > >> "mon_osd_adjust_down_out_interval": "true",
> > >> "mon_osd_adjust_heartbeat_grace": "true",
> > >> "mon_osd_auto_mark_auto_out_in": "true",
> > >> "mon_osd_auto_mark_in": "false",
> > >> "mon_osd_auto_mark_new_in": "true",
> > >> "mon_osd_backfillfull_ratio": "0.900000",
> > >> "mon_osd_blacklist_default_expire": "3600.000000",
> > >> "mon_osd_cache_size": "500",
> > >> "mon_osd_cache_size_min": "134217728",
> > >> "mon_osd_crush_smoke_test": "true",
> > >> "mon_osd_destroyed_out_interval": "600",
> > >> "mon_osd_down_out_interval": "3600",
> > >> "mon_osd_down_out_subtree_limit": "rack",
> > >> "mon_osd_err_op_age_ratio": "128.000000",
> > >> "mon_osd_force_trim_to": "0",
> > >> "mon_osd_full_ratio": "0.950000",
> > >> "mon_osd_initial_require_min_compat_client": "jewel",
> > >> "mon_osd_laggy_halflife": "3600",
> > >> "mon_osd_laggy_max_interval": "300",
> > >> "mon_osd_laggy_weight": "0.300000",
> > >> "mon_osd_mapping_pgs_per_chunk": "4096",
> > >> "mon_osd_max_creating_pgs": "1024",
> > >> "mon_osd_max_initial_pgs": "1024",
> > >> "mon_osd_min_down_reporters": "2",
> > >> "mon_osd_min_in_ratio": "0.750000",
> > >> "mon_osd_min_up_ratio": "0.300000",
> > >> "mon_osd_nearfull_ratio": "0.850000",
> > >> "mon_osd_prime_pg_temp": "true",
> > >> "mon_osd_prime_pg_temp_max_estimate": "0.250000",
> > >> "mon_osd_prime_pg_temp_max_time": "0.500000",
> > >> "mon_osd_report_timeout": "450",
> > >> "mon_osd_reporter_subtree_level": "host",
> > >> "mon_osd_snap_trim_queue_warn_on": "32768",
> > >> "mon_osd_warn_num_repaired": "10",
> > >> "mon_osd_warn_op_age": "32.000000",
> > >> "mon_osdmap_full_prune_enabled": "true",
> > >> "mon_osdmap_full_prune_interval": "10",
> > >> "mon_osdmap_full_prune_min": "10000",
> > >> "mon_osdmap_full_prune_txsize": "100",
> > >> "mon_pg_check_down_all_threshold": "0.500000",
> > >> "mon_pg_stuck_threshold": "60",
> > >> "mon_pg_warn_max_object_skew": "0.000000",
> > >> "mon_pg_warn_min_objects": "10000",
> > >> "mon_pg_warn_min_per_osd": "0",
> > >> "mon_pg_warn_min_pool_objects": "1000",
> > >> "mon_pool_quota_crit_threshold": "0",
> > >> "mon_pool_quota_warn_threshold": "0",
> > >> "mon_probe_timeout": "2.000000",
> > >> "mon_reweight_max_change": "0.050000",
> > >> "mon_reweight_max_osds": "4",
> > >> "mon_reweight_min_bytes_per_osd": "104857600",
> > >> "mon_reweight_min_pgs_per_osd": "10",
> > >> "mon_rocksdb_options":
> > >>
> "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true",
> > >> "mon_scrub_inject_crc_mismatch": "0.000000",
> > >> "mon_scrub_inject_missing_keys": "0.000000",
> > >> "mon_scrub_interval": "86400",
> > >> "mon_scrub_max_keys": "100",
> > >> "mon_scrub_timeout": "300",
> > >> "mon_session_timeout": "300",
> > >> "mon_smart_report_timeout": "5",
> > >> "mon_stat_smooth_intervals": "6",
> > >> "mon_subscribe_interval": "86400.000000",
> > >> "mon_sync_debug": "false",
> > >> "mon_sync_max_payload_keys": "2000",
> > >> "mon_sync_max_payload_size": "1048576",
> > >> "mon_sync_provider_kill_at": "0",
> > >> "mon_sync_requester_kill_at": "0",
> > >> "mon_sync_timeout": "60.000000",
> > >> "mon_target_pg_per_osd": "100",
> > >> "mon_tick_interval": "5",
> > >> "mon_timecheck_interval": "300.000000",
> > >> "mon_timecheck_skew_interval": "30.000000",
> > >> "mon_warn_on_cache_pools_without_hit_sets": "true",
> > >> "mon_warn_on_crush_straw_calc_version_zero": "true",
> > >> "mon_warn_on_insecure_global_id_reclaim": "true",
> > >> "mon_warn_on_insecure_global_id_reclaim_allowed": "true",
> > >> "mon_warn_on_legacy_crush_tunables": "true",
> > >> "mon_warn_on_misplaced": "false",
> > >> "mon_warn_on_msgr2_not_enabled": "false",
> > >> "mon_warn_on_osd_down_out_interval_zero": "true",
> > >> "mon_warn_on_pool_no_app": "true",
> > >> "mon_warn_on_pool_no_redundancy": "true",
> > >> "mon_warn_on_pool_pg_num_not_power_of_two": "true",
> > >> "mon_warn_on_slow_ping_ratio": "0.050000",
> > >> "mon_warn_on_slow_ping_time": "0.000000",
> > >> "mon_warn_on_too_few_osds": "true",
> > >> "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000",
> > >> "mon_warn_pg_not_scrubbed_ratio": "0.500000",
> > >> "monmap": "",
> > >> "ms_async_max_op_threads": "8",
> > >> "ms_async_op_threads": "4",
> > >> "ms_async_rdma_buffer_size": "131072",
> > >> "ms_async_rdma_cm": "false",
> > >> "ms_async_rdma_device_name": "",
> > >> "ms_async_rdma_dscp": "96",
> > >> "ms_async_rdma_enable_hugepage": "false",
> > >> "ms_async_rdma_gid_idx": "0",
> > >> "ms_async_rdma_local_gid": "",
> > >> "ms_async_rdma_polling_us": "1000",
> > >> "ms_async_rdma_port_num": "1",
> > >> "ms_async_rdma_receive_buffers": "32768",
> > >> "ms_async_rdma_receive_queue_len": "4096",
> > >> "ms_async_rdma_roce_ver": "1",
> > >> "ms_async_rdma_send_buffers": "1024",
> > >> "ms_async_rdma_sl": "3",
> > >> "ms_async_rdma_support_srq": "true",
> > >> "ms_async_rdma_type": "ib",
> > >> "ms_bind_before_connect": "false",
> > >> "ms_bind_ipv4": "true",
> > >> "ms_bind_ipv6": "false",
> > >> "ms_bind_msgr1": "true",
> > >> "ms_bind_msgr2": "false",
> > >> "ms_bind_port_max": "7300",
> > >> "ms_bind_port_min": "6800",
> > >> "ms_bind_prefer_ipv4": "false",
> > >> "ms_bind_retry_count": "3",
> > >> "ms_bind_retry_delay": "5",
> > >> "ms_blackhole_client": "false",
> > >> "ms_blackhole_mds": "false",
> > >> "ms_blackhole_mgr": "false",
> > >> "ms_blackhole_mon": "false",
> > >> "ms_blackhole_osd": "false",
> > >> "ms_client_mode": "crc secure",
> > >> "ms_cluster_mode": "crc secure",
> > >> "ms_cluster_type": "",
> > >> "ms_connection_idle_timeout": "900",
> > >> "ms_connection_ready_timeout": "10",
> > >> "ms_crc_data": "true",
> > >> "ms_crc_header": "true",
> > >> "ms_die_on_bad_msg": "false",
> > >> "ms_die_on_bug": "false",
> > >> "ms_die_on_old_message": "false",
> > >> "ms_die_on_skipped_message": "false",
> > >> "ms_die_on_unhandled_msg": "false",
> > >> "ms_dispatch_throttle_bytes": "104857600",
> > >> "ms_dpdk_coremask": "0xF",
> > >> "ms_dpdk_debug_allow_loopback": "false",
> > >> "ms_dpdk_gateway_ipv4_addr": "",
> > >> "ms_dpdk_host_ipv4_addr": "",
> > >> "ms_dpdk_hugepages": "",
> > >> "ms_dpdk_hw_flow_control": "true",
> > >> "ms_dpdk_hw_queue_weight": "1.000000",
> > >> "ms_dpdk_lro": "true",
> > >> "ms_dpdk_memory_channel": "4",
> > >> "ms_dpdk_netmask_ipv4_addr": "",
> > >> "ms_dpdk_pmd": "",
> > >> "ms_dpdk_port_id": "0",
> > >> "ms_dpdk_rx_buffer_count_per_core": "8192",
> > >> "ms_dump_corrupt_message_level": "1",
> > >> "ms_dump_on_send": "false",
> > >> "ms_initial_backoff": "0.200000",
> > >> "ms_inject_delay_max": "1.000000",
> > >> "ms_inject_delay_msg_type": "",
> > >> "ms_inject_delay_probability": "0.000000",
> > >> "ms_inject_delay_type": "",
> > >> "ms_inject_internal_delays": "0.000000",
> > >> "ms_inject_socket_failures": "0",
> > >> "ms_learn_addr_from_peer": "true",
> > >> "ms_max_accept_failures": "4",
> > >> "ms_max_backoff": "15.000000",
> > >> "ms_mon_client_mode": "secure crc",
> > >> "ms_mon_cluster_mode": "secure crc",
> > >> "ms_mon_service_mode": "secure crc",
> > >> "ms_pq_max_tokens_per_priority": "16777216",
> > >> "ms_pq_min_cost": "65536",
> > >> "ms_public_type": "",
> > >> "ms_service_mode": "crc secure",
> > >> "ms_tcp_listen_backlog": "512",
> > >> "ms_tcp_nodelay": "true",
> > >> "ms_tcp_prefetch_max_size": "4096",
> > >> "ms_tcp_rcvbuf": "0",
> > >> "ms_type": "async+posix",
> > >> "no_config_file": "false",
> > >> "objecter_completion_locks_per_session": "32",
> > >> "objecter_debug_inject_relock_delay": "false",
> > >> "objecter_inflight_op_bytes": "838860800",
> > >> "objecter_inflight_ops": "1024",
> > >> "objecter_inject_no_watch_ping": "false",
> > >> "objecter_retry_writes_after_first_reply": "false",
> > >> "objecter_tick_interval": "5.000000",
> > >> "objecter_timeout": "10.000000",
> > >> "objectstore_blackhole": "false",
> > >> "osd_agent_delay_time": "5.000000",
> > >> "osd_agent_hist_halflife": "1000",
> > >> "osd_agent_max_low_ops": "2",
> > >> "osd_agent_max_ops": "4",
> > >> "osd_agent_min_evict_effort": "0.100000",
> > >> "osd_agent_quantize_effort": "0.100000",
> > >> "osd_agent_slop": "0.020000",
> > >> "osd_allow_recovery_below_min_size": "true",
> > >> "osd_async_recovery_min_cost": "100",
> > >> "osd_auto_mark_unfound_lost": "false",
> > >> "osd_backfill_retry_interval": "30.000000",
> > >> "osd_backfill_scan_max": "512",
> > >> "osd_backfill_scan_min": "64",
> > >> "osd_backoff_on_degraded": "false",
> > >> "osd_backoff_on_peering": "false",
> > >> "osd_backoff_on_unfound": "true",
> > >> "osd_beacon_report_interval": "87",
> > >> "osd_bench_duration": "30",
> > >> "osd_bench_large_size_max_throughput": "104857600",
> > >> "osd_bench_max_block_size": "67108864"
> > >> "osd_bench_small_size_max_iops": "100",
> > >> "osd_blkin_trace_all": "false",
> > >> "osd_calc_pg_upmaps_aggressively": "true",
> > >> "osd_calc_pg_upmaps_local_fallback_retries": "100",
> > >> "osd_check_for_log_corruption": "false",
> > >> "osd_check_max_object_name_len_on_startup": "true",
> > >> "osd_class_default_list": "cephfs hello journal lock log numops otp
> > >> rbd refcount rgw rgw_gc timeindex user version cas",
> > >> "osd_class_dir": "/usr/lib/rados-classes",
> > >> "osd_class_load_list": "cephfs hello journal lock log numops otp
> rbd
> > >> refcount rgw rgw_gc timeindex user version cas",
> > >> "osd_class_update_on_start": "true",
> > >> "osd_client_message_cap": "256",
> > >> "osd_client_message_size_cap": "524288000",
> > >> "osd_client_op_priority": "63",
> > >> "osd_client_watch_timeout": "30",
> > >> "osd_command_max_records": "256",
> > >> "osd_command_thread_suicide_timeout": "900",
> > >> "osd_command_thread_timeout": "600",
> > >> "osd_copyfrom_max_chunk": "8388608",
> > >> "osd_crush_chooseleaf_type": "1",
> > >> "osd_crush_initial_weight": "-1.000000",
> > >> "osd_crush_update_on_start": "true",
> > >> "osd_crush_update_weight_set": "true",
> > >> "osd_data": "/var/lib/ceph/osd/ceph-51",
> > >> "osd_debug_crash_on_ignored_backoff": "false",
> > >> "osd_debug_deep_scrub_sleep": "0.000000",
> > >> "osd_debug_drop_ping_duration": "0",
> > >> "osd_debug_drop_ping_probability": "0.000000",
> > >> "osd_debug_feed_pullee": "-1",
> > >> "osd_debug_inject_copyfrom_error": "false",
> > >> "osd_debug_inject_dispatch_delay_duration": "0.100000",
> > >> "osd_debug_inject_dispatch_delay_probability": "0.000000",
> > >> "osd_debug_misdirected_ops": "false",
> > >> "osd_debug_no_acting_change": "false",
> > >> "osd_debug_no_purge_strays": "false",
> > >> "osd_debug_op_order": "false",
> > >> "osd_debug_pg_log_writeout": "false",
> > >> "osd_debug_pretend_recovery_active": "false",
> > >> "osd_debug_random_push_read_error": "0.000000",
> > >> "osd_debug_reject_backfill_probability": "0.000000",
> > >> "osd_debug_shutdown": "false",
> > >> "osd_debug_skip_full_check_in_backfill_reservation": "false",
> > >> "osd_debug_skip_full_check_in_recovery": "false",
> > >> "osd_debug_verify_cached_snaps": "false",
> > >> "osd_debug_verify_missing_on_start": "false",
> > >> "osd_debug_verify_snaps": "false",
> > >> "osd_debug_verify_stray_on_activate": "false",
> > >> "osd_deep_scrub_interval": "1209600.000000",
> > >> "osd_deep_scrub_keys": "1024",
> > >> "osd_deep_scrub_large_omap_object_key_threshold": "500000",
> > >> "osd_deep_scrub_large_omap_object_value_sum_threshold":
> "1073741824",
> > >> "osd_deep_scrub_randomize_ratio": "0.150000",
> > >> "osd_deep_scrub_stride": "524288",
> > >> "osd_deep_scrub_update_digest_min_age": "7200",
> > >> "osd_default_data_pool_replay_window": "45",
> > >> "osd_default_notify_timeout": "30",
> > >> "osd_delete_sleep": "0.000000",
> > >> "osd_delete_sleep_hdd": "5.000000",
> > >> "osd_delete_sleep_hybrid": "1.000000",
> > >> "osd_delete_sleep_ssd": "1.000000",
> > >> "osd_discard_disconnected_ops": "true",
> > >> "osd_enable_op_tracker": "true",
> > >> "osd_erasure_code_plugins": "jerasure lrc isa",
> > >> "osd_failsafe_full_ratio": "0.970000",
> > >> "osd_fast_fail_on_connection_refused": "true",
> > >> "osd_fast_info": "true",
> > >> "osd_fast_shutdown": "true",
> > >> "osd_fast_shutdown_notify_mon": "false",
> > >> "osd_find_best_info_ignore_history_les": "false",
> > >> "osd_force_auth_primary_missing_objects": "100",
> > >> "osd_force_recovery_pg_log_entries_factor": "1.300000",
> > >> "osd_function_tracing": "false",
> > >> "osd_heartbeat_grace": "60",
> > >> "osd_heartbeat_interval": "6",
> > >> "osd_heartbeat_min_healthy_ratio": "0.330000",
> > >> "osd_heartbeat_min_peers": "10",
> > >> "osd_heartbeat_min_size": "2000",
> > >> "osd_heartbeat_stale": "600",
> > >> "osd_heartbeat_use_min_delay_socket": "false",
> > >> "osd_hit_set_max_size": "100000",
> > >> "osd_hit_set_min_size": "1000",
> > >> "osd_hit_set_namespace": ".ceph-internal",
> > >> "osd_ignore_stale_divergent_priors": "false",
> > >> "osd_inject_bad_map_crc_probability": "0.000000",
> > >> "osd_inject_failure_on_pg_removal": "false",
> > >> "osd_journal": "/var/lib/ceph/osd/ceph-51/journal",
> > >> "osd_journal_flush_on_shutdown": "true",
> > >> "osd_journal_size": "5120",
> > >> "osd_kill_backfill_at": "0",
> > >> "osd_loop_before_reset_tphandle": "64",
> > >> "osd_map_cache_size": "50",
> > >> "osd_map_dedup": "true",
> > >> "osd_map_message_max": "40",
> > >> "osd_map_message_max_bytes": "10485760",
> > >> "osd_map_share_max_epochs": "40",
> > >> "osd_max_attr_name_len": "100",
> > >> "osd_max_attr_size": "0",
> > >> "osd_max_backfills": "8",
> > >> "osd_max_markdown_count": "5",
> > >> "osd_max_markdown_period": "600",
> > >> "osd_max_object_name_len": "2048",
> > >> "osd_max_object_namespace_len": "256",
> > >> "osd_max_object_size": "134217728",
> > >> "osd_max_omap_bytes_per_request": "1073741824",
> > >> "osd_max_omap_entries_per_request": "1024",
> > >> "osd_max_pg_blocked_by": "16",
> > >> "osd_max_pg_log_entries": "500",
> > >> "osd_max_pg_per_osd_hard_ratio": "3.500000",
> > >> "osd_max_pgls": "1024",
> > >> "osd_max_push_cost": "8388608",
> > >> "osd_max_push_objects": "10",
> > >> "osd_max_scrubs": "1",
> > >> "osd_max_snap_prune_intervals_per_epoch": "512",
> > >> "osd_max_trimming_pgs": "2",
> > >> "osd_max_write_op_reply_len": "32",
> > >> "osd_max_write_size": "90",
> > >> "osd_mclock_scheduler_anticipation_timeout": "0.000000",
> > >> "osd_mclock_scheduler_background_best_effort_lim": "999999",
> > >> "osd_mclock_scheduler_background_best_effort_res": "1",
> > >> "osd_mclock_scheduler_background_best_effort_wgt": "1",
> > >> "osd_mclock_scheduler_background_recovery_lim": "999999",
> > >> "osd_mclock_scheduler_background_recovery_res": "1",
> > >> "osd_mclock_scheduler_background_recovery_wgt": "1",
> > >> "osd_mclock_scheduler_client_lim": "999999",
> > >> "osd_mclock_scheduler_client_res": "1",
> > >> "osd_mclock_scheduler_client_wgt": "1",
> > >> "osd_memory_base": "805306368",
> > >> "osd_memory_cache_min": "134217728",
> > >> "osd_memory_cache_resize_interval": "1.000000",
> > >> "osd_memory_expected_fragmentation": "0.150000",
> > >> "osd_memory_target": "1073741824",
> > >> "osd_memory_target_cgroup_limit_ratio": "0.800000",
> > >> "osd_min_pg_log_entries": "500",
> > >> "osd_min_recovery_priority": "0",
> > >> "osd_mon_ack_timeout": "30.000000",
> > >> "osd_mon_heartbeat_interval": "30",
> > >> "osd_mon_heartbeat_stat_stale": "3600",
> > >> "osd_mon_report_interval": "5",
> > >> "osd_mon_report_max_in_flight": "2",
> > >> "osd_mon_shutdown_timeout": "5.000000",
> > >> "osd_num_cache_shards": "32",
> > >> "osd_num_op_tracker_shard": "32",
> > >> "osd_numa_auto_affinity": "true",
> > >> "osd_numa_node": "-1",
> > >> "osd_numa_prefer_iface": "true",
> > >> "osd_object_clean_region_max_num_intervals": "10",
> > >> "osd_objecter_finishers": "1",
> > >> "osd_objectstore": "bluestore",
> > >> "osd_objectstore_fuse": "false",
> > >> "osd_objectstore_tracing": "false",
> > >> "osd_op_complaint_time": "30.000000",
> > >> "osd_op_history_duration": "600",
> > >> "osd_op_history_size": "20",
> > >> "osd_op_history_slow_op_size": "20",
> > >> "osd_op_history_slow_op_threshold": "10.000000",
> > >> "osd_op_log_threshold": "5",
> > >> "osd_op_num_shards": "0",
> > >> "osd_op_num_shards_hdd": "5",
> > >> "osd_op_num_shards_ssd": "8",
> > >> "osd_op_num_threads_per_shard": "0",
> > >> "osd_op_num_threads_per_shard_hdd": "1",
> > >> "osd_op_num_threads_per_shard_ssd": "2",
> > >> "osd_op_pq_max_tokens_per_priority": "4194304",
> > >> "osd_op_pq_min_cost": "65536",
> > >> "osd_op_queue": "wpq",
> > >> "osd_op_queue_cut_off": "high",
> > >> "osd_op_thread_suicide_timeout": "150",
> > >> "osd_op_thread_timeout": "15",
> > >> "osd_open_classes_on_start": "true",
> > >> "osd_os_flags": "0",
> > >> "osd_peering_op_priority": "255",
> > >> "osd_pg_delete_cost": "1048576",
> > >> "osd_pg_delete_priority": "5",
> > >> "osd_pg_epoch_max_lag_factor": "2.000000",
> > >> "osd_pg_epoch_persisted_max_stale": "40",
> > >> "osd_pg_log_dups_tracked": "3000",
> > >> "osd_pg_log_trim_max": "10000",
> > >> "osd_pg_log_trim_min": "100",
> > >> "osd_pg_max_concurrent_snap_trims": "1",
> > >> "osd_pg_object_context_cache_count": "64",
> > >> "osd_pg_stat_report_interval_max": "500",
> > >> "osd_pool_default_cache_max_evict_check_size": "10",
> > >> "osd_pool_default_cache_min_evict_age": "0",
> > >> "osd_pool_default_cache_min_flush_age": "0",
> > >> "osd_pool_default_cache_target_dirty_high_ratio": "0.600000",
> > >> "osd_pool_default_cache_target_dirty_ratio": "0.400000",
> > >> "osd_pool_default_cache_target_full_ratio": "0.800000",
> > >> "osd_pool_default_crush_rule": "-1",
> > >> "osd_pool_default_ec_fast_read": "false",
> > >> "osd_pool_default_erasure_code_profile": "plugin=jerasure
> > >> technique=reed_sol_van k=2 m=2",
> > >> "osd_pool_default_flag_hashpspool": "true",
> > >> "osd_pool_default_flag_nodelete": "false",
> > >> "osd_pool_default_flag_nopgchange": "false",
> > >> "osd_pool_default_flag_nosizechange": "false",
> > >> "osd_pool_default_flags": "0",
> > >> "osd_pool_default_hit_set_bloom_fpp": "0.050000",
> > >> "osd_pool_default_min_size": "0",
> > >> "osd_pool_default_pg_autoscale_mode": "warn",
> > >> "osd_pool_default_pg_num": "32",
> > >> "osd_pool_default_pgp_num": "0",
> > >> "osd_pool_default_read_lease_ratio": "0.800000",
> > >> "osd_pool_default_size": "3",
> > >> "osd_pool_default_type": "replicated",
> > >> "osd_pool_erasure_code_stripe_unit": "4096",
> > >> "osd_pool_use_gmt_hitset": "true",
> > >> "osd_push_per_object_cost": "1000",
> > >> "osd_read_ec_check_for_errors": "false",
> > >> "osd_recover_clone_overlap": "true",
> > >> "osd_recover_clone_overlap_limit": "10",
> > >> "osd_recovery_cost": "20971520",
> > >> "osd_recovery_delay_start": "0.000000",
> > >> "osd_recovery_max_active": "2",
> > >> "osd_recovery_max_active_hdd": "3",
> > >> "osd_recovery_max_active_ssd": "10",
> > >> "osd_recovery_max_chunk": "8388608",
> > >> "osd_recovery_max_omap_entries_per_chunk": "8096",
> > >> "osd_recovery_max_single_start": "1",
> > >> "osd_recovery_op_priority": "3",
> > >> "osd_recovery_op_warn_multiple": "16",
> > >> "osd_recovery_priority": "5",
> > >> "osd_recovery_retry_interval": "30.000000",
> > >> "osd_recovery_sleep": "0.000000",
> > >> "osd_recovery_sleep_hdd": "0.025000",
> > >> "osd_recovery_sleep_hybrid": "0.025000",
> > >> "osd_recovery_sleep_ssd": "0.025000",
> > >> "osd_repair_during_recovery": "false",
> > >> "osd_requested_scrub_priority": "120",
> > >> "osd_rollback_to_cluster_snap": "",
> > >> "osd_scrub_auto_repair": "true",
> > >> "osd_scrub_auto_repair_num_errors": "5",
> > >> "osd_scrub_backoff_ratio": "0.660000",
> > >> "osd_scrub_begin_hour": "23",
> > >> "osd_scrub_begin_week_day": "0",
> > >> "osd_scrub_chunk_max": "25",
> > >> "osd_scrub_chunk_min": "5",
> > >> "osd_scrub_cost": "52428800",
> > >> "osd_scrub_during_recovery": "true",
> > >> "osd_scrub_end_hour": "9",
> > >> "osd_scrub_end_week_day": "7",
> > >> "osd_scrub_extended_sleep": "0.000000",
> > >> "osd_scrub_interval_randomize_ratio": "0.500000",
> > >> "osd_scrub_invalid_stats": "true",
> > >> "osd_scrub_load_threshold": "0.500000",
> > >> "osd_scrub_max_interval": "604800.000000",
> > >> "osd_scrub_max_preemptions": "5",
> > >> "osd_scrub_min_interval": "86400.000000",
> > >> "osd_scrub_priority": "5",
> > >> "osd_scrub_sleep": "0.050000",
> > >> "osd_shutdown_pgref_assert": "false",
> > >> "osd_skip_data_digest": "false",
> > >> "osd_smart_report_timeout": "5",
> > >> "osd_snap_trim_cost": "1048576",
> > >> "osd_snap_trim_priority": "5",
> > >> "osd_snap_trim_sleep": "0.000000",
> > >> "osd_snap_trim_sleep_hdd": "3.000000",
> > >> "osd_snap_trim_sleep_hybrid": "2.000000",
> > >> "osd_snap_trim_sleep_ssd": "0.100000",
> > >> "osd_stats_ack_timeout_decay": "0.900000",
> > >> "osd_stats_ack_timeout_factor": "2.000000",
> > >> "osd_target_pg_log_entries_per_osd": "300000",
> > >> "osd_target_transaction_size": "30",
> > >> "osd_tier_default_cache_hit_set_count": "4",
> > >> "osd_tier_default_cache_hit_set_grade_decay_rate": "20",
> > >> "osd_tier_default_cache_hit_set_period": "1200",
> > >> "osd_tier_default_cache_hit_set_search_last_n": "1",
> > >> "osd_tier_default_cache_hit_set_type": "bloom",
> > >> "osd_tier_default_cache_min_read_recency_for_promote": "1",
> > >> "osd_tier_default_cache_min_write_recency_for_promote": "1",
> > >> "osd_tier_default_cache_mode": "writeback",
> > >> "osd_tier_promote_max_bytes_sec": "5242880",
> > >> "osd_tier_promote_max_objects_sec": "25",
> > >> "osd_tracing": "false",
> > >> "osd_use_stale_snap": "false",
> > >> "osd_uuid": "00000000-0000-0000-0000-000000000000",
> > >> "osdc_blkin_trace_all": "false",
> > >> "paxos_kill_at": "0",
> > >> "paxos_max_join_drift": "10",
> > >> "paxos_min": "500",
> > >> "paxos_min_wait": "0.050000",
> > >> "paxos_propose_interval": "1.000000",
> > >> "paxos_service_trim_max": "500",
> > >> "paxos_service_trim_max_multiplier": "20",
> > >> "paxos_service_trim_min": "250",
> > >> "paxos_stash_full_interval": "25",
> > >> "paxos_trim_max": "500",
> > >> "paxos_trim_min": "250",
> > >> "perf": "true",
> > >> "pid_file": "",
> > >> "plugin_crypto_accelerator": "crypto_isal",
> > >> "plugin_dir": "/usr/lib/ceph",
> > >> "prompt":
> > >> "\u001b[01;33mCephFS:~\u001b[96m/\u001b[0m\u001b[01;33m>>>\u001b[00m
> ",
> > >> "public_addr": "-",
> > >> "public_addrv": "",
> > >> "public_bind_addr": "-",
> > >> "public_network": "10.28.254.253/16",
> > >> "public_network_interface": "",
> > >> "qat_compressor_enabled": "false",
> > >> "quiet": "false",
> > >> "rados_mon_op_timeout": "0",
> > >> "rados_osd_op_timeout": "0",
> > >> "rados_tracing": "false",
> > >> "rbd_atime_update_interval": "60",
> > >> "rbd_auto_exclusive_lock_until_manual_request": "true",
> > >> "rbd_balance_parent_reads": "false",
> > >> "rbd_balance_snap_reads": "false",
> > >> "rbd_blacklist_expire_seconds": "0",
> > >> "rbd_blacklist_on_break_lock": "true",
> > >> "rbd_blkin_trace_all": "false",
> > >> "rbd_cache": "true",
> > >> "rbd_cache_block_writes_upfront": "false",
> > >> "rbd_cache_max_dirty": "25165824",
> > >> "rbd_cache_max_dirty_age": "1.000000",
> > >> "rbd_cache_max_dirty_object": "0",
> > >> "rbd_cache_policy": "writearound",
> > >> "rbd_cache_size": "33554432",
> > >> "rbd_cache_target_dirty": "16777216",
> > >> "rbd_cache_writethrough_until_flush": "true",
> > >> "rbd_clone_copy_on_read": "false",
> > >> "rbd_compression_hint": "none",
> > >> "rbd_concurrent_management_ops": "10",
> > >> "rbd_config_pool_override_update_timestamp": "0",
> > >> "rbd_default_clone_format": "auto",
> > >> "rbd_default_data_pool": "",
> > >> "rbd_default_features": "61",
> > >> "rbd_default_format": "2",
> > >> "rbd_default_map_options": "",
> > >> "rbd_default_order": "22",
> > >> "rbd_default_pool": "rbd",
> > >> "rbd_default_stripe_count": "0",
> > >> "rbd_default_stripe_unit": "0",
> > >> "rbd_disable_zero_copy_writes": "true",
> > >> "rbd_discard_granularity_bytes": "65536",
> > >> "rbd_discard_on_zeroed_write_same": "true",
> > >> "rbd_enable_alloc_hint": "true",
> > >> "rbd_io_scheduler": "simple",
> > >> "rbd_io_scheduler_simple_max_delay": "0",
> > >> "rbd_journal_commit_age": "5.000000",
> > >> "rbd_journal_max_concurrent_object_sets": "0",
> > >> "rbd_journal_max_payload_bytes": "16384",
> > >> "rbd_journal_object_flush_age": "0.000000",
> > >> "rbd_journal_object_flush_bytes": "1048576",
> > >> "rbd_journal_object_flush_interval": "0",
> > >> "rbd_journal_object_max_in_flight_appends": "0",
> > >> "rbd_journal_object_writethrough_until_flush": "true",
> > >> "rbd_journal_order": "24",
> > >> "rbd_journal_pool": "",
> > >> "rbd_journal_splay_width": "4",
> > >> "rbd_localize_parent_reads": "false",
> > >> "rbd_localize_snap_reads": "false",
> > >> "rbd_mirror_concurrent_image_deletions": "1",
> > >> "rbd_mirror_concurrent_image_syncs": "5",
> > >> "rbd_mirror_delete_retry_interval": "30.000000",
> > >> "rbd_mirror_image_perf_stats_prio": "5",
> > >> "rbd_mirror_image_policy_migration_throttle": "300",
> > >> "rbd_mirror_image_policy_rebalance_timeout": "0.000000",
> > >> "rbd_mirror_image_policy_type": "simple",
> > >> "rbd_mirror_image_policy_update_throttle_interval": "1.000000",
> > >> "rbd_mirror_image_state_check_interval": "30",
> > >> "rbd_mirror_journal_commit_age": "5.000000",
> > >> "rbd_mirror_journal_poll_age": "5.000000",
> > >> "rbd_mirror_leader_heartbeat_interval": "5",
> > >> "rbd_mirror_leader_max_acquire_attempts_before_break": "3",
> > >> "rbd_mirror_leader_max_missed_heartbeats": "2",
> > >> "rbd_mirror_memory_autotune": "true",
> > >> "rbd_mirror_memory_base": "805306368",
> > >> "rbd_mirror_memory_cache_autotune_interval": "30.000000",
> > >> "rbd_mirror_memory_cache_min": "134217728",
> > >> "rbd_mirror_memory_cache_resize_interval": "5.000000",
> > >> "rbd_mirror_memory_expected_fragmentation": "0.150000",
> > >> "rbd_mirror_memory_target": "4294967296",
> > >> "rbd_mirror_perf_stats_prio": "5",
> > >> "rbd_mirror_pool_replayers_refresh_interval": "30",
> > >> "rbd_mirror_sync_point_update_age": "30.000000",
> > >> "rbd_mirroring_delete_delay": "0",
> > >> "rbd_mirroring_max_mirroring_snapshots": "3",
> > >> "rbd_mirroring_replay_delay": "0",
> > >> "rbd_mirroring_resync_after_disconnect": "false",
> > >> "rbd_move_parent_to_trash_on_remove": "false",
> > >> "rbd_move_to_trash_on_remove": "false",
> > >> "rbd_move_to_trash_on_remove_expire_seconds": "0",
> > >> "rbd_mtime_update_interval": "60",
> > >> "rbd_non_blocking_aio": "true",
> > >> "rbd_op_thread_timeout": "60",
> > >> "rbd_op_threads": "1",
> > >> "rbd_parent_cache_enabled": "false",
> > >> "rbd_qos_bps_burst": "0",
> > >> "rbd_qos_bps_limit": "0",
> > >> "rbd_qos_iops_burst": "0",
> > >> "rbd_qos_iops_limit": "0",
> > >> "rbd_qos_read_bps_burst": "0",
> > >> "rbd_qos_read_bps_limit": "0",
> > >> "rbd_qos_read_iops_burst": "0",
> > >> "rbd_qos_read_iops_limit": "0",
> > >> "rbd_qos_schedule_tick_min": "50",
> > >> "rbd_qos_write_bps_burst": "0",
> > >> "rbd_qos_write_bps_limit": "0",
> > >> "rbd_qos_write_iops_burst": "0",
> > >> "rbd_qos_write_iops_limit": "0",
> > >> "rbd_read_from_replica_policy": "default",
> > >> "rbd_readahead_disable_after_bytes": "52428800",
> > >> "rbd_readahead_max_bytes": "524288",
> > >> "rbd_readahead_trigger_requests": "10",
> > >> "rbd_request_timed_out_seconds": "30",
> > >> "rbd_rwl_enabled": "false",
> > >> "rbd_rwl_log_periodic_stats": "false",
> > >> "rbd_rwl_path": "/tmp",
> > >> "rbd_rwl_size": "1073741824",
> > >> "rbd_skip_partial_discard": "true",
> > >> "rbd_sparse_read_threshold_bytes": "65536",
> > >> "rbd_tracing": "false",
> > >> "rbd_validate_names": "true",
> > >> "rbd_validate_pool": "true",
> > >> "restapi_base_url": "",
> > >> "restapi_log_level": "",
> > >> "rgw_acl_grants_max_num": "100",
> > >> "rbd_mirror_memory_autotune": "true",
> > >> "rbd_mirror_memory_base": "805306368",
> > >> "rbd_mirror_memory_cache_autotune_interval": "30.000000",
> > >> "rbd_mirror_memory_cache_min": "134217728",
> > >> "rbd_mirror_memory_cache_resize_interval": "5.000000",
> > >> "rbd_mirror_memory_expected_fragmentation": "0.150000",
> > >> "rbd_mirror_memory_target": "4294967296",
> > >> "rbd_mirror_perf_stats_prio": "5",
> > >> "rbd_mirror_pool_replayers_refresh_interval": "30",
> > >> "rbd_mirror_sync_point_update_age": "30.000000",
> > >> "rbd_mirroring_delete_delay": "0",
> > >> "rbd_mirroring_max_mirroring_snapshots": "3",
> > >> "rbd_mirroring_replay_delay": "0",
> > >> "rbd_mirroring_resync_after_disconnect": "false",
> > >> "rbd_move_parent_to_trash_on_remove": "false",
> > >> "rbd_move_to_trash_on_remove": "false",
> > >> "rbd_move_to_trash_on_remove_expire_seconds": "0",
> > >> "rbd_mtime_update_interval": "60",
> > >> "rbd_non_blocking_aio": "true",
> > >> "rbd_op_thread_timeout": "60",
> > >> "rbd_op_threads": "1",
> > >> "rbd_parent_cache_enabled": "false",
> > >> "rbd_qos_bps_burst": "0",
> > >> "rbd_qos_bps_limit": "0",
> > >> "rbd_qos_iops_burst": "0",
> > >> "rbd_qos_iops_limit": "0",
> > >> "rbd_qos_read_bps_burst": "0",
> > >> "rbd_qos_read_bps_limit": "0",
> > >> "rbd_qos_read_iops_burst": "0",
> > >> "rbd_qos_read_iops_limit": "0",
> > >> "rbd_qos_schedule_tick_min": "50",
> > >> "rbd_qos_write_bps_burst": "0",
> > >> "rbd_qos_write_bps_limit": "0",
> > >> "rbd_qos_write_iops_burst": "0",
> > >> "rbd_qos_write_iops_limit": "0",
> > >> "rbd_read_from_replica_policy": "default",
> > >> "rbd_readahead_disable_after_bytes": "52428800",
> > >> "rbd_readahead_max_bytes": "524288",
> > >> "rbd_readahead_trigger_requests": "10",
> > >> "rbd_request_timed_out_seconds": "30",
> > >> "rbd_rwl_enabled": "false",
> > >> "rbd_rwl_log_periodic_stats": "false",
> > >> "rbd_rwl_path": "/tmp",
> > >> "rbd_rwl_size": "1073741824",
> > >> "rbd_skip_partial_discard": "true",
> > >> "rbd_sparse_read_threshold_bytes": "65536",
> > >> "rbd_tracing": "false",
> > >> "rbd_validate_names": "true",
> > >> "rbd_validate_pool": "true",
> > >> "restapi_base_url": "",
> > >> "restapi_log_level": "",
> > >> "rgw_acl_grants_max_num": "100",
> > >> "rgw_admin_entry": "admin",
> > >> "rgw_barbican_url": "",
> > >> "rgw_bucket_default_quota_max_objects": "-1",
> > >> "rgw_bucket_default_quota_max_size": "-1",
> > >> "rgw_bucket_index_max_aio": "128",
> > >> "rgw_bucket_quota_cache_size": "10000",
> > >> "rgw_bucket_quota_ttl": "600",
> > >> "rgw_cache_enabled": "true",
> > >> "rgw_cache_expiry_interval": "900",
> > >> "rgw_cache_lru_size": "10000",
> > >> "rgw_content_length_compat": "false",
> > >> "rgw_copy_obj_progress": "true",
> > >> "rgw_copy_obj_progress_every_bytes": "1048576",
> > >> "rgw_cors_rules_max_num": "100",
> > >> "rgw_cross_domain_policy": "<allow-access-from domain=\"*\"
> > >> secure=\"false\" />",
> > >> "rgw_crypt_default_encryption_key": "",
> > >> "rgw_crypt_require_ssl": "true",
> > >> "rgw_crypt_s3_kms_backend": "barbican",
> > >> "rgw_crypt_s3_kms_encryption_keys": "",
> > >> "rgw_crypt_suppress_logs": "true",
> > >> "rgw_crypt_vault_addr": "",
> > >> "rgw_crypt_vault_auth": "token",
> > >> "rgw_crypt_vault_namespace": "",
> > >> "rgw_crypt_vault_prefix": "",
> > >> "rgw_crypt_vault_secret_engine": "transit",
> > >> "rgw_crypt_vault_token_file": "",
> > >> "rgw_curl_low_speed_limit": "1024",
> > >> "rgw_curl_low_speed_time": "300",
> > >> "rgw_curl_wait_timeout_ms": "1000",
> > >> "rgw_data": "/var/lib/ceph/radosgw/ceph-51",
> > >> "rgw_data_log_changes_size": "1000",
> > >> "rgw_data_log_num_shards": "128",
> > >> "rgw_data_log_obj_prefix": "data_log",
> > >> "rgw_data_log_window": "30",
> > >> "rgw_data_notify_interval_msec": "200",
> > >> "rgw_default_realm_info_oid": "default.realm",
> > >> "rgw_default_region_info_oid": "default.region",
> > >> "rgw_default_zone_info_oid": "default.zone",
> > >> "rgw_default_zonegroup_info_oid": "default.zonegroup",
> > >> "rgw_defer_to_bucket_acls": "",
> > >> "rgw_delete_multi_obj_max_num": "1000",
> > >> "rgw_dmclock_admin_lim": "0.000000",
> > >> "rgw_dmclock_admin_res": "100.000000",
> > >> "rgw_dmclock_admin_wgt": "100.000000",
> > >> "rgw_dmclock_auth_lim": "0.000000",
> > >> "rgw_dmclock_auth_res": "200.000000",
> > >> "rgw_dmclock_auth_wgt": "100.000000",
> > >> "rgw_dmclock_data_lim": "0.000000",
> > >> "rgw_dmclock_data_res": "500.000000",
> > >> "rgw_dmclock_data_wgt": "500.000000",
> > >> "rgw_dmclock_metadata_lim": "0.000000",
> > >> "rgw_dmclock_metadata_res": "500.000000",
> > >> "rgw_dmclock_metadata_wgt": "500.000000",
> > >> "rgw_dns_name": "s3.ceph-enc.bitbahn.net",
> > >> "rgw_dns_s3website_name": "",
> > >> "rgw_dynamic_resharding": "true",
> > >> "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts,
> iam,
> > >> pubsub",
> > >> "rgw_enable_gc_threads": "true",
> > >> "rgw_enable_lc_threads": "true",
> > >> "rgw_enable_ops_log": "false",
> > >> "rgw_enable_quota_threads": "true",
> > >> "rgw_enable_static_website": "false",
> > >> "rgw_enable_usage_log": "false",
> > >> "rgw_enforce_swift_acls": "true",
> > >> "rgw_exit_timeout_secs": "120",
> > >> "rgw_expose_bucket": "false",
> > >> "rgw_extended_http_attrs": "",
> > >> "rgw_fcgi_socket_backlog": "1024",
> > >> "rgw_frontend_defaults": "beast
> > >> ssl_certificate=config://rgw/cert/$realm/$zone.crt
> > >> ssl_private_key=config://rgw/cert/$realm/$zone.key",
> > >> "rgw_frontends": "beast ssl_port=443 port=80
> > >> ssl_certificate=/etc/ceph/rgwcert.pem",
> > >> "rgw_gc_max_concurrent_io": "10",
> > >> "rgw_gc_max_deferred": "50",
> > >> "rgw_gc_max_deferred_entries_size": "3072",
> > >> "rgw_gc_max_objs": "32",
> > >> "rgw_gc_max_queue_size": "134213632",
> > >> "rgw_gc_max_trim_chunk": "16",
> > >> "rgw_gc_obj_min_wait": "7200",
> > >> "rgw_gc_processor_max_time": "3600",
> > >> "rgw_gc_processor_period": "3600",
> > >> "rgw_get_obj_max_req_size": "4194304",
> > >> "rgw_get_obj_window_size": "16777216",
> > >> "rgw_healthcheck_disabling_path": "",
> > >> "rgw_host": "",
> > >> "rgw_ignore_get_invalid_range": "false",
> > >> "rgw_init_timeout": "300",
> > >> "rgw_inject_notify_timeout_probability": "0.000000",
> > >> "rgw_keystone_accepted_admin_roles": "",
> > >> "rgw_keystone_accepted_roles": "Member, admin",
> > >> "rgw_keystone_admin_domain": "",
> > >> "rgw_keystone_admin_password": "",
> > >> "rgw_keystone_admin_password_path": "",
> > >> "rgw_keystone_admin_project": "",
> > >> "rgw_keystone_admin_tenant": "",
> > >> "rgw_keystone_admin_token": "",
> > >> "rgw_keystone_admin_token_path": "",
> > >> "rgw_keystone_admin_user": "",
> > >> "rgw_keystone_api_version": "2",
> > >> "rgw_keystone_barbican_domain": "",
> > >> "rgw_keystone_barbican_password": "",
> > >> "rgw_keystone_barbican_project": "",
> > >> "rgw_keystone_barbican_tenant": "",
> > >> "rgw_keystone_barbican_user": "",
> > >> "rgw_keystone_implicit_tenants": "false",
> > >> "rgw_keystone_token_cache_size": "10000",
> > >> "rgw_keystone_url": "",
> > >> "rgw_keystone_verify_ssl": "true",
> > >> "rgw_lc_debug_interval": "-1",
> > >> "rgw_lc_lock_max_time": "90",
> > >> "rgw_lc_max_objs": "32",
> > >> "rgw_lc_max_rules": "1000",
> > >> "rgw_lc_max_worker": "3",
> > >> "rgw_lc_max_wp_worker": "3",
> > >> "rgw_lc_thread_delay": "0",
> > >> "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com",
> > >> "rgw_ldap_dnattr": "uid",
> > >> "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com",
> > >> "rgw_ldap_searchfilter": "",
> > >> "rgw_ldap_secret": "/etc/openldap/secret",
> > >> "rgw_ldap_uri": "ldaps://<ldap.your.domain>",
> > >> "rgw_lifecycle_work_time": "00:00-06:00",
> > >> "rgw_list_bucket_min_readahead": "1000",
> > >> "rgw_list_buckets_max_chunk": "1000",
> > >> "rgw_log_http_headers": "",
> > >> "rgw_log_nonexistent_bucket": "false",
> > >> "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> > >> "rgw_log_object_name_utc": "false",
> > >> "rgw_max_attr_name_len": "0",
> > >> "rgw_max_attr_size": "0",
> > >> "rgw_max_attrs_num_in_req": "0",
> > >> "rgw_max_chunk_size": "4194304",
> > >> "rgw_max_concurrent_requests": "1024",
> > >> "rgw_max_dynamic_shards": "1999",
> > >> "rgw_max_listing_results": "1000",
> > >> "rgw_max_notify_retries": "3",
> > >> "rgw_max_objs_per_shard": "200000",
> > >> "rgw_max_put_param_size": "1048576",
> > >> "rgw_max_put_size": "5368709120",
> > >> "rgw_max_slo_entries": "1000",
> > >> "rgw_md_log_max_shards": "64",
> > >> "rgw_md_notify_interval_msec": "200",
> > >> "rgw_mime_types_file": "/etc/mime.types",
> > >> "rgw_mp_lock_max_time": "600",
> > >> "rgw_multipart_min_part_size": "5242880",
> > >> "rgw_multipart_part_upload_limit": "10000",
> > >> "rgw_nfs_fhcache_partitions": "3",
> > >> "rgw_nfs_fhcache_size": "2017",
> > >> "rgw_nfs_lru_lane_hiwat": "911",
> > >> "rgw_nfs_lru_lanes": "5",
> > >> "rgw_nfs_max_gc": "300",
> > >> "rgw_nfs_namespace_expire_secs": "300",
> > >> "rgw_nfs_run_gc_threads": "false",
> > >> "rgw_nfs_run_lc_threads": "false",
> > >> "rgw_nfs_run_quota_threads": "false",
> > >> "rgw_nfs_run_sync_thread": "false",
> > >> "rgw_nfs_s3_fast_attrs": "false",
> > >> "rgw_nfs_write_completion_interval_s": "10",
> > >> "rgw_num_async_rados_threads": "32",
> > >> "rgw_num_control_oids": "8",
> > >> "rgw_num_rados_handles": "1",
> > >> "rgw_numa_node": "-1",
> > >> "rgw_obj_stripe_size": "4194304",
> > >> "rgw_obj_tombstone_cache_size": "1000",
> > >> "rgw_objexp_chunk_size": "100",
> > >> "rgw_objexp_gc_interval": "600",
> > >> "rgw_objexp_hints_num_shards": "127",
> > >> "rgw_olh_pending_timeout_sec": "3600",
> > >> "rgw_op_thread_suicide_timeout": "0",
> > >> "rgw_op_thread_timeout": "600",
> > >> "rgw_op_tracing": "false",
> > >> "rgw_opa_token": "",
> > >> "rgw_opa_url": "",
> > >> "rgw_opa_verify_ssl": "true",
> > >> "rgw_ops_log_data_backlog": "5242880",
> > >> "rgw_ops_log_rados": "true",
> > >> "rgw_ops_log_socket_path": "",
> > >> "rgw_override_bucket_index_max_shards": "0",
> > >> "rgw_period_latest_epoch_info_oid": ".latest_epoch",
> > >> "rgw_period_push_interval": "2.000000",
> > >> "rgw_period_push_interval_max": "30.000000",
> > >> "rgw_period_root_pool": ".rgw.root",
> > >> "rgw_port": "",
> > >> "rgw_print_continue": "true",
> > >> "rgw_print_prohibited_content_length": "false",
> > >> "rgw_put_obj_max_window_size": "67108864",
> > >> "rgw_put_obj_min_window_size": "16777216",
> > >> "rgw_rados_pool_autoscale_bias": "4.000000",
> > >> "rgw_rados_pool_pg_num_min": "8",
> > >> "rgw_rados_pool_recovery_priority": "5",
> > >> "rgw_rados_tracing": "false",
> > >> "rgw_realm": "",
> > >> "rgw_realm_root_pool": ".rgw.root",
> > >> "rgw_region": "",
> > >> "rgw_region_root_pool": ".rgw.root",
> > >> "rgw_relaxed_region_enforcement": "false",
> > >> "rgw_relaxed_s3_bucket_names": "false",
> > >> "rgw_remote_addr_param": "REMOTE_ADDR",
> > >> "rgw_request_uri": "",
> > >> "rgw_reshard_batch_size": "64",
> > >> "rgw_reshard_bucket_lock_duration": "360",
> > >> "rgw_reshard_max_aio": "128",
> > >> "rgw_reshard_num_logs": "16",
> > >> "rgw_reshard_thread_interval": "600",
> > >> "rgw_resolve_cname": "false",
> > >> "rgw_rest_getusage_op_compat": "false",
> > >> "rgw_run_sync_thread": "true",
> > >> "rgw_s3_auth_order": "sts, external, local",
> > >> "rgw_s3_auth_use_keystone": "false",
> > >> "rgw_s3_auth_use_ldap": "false",
> > >> "rgw_s3_auth_use_rados": "true",
> > >> "rgw_s3_auth_use_sts": "false",
> > >> "rgw_s3_success_create_obj_status": "0",
> > >> "rgw_safe_max_objects_per_shard": "102400",
> > >> "rgw_scheduler_type": "throttler",
> > >> "rgw_script_uri": "",
> > >> "rgw_service_provider_name": "",
> > >> "rgw_shard_warning_threshold": "90.000000",
> > >> "rgw_socket_path": "",
> > >> "rgw_sts_client_id": "",
> > >> "rgw_sts_client_secret": "",
> > >> "rgw_sts_entry": "sts",
> > >> "rgw_sts_key": "sts",
> > >> "rgw_sts_max_session_duration": "43200",
> > >> "rgw_sts_token_introspection_url": "",
> > >> "rgw_swift_account_in_url": "false",
> > >> "rgw_swift_auth_entry": "auth",
> > >> "rgw_swift_auth_url": "",
> > >> "rgw_swift_custom_header": "",
> > >> "rgw_swift_enforce_content_length": "false",
> > >> "rgw_swift_need_stats": "true",
> > >> "rgw_swift_tenant_name": "",
> > >> "rgw_swift_token_expiration": "86400",
> > >> "rgw_swift_url": "",
> > >> "rgw_swift_url_prefix": "swift",
> > >> "rgw_swift_versioning_enabled": "false",
> > >> "rgw_sync_data_inject_err_probability": "0.000000",
> > >> "rgw_sync_lease_period": "120",
> > >> "rgw_sync_log_trim_concurrent_buckets": "4",
> > >> "rgw_sync_log_trim_interval": "1200",
> > >> "rgw_sync_log_trim_max_buckets": "16",
> > >> "rgw_sync_log_trim_min_cold_buckets": "4",
> > >> "rgw_sync_meta_inject_err_probability": "0.000000",
> > >> "rgw_sync_obj_etag_verify": "false",
> > >> "rgw_sync_trace_history_size": "4096",
> > >> "rgw_sync_trace_per_node_log_size": "32",
> > >> "rgw_sync_trace_servicemap_update_interval": "10",
> > >> "rgw_thread_pool_size": "512",
> > >> "rgw_torrent_comment": "",
> > >> "rgw_torrent_createby": "",
> > >> "rgw_torrent_encoding": "",
> > >> "rgw_torrent_flag": "false",
> > >> "rgw_torrent_origin": "",
> > >> "rgw_torrent_sha_unit": "524288",
> > >> "rgw_torrent_tracker": "",
> > >> "rgw_trust_forwarded_https": "false",
> > >> "rgw_usage_log_flush_threshold": "1024",
> > >> "rgw_usage_log_tick_interval": "30",
> > >> "rgw_usage_max_shards": "32",
> > >> "rgw_usage_max_user_shards": "1",
> > >> "rgw_use_opa_authz": "false",
> > >> "rgw_user_default_quota_max_objects": "-1",
> > >> "rgw_user_default_quota_max_size": "-1",
> > >> "rgw_user_max_buckets": "1000",
> > >> "rgw_user_quota_bucket_sync_interval": "180",
> > >> "rgw_user_quota_sync_idle_users": "false",
> > >> "rgw_user_quota_sync_interval": "86400",
> > >> "rgw_user_quota_sync_wait_time": "86400",
> > >> "rgw_user_unique_email": "true",
> > >> "rgw_verify_ssl": "true",
> > >> "rgw_website_routing_rules_max_num": "50",
> > >> "rgw_zone": "default",
> > >> "rgw_zone_root_pool": ".rgw.root",
> > >> "rgw_zonegroup": "",
> > >> "rgw_zonegroup_root_pool": ".rgw.root",
> > >> "rocksdb_block_size": "4096",
> > >> "rocksdb_bloom_bits_per_key": "20",
> > >> "rocksdb_cache_index_and_filter_blocks": "true",
> > >> "rocksdb_cache_index_and_filter_blocks_with_high_priority": "true",
> > >> "rocksdb_cache_row_ratio": "0.000000",
> > >> "rocksdb_cache_shard_bits": "4",
> > >> "rocksdb_cache_size": "536870912",
> > >> "rocksdb_cache_type": "binned_lru",
> > >> "rocksdb_collect_compaction_stats": "false",
> > >> "rocksdb_collect_extended_stats": "false",
> > >> "rocksdb_collect_memory_stats": "false",
> > >> "rocksdb_delete_range_threshold": "1048576",
> > >> "rocksdb_index_type": "binary_search",
> > >> "rocksdb_log_to_ceph_log": "true",
> > >> "rocksdb_metadata_block_size": "4096",
> > >> "rocksdb_partition_filters": "false",
> > >> "rocksdb_perf": "false",
> > >> "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false",
> > >> "rotating_keys_bootstrap_timeout": "30",
> > >> "rotating_keys_renewal_timeout": "10",
> > >> "run_dir": "/var/run/ceph",
> > >> "setgroup": "ceph",
> > >> "setuser": "ceph",
> > >> "setuser_match_path": "",
> > >> "target_max_misplaced_ratio": "0.050000",
> > >> "thp": "false",
> > >> "threadpool_default_timeout": "60",
> > >> "threadpool_empty_queue_max_wait": "2",
> > >> "throttler_perf_counter": "true",
> > >> "timing": "false"
> > >> }
> > >>
> > >>
> > >>
> > >>> On Thu, 6 Jan 2022 at 01:12, Igor Fedotov <igor.fedotov@xxxxxxxx>
> wrote:
> > >>>
> > >>> Hi Lee,
> > >>>
> > >>> could you please raise debug-bluestore and debug-osd to 20 (via ceph
> > >>> tell osd.N injectargs command) when OSD starts to eat up the RAM.
> Then
> > >>> drop it back to defaults after a few seconds (10s is enough) to avoid
> > >>> huge log size and share the resulting OSD log.
> > >>>
> > >>> Also I'm curious if you have any non-default settings for OSDs,
> please
> > >>> share if any.
> > >>>
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Igor
> > >>>
> > >>>
> > >>> On 1/6/2022 1:43 AM, Lee wrote:
> > >>>> The first OSD took 156Gb of Ram to boot.. :(
> > >>>>
> > >>>> Is there a easy way to stop Mempool pulling so much memory.
> > >>>>
> > >>>> On Wed, 5 Jan 2022 at 22:12, Mazzystr <mazzystr@xxxxxxxxx> wrote:
> > >>>>
> > >>>>> and that is exactly why I run osds containerized with limited cpu
> and
> > >>>>> memory as well as "bluestore cache size", "osd memory target", and
> "mds
> > >>>>> cache memory limit". Osd processes have become noisy neighbors in
> the
> > >>> last
> > >>>>> few versions.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Jan 5, 2022 at 1:47 PM Lee <lquince@xxxxxxxxx> wrote:
> > >>>>>
> > >>>>>> I'm not rushing,
> > >>>>>>
> > >>>>>> I have found the issue, Im am getting OOM errors as the OSD boots,
> > >>>>>> basically is starts to process the PG's and then the node runs
> out of
> > >>>>>> memory and the daemon kill's
> > >>>>>>
> > >>>>>> 2022-01-05 20:09:08 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:01.024+0000 7fce3c6bc700 10 osd.51 24448261 tick
> > >>>>>> 2022-01-05 20:09:10 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:01.060+0000 7fce3b441700 10 osd.51 24448261
> > >>>>>> tick_without_osd_lock
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:02.268+0000 7fce3c6bc700 10 osd.51 24448261
> > >>> do_waiters --
> > >>>>>> start
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:09.544+0000 7fce3c6bc700 10 osd.51 24448261
> > >>> do_waiters --
> > >>>>>> finish
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:10.260+0000 7fce1e407700 5 osd.51 24448261
> heartbeat
> > >>>>>> osd_stat(store_statfs(0x2258948000/0x40000000/0x3a38800000, data
> > >>>>>> 0x17919fd8c4/0x179feb4000, compress 0x0/0x0/0x0, omap 0xc9773,
> meta
> > >>>>>> 0x3ff3688d), peers [] op hist [])
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.060+0000 7fce3c6bc700 20 osd.51 24448261 tick
> > >>>>>> last_purged_snaps_scrub 2022-01-04T22:29:39.121925+0000 next
> > >>>>>> 2022-01-05T22:29:39.121925+0000
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.104+0000 7fce1e407700 20 osd.51 24448261
> > >>>>>> check_full_status cur ratio 0.410072, physical ratio 0.410072, new
> > >>> state
> > >>>>>> none
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.108+0000 7fce34c34700 20
> > >>>>>> bluestore(/var/lib/ceph/osd/ceph-51) deferred_try_submit 0 osrs, 0
> > >>> txcs
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.108+0000 7fce34c34700 5
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.160+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.216+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.264+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.400+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.536+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.640+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.644+0000 7fce1e407700 5 osd.51 24448261
> heartbeat
> > >>>>>> osd_stat(store_statfs(0x2258948000/0x40000000/0x3a38800000, data
> > >>>>>> 0x17919fd8c4/0x179feb4000, compress 0x0/0x0/0x0, omap 0xc9773,
> meta
> > >>>>>> 0x3ff3688d), peers [] op hist [])
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.712+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.688+0000 7fce1e407700 20 osd.51 24448261
> > >>>>>> check_full_status cur ratio 0.410072, physical ratio 0.410072, new
> > >>> state
> > >>>>>> none
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.480+0000 7fce3b441700 20
> > >>>>>> bluestore(/var/lib/ceph/osd/ceph-51) statfs
> > >>>>>> store_statfs(0x2258948000/0x40000000/0x3a38800000, data
> > >>>>>> 0x17919fd8c4/0x179feb4000, compress 0x0/0x0/0x0, omap 0xc9773,
> meta
> > >>>>>> 0x3ff3688d)
> > >>>>>> 2022-01-05 20:09:13 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:13.844+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:14 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:14.016+0000 7fce34c34700 20
> > >>>>>> bluestore.MempoolThread(0x55f42e762a98) _resize_shards cache_size:
> > >>>>>> 134217728 kv_alloc: 67108864 kv_used: 67082912 meta_alloc:
> 67108864
> > >>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>> 2022-01-05 20:09:38 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>> 2022-01-05T20:09:14.104+0000 7fce3c6bc700 10 osd.51 24448261 tick
> > >>>>>> 2022-01-05 20:10:37 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>> ceph-osd@51.service:
> > >>>>>> Main process exited, code=killed, status=9/KILL
> > >>>>>> 2022-01-05 20:10:37 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>> ceph-osd@51.service:
> > >>>>>> Failed with result 'signal'.
> > >>>>>> 2022-01-05 20:10:47 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>> ceph-osd@51.service:
> > >>>>>> Scheduled restart job, restart counter is at 1.
> > >>>>>> 2022-01-05 20:10:47 bb-ceph-enc-rm63-osd03-31 init.scope Stopped
> Ceph
> > >>>>>> object storage daemon osd.51.
> > >>>>>>
> > >>>>>> I have just increased the RAM physically in one of the node's
> removed
> > >>> the
> > >>>>>> other OSD's physically for now and managed to get one of the 3
> down to
> > >>>>>> come
> > >>>>>> up. Just stepping through each at the moment.
> > >>>>>>
> > >>>>>> Regards
> > >>>>>>
> > >>>>>> Lee
> > >>>>>>
> > >>>>>> On Wed, 5 Jan 2022 at 21:10, mhnx <morphinwithyou@xxxxxxxxx>
> wrote:
> > >>>>>>
> > >>>>>>> First of all, do not rush into bad decisions.
> > >>>>>>> Production is down and you wanna make it online but you should
> fix
> > >>> the
> > >>>>>>> problem and be sure first. If a second crash occurs in a healing
> > >>> state
> > >>>>>>> you will lose metadata.
> > >>>>>>> You don't need to debug first!
> > >>>>>>>
> > >>>>>>> You didn't mention your cluster status and we don't know what you
> > >>> have.
> > >>>>>>> We need some information;
> > >>>>>>> 1- ceph -s
> > >>>>>>> 2- ceph health detail
> > >>>>>>> 3- ceph df
> > >>>>>>> 4- tail /var/log/ceph/ceph-osd{crashed osd number}.log -n 1000
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Lee <lquince@xxxxxxxxx>, 5 Oca 2022 Çar, 23:14 tarihinde şunu
> yazdı:
> > >>>>>>>> Looking for some help as this is production effecting..
> > >>>>>>>>
> > >>>>>>>> We run a 3 Node cluster with a mix of 5xSSD,15xSATA and 5xSAS in
> > >>> each
> > >>>>>>> node.
> > >>>>>>>> Running 15.2.15. All using DB/WAL on NVME SSD except the SSD's
> > >>>>>>>>
> > >>>>>>>> Earlier today I increased the PG num from 32 to 128 on one of
> our
> > >>>>>> pools,
> > >>>>>>>> due to the status complaining. pretty normally really. 2-3 mins
> in I
> > >>>>>>>> watched in horror as SSD based OSD's crashed on all 3 nodes,
> > >>> refusing
> > >>>>>> to
> > >>>>>>>> restart.
> > >>>>>>>>
> > >>>>>>>> I've set debug_bluefs and bluestore to 20 it will get so far and
> > >>> then
> > >>>>>> the
> > >>>>>>>> daemon fails.
> > >>>>>>>>
> > >>>>>>>> 2022-01-05 19:39:23 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:23.335+0000 7f2794383700 20
> > >>>>>>>> bluestore(/var/lib/ceph/osd/ceph-51) deferred_try_submit 0
> osrs, 0
> > >>>>>> txcs
> > >>>>>>>> 2022-01-05 19:39:23 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:23.335+0000 7f2794383700 5
> > >>>>>>>> bluestore.MempoolThread(0x560433f0aa98) _resize_shards
> cache_size:
> > >>>>>>>> 134217728 kv_alloc: 67108864 kv_used: 67075728 meta_alloc:
> 67108864
> > >>>>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>>>> 2022-01-05 19:39:23 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:23.387+0000 7f2794383700 20
> > >>>>>>>> bluestore.MempoolThread(0x560433f0aa98) _resize_shards
> cache_size:
> > >>>>>>>> 134217728 kv_alloc: 67108864 kv_used: 67075728 meta_alloc:
> 67108864
> > >>>>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>>>> 2022-01-05 19:39:23 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:23.467+0000 7f2794383700 20
> > >>>>>>>> bluestore.MempoolThread(0x560433f0aa98) _resize_shards
> cache_size:
> > >>>>>>>> 134217728 kv_alloc: 67108864 kv_used: 67075728 meta_alloc:
> 67108864
> > >>>>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>>>> 2022-01-05 19:39:24 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:23.979+0000 7f2794383700 20
> > >>>>>>>> bluestore.MempoolThread(0x560433f0aa98) _resize_shards
> cache_size:
> > >>>>>>>> 134217728 kv_alloc: 67108864 kv_used: 67075728 meta_alloc:
> 67108864
> > >>>>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>>>> 2022-01-05 19:39:24 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:24.167+0000 7f2794383700 20
> > >>>>>>>> bluestore.MempoolThread(0x560433f0aa98) _resize_shards
> cache_size:
> > >>>>>>>> 134217728 kv_alloc: 67108864 kv_used: 67075728 meta_alloc:
> 67108864
> > >>>>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>>>> 2022-01-05 19:39:24 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:24.271+0000 7f2794383700 20
> > >>>>>>>> bluestore.MempoolThread(0x560433f0aa98) _resize_shards
> cache_size:
> > >>>>>>>> 134217728 kv_alloc: 67108864 kv_used: 67075728 meta_alloc:
> 67108864
> > >>>>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>>>> 2022-01-05 19:39:24 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:24.327+0000 7f2794383700 20
> > >>>>>>>> bluestore.MempoolThread(0x560433f0aa98) _resize_shards
> cache_size:
> > >>>>>>>> 134217728 kv_alloc: 67108864 kv_used: 67075728 meta_alloc:
> 67108864
> > >>>>>>>> meta_used: 75234 data_alloc: 67108864 data_used: 0
> > >>>>>>>> 2022-01-05 19:39:32 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>>> ceph-osd@51.service:
> > >>>>>>>> Main process exited, code=killed, status=9/KILL
> > >>>>>>>> 2022-01-05 19:39:32 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>>> ceph-osd@51.service:
> > >>>>>>>> Failed with result 'signal'.
> > >>>>>>>> 2022-01-05 19:39:42 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>>> ceph-osd@51.service:
> > >>>>>>>> Scheduled restart job, restart counter is at 1.
> > >>>>>>>>
> > >>>>>>>> I've run
> > >>>>>>>> ceph-bluestore-tool bluefs-bdev-sizes --path
> > >>> /var/lib/ceph/osd/ceph-51
> > >>>>>>>> inferring bluefs devices from bluestore path
> > >>>>>>>> 1 : device size 0x3a38800000 : own 0x[1bf2200000~254300000] =
> > >>>>>>> 0x254300000 :
> > >>>>>>>> using 0x3fd10000(1021 MiB) : bluestore has 0x1d83400000(118 GiB)
> > >>>>>>> available
> > >>>>>>>> Also fsck and repair all seems to be ok.
> > >>>>>>>>
> > >>>>>>>> The normal log looks like
> > >>>>>>>>
> > >>>>>>>> 2022-01-05 19:39:42 bb-ceph-enc-rm63-osd03-31 init.scope
> Starting
> > >>> Ceph
> > >>>>>>>> object storage daemon osd.51...
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.467+0000 7fca32943e00 0 set uid:gid to
> > >>>>>> 64045:64045
> > >>>>>>>> (ceph:ceph)
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.467+0000 7fca32943e00 0 ceph version
> 15.2.15
> > >>>>>>>> (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable),
> process
> > >>>>>>>> ceph-osd, pid 139577
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.467+0000 7fca32943e00 0 pidfile_write:
> ignore
> > >>>>>> empty
> > >>>>>>>> --pid-file
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1 bdev create path
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block type kernel
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1 bdev(0x55b4b234e000
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) open path
> > >>>>>>> /var/lib/ceph/osd/ceph-51/block
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1 bdev(0x55b4b234e000
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) open size 250056015872
> > >>> (0x3a38800000,
> > >>>>>>> 233
> > >>>>>>>> GiB) block_size 4096 (4 KiB) non-rotational discard not
> supported
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1
> > >>>>>>>> bluestore(/var/lib/ceph/osd/ceph-51) _set_cache_sizes cache_size
> > >>>>>>> 1073741824
> > >>>>>>>> meta 0.4 kv 0.4 data 0.2
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1 bdev create path
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block type kernel
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1 bdev(0x55b4b234e380
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) open path
> > >>>>>>> /var/lib/ceph/osd/ceph-51/block
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1 bdev(0x55b4b234e380
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) open size 250056015872
> > >>> (0x3a38800000,
> > >>>>>>> 233
> > >>>>>>>> GiB) block_size 4096 (4 KiB) non-rotational discard not
> supported
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1 bluefs
> add_block_device
> > >>>>>>> bdev 1
> > >>>>>>>> path /var/lib/ceph/osd/ceph-51/block size 233 GiB
> > >>>>>>>> 2022-01-05 19:39:46 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:46.491+0000 7fca32943e00 1 bdev(0x55b4b234e380
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) close
> > >>>>>>>> 2022-01-05 19:39:47 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:47.067+0000 7fca32943e00 0 starting osd.51
> > >>> osd_data
> > >>>>>>>> /var/lib/ceph/osd/ceph-51 /var/lib/ceph/osd/ceph-51/journal
> > >>>>>>>> 2022-01-05 19:39:47 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:47.159+0000 7fca32943e00 0 load: jerasure
> load:
> > >>> lrc
> > >>>>>>> load:
> > >>>>>>>> isa
> > >>>>>>>> 2022-01-05 19:39:47 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:47.159+0000 7fca32943e00 1 bdev create path
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block type kernel
> > >>>>>>>> 2022-01-05 19:39:47 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:47.159+0000 7fca32943e00 1 bdev(0x55b4b234e000
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) open path
> > >>>>>>> /var/lib/ceph/osd/ceph-51/block
> > >>>>>>>> 2022-01-05 19:39:47 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:47.163+0000 7fca32943e00 1 bdev(0x55b4b234e000
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) open size 250056015872
> > >>> (0x3a38800000,
> > >>>>>>> 233
> > >>>>>>>> GiB) block_size 4096 (4 KiB) non-rotational discard not
> supported
> > >>>>>>>> 2022-01-05 19:39:47 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:47.163+0000 7fca32943e00 1
> > >>>>>>>> bluestore(/var/lib/ceph/osd/ceph-51) _set_cache_sizes cache_size
> > >>>>>>> 1073741824
> > >>>>>>>> meta 0.4 kv 0.4 data 0.2
> > >>>>>>>> 2022-01-05 19:39:47 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:47.163+0000 7fca32943e00 1 bdev(0x55b4b234e000
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) close
> > >>>>>>>> 2022-01-05 19:39:48 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:48.619+0000 7fca32943e00 1
> > >>>>>>>> bluestore(/var/lib/ceph/osd/ceph-51) _open_alloc loaded 138 GiB
> in
> > >>>>>> 276582
> > >>>>>>>> extents available 129 GiB
> > >>>>>>>> 2022-01-05 19:39:48 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:48.619+0000 7fca32943e00 1 bluefs umount
> > >>>>>>>> 2022-01-05 19:39:48 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:48.619+0000 7fca32943e00 1 bdev(0x55b4b234e380
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) close
> > >>>>>>>> 2022-01-05 19:39:48 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:48.803+0000 7fca32943e00 1 bdev create path
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block type kernel
> > >>>>>>>> 2022-01-05 19:39:48 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:48.803+0000 7fca32943e00 1 bdev(0x55b4b234e380
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) open path
> > >>>>>>> /var/lib/ceph/osd/ceph-51/block
> > >>>>>>>> 2022-01-05 19:39:48 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:48.803+0000 7fca32943e00 1 bdev(0x55b4b234e380
> > >>>>>>>> /var/lib/ceph/osd/ceph-51/block) open size 250056015872
> > >>> (0x3a38800000,
> > >>>>>>> 233
> > >>>>>>>> GiB) block_size 4096 (4 KiB) non-rotational discard not
> supported
> > >>>>>>>> 2022-01-05 19:39:48 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:48.803+0000 7fca32943e00 1 bluefs
> add_block_device
> > >>>>>>> bdev 1
> > >>>>>>>> path /var/lib/ceph/osd/ceph-51/block size 233 GiB
> > >>>>>>>> 2022-01-05 19:39:48 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:48.803+0000 7fca32943e00 1 bluefs mount
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.087+0000 7fca32943e00 1
> > >>>>>>>> bluestore(/var/lib/ceph/osd/ceph-51) _open_db opened rocksdb
> path db
> > >>>>>>>> options
> > >>>>>>>>
> > >>>>>>
> > >>>
> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.087+0000 7fca32943e00 1
> > >>>>>>>> bluestore(/var/lib/ceph/osd/ceph-51) _upgrade_super from 4,
> latest 4
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.087+0000 7fca32943e00 1
> > >>>>>>>> bluestore(/var/lib/ceph/osd/ceph-51) _upgrade_super done
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.131+0000 7fca32943e00 0
> > >>>>>>>> /build/ceph-15.2.15/src/cls/cephfs/cls_cephfs.cc:198: loading
> > >>> cephfs
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.131+0000 7fca32943e00 0
> > >>>>>>>> /build/ceph-15.2.15/src/cls/hello/cls_hello.cc:312: loading
> > >>> cls_hello
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.135+0000 7fca32943e00 0 _get_class not
> > >>> permitted
> > >>>>>> to
> > >>>>>>>> load kvs
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.171+0000 7fca32943e00 0 _get_class not
> > >>> permitted
> > >>>>>> to
> > >>>>>>>> load lua
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.207+0000 7fca32943e00 0 _get_class not
> > >>> permitted
> > >>>>>> to
> > >>>>>>>> load queue
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.319+0000 7fca32943e00 0 _get_class not
> > >>> permitted
> > >>>>>> to
> > >>>>>>>> load sdk
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.319+0000 7fca32943e00 0 osd.51 24448261
> crush
> > >>> map
> > >>>>>>> has
> > >>>>>>>> features 288514051259236352, adjusting msgr requires for clients
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.319+0000 7fca32943e00 0 osd.51 24448261
> crush
> > >>> map
> > >>>>>>> has
> > >>>>>>>> features 288514051259236352 was 8705, adjusting msgr requires
> for
> > >>> mons
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.319+0000 7fca32943e00 0 osd.51 24448261
> crush
> > >>> map
> > >>>>>>> has
> > >>>>>>>> features 3314933000852226048, adjusting msgr requires for osds
> > >>>>>>>> 2022-01-05 19:39:49 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:39:49.319+0000 7fca32943e00 1 osd.51 24448261
> > >>>>>>>> check_osdmap_features require_osd_release unknown -> octopus
> > >>>>>>>> 2022-01-05 19:41:25 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:41:24.999+0000 7fca32943e00 0 osd.51 24448261
> > >>> load_pgs
> > >>>>>>>> opened 66 pgs
> > >>>>>>>> 2022-01-05 19:41:25 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:41:25.071+0000 7fca32943e00 -1 osd.51 24448261
> > >>>>>>>> log_to_monitors {default=true}
> > >>>>>>>> 2022-01-05 19:41:25 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:41:25.071+0000 7fca32943e00 -1 osd.51 24448261
> > >>>>>>>> log_to_monitors {default=true}
> > >>>>>>>> 2022-01-05 19:42:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:42:16.631+0000 7fca32943e00 0 osd.51 24448261
> done
> > >>> with
> > >>>>>>>> init, starting boot process
> > >>>>>>>> 2022-01-05 19:42:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:42:16.631+0000 7fca32943e00 1 osd.51 24448261
> > >>>>>> start_boot
> > >>>>>>>> 2022-01-05 19:42:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:42:16.635+0000 7fca14615700 1 osd.51 pg_epoch:
> > >>> 24448130
> > >>>>>>>> pg[44.17( v 24448128'27126321
> (24447767'27121032,24448128'27126321]
> > >>>>>>>> local-lis/les=24447864/24447865 n=2356 ec=4550661/4550661
> > >>>>>>>> lis/c=24447864/24447864 les/c/f=24447865/24447865/22709931
> > >>>>>> sis=24448130)
> > >>>>>>>> [51,48,15] r=0 lpr=24448130 pi=[24447864,24448130)/1
> > >>>>>>> crt=24448128'27126321
> > >>>>>>>> lcod 0'0 mlcod 0'0 unknown mbc={}] start_peering_interval up
> > >>>>>> [51,48,15]
> > >>>>>>> ->
> > >>>>>>>> [51,48,15], acting [51,48,15] -> [51,48,15], acting_primary 51
> ->
> > >>> 51,
> > >>>>>>>> up_primary 51 -> 51, role 0 -> 0, features acting
> > >>> 4540138292840890367
> > >>>>>>>> upacting 4540138292840890367
> > >>>>>>>> 2022-01-05 19:42:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:42:16.635+0000 7fca13613700 1 osd.51 pg_epoch:
> > >>> 24448130
> > >>>>>>>> pg[44.1( v 24448129'31648690
> (24447777'31643388,24448129'31648690]
> > >>>>>>>> local-lis/les=24447865/24447866 n=2314 ec=4550661/4550661
> > >>>>>>>> lis/c=24447865/24447865 les/c/f=24447866/24447866/22709931
> > >>>>>> sis=24448130)
> > >>>>>>>> [51,15,5] r=0 lpr=24448130 pi=[24447865,24448130)/1
> > >>>>>> crt=24448129'31648690
> > >>>>>>>> lcod 0'0 mlcod 0'0 unknown mbc={}] start_peering_interval up
> > >>>>>> [51,15,5] ->
> > >>>>>>>> [51,15,5], acting [51,15,5] -> [51,15,5], acting_primary 51 ->
> 51,
> > >>>>>>>> up_primary 51 -> 51, role 0 -> 0, features acting
> > >>> 4540138292840890367
> > >>>>>>>> upacting 4540138292840890367
> > >>>>>>>> 2022-01-05 19:42:16 bb-ceph-enc-rm63-osd03-31 osd.51
> > >>>>>>>> 2022-01-05T19:42:16.635+0000 7fca15617700 1 osd.51 pg_epoch:
> > >>> 24448130
> > >>>>>>>> pg[44.15( v 24448129'37939392
> (24447777'37936883,24448129'37939392]
> > >>>>>>>> local-lis/les=24448118/24448119 n=2350 ec=4550661/4550661
> > >>>>>>>> lis/c=24448118/24448118 les/c/f=24448119/24448119/22709931
> > >>>>>> sis=24448130)
> > >>>>>>>> [5,14,51] r=2 lpr=24448130 pi=[24448118,24448130)/1
> > >>>>>> crt=24448129'37939392
> > >>>>>>>> lcod 0'0 mlcod 0'0 unknown mbc={}] start_peering_interval up
> > >>>>>> [5,14,51] ->
> > >>>>>>>> [5,14,51], acting [5,14,51] -> [5,14,51], acting_primary 5 -> 5,
> > >>>>>>> up_primary
> > >>>>>>>> 5 -> 5, role 2 -> 2, features acting 4540138292840890367
> upacting
> > >>>>>>>> 4540138292840890367
> > >>>>>>>> 2022-01-05 19:42:51 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>>> ceph-osd@51.service:
> > >>>>>>>> Main process exited, code=killed, status=9/KILL
> > >>>>>>>> 2022-01-05 19:42:51 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>>> ceph-osd@51.service:
> > >>>>>>>> Failed with result 'signal'.
> > >>>>>>>> 2022-01-05 19:43:01 bb-ceph-enc-rm63-osd03-31 init.scope
> > >>>>>>> ceph-osd@51.service:
>
> > >>>>>>>> Scheduled restart job, restart counter is at 2.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> The problem I have this has basically taken the production and
> > >>>>>> metadata
> > >>>>>>> SSD
> > >>>>>>>> pool's down fully and all 3 copies are offline. And I cannot
> find a
> > >>>>>> way
> > >>>>>>> to
> > >>>>>>>> find out what is causing these to crash.
> > >>>>>>>>
> > >>>>>>>> Kind Regards
> > >>>>>>>>
> > >>>>>>>> Lee
> > >>>>>>>> _______________________________________________
> > >>>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> > >>>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >>>>>> _______________________________________________
> > >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> > >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >>>>>>
> > >>>> _______________________________________________
> > >>>> ceph-users mailing list -- ceph-users@xxxxxxx
> > >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >>>
> > >>> --
> > >>> Igor Fedotov
> > >>> Ceph Lead Developer
> > >>>
> > >>> Looking for help with your Ceph cluster? Contact us at
> https://croit.io
> > >>>
> > >>> croit GmbH, Freseniusstr. 31h, 81247 Munich
> > >>> CEO: Martin Verges - VAT-ID: DE310638492
> > >>> Com. register: Amtsgericht Munich HRB 231263
> > >>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
> > >>>
> > >>>
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
