Hi Eugen,
$ sudo ceph osd tree (output below):
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 2.05046 root default
-3 0.68349 host node01
0 hdd 0.14650 osd.0 up 1.00000 1.00000
4 hdd 0.04880 osd.4 up 1.00000 1.00000
8 hdd 0.04880 osd.8 up 1.00000 1.00000
10 hdd 0.04880 osd.10 up 1.00000 1.00000
14 hdd 0.39059 osd.14 DNE 0
-5 0.68349 host node02
2 hdd 0.14650 osd.2 up 1.00000 1.00000
5 hdd 0.04880 osd.5 up 1.00000 1.00000
7 hdd 0.04880 osd.7 up 1.00000 1.00000
9 hdd 0.04880 osd.9 up 1.00000 1.00000
12 hdd 0.39059 osd.12 up 1.00000 1.00000
-7 0.68349 host node03
1 hdd 0.14650 osd.1 up 1.00000 1.00000
3 hdd 0.04880 osd.3 up 1.00000 1.00000
6 hdd 0.04880 osd.6 up 1.00000 1.00000
11 hdd 0.04880 osd.11 up 1.00000 1.00000
13 hdd 0.39059 osd.13 up 1.00000 1.00000
Also, the output on manager node below:
2023-12-05T10:03:38.559+0200 7fb3fde06700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2023-12-05T10:03:38.559+0200 7fb3fde06700 -1 AuthRegistry(0x7fb3f8064310) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
2023-12-05T10:03:38.559+0200 7fb3fde06700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2023-12-05T10:03:38.559+0200 7fb3fde06700 -1 AuthRegistry(0x7fb3fde04fe0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
2023-12-05T10:03:38.559+0200 7fb3fce04700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:38.559+0200 7fb3fd605700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:41.560+0200 7fb3f7fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:41.560+0200 7fb3fce04700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:44.560+0200 7fb3f7fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:44.560+0200 7fb3fd605700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:47.560+0200 7fb3fd605700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:47.560+0200 7fb3f7fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:50.564+0200 7fb3fd605700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:50.564+0200 7fb3f7fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:53.560+0200 7fb3fce04700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:53.564+0200 7fb3f7fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:56.564+0200 7fb3fce04700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:56.564+0200 7fb3fd605700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:59.564+0200 7fb3fd605700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:03:59.564+0200 7fb3f7fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:04:02.564+0200 7fb3fce04700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:04:02.564+0200 7fb3fd605700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:04:05.564+0200 7fb3fce04700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:04:05.564+0200 7fb3f7fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2023-12-05T10:04:08.564+0200 7fb3fce04700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
It stucks after running "ceph mgr fail" command with all the above messages.
The mds daemon show the below when issuing "systemctl status ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mds.storage.node01.cjrvjc.service" (node01)
● ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mds.storage.node01.cjrvjc.service - Ceph mds.storage.node01.cjrvjc for be4304e4-b0d5-11ec-8c6a-2965d4229f37
Loaded: loaded (/etc/systemd/system/ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-12-05 10:16:41 EET; 7s ago
Main PID: 632331 (bash)
Tasks: 10 (limit: 72186)
Memory: 10.5M
CGroup: /system.slice/system-ceph\x2dbe4304e4\x2db0d5\x2d11ec\x2d8c6a\x2d2965d4229f37.slice/ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mds.storage.node01.cjrvjc.service
├─632331 /bin/bash /var/lib/ceph/be4304e4-b0d5-11ec-8c6a-2965d4229f37/mds.storage.node01.cjrvjc/unit.run
└─632356 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph-mds --init --name ceph-be4304e4-b0d5-11ec>
Dec 05 10:16:41 node01 systemd[1]: Started Ceph mds.storage.node01.cjrvjc for be4304e4-b0d5-11ec-8c6a-2965d4229f37.
Dec 05 10:16:42 node01 bash[632356]: debug 2023-12-05T08:16:42.166+0000 7fb7e5585ac0 0 set uid:gid to 167:167 (ceph:ceph)
Dec 05 10:16:42 node01 bash[632356]: debug 2023-12-05T08:16:42.166+0000 7fb7e5585ac0 0 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process ceph-md>
Dec 05 10:16:42 node01 bash[632356]: debug 2023-12-05T08:16:42.166+0000 7fb7e5585ac0 1 main not setting numa affinity
Dec 05 10:16:42 node01 bash[632356]: debug 2023-12-05T08:16:42.166+0000 7fb7e5585ac0 0 pidfile_write: ignore empty --pid-file
Dec 05 10:16:42 node01 bash[632356]: starting mds.storage.node01.cjrvjc at
Dec 05 10:16:42 node01 bash[632356]: debug 2023-12-05T08:16:42.174+0000 7fb7db80c700 1 mds.storage.node01.cjrvjc Updating MDS map to version 6 from mon.2
Dec 05 10:16:42 node01 bash[632356]: debug 2023-12-05T08:16:42.422+0000 7fb7db80c700 1 mds.storage.node01.cjrvjc Updating MDS map to version 7 from mon.2
Dec 05 10:16:42 node01 bash[632356]: debug 2023-12-05T08:16:42.422+0000 7fb7db80c700 1 mds.storage.node01.cjrvjc Monitors have assigned me to become a standby.
The mds daemon show the below when issuing "systemctl status ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mds.storage.node02.lyudbp.service" (node02)
● ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mds.storage.node02.lyudbp.service - Ceph mds.storage.node02.lyudbp for be4304e4-b0d5-11ec-8c6a-2965d4229f37
Loaded: loaded (/etc/systemd/system/ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-12-05 10:17:21 EET; 1s ago
Main PID: 612499 (bash)
Tasks: 10 (limit: 72186)
Memory: 10.5M
CGroup: /system.slice/system-ceph\x2dbe4304e4\x2db0d5\x2d11ec\x2d8c6a\x2d2965d4229f37.slice/ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mds.storage.node02.lyudbp.service
├─612499 /bin/bash /var/lib/ceph/be4304e4-b0d5-11ec-8c6a-2965d4229f37/mds.storage.node02.lyudbp/unit.run
└─612517 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph-mds --init --name ceph-be4304e4-b0d5-11ec>
Dec 05 10:17:21 node02 systemd[1]: Started Ceph mds.storage.node02.lyudbp for be4304e4-b0d5-11ec-8c6a-2965d4229f37.
Dec 05 10:17:22 node02 bash[612517]: debug 2023-12-05T08:17:22.181+0000 7fd6ec9f4ac0 0 set uid:gid to 167:167 (ceph:ceph)
Dec 05 10:17:22 node02 bash[612517]: debug 2023-12-05T08:17:22.181+0000 7fd6ec9f4ac0 0 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process ceph-md>
Dec 05 10:17:22 node02 bash[612517]: debug 2023-12-05T08:17:22.181+0000 7fd6ec9f4ac0 1 main not setting numa affinity
Dec 05 10:17:22 node02 bash[612517]: starting mds.storage.node02.lyudbp at
Dec 05 10:17:22 node02 bash[612517]: debug 2023-12-05T08:17:22.181+0000 7fd6ec9f4ac0 0 pidfile_write: ignore empty --pid-file
Dec 05 10:17:22 node02 bash[612517]: debug 2023-12-05T08:17:22.189+0000 7fd6e2c7b700 1 mds.storage.node02.lyudbp Updating MDS map to version 8 from mon.1
Dec 05 10:17:22 node02 bash[612517]: debug 2023-12-05T08:17:22.405+0000 7fd6e2c7b700 1 mds.storage.node02.lyudbp Updating MDS map to version 9 from mon.1
Dec 05 10:17:22 node02 bash[612517]: debug 2023-12-05T08:17:22.405+0000 7fd6e2c7b700 1 mds.storage.node02.lyudbp Monitors have assigned me to become a standby.
I also add below the output of "ceph health detail"
HEALTH_ERR 20 stray daemon(s) not managed by cephadm; 3 stray host(s) with 20 daemon(s) not managed by cephadm; 1/3 mons down, quorum node02,node03; 1/523510 objects unfound (0.000%); 3 nearfull osd(s); 1 osds exist in the crush map but not in the osdmap; Low space hindering backfill (add storage if this doesn't resolve itself): 20 pgs backfill_toofull; Possible data damage: 1 pg recovery_unfound; Degraded data redundancy: 74666/1570530 objects degraded (4.754%), 21 pgs degraded, 21 pgs undersized; 21 pgs not deep-scrubbed in time; 21 pgs not scrubbed in time; 3 pool(s) nearfull
[WRN] CEPHADM_STRAY_DAEMON: 20 stray daemon(s) not managed by cephadm
stray daemon mds.storage.node01.cjrvjc on host node01 not managed by cephadm
stray daemon mgr.node01.xlciyx on host node01 not managed by cephadm
stray daemon osd.0 on host node01 not managed by cephadm
stray daemon osd.10 on host node01 not managed by cephadm
stray daemon osd.4 on host node01 not managed by cephadm
stray daemon osd.8 on host node01 not managed by cephadm
stray daemon mds.storage.node02.lyudbp on host node02 not managed by cephadm
stray daemon mgr.node02.gudauu on host node02 not managed by cephadm
stray daemon mon.node02 on host node02 not managed by cephadm
stray daemon osd.12 on host node02 not managed by cephadm
stray daemon osd.2 on host node02 not managed by cephadm
stray daemon osd.5 on host node02 not managed by cephadm
stray daemon osd.7 on host node02 not managed by cephadm
stray daemon osd.9 on host node02 not managed by cephadm
stray daemon mon.node03 on host node03 not managed by cephadm
stray daemon osd.1 on host node03 not managed by cephadm
stray daemon osd.11 on host node03 not managed by cephadm
stray daemon osd.13 on host node03 not managed by cephadm
stray daemon osd.3 on host node03 not managed by cephadm
stray daemon osd.6 on host node03 not managed by cephadm
[WRN] CEPHADM_STRAY_HOST: 3 stray host(s) with 20 daemon(s) not managed by cephadm
stray host node01 has 6 stray daemons: ['mds.storage.node01.cjrvjc', 'mgr.node01.xlciyx', 'osd.0', 'osd.10', 'osd.4', 'osd.8']
stray host node02 has 8 stray daemons: ['mds.storage.node02.lyudbp', 'mgr.node02.gudauu', 'mon.node02', 'osd.12', 'osd.2', 'osd.5', 'osd.7', 'osd.9']
stray host node03 has 6 stray daemons: ['mon.node03', 'osd.1', 'osd.11', 'osd.13', 'osd.3', 'osd.6']
[WRN] MON_DOWN: 1/3 mons down, quorum node02,node03
mon.node01 (rank 0) addr [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] is down (out of quorum)
[WRN] OBJECT_UNFOUND: 1/523510 objects unfound (0.000%)
pg 2.2 has 1 unfound objects
[WRN] OSD_NEARFULL: 3 nearfull osd(s)
osd.0 is near full
osd.8 is near full
osd.10 is near full
[WRN] OSD_ORPHAN: 1 osds exist in the crush map but not in the osdmap
osd.14 exists in crush map but not in osdmap
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 20 pgs backfill_toofull
pg 3.2 is active+undersized+degraded+remapped+backfill_toofull, acting [13,12]
pg 3.c is active+undersized+degraded+remapped+backfill_toofull, acting [12,1]
pg 3.12 is active+undersized+degraded+remapped+backfill_toofull, acting [7,11]
pg 3.17 is active+undersized+degraded+remapped+backfill_toofull, acting [13,12]
pg 3.27 is active+undersized+degraded+remapped+backfill_toofull, acting [12,1]
pg 3.2a is active+undersized+degraded+remapped+backfill_toofull, acting [12,13]
pg 3.31 is active+undersized+degraded+remapped+backfill_toofull, acting [13,9]
pg 3.34 is active+undersized+degraded+remapped+backfill_toofull, acting [12,6]
pg 3.35 is active+undersized+degraded+remapped+backfill_toofull, acting [12,13]
pg 3.39 is active+undersized+degraded+remapped+backfill_toofull, acting [12,1]
pg 3.3b is active+undersized+degraded+remapped+backfill_toofull, acting [13,7]
pg 3.49 is active+undersized+degraded+remapped+backfill_toofull, acting [12,13]
pg 3.4a is active+undersized+degraded+remapped+backfill_toofull, acting [13,2]
pg 3.53 is active+undersized+degraded+remapped+backfill_toofull, acting [12,13]
pg 3.56 is active+undersized+degraded+remapped+backfill_toofull, acting [12,1]
pg 3.57 is active+undersized+degraded+remapped+backfill_toofull, acting [13,2]
pg 3.5d is active+undersized+degraded+remapped+backfill_toofull, acting [12,13]
pg 3.6c is active+undersized+degraded+remapped+backfill_toofull, acting [13,12]
pg 3.6d is active+undersized+degraded+remapped+backfill_toofull, acting [12,13]
pg 3.75 is active+undersized+degraded+remapped+backfill_toofull, acting [13,5]
[ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
pg 2.2 is active+recovery_unfound+undersized+degraded+remapped, acting [5,13], 1 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 74666/1570530 objects degraded (4.754%), 21 pgs degraded, 21 pgs undersized
pg 2.2 is stuck undersized for 2w, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,13]
pg 3.2 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [13,12]
pg 3.c is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,1]
pg 3.12 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [7,11]
pg 3.17 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [13,12]
pg 3.27 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,1]
pg 3.2a is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,13]
pg 3.31 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [13,9]
pg 3.34 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,6]
pg 3.35 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,13]
pg 3.39 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,1]
pg 3.3b is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [13,7]
pg 3.49 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,13]
pg 3.4a is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [13,2]
pg 3.53 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,13]
pg 3.56 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,1]
pg 3.57 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [13,2]
pg 3.5d is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,13]
pg 3.6c is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [13,12]
pg 3.6d is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,13]
pg 3.75 is stuck undersized for 2w, current state active+undersized+degraded+remapped+backfill_toofull, last acting [13,5]
[WRN] PG_NOT_DEEP_SCRUBBED: 21 pgs not deep-scrubbed in time
pg 3.75 not deep-scrubbed since 2023-11-09T21:26:58.057287+0000
pg 3.6d not deep-scrubbed since 2023-11-14T22:51:12.464463+0000
pg 3.6c not deep-scrubbed since 2023-11-11T16:54:13.940623+0000
pg 3.5d not deep-scrubbed since 2023-11-12T14:45:24.377322+0000
pg 3.57 not deep-scrubbed since 2023-11-12T11:13:12.897755+0000
pg 3.56 not deep-scrubbed since 2023-11-13T16:37:11.865479+0000
pg 3.53 not deep-scrubbed since 2023-11-11T07:31:11.837450+0000
pg 3.4a not deep-scrubbed since 2023-11-13T23:20:30.121413+0000
pg 3.49 not deep-scrubbed since 2023-11-15T00:10:04.825296+0000
pg 3.3b not deep-scrubbed since 2023-11-13T20:32:17.338096+0000
pg 3.39 not deep-scrubbed since 2023-11-15T06:01:18.346350+0000
pg 3.35 not deep-scrubbed since 2023-11-08T17:47:01.511603+0000
pg 3.34 not deep-scrubbed since 2023-11-15T19:45:02.148231+0000
pg 3.31 not deep-scrubbed since 2023-11-15T15:34:01.510935+0000
pg 3.17 not deep-scrubbed since 2023-11-15T03:29:14.419442+0000
pg 3.12 not deep-scrubbed since 2023-11-09T09:41:32.171837+0000
pg 2.2 not deep-scrubbed since 2023-11-10T03:02:25.248648+0000
pg 3.2 not deep-scrubbed since 2023-11-14T20:25:27.750532+0000
pg 3.c not deep-scrubbed since 2023-11-15T18:47:44.742320+0000
pg 3.27 not deep-scrubbed since 2023-11-14T16:33:14.652728+0000
pg 3.2a not deep-scrubbed since 2023-11-15T18:01:21.875230+0000
[WRN] PG_NOT_SCRUBBED: 21 pgs not scrubbed in time
pg 3.75 not scrubbed since 2023-11-14T23:02:21.867641+0000
pg 3.6d not scrubbed since 2023-11-14T22:51:12.464463+0000
pg 3.6c not scrubbed since 2023-11-15T22:35:52.110113+0000
pg 3.5d not scrubbed since 2023-11-15T06:14:24.294473+0000
pg 3.57 not scrubbed since 2023-11-15T06:58:50.453749+0000
pg 3.56 not scrubbed since 2023-11-14T22:27:28.762497+0000
pg 3.53 not scrubbed since 2023-11-15T12:50:43.604679+0000
pg 3.4a not scrubbed since 2023-11-15T07:17:50.225197+0000
pg 3.49 not scrubbed since 2023-11-15T00:10:04.825296+0000
pg 3.3b not scrubbed since 2023-11-14T23:39:36.602972+0000
pg 3.39 not scrubbed since 2023-11-15T06:01:18.346350+0000
pg 3.35 not scrubbed since 2023-11-15T06:29:59.408409+0000
pg 3.34 not scrubbed since 2023-11-15T19:45:02.148231+0000
pg 3.31 not scrubbed since 2023-11-15T15:34:01.510935+0000
pg 3.17 not scrubbed since 2023-11-15T03:29:14.419442+0000
pg 3.12 not scrubbed since 2023-11-15T20:05:23.103069+0000
pg 2.2 not scrubbed since 2023-11-15T05:46:04.363718+0000
pg 3.2 not scrubbed since 2023-11-14T20:25:27.750532+0000
pg 3.c not scrubbed since 2023-11-15T18:47:44.742320+0000
pg 3.27 not scrubbed since 2023-11-15T21:09:57.747494+0000
pg 3.2a not scrubbed since 2023-11-15T18:01:21.875230+0000
[WRN] POOL_NEARFULL: 3 pool(s) nearfull
pool '.mgr' is nearfull
pool 'cephfs.storage.meta' is nearfull
pool 'cephfs.storage.data' is nearfull
Any ideas ?
Thanks,
Manolis Daramas
-----Original Message-----
From: Eugen Block <eblock@xxxxxx>
Sent: Tuesday, November 21, 2023 1:10 PM
To: ceph-users@xxxxxxx
Subject: Re: After hardware failure tried to recover ceph and followed instructions for recovery using OSDS
Hi,
I guess you could just redeploy the third MON which fails to start
(after the orchestrator is responding again) unless you figured it out
already. What is it logging?
> 1 osds exist in the crush map but not in the osdmap
This could be due to the input/output error, but it's just a guess:
> osd.10 : 9225 osdmaps trimmed, 0 osdmaps added.
> Mount failed with '(5) Input/output error'
Can you add the 'ceph osd tree' output?
> # ceph fs ls (output below):
> No filesystems enabled
Ceph doesn't report active MDS daemons, there are two processes
listed, one on node01, the other on node02. What are those daemons
logging?
> It looks like that we have a problem with the orchestrator now
> (we've lost cephadm orchestrator) and we also cannot see the
> filesystem.
Depending on the cluster status the orchestrator might not behave as
expected, and HEALTH_ERR isn't too good, of course. But you could try
to do a 'ceph mgr fail' and see if it reacts again.
Zitat von Manolis Daramas <mdaramas@xxxxxxxxxxxx>:
> Hello everyone,
>
> We had a recent power failure on a server which hosts a 3-node ceph
> cluster (with Ubuntu 20.04 and Ceph version 17.2.7) and we think
> that we may have lost some of our data if not all of them.
>
> We have followed the instructions on
> https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#recovery-using-osds but with no
> luck.
>
> We have kept a backup of store.db folder on all 3 nodes prior the
> below steps.
>
> We have stopped ceph.target on all 3 nodes.
>
> We have run the first part of the script and we have altered it
> according to our configuration:
>
> ms=/root/mon-store
> mkdir $ms
>
> hosts="node01 node02 node03"
> # collect the cluster map from stopped OSDs
> for host in $hosts; do
> rsync -avz $ms/. root@$host:$ms.remote
> rm -rf $ms
> ssh root@$host <<EOF
> for osd in /var/lib/ceph/be4304e4-b0d5-11ec-8c6a-2965d4229f37/osd*; do
> ceph-objectstore-tool --data-path \$osd --no-mon-config --op
> update-mon-db --mon-store-path $ms.remote
> done
> EOF
> rsync -avz root@$host:$ms.remote/. $ms
> done
>
> and the results were:
>
> for node01
>
> sd.0 : 0 osdmaps trimmed, 673 osdmaps added.
> osd.10 : 9225 osdmaps trimmed, 0 osdmaps added.
> Mount failed with '(5) Input/output error'
> osd.4 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.8 : 0 osdmaps trimmed, 0 osdmaps added.
> receiving incremental file list
> created directory /root/mon-store
> ./
> kv_backend
> store.db/
> store.db/000008.sst
> store.db/000014.sst
> store.db/000020.sst
> store.db/000022.log
> store.db/CURRENT
> store.db/IDENTITY
> store.db/LOCK
> store.db/MANIFEST-000021
> store.db/OPTIONS-000018
> store.db/OPTIONS-000024
>
> sent 248 bytes received 286,474 bytes 191,148.00 bytes/sec
> total size is 7,869,025 speedup is 27.44
> sending incremental file list
> created directory /root/mon-store.remote
> ./
> kv_backend
> store.db/
> store.db/000008.sst
> store.db/000014.sst
> store.db/000020.sst
> store.db/000022.log
> store.db/CURRENT
> store.db/IDENTITY
> store.db/LOCK
> store.db/MANIFEST-000021
> store.db/OPTIONS-000018
> store.db/OPTIONS-000024
>
> sent 286,478 bytes received 285 bytes 191,175.33 bytes/sec
> total size is 7,869,025 speedup is 27.44
>
> for node02
>
> osd.12 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.2 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.5 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.7 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.9 : 0 osdmaps trimmed, 0 osdmaps added.
> receiving incremental file list
> created directory /root/mon-store
> ./
> kv_backend
> store.db/
> store.db/000008.sst
> store.db/000014.sst
> store.db/000020.sst
> store.db/000026.sst
> store.db/000032.sst
> store.db/000038.sst
> store.db/000044.sst
> store.db/000050.sst
> store.db/000052.log
> store.db/CURRENT
> store.db/IDENTITY
> store.db/LOCK
> store.db/MANIFEST-000051
> store.db/OPTIONS-000048
> store.db/OPTIONS-000054
>
> sent 343 bytes received 291,082 bytes 194,283.33 bytes/sec
> total size is 7,875,746 speedup is 27.02
> sending incremental file list
> created directory /root/mon-store.remote
> ./
> kv_backend
> store.db/
> store.db/000008.sst
> store.db/000014.sst
> store.db/000020.sst
> store.db/000026.sst
> store.db/000032.sst
> store.db/000038.sst
> store.db/000044.sst
> store.db/000050.sst
> store.db/000052.log
> store.db/CURRENT
> store.db/IDENTITY
> store.db/LOCK
> store.db/MANIFEST-000051
> store.db/OPTIONS-000048
> store.db/OPTIONS-000054
>
> sent 291,078 bytes received 380 bytes 582,916.00 bytes/sec
> total size is 7,875,746 speedup is 27.02
>
> for node03
>
> osd.1 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.11 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.13 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.3 : 0 osdmaps trimmed, 0 osdmaps added.
> osd.6 : 0 osdmaps trimmed, 0 osdmaps added.
> receiving incremental file list
> created directory /root/mon-store
> ./
> kv_backend
> store.db/
> store.db/000008.sst
> store.db/000014.sst
> store.db/000020.sst
> store.db/000026.sst
> store.db/000032.sst
> store.db/000038.sst
> store.db/000044.sst
> store.db/000050.sst
> store.db/000056.sst
> store.db/000062.sst
> store.db/000068.sst
> store.db/000074.sst
> store.db/000080.sst
> store.db/000082.log
> store.db/CURRENT
> store.db/IDENTITY
> store.db/LOCK
> store.db/MANIFEST-000081
> store.db/OPTIONS-000078
> store.db/OPTIONS-000084
>
> sent 438 bytes received 295,659 bytes 592,194.00 bytes/sec
> total size is 7,882,477 speedup is 26.62
>
> Then we have run the (in order to rebuild the monstore DB and fix it):
>
>
> ceph-monstore-tool /root/mon-store rebuild -- --keyring
> /etc/ceph/ceph.client.admin.keyring --mon-ids node01 node02 node03
>
>
>
> and the output is below:
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: RocksDB version: 6.15.5
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Git sha
> rocksdb_build_git_sha:@0@
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compile date
> Oct 25 2023
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB SUMMARY
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB Session ID:
> OS2T69IQ02SU5OKHBI40
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: CURRENT file: CURRENT
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: IDENTITY file:
> IDENTITY
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: MANIFEST file:
> MANIFEST-000081 size: 1083 Bytes
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: SST files in
> /root/mon-store/store.db dir, Total Num: 13, files: 000008.sst
> 000014.sst 000020.sst 000026.sst 000032.sst 000038.sst 000044.sst
> 000050.sst 000056.sst
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Write Ahead
> Log file in /root/mon-store/store.db: 000082.log size: 244 ;
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.error_if_exists: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.create_if_missing: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.paranoid_checks: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.track_and_verify_wals_in_manifest: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.env: 0x56017c8d1c20
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.fs: Posix File System
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.info_log: 0x56017d4c3860
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_file_opening_threads: 16
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.statistics: (nil)
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.use_fsync: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_log_file_size: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_manifest_file_size: 1073741824
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.log_file_time_to_roll: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.keep_log_file_num: 1000
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.recycle_log_file_num: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.allow_fallocate: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.allow_mmap_reads: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.allow_mmap_writes: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.use_direct_reads: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.use_direct_io_for_flush_and_compaction: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.create_missing_column_families: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.db_log_dir:
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.wal_dir: /root/mon-store/store.db
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.table_cache_numshardbits: 6
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.WAL_ttl_seconds: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.WAL_size_limit_MB: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_write_batch_group_size_bytes: 1048576
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.manifest_preallocation_size: 4194304
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.is_fd_close_on_exec: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.advise_random_on_open: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.db_write_buffer_size: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.write_buffer_manager: 0x56017d1f6a20
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.access_hint_on_compaction_start: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.new_table_reader_for_compaction_inputs: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.random_access_max_buffer_size: 1048576
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.use_adaptive_mutex: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.rate_limiter: (nil)
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.sst_file_manager.rate_bytes_per_sec: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.wal_recovery_mode: 2
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.enable_thread_tracking: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.enable_pipelined_write: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.unordered_write: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.allow_concurrent_memtable_write: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.enable_write_thread_adaptive_yield: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.write_thread_max_yield_usec: 100
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.write_thread_slow_yield_usec: 3
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.row_cache: None
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.wal_filter: None
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.avoid_flush_during_recovery: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.allow_ingest_behind: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.preserve_deletes: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.two_write_queues: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.manual_wal_flush: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.atomic_flush: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.avoid_unnecessary_blocking_io: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.persist_stats_to_disk: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.write_dbid_to_manifest: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.log_readahead_size: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.file_checksum_gen_factory: Unknown
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.best_efforts_recovery: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bgerror_resume_count: 2147483647
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bgerror_resume_retry_interval: 1000000
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.allow_data_in_errors: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.db_host_id: __hostname__
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_background_jobs: 2
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_background_compactions: -1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_subcompactions: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.avoid_flush_during_shutdown: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.writable_file_max_buffer_size: 1048576
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.delayed_write_rate : 16777216
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_total_wal_size: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.delete_obsolete_files_period_micros: 21600000000
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.stats_dump_period_sec: 600
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.stats_persist_period_sec: 600
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.stats_history_buffer_size: 1048576
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_open_files: -1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bytes_per_sync: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.wal_bytes_per_sync: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.strict_bytes_per_sync: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_readahead_size: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_background_flushes: -1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compression
> algorithms supported:
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> kZSTDNotFinalCompression supported: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTD supported: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> kXpressCompression supported: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> kLZ4HCCompression supported: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> kLZ4Compression supported: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> kBZip2Compression supported: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> kZlibCompression supported: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> kSnappyCompression supported: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Fast CRC32
> supported: Supported on x86
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> [db/version_set.cc:4724] Recovering from manifest file:
> /root/mon-store/store.db/MANIFEST-000081
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> [db/column_family.cc:595] --------------- Options for column family
> [default]:
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.comparator: leveldb.BytewiseComparator
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.merge_operator:
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_filter: None
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_filter_factory: None
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.sst_partitioner_factory: None
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.memtable_factory: SkipListFactory
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.table_factory: BlockBasedTable
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> table_factory options: flush_block_policy_factory:
> FlushBlockBySizePolicyFactory (0x56017d234f80)
>
> cache_index_and_filter_blocks: 1
>
> cache_index_and_filter_blocks_with_high_priority: 0
>
> pin_l0_filter_and_index_blocks_in_cache: 0
>
> pin_top_level_index_and_filter: 1
>
> index_type: 0
>
> data_block_index_type: 0
>
> index_shortening: 1
>
> data_block_hash_table_util_ratio: 0.750000
>
> hash_index_allow_collision: 1
>
> checksum: 1
>
> no_block_cache: 0
>
> block_cache: 0x56017d22f610
>
> block_cache_name: BinnedLRUCache
>
> block_cache_options:
>
> capacity : 536870912
>
> num_shard_bits : 4
>
> strict_capacity_limit : 0
>
> high_pri_pool_ratio: 0.000
>
> block_cache_compressed: (nil)
>
> persistent_cache: (nil)
>
> block_size: 4096
>
> block_size_deviation: 10
>
> block_restart_interval: 16
>
> index_block_restart_interval: 1
>
> metadata_block_size: 4096
>
> partition_filters: 0
>
> use_delta_encoding: 1
>
> filter_policy: rocksdb.BuiltinBloomFilter
>
> whole_key_filtering: 1
>
> verify_compression: 0
>
> read_amp_bytes_per_bit: 0
>
> format_version: 4
>
> enable_index_compression: 1
>
> block_align: 0
>
>
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.write_buffer_size: 33554432
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_write_buffer_number: 2
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compression: NoCompression
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bottommost_compression: Disabled
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.prefix_extractor: nullptr
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.memtable_insert_with_hint_prefix_extractor: nullptr
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.num_levels: 7
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.min_write_buffer_number_to_merge: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_write_buffer_number_to_maintain: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_write_buffer_size_to_maintain: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bottommost_compression_opts.window_bits: -14
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bottommost_compression_opts.level: 32767
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bottommost_compression_opts.strategy: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bottommost_compression_opts.max_dict_bytes: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bottommost_compression_opts.zstd_max_train_bytes: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bottommost_compression_opts.parallel_threads: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bottommost_compression_opts.enabled: false
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compression_opts.window_bits: -14
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compression_opts.level: 32767
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compression_opts.strategy: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compression_opts.max_dict_bytes: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compression_opts.zstd_max_train_bytes: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compression_opts.parallel_threads: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compression_opts.enabled: false
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.level0_file_num_compaction_trigger: 4
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.level0_slowdown_writes_trigger: 20
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.level0_stop_writes_trigger: 36
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.target_file_size_base: 67108864
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.target_file_size_multiplier: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_base: 268435456
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.level_compaction_dynamic_level_bytes: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_multiplier: 10.000000
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_multiplier_addtl[0]: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_multiplier_addtl[1]: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_multiplier_addtl[2]: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_multiplier_addtl[3]: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_multiplier_addtl[4]: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_multiplier_addtl[5]: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_bytes_for_level_multiplier_addtl[6]: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_sequential_skip_in_iterations: 8
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_compaction_bytes: 1677721600
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.arena_block_size: 4194304
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.soft_pending_compaction_bytes_limit: 68719476736
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.hard_pending_compaction_bytes_limit: 274877906944
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.rate_limit_delay_max_milliseconds: 100
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.disable_auto_compactions: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_style: kCompactionStyleLevel
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_pri: kMinOverlappingRatio
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_options_universal.size_ratio: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_options_universal.min_merge_width: 2
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_options_universal.max_merge_width: 4294967295
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_options_universal.max_size_amplification_percent:
> 200
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_options_universal.compression_size_percent: -1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_options_universal.stop_style:
> kCompactionStopStyleTotalSize
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_options_fifo.max_table_files_size: 1073741824
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.compaction_options_fifo.allow_compaction: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.table_properties_collectors:
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.inplace_update_support: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.inplace_update_num_locks: 10000
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.memtable_prefix_bloom_size_ratio: 0.000000
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.memtable_whole_key_filtering: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.memtable_huge_page_size: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.bloom_locality: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.max_successive_merges: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.optimize_filters_for_hits: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.paranoid_file_checks: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.force_consistency_checks: 1
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.report_bg_io_stats: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.ttl: 2592000
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.periodic_compaction_seconds: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.enable_blob_files: false
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.min_blob_size: 0
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.blob_file_size: 268435456
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.blob_compression_type: NoCompression
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.enable_blob_garbage_collection: false
>
>
>
> 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
> Options.blob_garbage_collection_age_cutoff: 0.250000
>
>
>
> 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb:
> [db/version_set.cc:4764] Recovered from manifest
> file:/root/mon-store/store.db/MANIFEST-000081
> succeeded,manifest_file_number is 81, next_file_number is 83,
> last_sequence is 21183, log_number is 77,prev_log_number is
> 0,max_column_family is 0,min_log_number_to_keep is 0
>
>
>
>
>
> 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb:
> [db/version_set.cc:4779] Column family [default] (ID 0), log number
> is 77
>
>
>
>
>
> 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb:
> [db/version_set.cc:4082] Creating manifest 85
>
>
>
>
>
> 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784162798, "job": 1, "event":
> "recovery_started", "wal_files": [82]}
>
>
>
> 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb:
> [db/db_impl/db_impl_open.cc:845] Recovering log #82 mode 2
>
>
>
> 2023-11-17T12:26:24.160+0200 7f482b393600 3 rocksdb:
> [table/block_based/filter_policy.cc:991] Using legacy Bloom filter
> with high (20) bits/key. Dramatic filter space and/or accuracy
> improvement is available with format_version>=5.
>
>
>
> 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784163944, "cf_name": "default", "job": 1,
> "event": "table_file_creation", "file_number": 86, "file_size":
> 1266, "file_checksum": "", "file_checksum_func_name": "Unknown",
> "table_properties": {"data_size": 238, "index_size": 40,
> "index_partitions": 0, "top_level_index_size": 0,
> "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1,
> "filter_size": 69, "raw_key_size": 72, "raw_average_key_size": 24,
> "raw_value_size": 148, "raw_average_value_size": 49,
> "num_data_blocks": 1, "num_entries": 3, "num_deletions": 0,
> "num_merge_operands": 0, "num_range_deletions": 0, "format_version":
> 0, "fixed_key_len": 0, "filter_policy":
> "rocksdb.BuiltinBloomFilter", "column_family_name": "default",
> "column_family_id": 0, "comparator": "leveldb.BytewiseComparator",
> "merge_operator": "", "prefix_extractor_name": "nullptr",
> "property_collectors": "[]", "compression": "NoCompression",
> "compression_options": "wind
> ow_bits=-14; level=32767; strategy=0; max_dict_bytes=0;
> zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216784,
> "oldest_key_time": 0, "file_creation_time": 0, "db_id":
> "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id":
> "OS2T69IQ02SU5OKHBI40"}}
>
>
>
> 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb:
> [db/version_set.cc:4082] Creating manifest 87
>
>
>
>
>
> 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784166273, "job": 1, "event":
> "recovery_finished"}
>
>
>
> 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb:
> [db/column_family.cc:983] [default] Increasing compaction threads
> because we have 14 level-0 files
>
>
>
> 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000082.log immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb:
> [db/db_impl/db_impl_open.cc:1700] SstFileManager instance
> 0x56017d230700
>
>
>
> 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: DB pointer
> 0x56017df56000
>
>
>
> adding auth for 'client.admin':
> auth(key=AQCsdUViHYjTGBAAf7/1KYZjb0h3x3EOywqbbQ==) with
> caps({mds=allow *,mgr=allow *,mon=allow *,osd=allow *})
>
> 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb:
> [db/compaction/compaction_job.cc:1881] [default] [JOB 3] Compacting
> 14@0 files to L6, score 3.50
>
>
>
> 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb:
> [db/compaction/compaction_job.cc:1887] [default] Compaction start
> summary: Base version 3 Base level 0, inputs: [86(1266B) 80(1266B)
> 74(1267B) 68(1267B) 62(1266B) 56(1265B) 50(1265B) 44(1265B)
> 38(1265B) 32(1266B) 26(1265B) 20(1265B) 14(283KB) 8(7387KB)]
>
>
>
>
>
> 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784169200, "job": 3, "event":
> "compaction_started", "compaction_reason": "LevelL0FilesNum",
> "files_L0": [86, 80, 74, 68, 62, 56, 50, 44, 38, 32, 26, 20, 14, 8],
> "score": 3.5, "input_data_size": 7870219}
>
>
>
> 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb:
> [db/db_impl/db_impl.cc:901] ------- DUMPING STATS -------
>
>
>
> 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb:
> [db/db_impl/db_impl.cc:903]
>
> ** DB Stats **
>
> Uptime(secs): 0.0 total, 0.0 interval
>
> Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per
> commit group, ingest: 0.00 GB, 0.00 MB/s
>
> Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written:
> 0.00 GB, 0.00 MB/s
>
> Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>
> Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per
> commit group, ingest: 0.00 MB, 0.00 MB/s
>
> Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00
> MB, 0.00 MB/s
>
> Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>
>
> ** Compaction Stats [default] **
>
> Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB)
> Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
> CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0
> 0.0 0.0 1.0 0.0 1.1 0.00 0.00
> 1 0.001 0 0
>
> Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0
> 0.0 0.0 1.0 0.0 1.1 0.00 0.00
> 1 0.001 0 0
>
> Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
> 0.0 0.0 1.0 0.0 1.1 0.00 0.00
> 1 0.001 0 0
>
>
>
> ** Compaction Stats [default] **
>
> Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB)
> Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
> CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
> 0.0 0.0 0.0 0.0 1.1 0.00 0.00
> 1 0.001 0 0
>
> Uptime(secs): 0.0 total, 0.0 interval
>
> Flush(GB): cumulative 0.000, interval 0.000
>
> AddFile(GB): cumulative 0.000, interval 0.000
>
> AddFile(Total Files): cumulative 0, interval 0
>
> AddFile(L0 Files): cumulative 0, interval 0
>
> AddFile(Keys): cumulative 0, interval 0
>
> Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read,
> 0.00 MB/s read, 0.0 seconds
>
> Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read,
> 0.00 MB/s read, 0.0 seconds
>
> Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction,
> 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for
> pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0
> memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>
>
> ** File Read Latency Histogram By Level [default] **
>
>
>
> ** Compaction Stats [default] **
>
> Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB)
> Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
> CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0
> 0.0 0.0 1.0 0.0 1.1 0.00 0.00
> 1 0.001 0 0
>
> Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0
> 0.0 0.0 1.0 0.0 1.1 0.00 0.00
> 1 0.001 0 0
>
> Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
> 0.0 0.0 0.0 0.0 0.0 0.00 0.00
> 0 0.000 0 0
>
>
>
> ** Compaction Stats [default] **
>
> Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB)
> Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
> CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
> 0.0 0.0 0.0 0.0 1.1 0.00 0.00
> 1 0.001 0 0
>
> Uptime(secs): 0.0 total, 0.0 interval
>
> Flush(GB): cumulative 0.000, interval 0.000
>
> AddFile(GB): cumulative 0.000, interval 0.000
>
> AddFile(Total Files): cumulative 0, interval 0
>
> AddFile(L0 Files): cumulative 0, interval 0
>
> AddFile(Keys): cumulative 0, interval 0
>
> Cumulative compaction: 0.00 GB write, 0.10 MB/s write, 0.00 GB read,
> 0.00 MB/s read, 0.0 seconds
>
> Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read,
> 0.00 MB/s read, 0.0 seconds
>
> Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction,
> 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for
> pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0
> memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>
>
> ** File Read Latency Histogram By Level [default] **
>
>
>
>
>
> 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb:
> [db/compaction/compaction_job.cc:1516] [default] [JOB 3] Generated
> table #91: 1366 keys, 7566988 bytes
>
>
>
> 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784213586, "cf_name": "default", "job": 3,
> "event": "table_file_creation", "file_number": 91, "file_size":
> 7566988, "file_checksum": "", "file_checksum_func_name": "Unknown",
> "table_properties": {"data_size": 7541895, "index_size": 20610,
> "index_partitions": 0, "top_level_index_size": 0,
> "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1,
> "filter_size": 3525, "raw_key_size": 29308, "raw_average_key_size":
> 21, "raw_value_size": 7503048, "raw_average_value_size": 5492,
> "num_data_blocks": 764, "num_entries": 1366, "num_deletions": 0,
> "num_merge_operands": 0, "num_range_deletions": 0, "format_version":
> 0, "fixed_key_len": 0, "filter_policy":
> "rocksdb.BuiltinBloomFilter", "column_family_name": "default",
> "column_family_id": 0, "comparator": "leveldb.BytewiseComparator",
> "merge_operator": "", "prefix_extractor_name": "nullptr",
> "property_collectors": "[]", "compression": "NoCompression", "c
> ompression_options": "window_bits=-14; level=32767; strategy=0;
> max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ",
> "creation_time": 1700216681, "oldest_key_time": 0,
> "file_creation_time": 1700216784, "db_id":
> "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id":
> "OS2T69IQ02SU5OKHBI40"}}
>
>
>
> 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb:
> [db/compaction/compaction_job.cc:1594] [default] [JOB 3] Compacted
> 14@0 files to L6 => 7566988 bytes
>
>
>
> 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb:
> [db/version_set.cc:3457] More existing levels in DB than needed.
> max_bytes_for_level_multiplier may not be guaranteed.
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log
> Time 2023/11/17-12:26:24.215298)
> [db/compaction/compaction_job.cc:812] [default] compacted to: base
> level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0
> 0 0 0 1] max score 0.00, MB/sec: 177.1 rd, 170.3 wr, level 6, files
> in(14, 0) out(1) MB in(7.5, 0.0) out(7.2), read-write-amplify(2.0)
> write-amplify(1.0) OK, records in: 19842, records dropped: 18476
> output_compression: NoCompression
>
>
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log
> Time 2023/11/17-12:26:24.215314) EVENT_LOG_v1 {"time_micros":
> 1700216784215306, "job": 3, "event": "compaction_finished",
> "compaction_time_micros": 44437, "compaction_time_cpu_micros":
> 40923, "output_level": 6, "num_output_files": 1,
> "total_output_size": 7566988, "num_input_records": 19842,
> "num_output_records": 1366, "num_subcompactions": 1,
> "output_compression": "NoCompression",
> "num_single_delete_mismatches": 0, "num_single_delete_fallthrough":
> 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000086.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215520, "job": 3, "event":
> "table_file_deletion", "file_number": 86}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000080.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215570, "job": 3, "event":
> "table_file_deletion", "file_number": 80}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000074.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215603, "job": 3, "event":
> "table_file_deletion", "file_number": 74}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000068.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215641, "job": 3, "event":
> "table_file_deletion", "file_number": 68}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000062.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215672, "job": 3, "event":
> "table_file_deletion", "file_number": 62}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000056.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215708, "job": 3, "event":
> "table_file_deletion", "file_number": 56}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000050.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215739, "job": 3, "event":
> "table_file_deletion", "file_number": 50}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000044.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215772, "job": 3, "event":
> "table_file_deletion", "file_number": 44}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000038.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215804, "job": 3, "event":
> "table_file_deletion", "file_number": 38}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000032.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215831, "job": 3, "event":
> "table_file_deletion", "file_number": 32}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000026.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215858, "job": 3, "event":
> "table_file_deletion", "file_number": 26}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000020.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215888, "job": 3, "event":
> "table_file_deletion", "file_number": 20}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000014.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784215952, "job": 3, "event":
> "table_file_deletion", "file_number": 14}
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
> [file/delete_scheduler.cc:69] Deleted file
> /root/mon-store/store.db/000008.sst immediately, rate_bytes_per_sec
> 0, total_trash_size 0 max_trash_db_ratio 0.250000
>
>
>
> 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1700216784216804, "job": 3, "event":
> "table_file_deletion", "file_number": 8}
>
>
>
> update_mkfs generating seed initial monmap
>
> epoch 0
>
> fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37
>
> last_changed 2023-11-17T12:26:24.222814+0200
>
> created 2023-11-17T12:26:24.222814+0200
>
> min_mon_release 0 (unknown)
>
> election_strategy: 1
>
> 0: [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon.node01
>
> 1: [v2:10.40.99.12:3300/0,v1:10.40.99.12:6789/0] mon.node02
>
> 2: [v2:10.40.99.13:3300/0,v1:10.40.99.13:6789/0] mon.node03
>
> 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb:
> [db/db_impl/db_impl.cc:446] Shutdown: canceling all background work
>
>
>
> 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb:
> [db/db_impl/db_impl.cc:625] Shutdown complete
>
>
>
>
> Then we copied the /root/mon-store/store.db folder across on all 3
> nodes and tried to start ceph.target service again.
>
> The output on node01 is below:
>
> d31781fa6b4c quay.io/ceph/ceph
> "/usr/bin/ceph-mds -..." 55 minutes ago Up 55 minutes
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node01-cjrvjc
> e385c32651d2 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-10
> 904f522c4cb5 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-0
> 033edf99a98e quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-4
> 70344a6e87a0 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-8
> 905b782aedcf quay.io/prometheus/prometheus:v2.43.0
> "/bin/prometheus --c..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-prometheus-node01
> ff191654eb3e quay.io/prometheus/node-exporter:v1.5.0
> "/bin/node_exporter ..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node01
> 459c46f4bdb7 quay.io/ceph/ceph
> "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node01-xlciyx
> cacfe8abcbbf quay.io/ceph/ceph
> "/usr/bin/ceph-crash..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node01
> e216ef2af166 quay.io/prometheus/alertmanager:v0.25.0
> "/bin/alertmanager -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-alertmanager-node01
> d3238b2285d1 quay.io/ceph/ceph-grafana:9.4.7 "/bin/sh -c
> 'grafana..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-grafana-node01
>
> The output on node02 is below:
>
> 2aec62685dee quay.io/ceph/ceph
> "/usr/bin/ceph-mds -..." 54 minutes ago Up 54 minutes
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node02-lyudbp
> 249b04f32f8c quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-5
> a2c96f56b517 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-2
> 87496d374a29 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-12
> 55fe47765917 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-9
> 76171e25dbde quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-7
> 220472e8c1bf quay.io/ceph/ceph
> "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node02-gudauu
> 0c783e73e543 quay.io/prometheus/node-exporter:v1.5.0
> "/bin/node_exporter ..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node02
> 4e638003fa2e quay.io/ceph/ceph
> "/usr/bin/ceph-crash..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node02
> 42719d5cfdbf quay.io/ceph/ceph
> "/usr/bin/ceph-mon -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node02
>
> The output on node03 is below:
>
> 7e5879dce643 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-11
> d53996ff33b9 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-3
> e1ac5a8b87d3 quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-1
> f4cda871218d quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-13
> 969e670dc47c quay.io/ceph/ceph
> "/usr/bin/ceph-osd -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-6
> a49e91a7bb8e quay.io/prometheus/node-exporter:v1.5.0
> "/bin/node_exporter ..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node03
> 835c3893a3f4 quay.io/ceph/ceph
> "/usr/bin/ceph-crash..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node03
> bfa6f5b989ea quay.io/ceph/ceph
> "/usr/bin/ceph-mon -..." 2 days ago Up 2 days
> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node03
>
>
> # ceph -s (output below):
>
> cluster:
> id: be4304e4-b0d5-11ec-8c6a-2965d4229f37
> health: HEALTH_ERR
> 20 stray daemon(s) not managed by cephadm
> 3 stray host(s) with 20 daemon(s) not managed by cephadm
> 1/3 mons down, quorum node02,node03
> 1/523510 objects unfound (0.000%)
> 3 nearfull osd(s)
> 1 osds exist in the crush map but not in the osdmap
> Low space hindering backfill (add storage if this
> doesn't resolve itself): 20 pgs backfill_toofull
> Possible data damage: 1 pg recovery_unfound
> Degraded data redundancy: 74666/1570530 objects degraded
> (4.754%), 21 pgs degraded, 21 pgs undersized
> 3 pool(s) nearfull
>
> services:
> mon: 3 daemons, quorum node02,node03 (age 2d), out of quorum: node01
> mgr: node01.xlciyx(active, since 2d), standbys: node02.gudauu
> osd: 14 osds: 14 up (since 2d), 14 in (since 3d); 21 remapped pgs
>
> data:
> pools: 3 pools, 161 pgs
> objects: 523.51k objects, 299 GiB
> usage: 1014 GiB used, 836 GiB / 1.8 TiB avail
> pgs: 74666/1570530 objects degraded (4.754%)
> 1/523510 objects unfound (0.000%)
> 140 active+clean
> 20 active+undersized+degraded+remapped+backfill_toofull
>
> 1. active+recovery_unfound+undersized+degraded+remapped
>
> # ceph fs ls (output below):
> No filesystems enabled
>
> It looks like that we have a problem with the orchestrator now
> (we've lost cephadm orchestrator) and we also cannot see the
> filesystem.
>
>
> May you please assist since we are not able to mount the filesystem ?
>
>
> Thank you,
>
> Manolis Daramas
>
>
> Under the General Data Protection Regulation (GDPR) (EU) 2016/679,
> Motivian as Data Controller has a legal duty to protect any
> information collected from you via email. Information contained in
> this email and any attachments may be privileged or confidential and
> intended for the exclusive use of the original recipient. If you
> have received this email by mistake, please advise the sender
> immediately and delete the email, including emptying your deleted
> email box. Information included in this email is reserved to named
> addressee's eyes only. You may not share this message or any of its
> attachments to anyone. Please note that as the recipient, it is your
> responsibility to check the email for malicious software. Motivian
> puts the security of the client at a high priority. Therefore, we
> have put efforts into ensuring that the message is error and
> virus-free. Unfortunately, full security of the email cannot be
> ensured as, despite our efforts, the data included in emails could
> be infected,
> intercepted, or corrupted. Therefore, the recipient should check
> the email for threats with proper software, as the sender does not
> accept liability for any damage inflicted by viewing the content of
> this email.
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
Under the General Data Protection Regulation (GDPR) (EU) 2016/679, Motivian as Data Controller has a legal duty to protect any information collected from you via email. Information contained in this email and any attachments may be privileged or confidential and intended for the exclusive use of the original recipient. If you have received this email by mistake, please advise the sender immediately and delete the email, including emptying your deleted email box. Information included in this email is reserved to named addressee’s eyes only. You may not share this message or any of its attachments to anyone. Please note that as the recipient, it is your responsibility to check the email for malicious software. Motivian puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.
Under the General Data Protection Regulation (GDPR) (EU) 2016/679, Motivian as Data Controller has a legal duty to protect any information collected from you via email. Information contained in this email and any attachments may be privileged or confidential and intended for the exclusive use of the original recipient. If you have received this email by mistake, please advise the sender immediately and delete the email, including emptying your deleted email box. Information included in this email is reserved to named addressee’s eyes only. You may not share this message or any of its attachments to anyone. Please note that as the recipient, it is your responsibility to check the email for malicious software. Motivian puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
