Re: Is it normal Ceph reports "Degraded data redundancy" in normal use?

Kai Stian Olstad <ceph+list@xxxxxxxxxx> · Fri, 17 Sep 2021 10:32:25 +0200

On 16.09.2021 15:51, Josh Baergen wrote:
I assume it's the balancer module. If you write lots of data quickly
into the cluster the distribution can vary and the balancer will try
to even out the placement.

The balancer won't cause degradation, only misplaced objects.

Since I'm trying to test different erasure encoding plugin and technique 
I don't want the balancer active.
So I tried setting it to none as Eguene suggested, and to my surprise I 
did not get any degraded messages at all, and the cluster was in 
HEALTH_OK the whole time.

    Degraded data redundancy: 260/11856050 objects degraded
(0.014%), 1 pg degraded

That status definitely indicates that something is wrong. Check your
cluster logs on your mons (/var/log/ceph/ceph.log) for the cause; my
guess is that you have OSDs flapping (rapidly going down and up again)
due to either overload (disk or network) or some sort of
misconfiguration.

So I enabled the balancer and run the rados bench again and the degraded 
messages is back.

I guess the equivalent log to /var/log/ceph/ceph.log in Cephadm is
  journalctl -u 
ceph-b321e76e-da3a-11eb-b75c-4f948441dcd@xxxxxxxx-mon-1.service

There are no messages about osd being marked down, so I don't understand 
why this is happening.
I probably need to raise some verbose value.

I have attach the log from journalctl, it start at 06:30:00 when I 
started the rados bench and included a few lines after the first degrade 
message at 06:31.06.
Just be aware that 15 OSD is set to out, since I have some problem with 
the a HBA on one host, all test has been done with those 15 OSD in 
status out.

--
Kai Stian Olstad
Sep 17 06:30:00 pech-mon-1 conmon[1337]: debug 2021-09-17T06:29:59.994+0000 7f66b232d700  0 log_channel(cluster) log [INF] : overall HEALTH_OK
Sep 17 06:30:00 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:29:59.317530+0000 mgr.pech-mon-1.ptrsea
Sep 17 06:30:00 pech-mon-1 conmon[1337]:  (mgr.245802) 345745 : cluster [DBG] pgmap v347889: 1025 pgs: 1025 active+clean; 0 B data, 73 TiB used, 2.8 PiB / 2.9 PiB avail
Sep 17 06:30:00 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:00.000143+0000 mon.pech-mon-1 (mon.0) 1166236 : 
Sep 17 06:30:00 pech-mon-1 conmon[1337]: cluster [INF] overall HEALTH_OK
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.318+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.6d", "id": [293, 327]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.318+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.6d", "id": [293, 327]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.318+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.144", "id": [307, 351]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.318+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.144", "id": [307, 351]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.17d", "id": [144, 136]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.17d", "id": [144, 136]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1a2", "id": [199, 189]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1a2", "id": [199, 189]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1e1", "id": [289, 344]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1e1", "id": [289, 344]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.237", "id": [282, 292]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.237", "id": [282, 292]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.2b5", "id": [305, 327]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.2b5", "id": [305, 327]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.304", "id": [290, 327]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.304", "id": [290, 327]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.309", "id": [311, 344]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.309", "id": [311, 344]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.318", "id": [310, 299]} v 0) v1
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.322+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.318", "id": [310, 299]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.370+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13996 do_prune osdmap full prune enabled
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.382+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e13997 e13997: 355 total, 355 up, 340 in
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.6d", "id": [293, 327]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.144", "id": [307, 351]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.17d", "id": [144, 136]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1a2", "id": [199, 189]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1e1", "id": [289, 344]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.237", "id": [282, 292]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.2b5", "id": [305, 327]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.304", "id": [290, 327]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.309", "id": [311, 344]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.318", "id": [310, 299]}]': finished
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.386+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e13997: 355 total, 355 up, 340 in
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 2021-09-17T06
Sep 17 06:30:01 pech-mon-1 conmon[1337]: :30:01.324212+0000 mon.pech-mon-1 (mon.0) 1166237 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.6d", "id": [293, 327]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 2021-09-17T06:
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 30:01.324703+0000 mon.pech-mon-1 (mon.0) 1166238 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.144", "id": [307, 351]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 2021-
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 09-17T06:30:01.325129+0000 mon.pech-mon-1 (mon.0) 1166239 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.17d", "id": [144, 136]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:01.325506+0000 mon.pech-mon-1
Sep 17 06:30:01 pech-mon-1 conmon[1337]:  (mon.0) 1166240 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1a2", "id": [199, 189]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 2021-09-17T06:30:01.325869+0000 mon.pech-mon-1 (mon.0) 1166241 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1e1", "id": [289, 344]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:01.326215+0000 mon.pech-mon-1 (mon.
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 0) 1166242 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.237", "id": [282, 292]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 2021-
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 09-17T06:30:01.326598+0000 mon.pech-mon-1 (mon.0) 1166243 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.2b5", "id": [305, 327]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:01.326932+0000 
Sep 17 06:30:01 pech-mon-1 conmon[1337]: mon.pech-mon-1 (mon.0) 1166244 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.304", "id": [290, 327]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit
Sep 17 06:30:01 pech-mon-1 conmon[1337]:  2021-09-17T06:30:01.327291+0000 mon.pech-mon-1 (mon.0) 1166245 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.309", "id": [311, 344]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 
Sep 17 06:30:01 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:01
Sep 17 06:30:01 pech-mon-1 conmon[1337]: .327659+0000 mon.pech-mon-1 (mon.0) 1166246 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.318", "id": [310, 299]}]: dispatch
Sep 17 06:30:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:01.402+0000 7f66b0329700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b0329700' had timed out after 0.000000000s
Sep 17 06:30:01 pech-mon-1 conmon[1337]: 
Sep 17 06:30:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:02.390+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13997 do_prune osdmap full prune enabled
Sep 17 06:30:02 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 30:01.330036+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345746 : cluster [DBG] pgmap v347890: 1025 pgs: 1025 active+clean; 0 B data, 73 TiB used, 2.8 PiB / 2.9 PiB avail
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09-
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 17T06:30:01.389867+0000 mon.pech-mon-1 (mon.0) 1166247 : audit [INF]
Sep 17 06:30:02 pech-mon-1 conmon[1337]:  from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.6d", "id": [293, 327]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 09-17T06:30:01.389905+0000 mon.pech-mon-1 (mon.0) 1166248 : audit 
Sep 17 06:30:02 pech-mon-1 conmon[1337]: [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.144", "id": [307, 351]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09
Sep 17 06:30:02 pech-mon-1 conmon[1337]: -17T06:30:01.389951+0000 mon.pech-mon-1 (mon.0) 1166249 : audit [INF]
Sep 17 06:30:02 pech-mon-1 conmon[1337]:  from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.17d", "id": [144, 136]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09-
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 17T06:30:01.389992+0000 mon.pech-mon-1 (mon.0) 1166250 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1a2", "id": [199, 189]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:01.390014
Sep 17 06:30:02 pech-mon-1 conmon[1337]: +0000 mon.pech-mon-1 (mon.0) 1166251 : audit
Sep 17 06:30:02 pech-mon-1 conmon[1337]:  [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1e1", "id": [289, 344]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09-17T06:
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 30:01.390071+0000 mon.pech-mon-1 (mon.
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 0) 1166252 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.237", "id": [282, 292]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:01.
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 390106+0000 mon.pech-mon-1 (mon.0) 1166253 : audit [INF] 
Sep 17 06:30:02 pech-mon-1 conmon[1337]: from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.2b5", "id": [305, 327]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09-17T
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 06:30:01.390138+0000 mon.pech-mon-1 (mon.0) 1166254 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.304", "id": [290, 327]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09-17T06:
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 30:01.390213+0000 mon.pech-mon-1 (mon.0) 1166255 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.309", "id": [311, 344]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:
Sep 17 06:30:02 pech-mon-1 conmon[1337]: 01.390234+0000 mon.pech-mon-1 (mon.0) 1166256 : audit [INF]
Sep 17 06:30:02 pech-mon-1 conmon[1337]:  from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.318", "id": [310, 299]}]': finished
Sep 17 06:30:02 pech-mon-1 conmon[1337]: cluster 2021-09-17
Sep 17 06:30:02 pech-mon-1 conmon[1337]: T06:30:01.390252+0000 mon.pech-mon-1 (mon.0) 1166257 : cluster [DBG] 
Sep 17 06:30:02 pech-mon-1 conmon[1337]: osdmap e13997: 355 total, 355 up, 340 in
Sep 17 06:30:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:02.406+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e13998 e13998: 355 total, 355 up, 340 in
Sep 17 06:30:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:02.410+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e13998: 355 total, 355 up, 340 in
Sep 17 06:30:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:02.426+0000 7f66b1b2c700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b1b2c700' had timed out after 0.000000000s
Sep 17 06:30:03 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:03.406+0000 7f66b232d700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)
Sep 17 06:30:03 pech-mon-1 conmon[1337]: cluster 2021-09
Sep 17 06:30:03 pech-mon-1 conmon[1337]: -17T06:30:02.413839+0000 mon.pech-mon-1 (mon.0) 1166258 : cluster [DBG] osdmap e13998: 355 total, 355 up, 340 in
Sep 17 06:30:04 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:04.142+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:04 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:
Sep 17 06:30:04 pech-mon-1 conmon[1337]: 30:03.334124+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345747 : cluster [DBG] pgmap v347893: 1025 pgs: 5 peering, 1020 active+clean; 0 B data, 73 TiB used, 2.8 PiB / 2.9 PiB avail
Sep 17 06:30:04 pech-mon-1 conmon[1337]: cluster 2021
Sep 17 06:30:04 pech-mon-1 conmon[1337]: -09-17T06:30:03.411664+0000 mon.pech-mon-1 (mon.0) 
Sep 17 06:30:04 pech-mon-1 conmon[1337]: 1166259 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)
Sep 17 06:30:05 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:05.342736+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 
Sep 17 06:30:05 pech-mon-1 conmon[1337]: 345748 : cluster [DBG] pgmap v347894: 1025 pgs: 5 peering, 1020 active+clean; 3.2 GiB data, 73 TiB used, 2.8 PiB / 2.9 PiB avail; 410 MiB/s wr, 102 op/s
Sep 17 06:30:07 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:07.403+0000 7f66b232d700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering)
Sep 17 06:30:07 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:07.403+0000 7f66b232d700  0 log_channel(cluster) log [INF] : Cluster is now healthy
Sep 17 06:30:07 pech-mon-1 conmon[1337]: cluster 2021-09-17T
Sep 17 06:30:07 pech-mon-1 conmon[1337]: 06:30:07.354342+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345749 : cluster [DBG] pgmap v347895: 1025 pgs: 1025 active+clean; 16 GiB data, 73 TiB used, 2.8 PiB / 2.9 PiB avail; 2.0 GiB/s wr, 498 op/s
Sep 17 06:30:07 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:07.405439+0000 
Sep 17 06:30:07 pech-mon-1 conmon[1337]: mon.pech-mon-1 (mon.0) 1166260 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering)
Sep 17 06:30:07 pech-mon-1 conmon[1337]: cluster 2021-
Sep 17 06:30:07 pech-mon-1 conmon[1337]: 09-17T06:30:07.405475+0000 mon.pech-mon-1 (mon.
Sep 17 06:30:07 pech-mon-1 conmon[1337]: 0) 1166261 : cluster [INF] Cluster is now healthy
Sep 17 06:30:09 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:09.143+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:10 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:09.
Sep 17 06:30:10 pech-mon-1 conmon[1337]: 358760+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345750 : cluster [DBG] pgmap v347896: 1025 pgs: 1025 active+clean; 16 GiB data, 73 TiB used, 2.8 PiB / 2.9 PiB avail; 1.9 GiB/s wr, 498 op/s
Sep 17 06:30:11 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:11
Sep 17 06:30:11 pech-mon-1 conmon[1337]: .370694+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345751 : cluster [DBG] pgmap v347897: 1025 pgs: 1025 active+clean; 47 GiB data, 73 TiB used, 2.8 PiB / 2.9 PiB avail; 4.7 GiB/s wr, 1.20k op/s
Sep 17 06:30:12 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:12.887+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/trash_purge_schedule"} v 0) v1
Sep 17 06:30:12 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:12.887+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/trash_purge_schedule"}]: dispatch
Sep 17 06:30:12 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:12.892579+0000 mon.pech-mon-1 (mon
Sep 17 06:30:12 pech-mon-1 conmon[1337]: .0) 1166262 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/trash_purge_schedule"}]: dispatch
Sep 17 06:30:13 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:13.377736+0000 
Sep 17 06:30:13 pech-mon-1 conmon[1337]: mgr.pech-mon-1.ptrsea (mgr.245802) 345752 : cluster [DBG] pgmap v347898: 1025 pgs: 1025 active+clean; 63 GiB data, 73 TiB used, 2.8 PiB / 2.9 PiB avail; 5.7 GiB/s wr, 1.46k op/s
Sep 17 06:30:14 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:14.007+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:14 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:14.007+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/3403936623' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:14 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:14.147+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:14 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:14.010958+0000 mon.pech-mon-1 (mon.0) 1166263 : audit [DBG] from='client.? 10.0.1.10:0/3403936623' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:15 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:15.386812+0000 mgr.pech-mon-1.ptrsea (
Sep 17 06:30:15 pech-mon-1 conmon[1337]: mgr.245802) 345753 : cluster [DBG] pgmap v347899: 1025 pgs: 1025 active+clean; 81 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 6.8 GiB/s wr, 1.73k op/s
Sep 17 06:30:17 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:17.396862+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345754
Sep 17 06:30:17 pech-mon-1 conmon[1337]:  : cluster [DBG] pgmap v347900: 1025 pgs: 1025 active+clean; 109 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 8.8 GiB/s wr, 2.25k op/s
Sep 17 06:30:17 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:17.499+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:17 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:17.499+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/1300851108' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:18 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:17.503140+0000 mon.pech-mon-1 (mon.0) 1166264
Sep 17 06:30:18 pech-mon-1 conmon[1337]:  : audit [DBG] from='client.? 10.0.1.10:0/1300851108' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:19 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:19.147+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:20 pech-mon-1 conmon[1337]: cluster 2021-09-17T06
Sep 17 06:30:20 pech-mon-1 conmon[1337]: :30:19.401219+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345755 : cluster [DBG] pgmap v347901: 1025 pgs: 1025 active+clean; 109 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 7.8 GiB/s wr, 1.99k op/s
Sep 17 06:30:20 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:20.983+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:20 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:20.983+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/2409590211' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:21 pech-mon-1 conmon[1337]: audit 2021-09-17T
Sep 17 06:30:21 pech-mon-1 conmon[1337]: 06:30:20.987618+0000 mon.pech-mon-1 (mon.0) 1166265 : audit [DBG] from='client.? 10.0.1.10:0/2409590211' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:22 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:21.413470
Sep 17 06:30:22 pech-mon-1 conmon[1337]: +0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345756 : cluster [DBG] pgmap v347902: 1025 pgs: 1025 active+clean; 140 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 10 GiB/s wr, 2.64k op/s
Sep 17 06:30:24 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:24.151+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:24 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:24.679+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:24 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:24.679+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/1961226959' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:25 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:23.421653
Sep 17 06:30:25 pech-mon-1 conmon[1337]: +0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345757 : cluster [DBG] pgmap v347903: 1025 pgs: 1025 active+clean; 156 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.0 GiB/s wr, 2.31k op/s
Sep 17 06:30:25 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:24.683987+0000 mon.pech-mon-1 (mon.0) 1166266 : audit [DBG] from='client.? 10.0.1.10:0/1961226959' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:25 pech-mon-1 conmon[1337]: 
Sep 17 06:30:27 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:
Sep 17 06:30:27 pech-mon-1 conmon[1337]: 30:25.431228+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345758 : cluster [DBG] pgmap v347904: 1025 pgs: 1025 active+clean; 176 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.4 GiB/s wr, 2.40k op/s
Sep 17 06:30:28 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:28.279+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:28 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:28.279+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/751451053' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:29 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:29.151+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:29 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30
Sep 17 06:30:29 pech-mon-1 conmon[1337]: :27.441983+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345759 : cluster [DBG] pgmap v347905: 1025 pgs: 1025 active+clean; 202 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 10 GiB/s wr, 2.56k op/s
Sep 17 06:30:29 pech-mon-1 conmon[1337]: audit 2021-09-17T
Sep 17 06:30:29 pech-mon-1 conmon[1337]: 06:30:28.282237+0000 mon.pech-mon-1 (mon.0) 1166267 : audit [DBG] from='client.? 10.0.1.10:0/751451053' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:31 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:
Sep 17 06:30:31 pech-mon-1 conmon[1337]: 30:29.446201+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345760 : cluster [DBG] pgmap v347906: 1025 pgs: 1025 active+clean; 202 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 7.7 GiB/s wr, 1.97k op/s
Sep 17 06:30:31 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:31.884+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:31 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:31.884+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/307298187' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:32 pech-mon-1 conmon[1337]: audit 2021-09-17
Sep 17 06:30:32 pech-mon-1 conmon[1337]: T06:30:31.888795+0000 mon.pech-mon-1 (mon.0) 1166268 : audit [DBG] from='client.? 10.0.1.10:0/307298187' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:33 pech-mon-1 conmon[1337]: cluster 2021-09-17T06
Sep 17 06:30:33 pech-mon-1 conmon[1337]: :30:31.458380+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345761 : cluster [DBG] pgmap v347907: 1025 pgs: 1025 active+clean; 233 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 10 GiB/s wr, 2.62k op/s
Sep 17 06:30:34 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:34.152+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:35 pech-mon-1 conmon[1337]: cluster
Sep 17 06:30:35 pech-mon-1 conmon[1337]:  2021-09-17T06:30:33.466930+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345762 : cluster [DBG] pgmap v347908: 1025 pgs: 1025 active+clean; 249 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.0 GiB/s wr, 2.30k op/s
Sep 17 06:30:35 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:35.488+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:35 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:35.488+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/1844333395' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:36 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:35.491790+0000 mon.pech-mon-1 (
Sep 17 06:30:36 pech-mon-1 conmon[1337]: mon.0) 1166269 : audit [DBG] from='client.? 10.0.1.10:0/1844333395' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.208+0000 7f66ab31f700  4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #335784. Immutable memtables: 0.
Sep 17 06:30:36 pech-mon-1 conmon[1337]: 
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.208+0000 7f66b5333700  4 rocksdb: (Original Log Time 2021/09/17-06:30:36.211770) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.208+0000 7f66b5333700  4 rocksdb: [flush_job.cc:321] [default] [JOB 92091] Flushing memtable with next log file: 335784
Sep 17 06:30:36 pech-mon-1 conmon[1337]: 
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.208+0000 7f66b5333700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860236211798, "job": 92091, "event": "flush_started", "num_memtables": 1, "num_entries": 2255, "num_deletes": 251, "total_data_size": 17291399, "memory_usage": 17465992, "flush_reason": "Manual Compaction"}
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.208+0000 7f66b5333700  4 rocksdb: [flush_job.cc:350] [default] [JOB 92091] Level-0 flush table #335785: started
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.304+0000 7f66b5333700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860236306647, "cf_name": "default", "job": 92091, "event": "table_file_creation", "file_number": 335785, "file_size": 17252489, "table_properties": {"data_size": 17233677, "index_size": 15719, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 2245, "raw_key_size": 18782, "raw_average_key_size": 21, "raw_value_size": 17210305, "raw_average_value_size": 20035, "num_data_blocks": 438, "num_entries": 859, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1631860053, "oldest_key_time": 1631860053, "file_creation_time": 1631860236}}
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.304+0000 7f66b5333700  4 rocksdb: [flush_job.cc:401] [default] [JOB 92091] Level-0 flush table #335785: 17252489 bytes OK
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.304+0000 7f66b5333700  4 rocksdb: (Original Log Time 2021/09/17-06:30:36.306954) [memtable_list.cc:447] [default] Level-0 commit table #335785 started
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.304+0000 7f66b5333700  4 rocksdb: (Original Log Time 2021/09/17-06:30:36.307309) [memtable_list.cc:503] [default] Level-0 commit table #335785: memtable #1 done
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.304+0000 7f66b5333700  4 rocksdb: (Original Log Time 2021/09/17-06:30:36.307331) EVENT_LOG_v1 {"time_micros": 1631860236307322, "job": 92091, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 3], "immutable_memtables": 0}
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.304+0000 7f66b5333700  4 rocksdb: (Original Log Time 2021/09/17-06:30:36.307354) [db_impl/db_impl_compaction_flush.cc:205] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 3] max score 0.25
Sep 17 06:30:36 pech-mon-1 conmon[1337]: 
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.304+0000 7f66b5333700  4 rocksdb: [db_impl/db_impl_files.cc:353] [JOB 92091] Try to delete WAL files size 17281903, prev total WAL file size 17281903, number of live WAL files 2.
Sep 17 06:30:36 pech-mon-1 conmon[1337]: 
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.308+0000 7f66ab31f700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.308+0000 7f66b5b34700  4 rocksdb: (Original Log Time 2021/09/17-06:30:36.311638) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-6 from 'paxos .. 'paxos; will stop at (end)
Sep 17 06:30:36 pech-mon-1 conmon[1337]: 
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.308+0000 7f66b5b34700  4 rocksdb: [compaction/compaction_job.cc:1676] [default] [JOB 92092] Compacting 1@0 + 3@6 files to L6, score -1.00
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.308+0000 7f66b5b34700  4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 92091 Base level 0, inputs: [335785(16MB)], [335781(64MB) 335782(64MB) 335783(26MB)]
Sep 17 06:30:36 pech-mon-1 conmon[1337]: 
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.308+0000 7f66b5b34700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860236311671, "job": 92092, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [335785], "files_L6": [335781, 335782, 335783], "score": -1, "input_data_size": 179801052}
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.716+0000 7f66b5b34700  4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 92092] Generated table #335786: 4764 keys, 67409667 bytes
Sep 17 06:30:36 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:36.716+0000 7f66b5b34700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860236717755, "cf_name": "default", "job": 92092, "event": "table_file_creation", "file_number": 335786, "file_size": 67409667, "table_properties": {"data_size": 67338223, "index_size": 58627, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 11973, "raw_key_size": 162225, "raw_average_key_size": 34, "raw_value_size": 67206451, "raw_average_value_size": 14107, "num_data_blocks": 1560, "num_entries": 4764, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1625123153, "oldest_key_time": 0, "file_creation_time": 1631860236}}
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.116+0000 7f66b5b34700  4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 92092] Generated table #335787: 401 keys, 67319589 bytes
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.116+0000 7f66b5b34700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860237117912, "cf_name": "default", "job": 92092, "event": "table_file_creation", "file_number": 335787, "file_size": 67319589, "table_properties": {"data_size": 67302065, "index_size": 15589, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 1093, "raw_key_size": 10025, "raw_average_key_size": 25, "raw_value_size": 67284822, "raw_average_value_size": 167792, "num_data_blocks": 401, "num_entries": 401, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1625123153, "oldest_key_time": 0, "file_creation_time": 1631860236}}
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.328+0000 7f66b5b34700  4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 92092] Generated table #335788: 621 keys, 36859801 bytes
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.328+0000 7f66b5b34700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860237332353, "cf_name": "default", "job": 92092, "event": "table_file_creation", "file_number": 335788, "file_size": 36859801, "table_properties": {"data_size": 36838436, "index_size": 18917, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 1605, "raw_key_size": 13522, "raw_average_key_size": 21, "raw_value_size": 36816126, "raw_average_value_size": 59285, "num_data_blocks": 535, "num_entries": 621, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1625123153, "oldest_key_time": 0, "file_creation_time": 1631860237}}
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.332+0000 7f66b5b34700  4 rocksdb: [compaction/compaction_job.cc:1415] [default] [JOB 92092] Compacted 1@0 + 3@6 files to L6 => 171589057 bytes
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.332+0000 7f66b5b34700  4 rocksdb: (Original Log Time 2021/09/17-06:30:37.333711) [compaction/compaction_job.cc:760] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 3] max score 0.00, MB/sec: 176.1 rd, 168.1 wr, level 6, files in(1, 3) out(3) MB in(16.5, 155.0) out(163.6), read-write-amplify(20.4) write-amplify(9.9) OK, records in: 6312, records dropped: 526 output_compression: NoCompression
Sep 17 06:30:37 pech-mon-1 conmon[1337]: 
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.332+0000 7f66b5b34700  4 rocksdb: (Original Log Time 2021/09/17-06:30:37.333754) EVENT_LOG_v1 {"time_micros": 1631860237333736, "job": 92092, "event": "compaction_finished", "compaction_time_micros": 1020775, "compaction_time_cpu_micros": 158158, "output_level": 6, "num_output_files": 3, "total_output_size": 171589057, "num_input_records": 6312, "num_output_records": 5786, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 3]}
Sep 17 06:30:37 pech-mon-1 conmon[1337]: cluster 2021
Sep 17 06:30:37 pech-mon-1 conmon[1337]: -09-17T06:30:35.477012+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345763 : cluster [DBG] 
Sep 17 06:30:37 pech-mon-1 conmon[1337]: pgmap v347909: 1025 pgs: 1025 active+clean; 271 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.6 GiB/s wr, 2.44k op/s
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.336+0000 7f66b5b34700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860237337916, "job": 92092, "event": "table_file_deletion", "file_number": 335785}
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.340+0000 7f66b5b34700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860237344269, "job": 92092, "event": "table_file_deletion", "file_number": 335783}
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.348+0000 7f66b5b34700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860237350867, "job": 92092, "event": "table_file_deletion", "file_number": 335782}
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.352+0000 7f66b5b34700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1631860237356916, "job": 92092, "event": "table_file_deletion", "file_number": 335781}
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.356+0000 7f66ab31f700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.356+0000 7f66ab31f700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.356+0000 7f66ab31f700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.356+0000 7f66ab31f700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
Sep 17 06:30:37 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:37.356+0000 7f66ab31f700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
Sep 17 06:30:39 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:39.160+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:39 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:39.196+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:39 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:39.196+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/61226283' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:39 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:37.487936+0000 mgr.pech-mon-1.ptrsea
Sep 17 06:30:39 pech-mon-1 conmon[1337]:  (mgr.245802) 345764 : cluster [DBG] pgmap v347910: 1025 pgs: 1025 active+clean; 295 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.9 GiB/s wr, 2.53k op/s
Sep 17 06:30:39 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:39.198179+0000 mon.pech-mon-1 (
Sep 17 06:30:39 pech-mon-1 conmon[1337]: mon.0) 1166270 : audit [DBG] from='client.? 10.0.1.10:0/61226283' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:41 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:39.492392+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 
Sep 17 06:30:41 pech-mon-1 conmon[1337]: 345765 : cluster [DBG] pgmap v347911: 1025 pgs: 1025 active+clean; 295 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 7.7 GiB/s wr, 1.98k op/s
Sep 17 06:30:42 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:42.688+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:42 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:42.688+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/4069355450' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:43 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:43.036+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/mirror_snapshot_schedule"} v 0) v1
Sep 17 06:30:43 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:43.036+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/mirror_snapshot_schedule"}]: dispatch
Sep 17 06:30:43 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30
Sep 17 06:30:43 pech-mon-1 conmon[1337]: :41.502974+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345766 : cluster [DBG] pgmap v347912: 1025 pgs: 1025 active+clean; 327 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 10 GiB/s wr, 2.64k op/s
Sep 17 06:30:43 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:42.691911+0000 mon.pech-mon-1 (mon.0
Sep 17 06:30:43 pech-mon-1 conmon[1337]: ) 1166271 : audit [DBG] from='client.? 10.0.1.10:0/4069355450' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:43 pech-mon-1 conmon[1337]: audit 2021
Sep 17 06:30:43 pech-mon-1 conmon[1337]: -09-17T06:30:43.038380+0000 mon.pech-mon-1 (
Sep 17 06:30:43 pech-mon-1 conmon[1337]: mon.0) 1166272 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/mirror_snapshot_schedule"}]: dispatch
Sep 17 06:30:44 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:44.164+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:45 pech-mon-1 conmon[1337]: cluster 2021-
Sep 17 06:30:45 pech-mon-1 conmon[1337]: 09-17T06:30:43.511604+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345767 : cluster [DBG] pgmap v347913: 1025 pgs: 1025 active+clean; 343 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.1 GiB/s wr, 2.33k op/s
Sep 17 06:30:46 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:46.308+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:46 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:46.308+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/512103694' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:46 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:46.309932+0000 mon.pech-mon-1 (mon.0) 1166273 : audit [DBG]
Sep 17 06:30:46 pech-mon-1 conmon[1337]:  from='client.? 10.0.1.10:0/512103694' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:47 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:45.517634
Sep 17 06:30:47 pech-mon-1 conmon[1337]: +0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345768 : cluster [DBG] pgmap v347914: 1025 pgs: 1025 active+clean; 365 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.6 GiB/s wr, 2.45k op/s
Sep 17 06:30:48 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:48.488+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "config dump", "format": "json"} v 0) v1
Sep 17 06:30:48 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:48.488+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
Sep 17 06:30:48 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:48.488+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "config generate-minimal-conf"} v 0) v1
Sep 17 06:30:48 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:48.488+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.008+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "config rm", "who": "osd/host:pech-hd-006", "name": "osd_memory_target"} v 0) v1
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.008+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config rm", "who": "osd/host:pech-hd-006", "name": "osd_memory_target"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.028+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "config rm", "who": "osd/host:pech-hd-007", "name": "osd_memory_target"} v 0) v1
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.028+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config rm", "who": "osd/host:pech-hd-007", "name": "osd_memory_target"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.116+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "config rm", "who": "osd/host:pech-hd-004", "name": "osd_memory_target"} v 0) v1
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.116+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config rm", "who": "osd/host:pech-hd-004", "name": "osd_memory_target"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.164+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:49 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:47.527913+0000 mgr.pech-mon-1.ptrsea (
Sep 17 06:30:49 pech-mon-1 conmon[1337]: mgr.245802) 345769 : cluster [DBG] pgmap v347915: 1025 pgs: 1025 active+clean; 388 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.7 GiB/s wr, 2.49k op/s
Sep 17 06:30:49 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:48.489569+0000 mon.pech-mon-1 (mon.
Sep 17 06:30:49 pech-mon-1 conmon[1337]: 0) 1166274 : audit [DBG] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:48.491119+0000 mon.pech-mon-1 (mon.0
Sep 17 06:30:49 pech-mon-1 conmon[1337]: ) 1166275 : audit [DBG] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: audit 2021-09-
Sep 17 06:30:49 pech-mon-1 conmon[1337]: 17T06:30:49.009216+0000 mon.pech-mon-1 (mon.0) 1166276 : audit [INF]
Sep 17 06:30:49 pech-mon-1 conmon[1337]:  from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config rm", "who": "osd/host:pech-hd-006", "name": "osd_memory_target"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: audit 2021-
Sep 17 06:30:49 pech-mon-1 conmon[1337]: 09-17T06:30:49.029285+0000 mon.pech-mon-1 (mon.0) 1166277 : audit
Sep 17 06:30:49 pech-mon-1 conmon[1337]:  [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config rm", "who": "osd/host:pech-hd-007", "name": "osd_memory_target"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: audit 2021
Sep 17 06:30:49 pech-mon-1 conmon[1337]: -09-17T06:30:49.119027+0000 mon.pech-mon-1 (mon.0) 1166278 : 
Sep 17 06:30:49 pech-mon-1 conmon[1337]: audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "config rm", "who": "osd/host:pech-hd-004", "name": "osd_memory_target"}]: dispatch
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.796+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:49 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:49.796+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/905904831' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:50 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:50.172+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command([{prefix=config-key set, key=mgr/cephadm/osd_remove_queue}] v 0) v1
Sep 17 06:30:50 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:50.176+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' 
Sep 17 06:30:50 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:50.184+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd tree", "states": ["destroyed"], "format": "json"} v 0) v1
Sep 17 06:30:50 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:50.184+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch
Sep 17 06:30:50 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:50.212+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd tree", "states": ["destroyed"], "format": "json"} v 0) v1
Sep 17 06:30:50 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:50.212+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch
Sep 17 06:30:50 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30
Sep 17 06:30:50 pech-mon-1 conmon[1337]: :49.802268+0000 mon.pech-mon-1 (mon.0) 1166279 : audit [DBG] from='client.? 10.0.1.10:0/905904831' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:50 pech-mon-1 conmon[1337]: audit 2021-09
Sep 17 06:30:50 pech-mon-1 conmon[1337]: -17T06:30:50.182087+0000 mon.pech-mon-1 (mon.0) 1166280 : audit
Sep 17 06:30:50 pech-mon-1 conmon[1337]:  [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' 
Sep 17 06:30:50 pech-mon-1 conmon[1337]: audit 2021-09-17T06
Sep 17 06:30:50 pech-mon-1 conmon[1337]: :30:50.191420+0000 mon.pech-mon-1 (mon
Sep 17 06:30:50 pech-mon-1 conmon[1337]: .0) 1166281 : audit [DBG] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch
Sep 17 06:30:50 pech-mon-1 conmon[1337]: audit 2021-
Sep 17 06:30:50 pech-mon-1 conmon[1337]: 09-17T06:30:50.218554
Sep 17 06:30:50 pech-mon-1 conmon[1337]: +0000 mon.pech-mon-1 (mon.0) 1166282 : audit
Sep 17 06:30:50 pech-mon-1 conmon[1337]:  [DBG] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch
Sep 17 06:30:51 pech-mon-1 conmon[1337]: cluster 2021-09
Sep 17 06:30:51 pech-mon-1 conmon[1337]: -17T06:30:49.531412+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345770 : cluster [DBG] pgmap v347916: 1025 pgs: 1025 active+clean; 388 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 7.7 GiB/s wr, 1.97k op/s
Sep 17 06:30:53 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:53.265+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:53 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:53.265+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/1464031021' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:53 pech-mon-1 conmon[1337]: cluster 
Sep 17 06:30:53 pech-mon-1 conmon[1337]: 2021-09-17T06:30:51.543902+0000 mgr.pech-mon-1.ptrsea (
Sep 17 06:30:53 pech-mon-1 conmon[1337]: mgr.245802) 345771
Sep 17 06:30:53 pech-mon-1 conmon[1337]:  : cluster [DBG] pgmap v347917: 1025 pgs: 1025 active+clean; 421 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 10 GiB/s wr, 2.67k op/s
Sep 17 06:30:53 pech-mon-1 conmon[1337]: audit 
Sep 17 06:30:53 pech-mon-1 conmon[1337]: 2021-09-
Sep 17 06:30:53 pech-mon-1 conmon[1337]: 17T06:30:
Sep 17 06:30:53 pech-mon-1 conmon[1337]: 53.271453+0000 
Sep 17 06:30:53 pech-mon-1 conmon[1337]: mon.pech-mon-1 (mon.0
Sep 17 06:30:53 pech-mon-1 conmon[1337]: ) 1166283 : audit 
Sep 17 06:30:53 pech-mon-1 conmon[1337]: [DBG] from='client.? 10.0.1.10:0/1464031021' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:54 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:54.169+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:55 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:53.552235
Sep 17 06:30:55 pech-mon-1 conmon[1337]: +0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345772 : cluster [DBG] pgmap v347918: 1025 pgs: 1025 active+clean; 435 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.0 GiB/s wr, 2.30k op/s
Sep 17 06:30:56 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:56.881+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:30:56 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:56.881+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/298451163' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:57 pech-mon-1 conmon[1337]: cluster 2021-09-
Sep 17 06:30:57 pech-mon-1 conmon[1337]: 17T06:30:55.562124+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345773 : cluster [DBG] pgmap v347919: 1025 pgs: 1025 active+clean; 458 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.6 GiB/s wr, 2.44k op/s
Sep 17 06:30:57 pech-mon-1 conmon[1337]: audit 2021-09-17T06:30:56.
Sep 17 06:30:57 pech-mon-1 conmon[1337]: 888866+0000 mon.pech-mon-1 (mon.0) 1166284 : audit [DBG] from='client.? 10.0.1.10:0/298451163' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:30:59 pech-mon-1 conmon[1337]: debug 2021-09-17T06:30:59.165+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:30:59 pech-mon-1 conmon[1337]: cluster 2021-09-17T
Sep 17 06:30:59 pech-mon-1 conmon[1337]: 06:30:57.572631+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345774 : cluster [DBG] pgmap v347920: 1025 pgs: 1025 active+clean; 482 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.7 GiB/s wr, 2.49k op/s
Sep 17 06:31:00 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:00.389+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:31:00 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:00.389+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/1321627932' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:00 pech-mon-1 conmon[1337]: audit 
Sep 17 06:31:00 pech-mon-1 conmon[1337]: 2021-09-17T06:31:00.396365+0000 mon.pech-mon-1 (mon.0) 1166285 : audit [DBG] from='client.? 10.0.1.10:0/1321627932' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:30:59.577378+0000
Sep 17 06:31:01 pech-mon-1 conmon[1337]:  mgr.pech-mon-1.ptrsea (mgr.245802) 345775 : cluster [DBG] pgmap v347921: 1025 pgs: 1025 active+clean; 482 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 7.7 GiB/s wr, 1.98k op/s
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 
Sep 17 06:31:01 pech-mon-1 conmon[1337]: 2021-09-17T06:31:01.465+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.4e", "id": [156, 208]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.4e", "id": [156, 208]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.12c", "id": [195, 204]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.12c", "id": [195, 204]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.175", "id": [196, 176]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.175", "id": [196, 176]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1bc", "id": [330, 299]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1bc", "id": [330, 299]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1f8", "id": [200, 136]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1f8", "id": [200, 136]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.32a", "id": [237, 188]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.32a", "id": [237, 188]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.341", "id": [162, 230]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.341", "id": [162, 230]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.344", "id": [145, 204]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.465+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.344", "id": [145, 204]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.469+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.369", "id": [212, 312]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.469+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.369", "id": [212, 312]}]: dispatch
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.469+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.37f", "id": [165, 176]} v 0) v1
Sep 17 06:31:01 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:01.469+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.37f", "id": [165, 176]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.413+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13998 do_prune osdmap full prune enabled
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021-09-17T06:31:
Sep 17 06:31:02 pech-mon-1 conmon[1337]: 01.471138+0000 mon.pech-mon-1 (mon.0) 1166286 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.4e", "id": [156, 208]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 
Sep 17 06:31:02 pech-mon-1 conmon[1337]: 2021-09-17T06:31:01.471435+0000 mon.pech-mon-1 (mon.0) 1166287 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.12c", "id": [195, 204]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021-09-17T06:31
Sep 17 06:31:02 pech-mon-1 conmon[1337]: :01.471664+0000 mon.pech-mon-1 (mon.0) 1166288 : audit 
Sep 17 06:31:02 pech-mon-1 conmon[1337]: [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.175", "id": [196, 176]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021-09-17T
Sep 17 06:31:02 pech-mon-1 conmon[1337]: 06:31:01.471885+0000 mon.pech-mon-1 (mon.0) 
Sep 17 06:31:02 pech-mon-1 conmon[1337]: 1166289 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1bc", "id": [330, 299]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021-09-17T06
Sep 17 06:31:02 pech-mon-1 conmon[1337]: :31:01.472099+0000 mon.pech-mon-1 (mon.0) 1166290
Sep 17 06:31:02 pech-mon-1 conmon[1337]:  : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1f8", "id": [200, 136]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021
Sep 17 06:31:02 pech-mon-1 conmon[1337]: -09-17T06:31:01.472311+0000 mon.pech-mon-1 (mon.0) 1166291 : 
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.32a", "id": [237, 188]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021-
Sep 17 06:31:02 pech-mon-1 conmon[1337]: 09-17T06:31:01.472548+0000 mon.pech-mon-1 (mon.0) 1166292 : audit
Sep 17 06:31:02 pech-mon-1 conmon[1337]:  [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.341", "id": [162, 230]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021-09
Sep 17 06:31:02 pech-mon-1 conmon[1337]: -17T06:31:01.472822+0000 mon.pech-mon-1 (mon.0) 1166293 : audit 
Sep 17 06:31:02 pech-mon-1 conmon[1337]: [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.344", "id": [145, 204]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021-09-
Sep 17 06:31:02 pech-mon-1 conmon[1337]: 17T06:31:01.473114+0000 mon.pech-mon-1 (mon.0) 1166294 : audit [INF]
Sep 17 06:31:02 pech-mon-1 conmon[1337]:  from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.369", "id": [212, 312]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: audit 2021-09-17
Sep 17 06:31:02 pech-mon-1 conmon[1337]: T06:31:01.473390+0000 mon.pech-mon-1 (mon.0) 1166295 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.37f", "id": [165, 176]}]: dispatch
Sep 17 06:31:02 pech-mon-1 conmon[1337]: 
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.429+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e13999 e13999: 355 total, 355 up, 340 in
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.4e", "id": [156, 208]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.12c", "id": [195, 204]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.175", "id": [196, 176]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1bc", "id": [330, 299]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1f8", "id": [200, 136]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.32a", "id": [237, 188]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.341", "id": [162, 230]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.344", "id": [145, 204]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.369", "id": [212, 312]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 
Sep 17 06:31:02 pech-mon-1 conmon[1337]: 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.37f", "id": [165, 176]}]': finished
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.433+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e13999: 355 total, 355 up, 340 in
Sep 17 06:31:02 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:02.449+0000 7f66b0b2a700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b0b2a700' had timed out after 0.000000000s
Sep 17 06:31:03 pech-mon-1 conmon[1337]: cluster 
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 2021-09-17T06:31:01.589730+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345776 : cluster [DBG] pgmap v347922: 1025 pgs: 1025 active+clean; 516 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 11 GiB/s wr, 2.70k op/s
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 2021-09-17T06:31:02.437726+0000 mon.pech-mon-1
Sep 17 06:31:03 pech-mon-1 conmon[1337]:  (mon.0) 1166296 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.4e", "id": [156, 208]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 2021-09-17T06:31:02.437806+0000 
Sep 17 06:31:03 pech-mon-1 conmon[1337]: mon.pech-mon-1 (mon.0) 1166297 : audit [INF]
Sep 17 06:31:03 pech-mon-1 conmon[1337]:  from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.12c", "id": [195, 204]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 2021-09-17T06:
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 31:02.437886+0000 mon.pech-mon-1 (mon.
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 0) 1166298 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.175", "id": [196, 176]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 2021-09-17
Sep 17 06:31:03 pech-mon-1 conmon[1337]: T06:31:02.437952+0000 
Sep 17 06:31:03 pech-mon-1 conmon[1337]: mon.pech-mon-1 (mon.0) 1166299 : audit [INF]
Sep 17 06:31:03 pech-mon-1 conmon[1337]:  from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1bc", "id": [330, 299]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit
Sep 17 06:31:03 pech-mon-1 conmon[1337]:  2021-09-17T06:31:02.438000+0000 mon.pech-mon-1 (mon.0) 1166300 : audit 
Sep 17 06:31:03 pech-mon-1 conmon[1337]: [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1f8", "id": [200, 136]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 2021-09-
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 17T06:31:02.438048+0000 mon.pech-mon-1 (mon.0) 1166301 : 
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.32a", "id": [237, 188]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 2021-09-17T06:31
Sep 17 06:31:03 pech-mon-1 conmon[1337]: :02.438095+0000 mon.pech-mon-1 (mon.0
Sep 17 06:31:03 pech-mon-1 conmon[1337]: ) 1166302 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.341", "id": [162, 230]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 2021-09-17T
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 06:31:02.438141+0000 mon.pech-mon-1 (mon
Sep 17 06:31:03 pech-mon-1 conmon[1337]: .0) 1166303 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.344", "id": [145, 204]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 2021-09-17T06:
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 31:02.438189+0000 mon.pech-mon-1 (mon.0) 1166304
Sep 17 06:31:03 pech-mon-1 conmon[1337]:  : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.369", "id": [212, 312]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: audit 2021-09-17T06:31
Sep 17 06:31:03 pech-mon-1 conmon[1337]: :02.438234+0000 mon.pech-mon-1 (mon.0
Sep 17 06:31:03 pech-mon-1 conmon[1337]: ) 1166305 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.37f", "id": [165, 176]}]': finished
Sep 17 06:31:03 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:
Sep 17 06:31:03 pech-mon-1 conmon[1337]: 31:02.438269+0000 mon.pech-mon-1 (mon.0) 1166306 : cluster 
Sep 17 06:31:03 pech-mon-1 conmon[1337]: [DBG] osdmap e13999: 355 total, 355 up, 340 in
Sep 17 06:31:03 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:03.429+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e13999 do_prune osdmap full prune enabled
Sep 17 06:31:03 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:03.437+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e14000 e14000: 355 total, 355 up, 340 in
Sep 17 06:31:03 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:03.441+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e14000: 355 total, 355 up, 340 in
Sep 17 06:31:03 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:03.461+0000 7f66b1b2c700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b1b2c700' had timed out after 0.000000000s
Sep 17 06:31:03 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:03.913+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:31:03 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:03.913+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/1627364461' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:04 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:04.169+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14000 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:31:04 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:04.437+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14000 do_prune osdmap full prune enabled
Sep 17 06:31:04 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:04.449+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e14001 e14001: 355 total, 355 up, 340 in
Sep 17 06:31:04 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:04.453+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e14001: 355 total, 355 up, 340 in
Sep 17 06:31:04 pech-mon-1 conmon[1337]: cluster 
Sep 17 06:31:04 pech-mon-1 conmon[1337]: 2021-09-17T06:31:03.447173+0000 mon.pech-mon-1 (mon.0) 1166307 : cluster [DBG] osdmap e14000: 355 total, 355 up, 340 in
Sep 17 06:31:04 pech-mon-1 conmon[1337]: cluster 2021
Sep 17 06:31:04 pech-mon-1 conmon[1337]: -09-17T06:31:03.592867+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345777 : cluster [DBG] pgmap v347925: 1025 pgs: 1025 active+clean; 528 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 8.8 GiB/s wr, 2.25k op/s
Sep 17 06:31:04 pech-mon-1 conmon[1337]: audit 
Sep 17 06:31:04 pech-mon-1 conmon[1337]: 2021-09-17T06:31:03.920820+0000 mon.pech-mon-1 (mon.0) 1166308 : audit [DBG] from='client.? 10.0.1.10:0/1627364461' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:04 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:04.473+0000 7f66b1b2c700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b1b2c700' had timed out after 0.000000000s
Sep 17 06:31:05 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:31:04.459669+0000 mon.pech-mon-1 (mon.0) 1166309 : cluster [DBG] osdmap e14001: 355 total, 355 up, 340 in
Sep 17 06:31:06 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:06.461+0000 7f66b232d700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 274/704680 objects degraded (0.039%), 2 pgs degraded (PG_DEGRADED)
Sep 17 06:31:06 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:31:05.602855+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345778 : cluster [DBG] pgmap v347927: 1025 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1020 active+clean; 551 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 11 GiB/s wr, 2.92k op/s; 274/704680 objects degraded (0.039%); 386/704680 objects misplaced (0.055%); 14 MiB/s, 3 objects/s recovering
Sep 17 06:31:07 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:07.397+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:31:07 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:07.397+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/612477070' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:07 pech-mon-1 conmon[1337]: cluster 2021-09-17T
Sep 17 06:31:07 pech-mon-1 conmon[1337]: 06:31:06.467305+0000 mon.pech-mon-1 (mon.0) 1166310 : cluster [WRN] Health check failed: Degraded data redundancy: 274/704680 objects degraded (0.039%), 2 pgs degraded (PG_DEGRADED)
Sep 17 06:31:07 pech-mon-1 conmon[1337]: audit 2021-09-17T06
Sep 17 06:31:07 pech-mon-1 conmon[1337]: :31:07.404736+0000 mon.pech-mon-1 (mon.0) 1166311 : audit [DBG] from='client.? 10.0.1.10:0/612477070' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:08 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:31:07.613188+0000 
Sep 17 06:31:08 pech-mon-1 conmon[1337]: mgr.pech-mon-1.ptrsea (mgr.245802) 345779 : cluster [DBG] pgmap v347928: 1025 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1015 active+clean; 574 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.7 GiB/s wr, 2.47k op/s; 660/734700 objects degraded (0.090%); 596/734700 objects misplaced (0.081%); 49 MiB/s, 12 objects/s recovering
Sep 17 06:31:09 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:09.169+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14001 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:31:10 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:10.893+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:31:10 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:10.893+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/137858819' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:11 pech-mon-1 conmon[1337]: cluster 2021-09-17T06
Sep 17 06:31:11 pech-mon-1 conmon[1337]: :31:09.617576+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345780 : cluster [DBG] pgmap v347929: 1025 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1015 active+clean; 574 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 6.4 GiB/s wr, 1.63k op/s; 660/734700 objects degraded (0.090%); 596/734700 objects misplaced (0.081%); 41 MiB/s, 10 objects/s recovering
Sep 17 06:31:11 pech-mon-1 conmon[1337]: audit 2021-09-17T06:
Sep 17 06:31:11 pech-mon-1 conmon[1337]: 31:10.897521+0000 mon.pech-mon-1 (mon.0) 1166312 : audit [DBG] from='client.? 10.0.1.10:0/137858819' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:12 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:12.177+0000 7f66b232d700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 660/775905 objects degraded (0.085%), 5 pgs degraded (PG_DEGRADED)
Sep 17 06:31:12 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:
Sep 17 06:31:12 pech-mon-1 conmon[1337]: 31:12.182739+0000 mon.pech-mon-1 (mon.0) 1166313 : cluster [WRN] Health check update: Degraded data redundancy: 660/775905 objects degraded (0.085%), 5 pgs degraded (PG_DEGRADED)
Sep 17 06:31:12 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:12.925+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/trash_purge_schedule"} v 0) v1
Sep 17 06:31:12 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:12.925+0000 7f66afb28700  0 log_channel(audit) log [INF] : from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/trash_purge_schedule"}]: dispatch
Sep 17 06:31:13 pech-mon-1 conmon[1337]: cluster 2021-
Sep 17 06:31:13 pech-mon-1 conmon[1337]: 09-17T06:31:11.630011+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345781 : cluster [DBG] pgmap v347930: 1025 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1015 active+clean; 606 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.5 GiB/s wr, 2.44k op/s; 660/775905 objects degraded (0.085%); 416/775905 objects misplaced (0.054%); 124 MiB/s, 30 objects/s recovering
Sep 17 06:31:13 pech-mon-1 conmon[1337]: audit 2021-09-17T06:31:12
Sep 17 06:31:13 pech-mon-1 conmon[1337]: .927025+0000 mon.pech-mon-1 (mon.0) 1166314 : audit [INF] from='mgr.245802 10.0.1.10:0/136830414' entity='mgr.pech-mon-1.ptrsea' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/pech-mon-1.ptrsea/trash_purge_schedule"}]: dispatch
Sep 17 06:31:14 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:14.169+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14001 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:31:14 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:14.610+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:31:14 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:14.610+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/1542833761' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:15 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:31
Sep 17 06:31:15 pech-mon-1 conmon[1337]: :13.637611+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345782 : cluster [DBG] pgmap v347931: 1025 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1015 active+clean; 619 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 9.0 GiB/s wr, 2.29k op/s; 660/791520 objects degraded (0.083%); 305/791520 objects misplaced (0.039%); 145 MiB/s, 36 objects/s recovering
Sep 17 06:31:15 pech-mon-1 conmon[1337]: audit 2021-09-17T06:31:14.613690+0000
Sep 17 06:31:15 pech-mon-1 conmon[1337]:  mon.pech-mon-1 (mon.0) 1166315 : audit [DBG] from='client.? 10.0.1.10:0/1542833761' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:16 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:16.242+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14001 do_prune osdmap full prune enabled
Sep 17 06:31:16 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:16.250+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e14002 e14002: 355 total, 355 up, 340 in
Sep 17 06:31:16 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:16.254+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e14002: 355 total, 355 up, 340 in
Sep 17 06:31:16 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:16.278+0000 7f66b0b2a700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b0b2a700' had timed out after 0.000000000s
Sep 17 06:31:17 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:31:
Sep 17 06:31:17 pech-mon-1 conmon[1337]: 15.646766+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345783 : cluster [DBG] pgmap v347932: 1025 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1015 active+clean; 643 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 10 GiB/s wr, 2.61k op/s; 660/822455 objects degraded (0.080%); 152/822455 objects misplaced (0.018%); 185 MiB/s, 46 objects/s recovering
Sep 17 06:31:17 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:31:16.259204+0000 
Sep 17 06:31:17 pech-mon-1 conmon[1337]: mon.pech-mon-1 (mon.0) 1166316 : cluster [DBG] osdmap e14002: 355 total, 355 up, 340 in
Sep 17 06:31:17 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:17.262+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14002 do_prune osdmap full prune enabled
Sep 17 06:31:17 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:17.270+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e14003 e14003: 355 total, 355 up, 340 in
Sep 17 06:31:17 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:17.274+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e14003: 355 total, 355 up, 340 in
Sep 17 06:31:17 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:17.290+0000 7f66b0b2a700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b0b2a700' had timed out after 0.000000000s
Sep 17 06:31:18 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:18.122+0000 7f66afb28700  0 mon.pech-mon-1@0(leader) e7 handle_command mon_command({"prefix": "status"} v 0) v1
Sep 17 06:31:18 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:18.122+0000 7f66afb28700  0 log_channel(audit) log [DBG] : from='client.? 10.0.1.10:0/3185799594' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:18 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:18.270+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14003 do_prune osdmap full prune enabled
Sep 17 06:31:18 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:18.278+0000 7f66b232d700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 535/851005 objects degraded (0.063%), 4 pgs degraded (PG_DEGRADED)
Sep 17 06:31:18 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:18.282+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e14004 e14004: 355 total, 355 up, 340 in
Sep 17 06:31:18 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:18.286+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e14004: 355 total, 355 up, 340 in
Sep 17 06:31:18 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:31:17.278500+0000 mon.pech-mon-1
Sep 17 06:31:18 pech-mon-1 conmon[1337]:  (mon.0) 1166317 : cluster [DBG] osdmap e14003: 355 total, 355 up, 340 in
Sep 17 06:31:18 pech-mon-1 conmon[1337]: audit 2021-09-17T06:31:18.128080+0000 mon.pech-mon-1 (mon.0) 1166318 : audit [DBG] from='client.? 10.0.1.10:0/3185799594' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
Sep 17 06:31:18 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:18.306+0000 7f66b1b2c700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b1b2c700' had timed out after 0.000000000s
Sep 17 06:31:19 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:19.174+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14004 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 75497472 full_alloc: 549453824 kv_alloc: 390070272
Sep 17 06:31:19 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:19.294+0000 7f66b232d700  1 mon.pech-mon-1@0(leader).osd e14004 do_prune osdmap full prune enabled
Sep 17 06:31:19 pech-mon-1 conmon[1337]: cluster 2021-09-17T06:
Sep 17 06:31:19 pech-mon-1 conmon[1337]: 31:17.650551+0000 mgr.pech-mon-1.ptrsea (mgr.245802) 345784 : cluster [DBG] pgmap v347935: 1025 pgs: 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 1015 active+clean; 665 GiB data, 74 TiB used, 2.8 PiB / 2.9 PiB avail; 11 GiB/s wr, 2.90k op/s; 535/851005 objects degraded (0.063%); 172/851005 objects misplaced (0.020%); 274 MiB/s, 68 objects/s recovering
Sep 17 06:31:19 pech-mon-1 conmon[1337]: cluster 2021-09-17
Sep 17 06:31:19 pech-mon-1 conmon[1337]: T06:31:18.282938+0000 mon.pech-mon-1 (mon.0) 1166319 : cluster [WRN] 
Sep 17 06:31:19 pech-mon-1 conmon[1337]: Health check update: Degraded data redundancy: 535/851005 objects degraded (0.063%), 4 pgs degraded (PG_DEGRADED)
Sep 17 06:31:19 pech-mon-1 conmon[1337]: cluster
Sep 17 06:31:19 pech-mon-1 conmon[1337]:  2021-09-17T06:31:18.291235
Sep 17 06:31:19 pech-mon-1 conmon[1337]: +0000 mon.pech-mon-1 (mon.0) 1166320 : cluster 
Sep 17 06:31:19 pech-mon-1 conmon[1337]: [DBG] osdmap e14004: 355 total, 355 up, 340 in
Sep 17 06:31:19 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:19.310+0000 7f66ae325700  1 mon.pech-mon-1@0(leader).osd e14005 e14005: 355 total, 355 up, 340 in
Sep 17 06:31:19 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:19.310+0000 7f66ae325700  0 log_channel(cluster) log [DBG] : osdmap e14005: 355 total, 355 up, 340 in
Sep 17 06:31:19 pech-mon-1 conmon[1337]: debug 2021-09-17T06:31:19.326+0000 7f66b1b2c700  1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f66b1b2c700' had timed out after 0.000000000s
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx