A few additional findings. I'm of the impression that initial deployment
and config of RGW on quincy (net-new) creates a default Realm. For
reasons I don't understand, my default realm no longer exists.
Could this be the issue with RGWs not being manageable / detected by my
cluster?
Under what circumstances would a Realm disappear?
How to get this fixed?
# radosgw-admin realm list
{
"default_info": "",
"realms": []
}
# radosgw-admin realm default
failed to init realm: (2) No such file or directory
# radosgw-admin realm get-default
No default realm is set
# radosgw-admin realm list-periods
failed to read realm: (2) No such file or directory
# rados ls -p .rgw.root
zonegroup_info.45518452-8aa6-41b4-99f0-059b255c31cd
zone_info.743ea532-f5bc-4cca-891b-c27a586d5129
zone_names.default
zonegroups_names.default
On Sat, Dec 31, 2022 at 3:15 PM Deep Dish <deeepdish@xxxxxxxxx> wrote:
> Hi Pavin,
>
> Happy New Year!
>
> Many thanks for the commands. I managed to get the cluster into green
> status with the repeer command.
>
> Still had some slow MDS ops, I decided to purge a few older backups in a
> repository that's backed by one of the cephfs volumes. This resolved all
> slow OPS -- there must have been an inconsistent file causing an issue, not
> a big deal as it's a backup (likely manifested itself when cluster was
> rebuilding / rebalancing).
>
> I'm still having an issue in getting RGWs to be manageable / detected by
> the cluster (any ideas appreciated):
> - Increased debug logging to 5/5 across debug_rgw,debug_rgw_datacache,
> debug_rgw_sync
> - Redeployed RGW service (removed and recreated service);
> - All GWs show similar log entries -- There's a failed init realm ID error
> (could that be it?):
>
> 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 0 deferred set uid:gid to
> 167:167 (ceph:ceph)
>
> 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 0 ceph version 17.2.5
> (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
> radosgw, pid 2
>
> 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 0 framework: beast
>
> 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 0 framework conf key: port,
> val: 80
>
> 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 1 radosgw_Main not setting
> numa affinity
>
> 2022-12-31T20:02:55.574+0000 7fea6d76a5c0 1 rgw_d3n:
> rgw_d3n_l1_local_datacache_enabled=0
>
> 2022-12-31T20:02:55.574+0000 7fea6d76a5c0 1 D3N datacache enabled: 0
>
> 2022-12-31T20:02:55.590+0000 7fea6d76a5c0 4 rgw main: RGWPeriod::init
> failed to init realm id : (2) No such file or directory
>
> 2022-12-31T20:02:55.686+0000 7fea6d76a5c0 4 rgw main: Realm:
> ()
>
> 2022-12-31T20:02:55.686+0000 7fea6d76a5c0 4 rgw main: ZoneGroup: default
> (45518452-8aa6-41b4-99f0-059b255c31cd)
>
> 2022-12-31T20:02:55.686+0000 7fea6d76a5c0 4 rgw main: Zone: default
> (743ea532-f5bc-4cca-891b-c27a586d5129)
>
> 2022-12-31T20:02:56.010+0000 7fea6d76a5c0 2 all 8 watchers are set,
> enabling cache
>
> 2022-12-31T20:02:56.026+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:02:57.170+0000 7fea4eab1700 2 garbage collection: garbage
> collection: start
>
> 2022-12-31T20:02:57.170+0000 7fea4e2b0700 2 rgw object expirer Worker
> thread: object expiration: start
>
> 2022-12-31T20:02:57.170+0000 7fea48aa5700 5 lifecycle: schedule life
> cycle next start time: Sun Jan 1 00:00:00 2023
>
> 2022-12-31T20:02:57.170+0000 7fea46aa1700 5 lifecycle: schedule life
> cycle next start time: Sun Jan 1 00:00:00 2023
>
> 2022-12-31T20:02:57.170+0000 7fea44a9d700 5 lifecycle: schedule life
> cycle next start time: Sun Jan 1 00:00:00 2023
>
> 2022-12-31T20:02:59.186+0000 7fea4eab1700 2 garbage collection: garbage
> collection: stop
>
> 2022-12-31T20:03:18.029+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:03:40.029+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:04:02.028+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:04:24.027+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:04:46.031+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:05:08.030+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:05:30.030+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:05:52.029+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:06:14.029+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:06:36.028+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:06:58.032+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:07:20.031+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:07:42.031+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:08:04.030+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:08:26.029+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:08:48.029+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:09:10.032+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:09:32.028+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
> 2022-12-31T20:09:54.027+0000 7fea52ab9700 2 rgw data changes log:
> RGWDataChangesLog::ChangesRenewThread: start
>
>
>
> On Sat, Dec 31, 2022 at 1:22 AM Pavin Joseph <me@xxxxxxxxxxxxxxx> wrote:
>
>> Hey there,
>>
>> Sorry for the late reply.
>> If the pg issue isn't solved yet, could you run these:
>>
>> ceph pg repeer <pgid>
>> ceph pg repair <pgid>
>>
>> Pavin.
>>
>> On 29-Dec-22 4:08 AM, Deep Dish wrote:
>> > Hi Pavin,
>> >
>> > The following are additional developments.. There's one PG that's
>> > stuck and unable to recover. I've attached relevant ceph -s / health
>> > detail and pg stat outputs below.
>> >
>> > - There were some remaining lock files as suggested in /var/run/ceph/
>> > pertaining to rgw. I removed the service, deleted any stale lock files
>> > and redeployed the RGWs. All started with the common log entries
>> across
>> > all:
>> >
>> > 7ff5d9aaf5c0 0 deferred set uid:gid to 167:167 (ceph:ceph)
>> >
>> > 7ff5d9aaf5c0 0 ceph version 17.2.5
>> > (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
>> > radosgw, pid 2
>> >
>> > 7ff5d9aaf5c0 0 framework: beast
>> >
>> > 7ff5d9aaf5c0 0 framework conf key: port, val: 80
>> >
>> > 7ff5d9aaf5c0 1 radosgw_Main not setting numa affinity
>> >
>> > 7ff5d9aaf5c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0
>> >
>> > 7ff5d9aaf5c0 1 D3N datacache enabled: 0
>> >
>> > No additional log entries are recorded since starting them post
>> > re-deployment as per above.
>> >
>> > The cluster settled, there is no recovery activity. There is one pg
>> that's
>> > stuck and I have a hunch that it's impacting MDS and RGW processes as
>> > stated in the thread. PG is stuck as as active+remapped+backfilling:
>> >
>> >
>> >
>> > data:
>> >
>> > volumes: 2/2 healthy
>> >
>> > pools: 16 pools, 1504 pgs
>> >
>> > objects: 24.49M objects, 79 TiB
>> >
>> > usage: 119 TiB used, 390 TiB / 508 TiB avail
>> >
>> > pgs: 65210/146755179 objects misplaced (0.044%)
>> >
>> > 1503 active+clean
>> >
>> > 1 active+remapped+backfilling
>> >
>> >
>> >
>> > progress:
>> >
>> > Global Recovery Event (6h)
>> >
>> > [===========================.] (remaining: 73s)
>> >
>> > # ceph health detail
>> >
>> > HEALTH_WARN 1 MDSs report slow metadata IOs; 1 pgs not deep-scrubbed in
>> > time; 1 pgs not scrubbed in time
>> >
>> > [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
>> >
>> > mds.fs01.ceph02mon02.wicrdz(mds.0): 5 slow metadata IOs are
>> blocked >
>> > 30 secs, oldest blocked for 74436 secs
>> >
>> > [WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
>> >
>> > pg 14.ff not deep-scrubbed since 2022-12-14T19:35:51.893008+0000
>> >
>> > [WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
>> >
>> > pg 14.ff not scrubbed since 2022-12-17T06:33:40.577932+0000
>> >
>> >
>> >
>> > From the following pg query:
>> >
>> > - "pgid": "14.ffs0" is stuck as peering (osd 5)
>> > - "pgid": "14.ffs4" is stuck as unknown (osd 18)
>> > - "pgid": "14.ffs5" is stuck as unknown (osd 24)
>> > - "pgid": "14.ffs3" is stuck as unknown (osd 42)
>> > - "pgid": "14.ffs2" is stick as unknown (osd 58)
>> > - "pgid": "14.ffs1" is marked as active+clean (osd 36)
>> >
>> > # ceph pg 14.ff query
>> >
>> > {
>> >
>> > "snap_trimq": "[]",
>> >
>> > "snap_trimq_len": 0,
>> >
>> > "state": "active+remapped+backfilling",
>> >
>> > "epoch": 19594,
>> >
>> > "up": [
>> >
>> > 5,
>> >
>> > 36,
>> >
>> > 58,
>> >
>> > 42,
>> >
>> > 18,
>> >
>> > 24
>> >
>> > ],
>> >
>> > "acting": [
>> >
>> > 50,
>> >
>> > 36,
>> >
>> > 5,
>> >
>> > 26,
>> >
>> > 15,
>> >
>> > 46
>> >
>> > ],
>> >
>> > "backfill_targets": [
>> >
>> > "5(0)",
>> >
>> > "18(4)",
>> >
>> > "24(5)",
>> >
>> > "42(3)",
>> >
>> > "58(2)"
>> >
>> > ],
>> >
>> > "acting_recovery_backfill": [
>> >
>> > "5(0)",
>> >
>> > "5(2)",
>> >
>> > "15(4)",
>> >
>> > "18(4)",
>> >
>> > "24(5)",
>> >
>> > "26(3)",
>> >
>> > "36(1)",
>> >
>> > "42(3)",
>> >
>> > "46(5)",
>> >
>> > "50(0)",
>> >
>> > "58(2)"
>> >
>> > ],
>> >
>> > "info": {
>> >
>> > "pgid": "14.ffs0",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "19550'35077",
>> >
>> > "log_tail": "13761'32157",
>> >
>> > "last_user_version": 35077,
>> >
>> > "last_backfill": "MAX",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "19550'35077",
>> >
>> > "reported_seq": 396919,
>> >
>> > "reported_epoch": 19594,
>> >
>> > "state": "active+remapped+backfilling",
>> >
>> > "last_fresh": "2022-12-28T22:03:20.278478+0000",
>> >
>> > "last_change": "2022-12-26T21:27:51.600940+0000",
>> >
>> > "last_active": "2022-12-28T22:03:20.278478+0000",
>> >
>> > "last_peered": "2022-12-28T22:03:20.278478+0000",
>> >
>> > "last_clean": "2022-12-26T21:27:45.471954+0000",
>> >
>> > "last_became_active": "2022-12-26T21:27:51.085966+0000",
>> >
>> > "last_became_peered": "2022-12-26T21:27:51.085966+0000",
>> >
>> > "last_unstale": "2022-12-28T22:03:20.278478+0000",
>> >
>> > "last_undegraded": "2022-12-28T22:03:20.278478+0000",
>> >
>> > "last_fullsized": "2022-12-28T22:03:20.278478+0000",
>> >
>> > "mapping_epoch": 16615,
>> >
>> > "log_start": "13761'32157",
>> >
>> > "ondisk_log_start": "13761'32157",
>> >
>> > "created": 4537,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 8,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "objects_scrubbed": 16227,
>> >
>> > "log_size": 2920,
>> >
>> > "ondisk_log_size": 2920,
>> >
>> > "stats_invalid": true,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 14,
>> >
>> > "scrub_schedule": "queued for deep scrub",
>> >
>> > "scrub_duration": 13.320415128,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 56709530650,
>> >
>> > "num_objects": 13548,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 81288,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 0,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 65210,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 13548,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 67760,
>> >
>> > "num_read_kb": 177798674,
>> >
>> > "num_write": 21231,
>> >
>> > "num_write_kb": 70024901,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 34481,
>> >
>> > "num_bytes_recovered": 144364295675,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [
>> >
>> > 5,
>> >
>> > 36,
>> >
>> > 58,
>> >
>> > 42,
>> >
>> > 18,
>> >
>> > 24
>> >
>> > ],
>> >
>> > "acting": [
>> >
>> > 50,
>> >
>> > 36,
>> >
>> > 5,
>> >
>> > 26,
>> >
>> > 15,
>> >
>> > 46
>> >
>> > ],
>> >
>> > "avail_no_missing": [
>> >
>> > "50(0)",
>> >
>> > "5(2)",
>> >
>> > "15(4)",
>> >
>> > "26(3)",
>> >
>> > "36(1)",
>> >
>> > "46(5)"
>> >
>> > ],
>> >
>> > "object_location_counts": [
>> >
>> > {
>> >
>> > "shards": "5(2),15(4),26(3),36(1),46(5),50(0)",
>> >
>> > "objects": 13548
>> >
>> > }
>> >
>> > ],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": 5,
>> >
>> > "acting_primary": 50,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 0,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > "peer_info": [
>> >
>> > {
>> >
>> > "peer": "5(0)",
>> >
>> > "pgid": "14.ffs0",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 0,
>> >
>> > "last_backfill": "14:ff09a915:::10000001249.00000353:head",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "0'0",
>> >
>> > "reported_seq": 2,
>> >
>> > "reported_epoch": 16614,
>> >
>> > "state": "peering",
>> >
>> > "last_fresh": "2022-12-26T21:27:47.993646+0000",
>> >
>> > "last_change": "2022-12-26T21:27:47.769141+0000",
>> >
>> > "last_active": "0.000000",
>> >
>> > "last_peered": "0.000000",
>> >
>> > "last_clean": "0.000000",
>> >
>> > "last_became_active": "0.000000",
>> >
>> > "last_became_peered": "0.000000",
>> >
>> > "last_unstale": "2022-12-26T21:27:47.993646+0000",
>> >
>> > "last_undegraded": "2022-12-26T21:27:47.993646+0000",
>> >
>> > "last_fullsized": "2022-12-26T21:27:47.993646+0000",
>> >
>> > "mapping_epoch": 16615,
>> >
>> > "log_start": "0'0",
>> >
>> > "ondisk_log_start": "0'0",
>> >
>> > "created": 4537,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 0,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "objects_scrubbed": 0,
>> >
>> > "log_size": 0,
>> >
>> > "ondisk_log_size": 0,
>> >
>> > "stats_invalid": false,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 0,
>> >
>> > "scrub_schedule": "queued for deep scrub",
>> >
>> > "scrub_duration": 0,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 2115497390,
>> >
>> > "num_objects": 506,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 0,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 13042,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 506,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 0,
>> >
>> > "num_read_kb": 0,
>> >
>> > "num_write": 42,
>> >
>> > "num_write_kb": 172032,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 0,
>> >
>> > "num_bytes_recovered": 0,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [
>> >
>> > 5,
>> >
>> > 36,
>> >
>> > 58,
>> >
>> > 42,
>> >
>> > 18,
>> >
>> > 24
>> >
>> > ],
>> >
>> > "acting": [
>> >
>> > 50,
>> >
>> > 36,
>> >
>> > 5,
>> >
>> > 26,
>> >
>> > 15,
>> >
>> > 46
>> >
>> > ],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [
>> >
>> > 24
>> >
>> > ],
>> >
>> > "up_primary": 5,
>> >
>> > "acting_primary": 50,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 1,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "5(2)",
>> >
>> > "pgid": "14.ffs2",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 33714,
>> >
>> > "last_backfill": "MAX",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "16465'33714",
>> >
>> > "reported_seq": 388679,
>> >
>> > "reported_epoch": 16465,
>> >
>> > "state": "active+clean",
>> >
>> > "last_fresh": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_change": "2022-12-26T02:28:03.455705+0000",
>> >
>> > "last_active": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_peered": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_clean": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_became_active":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_became_peered":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_unstale": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_undegraded": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_fullsized": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "mapping_epoch": 16615,
>> >
>> > "log_start": "12957'31152",
>> >
>> > "ondisk_log_start": "12957'31152",
>> >
>> > "created": 4537,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 8,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "objects_scrubbed": 16227,
>> >
>> > "log_size": 2562,
>> >
>> > "ondisk_log_size": 2562,
>> >
>> > "stats_invalid": true,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 14,
>> >
>> > "scrub_schedule": "queued for deep scrub",
>> >
>> > "scrub_duration": 13.320415128,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 51491812456,
>> >
>> > "num_objects": 12301,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 73806,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 0,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 12301,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 66867,
>> >
>> > "num_read_kb": 177079789,
>> >
>> > "num_write": 19946,
>> >
>> > "num_write_kb": 64929471,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 34017,
>> >
>> > "num_bytes_recovered": 142424959053,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [
>> >
>> > 5,
>> >
>> > 36,
>> >
>> > 58,
>> >
>> > 42,
>> >
>> > 18,
>> >
>> > 24
>> >
>> > ],
>> >
>> > "acting": [
>> >
>> > 50,
>> >
>> > 36,
>> >
>> > 5,
>> >
>> > 26,
>> >
>> > 15,
>> >
>> > 46
>> >
>> > ],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": 5,
>> >
>> > "acting_primary": 50,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 0,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "15(4)",
>> >
>> > "pgid": "14.ffs4",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 33714,
>> >
>> > "last_backfill": "MAX",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "16465'33714",
>> >
>> > "reported_seq": 388679,
>> >
>> > "reported_epoch": 16465,
>> >
>> > "state": "active+clean",
>> >
>> > "last_fresh": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_change": "2022-12-26T02:28:03.455705+0000",
>> >
>> > "last_active": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_peered": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_clean": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_became_active":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_became_peered":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_unstale": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_undegraded": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_fullsized": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "mapping_epoch": 16615,
>> >
>> > "log_start": "12957'31152",
>> >
>> > "ondisk_log_start": "12957'31152",
>> >
>> > "created": 4537,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 8,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "objects_scrubbed": 16227,
>> >
>> > "log_size": 2562,
>> >
>> > "ondisk_log_size": 2562,
>> >
>> > "stats_invalid": true,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 14,
>> >
>> > "scrub_schedule": "queued for deep scrub",
>> >
>> > "scrub_duration": 13.320415128,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 51491812456,
>> >
>> > "num_objects": 12301,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 73806,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 0,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 12301,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 66867,
>> >
>> > "num_read_kb": 177079789,
>> >
>> > "num_write": 19946,
>> >
>> > "num_write_kb": 64929471,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 34017,
>> >
>> > "num_bytes_recovered": 142424959053,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [
>> >
>> > 5,
>> >
>> > 36,
>> >
>> > 58,
>> >
>> > 42,
>> >
>> > 18,
>> >
>> > 24
>> >
>> > ],
>> >
>> > "acting": [
>> >
>> > 50,
>> >
>> > 36,
>> >
>> > 5,
>> >
>> > 26,
>> >
>> > 15,
>> >
>> > 46
>> >
>> > ],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": 5,
>> >
>> > "acting_primary": 50,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 0,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "18(4)",
>> >
>> > "pgid": "14.ffs4",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 0,
>> >
>> > "last_backfill": "14:ff09a915:::10000001249.00000353:head",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "0'0",
>> >
>> > "reported_seq": 0,
>> >
>> > "reported_epoch": 0,
>> >
>> > "state": "unknown",
>> >
>> > "last_fresh": "0.000000",
>> >
>> > "last_change": "0.000000",
>> >
>> > "last_active": "0.000000",
>> >
>> > "last_peered": "0.000000",
>> >
>> > "last_clean": "0.000000",
>> >
>> > "last_became_active": "0.000000",
>> >
>> > "last_became_peered": "0.000000",
>> >
>> > "last_unstale": "0.000000",
>> >
>> > "last_undegraded": "0.000000",
>> >
>> > "last_fullsized": "0.000000",
>> >
>> > "mapping_epoch": 0,
>> >
>> > "log_start": "0'0",
>> >
>> > "ondisk_log_start": "0'0",
>> >
>> > "created": 0,
>> >
>> > "last_epoch_clean": 0,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 0,
>> >
>> > "last_scrub": "0'0",
>> >
>> > "last_scrub_stamp": "0.000000",
>> >
>> > "last_deep_scrub": "0'0",
>> >
>> > "last_deep_scrub_stamp": "0.000000",
>> >
>> > "last_clean_scrub_stamp": "0.000000",
>> >
>> > "objects_scrubbed": 0,
>> >
>> > "log_size": 0,
>> >
>> > "ondisk_log_size": 0,
>> >
>> > "stats_invalid": false,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 0,
>> >
>> > "scrub_schedule": "--",
>> >
>> > "scrub_duration": 0,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 2115497390,
>> >
>> > "num_objects": 506,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 0,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 13042,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 506,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 0,
>> >
>> > "num_read_kb": 0,
>> >
>> > "num_write": 42,
>> >
>> > "num_write_kb": 172032,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 0,
>> >
>> > "num_bytes_recovered": 0,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [],
>> >
>> > "acting": [],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": -1,
>> >
>> > "acting_primary": -1,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 1,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "24(5)",
>> >
>> > "pgid": "14.ffs5",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 0,
>> >
>> > "last_backfill": "14:ff09a915:::10000001249.00000353:head",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "0'0",
>> >
>> > "reported_seq": 0,
>> >
>> > "reported_epoch": 0,
>> >
>> > "state": "unknown",
>> >
>> > "last_fresh": "0.000000",
>> >
>> > "last_change": "0.000000",
>> >
>> > "last_active": "0.000000",
>> >
>> > "last_peered": "0.000000",
>> >
>> > "last_clean": "0.000000",
>> >
>> > "last_became_active": "0.000000",
>> >
>> > "last_became_peered": "0.000000",
>> >
>> > "last_unstale": "0.000000",
>> >
>> > "last_undegraded": "0.000000",
>> >
>> > "last_fullsized": "0.000000",
>> >
>> > "mapping_epoch": 0,
>> >
>> > "log_start": "0'0",
>> >
>> > "ondisk_log_start": "0'0",
>> >
>> > "created": 0,
>> >
>> > "last_epoch_clean": 0,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 0,
>> >
>> > "last_scrub": "0'0",
>> >
>> > "last_scrub_stamp": "0.000000",
>> >
>> > "last_deep_scrub": "0'0",
>> >
>> > "last_deep_scrub_stamp": "0.000000",
>> >
>> > "last_clean_scrub_stamp": "0.000000",
>> >
>> > "objects_scrubbed": 0,
>> >
>> > "log_size": 0,
>> >
>> > "ondisk_log_size": 0,
>> >
>> > "stats_invalid": false,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 0,
>> >
>> > "scrub_schedule": "--",
>> >
>> > "scrub_duration": 0,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 2115497390,
>> >
>> > "num_objects": 506,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 0,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 13042,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 506,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 0,
>> >
>> > "num_read_kb": 0,
>> >
>> > "num_write": 42,
>> >
>> > "num_write_kb": 172032,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 0,
>> >
>> > "num_bytes_recovered": 0,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [],
>> >
>> > "acting": [],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": -1,
>> >
>> > "acting_primary": -1,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 1,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "26(3)",
>> >
>> > "pgid": "14.ffs3",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 33714,
>> >
>> > "last_backfill": "MAX",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "16465'33714",
>> >
>> > "reported_seq": 388679,
>> >
>> > "reported_epoch": 16465,
>> >
>> > "state": "active+clean",
>> >
>> > "last_fresh": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_change": "2022-12-26T02:28:03.455705+0000",
>> >
>> > "last_active": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_peered": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_clean": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_became_active":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_became_peered":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_unstale": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_undegraded": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_fullsized": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "mapping_epoch": 16615,
>> >
>> > "log_start": "12957'31152",
>> >
>> > "ondisk_log_start": "12957'31152",
>> >
>> > "created": 4537,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 8,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "objects_scrubbed": 16227,
>> >
>> > "log_size": 2562,
>> >
>> > "ondisk_log_size": 2562,
>> >
>> > "stats_invalid": true,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 14,
>> >
>> > "scrub_schedule": "queued for deep scrub",
>> >
>> > "scrub_duration": 13.320415128,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 51491812456,
>> >
>> > "num_objects": 12301,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 73806,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 0,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 12301,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 66867,
>> >
>> > "num_read_kb": 177079789,
>> >
>> > "num_write": 19946,
>> >
>> > "num_write_kb": 64929471,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 34017,
>> >
>> > "num_bytes_recovered": 142424959053,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [
>> >
>> > 5,
>> >
>> > 36,
>> >
>> > 58,
>> >
>> > 42,
>> >
>> > 18,
>> >
>> > 24
>> >
>> > ],
>> >
>> > "acting": [
>> >
>> > 50,
>> >
>> > 36,
>> >
>> > 5,
>> >
>> > 26,
>> >
>> > 15,
>> >
>> > 46
>> >
>> > ],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": 5,
>> >
>> > "acting_primary": 50,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 0,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "36(1)",
>> >
>> > "pgid": "14.ffs1",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 33714,
>> >
>> > "last_backfill": "MAX",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "16465'33714",
>> >
>> > "reported_seq": 388679,
>> >
>> > "reported_epoch": 16465,
>> >
>> > "state": "active+clean",
>> >
>> > "last_fresh": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_change": "2022-12-26T02:28:03.455705+0000",
>> >
>> > "last_active": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_peered": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_clean": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_became_active":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_became_peered":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_unstale": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_undegraded": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_fullsized": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "mapping_epoch": 16615,
>> >
>> > "log_start": "12957'31152",
>> >
>> > "ondisk_log_start": "12957'31152",
>> >
>> > "created": 4537,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 8,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "objects_scrubbed": 16227,
>> >
>> > "log_size": 2562,
>> >
>> > "ondisk_log_size": 2562,
>> >
>> > "stats_invalid": true,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 14,
>> >
>> > "scrub_schedule": "queued for deep scrub",
>> >
>> > "scrub_duration": 13.320415128,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 51491812456,
>> >
>> > "num_objects": 12301,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 73806,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 0,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 12301,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 66867,
>> >
>> > "num_read_kb": 177079789,
>> >
>> > "num_write": 19946,
>> >
>> > "num_write_kb": 64929471,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 34017,
>> >
>> > "num_bytes_recovered": 142424959053,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [
>> >
>> > 5,
>> >
>> > 36,
>> >
>> > 58,
>> >
>> > 42,
>> >
>> > 18,
>> >
>> > 24
>> >
>> > ],
>> >
>> > "acting": [
>> >
>> > 50,
>> >
>> > 36,
>> >
>> > 5,
>> >
>> > 26,
>> >
>> > 15,
>> >
>> > 46
>> >
>> > ],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": 5,
>> >
>> > "acting_primary": 50,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 0,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "42(3)",
>> >
>> > "pgid": "14.ffs3",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 0,
>> >
>> > "last_backfill": "14:ff09a915:::10000001249.00000353:head",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "0'0",
>> >
>> > "reported_seq": 0,
>> >
>> > "reported_epoch": 0,
>> >
>> > "state": "unknown",
>> >
>> > "last_fresh": "0.000000",
>> >
>> > "last_change": "0.000000",
>> >
>> > "last_active": "0.000000",
>> >
>> > "last_peered": "0.000000",
>> >
>> > "last_clean": "0.000000",
>> >
>> > "last_became_active": "0.000000",
>> >
>> > "last_became_peered": "0.000000",
>> >
>> > "last_unstale": "0.000000",
>> >
>> > "last_undegraded": "0.000000",
>> >
>> > "last_fullsized": "0.000000",
>> >
>> > "mapping_epoch": 0,
>> >
>> > "log_start": "0'0",
>> >
>> > "ondisk_log_start": "0'0",
>> >
>> > "created": 0,
>> >
>> > "last_epoch_clean": 0,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 0,
>> >
>> > "last_scrub": "0'0",
>> >
>> > "last_scrub_stamp": "0.000000",
>> >
>> > "last_deep_scrub": "0'0",
>> >
>> > "last_deep_scrub_stamp": "0.000000",
>> >
>> > "last_clean_scrub_stamp": "0.000000",
>> >
>> > "objects_scrubbed": 0,
>> >
>> > "log_size": 0,
>> >
>> > "ondisk_log_size": 0,
>> >
>> > "stats_invalid": false,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 0,
>> >
>> > "scrub_schedule": "--",
>> >
>> > "scrub_duration": 0,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 2115497390,
>> >
>> > "num_objects": 506,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 0,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 13042,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 506,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 0,
>> >
>> > "num_read_kb": 0,
>> >
>> > "num_write": 42,
>> >
>> > "num_write_kb": 172032,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 0,
>> >
>> > "num_bytes_recovered": 0,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [],
>> >
>> > "acting": [],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": -1,
>> >
>> > "acting_primary": -1,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 1,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "46(5)",
>> >
>> > "pgid": "14.ffs5",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 33714,
>> >
>> > "last_backfill": "MAX",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "16465'33714",
>> >
>> > "reported_seq": 388679,
>> >
>> > "reported_epoch": 16465,
>> >
>> > "state": "active+clean",
>> >
>> > "last_fresh": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_change": "2022-12-26T02:28:03.455705+0000",
>> >
>> > "last_active": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_peered": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_clean": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_became_active":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_became_peered":
>> "2022-12-26T02:28:03.455192+0000",
>> >
>> > "last_unstale": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_undegraded": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "last_fullsized": "2022-12-26T16:10:12.977587+0000",
>> >
>> > "mapping_epoch": 16615,
>> >
>> > "log_start": "12957'31152",
>> >
>> > "ondisk_log_start": "12957'31152",
>> >
>> > "created": 4537,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 8,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "objects_scrubbed": 16227,
>> >
>> > "log_size": 2562,
>> >
>> > "ondisk_log_size": 2562,
>> >
>> > "stats_invalid": true,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 14,
>> >
>> > "scrub_schedule": "queued for deep scrub",
>> >
>> > "scrub_duration": 13.320415128,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 51491812456,
>> >
>> > "num_objects": 12301,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 73806,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 0,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 12301,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 66867,
>> >
>> > "num_read_kb": 177079789,
>> >
>> > "num_write": 19946,
>> >
>> > "num_write_kb": 64929471,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 34017,
>> >
>> > "num_bytes_recovered": 142424959053,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [
>> >
>> > 5,
>> >
>> > 36,
>> >
>> > 58,
>> >
>> > 42,
>> >
>> > 18,
>> >
>> > 24
>> >
>> > ],
>> >
>> > "acting": [
>> >
>> > 50,
>> >
>> > 36,
>> >
>> > 5,
>> >
>> > 26,
>> >
>> > 15,
>> >
>> > 46
>> >
>> > ],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": 5,
>> >
>> > "acting_primary": 50,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 0,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "peer": "58(2)",
>> >
>> > "pgid": "14.ffs2",
>> >
>> > "last_update": "19550'35077",
>> >
>> > "last_complete": "17223'34381",
>> >
>> > "log_tail": "12957'31152",
>> >
>> > "last_user_version": 0,
>> >
>> > "last_backfill": "14:ff09a915:::10000001249.00000353:head",
>> >
>> > "purged_snaps": [],
>> >
>> > "history": {
>> >
>> > "epoch_created": 4537,
>> >
>> > "epoch_pool_created": 2032,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "last_interval_started": 16615,
>> >
>> > "last_epoch_clean": 14655,
>> >
>> > "last_interval_clean": 14654,
>> >
>> > "last_epoch_split": 4537,
>> >
>> > "last_epoch_marked_full": 0,
>> >
>> > "same_up_since": 16613,
>> >
>> > "same_interval_since": 16615,
>> >
>> > "same_primary_since": 16615,
>> >
>> > "last_scrub": "3817'25569",
>> >
>> > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
>> >
>> > "last_deep_scrub": "3756'21592",
>> >
>> > "last_deep_scrub_stamp":
>> "2022-12-14T19:35:51.893008+0000",
>> >
>> > "last_clean_scrub_stamp":
>> "2022-12-17T06:33:40.577932+0000",
>> >
>> > "prior_readable_until_ub": 0
>> >
>> > },
>> >
>> > "stats": {
>> >
>> > "version": "0'0",
>> >
>> > "reported_seq": 0,
>> >
>> > "reported_epoch": 0,
>> >
>> > "state": "unknown",
>> >
>> > "last_fresh": "0.000000",
>> >
>> > "last_change": "0.000000",
>> >
>> > "last_active": "0.000000",
>> >
>> > "last_peered": "0.000000",
>> >
>> > "last_clean": "0.000000",
>> >
>> > "last_became_active": "0.000000",
>> >
>> > "last_became_peered": "0.000000",
>> >
>> > "last_unstale": "0.000000",
>> >
>> > "last_undegraded": "0.000000",
>> >
>> > "last_fullsized": "0.000000",
>> >
>> > "mapping_epoch": 0,
>> >
>> > "log_start": "0'0",
>> >
>> > "ondisk_log_start": "0'0",
>> >
>> > "created": 0,
>> >
>> > "last_epoch_clean": 0,
>> >
>> > "parent": "0.0",
>> >
>> > "parent_split_bits": 0,
>> >
>> > "last_scrub": "0'0",
>> >
>> > "last_scrub_stamp": "0.000000",
>> >
>> > "last_deep_scrub": "0'0",
>> >
>> > "last_deep_scrub_stamp": "0.000000",
>> >
>> > "last_clean_scrub_stamp": "0.000000",
>> >
>> > "objects_scrubbed": 0,
>> >
>> > "log_size": 0,
>> >
>> > "ondisk_log_size": 0,
>> >
>> > "stats_invalid": false,
>> >
>> > "dirty_stats_invalid": false,
>> >
>> > "omap_stats_invalid": false,
>> >
>> > "hitset_stats_invalid": false,
>> >
>> > "hitset_bytes_stats_invalid": false,
>> >
>> > "pin_stats_invalid": false,
>> >
>> > "manifest_stats_invalid": false,
>> >
>> > "snaptrimq_len": 0,
>> >
>> > "last_scrub_duration": 0,
>> >
>> > "scrub_schedule": "--",
>> >
>> > "scrub_duration": 0,
>> >
>> > "objects_trimmed": 0,
>> >
>> > "snaptrim_duration": 0,
>> >
>> > "stat_sum": {
>> >
>> > "num_bytes": 2115497390,
>> >
>> > "num_objects": 506,
>> >
>> > "num_object_clones": 0,
>> >
>> > "num_object_copies": 0,
>> >
>> > "num_objects_missing_on_primary": 0,
>> >
>> > "num_objects_missing": 13042,
>> >
>> > "num_objects_degraded": 0,
>> >
>> > "num_objects_misplaced": 0,
>> >
>> > "num_objects_unfound": 0,
>> >
>> > "num_objects_dirty": 506,
>> >
>> > "num_whiteouts": 0,
>> >
>> > "num_read": 0,
>> >
>> > "num_read_kb": 0,
>> >
>> > "num_write": 42,
>> >
>> > "num_write_kb": 172032,
>> >
>> > "num_scrub_errors": 0,
>> >
>> > "num_shallow_scrub_errors": 0,
>> >
>> > "num_deep_scrub_errors": 0,
>> >
>> > "num_objects_recovered": 0,
>> >
>> > "num_bytes_recovered": 0,
>> >
>> > "num_keys_recovered": 0,
>> >
>> > "num_objects_omap": 0,
>> >
>> > "num_objects_hit_set_archive": 0,
>> >
>> > "num_bytes_hit_set_archive": 0,
>> >
>> > "num_flush": 0,
>> >
>> > "num_flush_kb": 0,
>> >
>> > "num_evict": 0,
>> >
>> > "num_evict_kb": 0,
>> >
>> > "num_promote": 0,
>> >
>> > "num_flush_mode_high": 0,
>> >
>> > "num_flush_mode_low": 0,
>> >
>> > "num_evict_mode_some": 0,
>> >
>> > "num_evict_mode_full": 0,
>> >
>> > "num_objects_pinned": 0,
>> >
>> > "num_legacy_snapsets": 0,
>> >
>> > "num_large_omap_objects": 0,
>> >
>> > "num_objects_manifest": 0,
>> >
>> > "num_omap_bytes": 0,
>> >
>> > "num_omap_keys": 0,
>> >
>> > "num_objects_repaired": 0
>> >
>> > },
>> >
>> > "up": [],
>> >
>> > "acting": [],
>> >
>> > "avail_no_missing": [],
>> >
>> > "object_location_counts": [],
>> >
>> > "blocked_by": [],
>> >
>> > "up_primary": -1,
>> >
>> > "acting_primary": -1,
>> >
>> > "purged_snaps": []
>> >
>> > },
>> >
>> > "empty": 0,
>> >
>> > "dne": 0,
>> >
>> > "incomplete": 1,
>> >
>> > "last_epoch_started": 16616,
>> >
>> > "hit_set_history": {
>> >
>> > "current_last_update": "0'0",
>> >
>> > "history": []
>> >
>> > }
>> >
>> > }
>> >
>> > ],
>> >
>> > "recovery_state": [
>> >
>> > {
>> >
>> > "name": "Started/Primary/Active",
>> >
>> > "enter_time": "2022-12-26T21:27:49.970477+0000",
>> >
>> > "might_have_unfound": [],
>> >
>> > "recovery_progress": {
>> >
>> > "backfill_targets": [
>> >
>> > "5(0)",
>> >
>> > "18(4)",
>> >
>> > "24(5)",
>> >
>> > "42(3)",
>> >
>> > "58(2)"
>> >
>> > ],
>> >
>> > "waiting_on_backfill": [],
>> >
>> > "last_backfill_started":
>> > "14:ff09b920:::10000005377.00000209:head",
>> >
>> > "backfill_info": {
>> >
>> > "begin": "14:ff09bed4:::10000001a7f.0000182d:head",
>> >
>> > "end": "14:ff0a8fd7:::1000000072c.000001a0:head",
>> >
>> > "objects": [
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09bed4:::10000001a7f.0000182d:head",
>> >
>> > "version": "3801'24969"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09c414:::10000004daf.00001063:head",
>> >
>> > "version": "6812'28471"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09c530:::10000003957.00000189:head",
>> >
>> > "version": "9167'29571"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09c607:::100000011db.00000977:head",
>> >
>> > "version": "3703'16620"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09c922:::10000001230.000001e7:head",
>> >
>> > "version": "3728'18321"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09cb55:::1000000454e.000009a3:head",
>> >
>> > "version": "3706'17783"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09ce4d:::10000001316.00000154:head",
>> >
>> > "version": "3798'23751"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09d097:::10000003c4f.00000085:head",
>> >
>> > "version": "3822'26172"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09d449:::1000000188a.0000034a:head",
>> >
>> > "version": "4627'27150"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09d6b4:::10000003c4a.00000b0f:head",
>> >
>> > "version": "3822'26170"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09de4e:::10000001b63.000004c5:head",
>> >
>> > "version": "3608'11396"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09e1e3:::10000002a02.000016cb:head",
>> >
>> > "version": "12960'31316"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09f059:::10000002c09.00000095:head",
>> >
>> > "version": "13226'31609"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff09f069:::1000000279e.0000012f:head",
>> >
>> > "version": "13274'31844"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a0029:::10000001cd0.000000a2:head",
>> >
>> > "version": "3702'16325"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a083b:::100000016c6.0000013e:head",
>> >
>> > "version": "3497'8495"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a0d58:::1000000468d.00001cba:head",
>> >
>> > "version": "9168'29618"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a0dfd:::10000000540.0000064a:head",
>> >
>> > "version": "3493'8171"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a1149:::10000005049.00000b52:head",
>> >
>> > "version": "3529'10013"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a1cdb:::100000009df.0000afec:head",
>> >
>> > "version": "2106'380"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a1e1a:::1000000390b.00000106:head",
>> >
>> > "version": "6812'28456"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a2180:::10000003868.000000b9:head",
>> >
>> > "version": "3706'17681"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a2438:::1000000133e.00000024:head",
>> >
>> > "version": "3798'24057"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a280e:::10000002ccc.000004af:head",
>> >
>> > "version": "15259'33150"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a2f43:::10000003d40.00000665:head",
>> >
>> > "version": "10508'29957"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a315f:::100000011db.00000caa:head",
>> >
>> > "version": "3703'16628"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a3347:::10000004576.00001fb9:head",
>> >
>> > "version": "3732'19654"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a3948:::100000032eb.0000011a:head",
>> >
>> > "version": "3612'11760"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a3af1:::1000000275d.0000027a:head",
>> >
>> > "version": "12960'31292"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a4073:::100000018fe.0000355d:head",
>> >
>> > "version": "3498'8949"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a44db:::10000001a11.00000203:head",
>> >
>> > "version": "3755'21336"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a4888:::1000000246e.0000360a:head",
>> >
>> > "version": "13794'32296"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a4e86:::100000054b1.000019c1:head",
>> >
>> > "version": "12890'30650"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a51c6:::10000003970.0000096e:head",
>> >
>> > "version": "10636'30139"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a5302:::10000001871.000018eb:head",
>> >
>> > "version": "3817'25516"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a56ba:::10000002848.00000266:head",
>> >
>> > "version": "13855'32461"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a61f3:::10000000759.00000030:head",
>> >
>> > "version": "3703'17434"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a68e3:::10000001c17.0000012c:head",
>> >
>> > "version": "3653'13629"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a722a:::10000003e74.00000091:head",
>> >
>> > "version": "3778'22970"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a7557:::100000044fe.00001664:head",
>> >
>> > "version": "3660'14671"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a8f78:::10000001821.00000844:head",
>> >
>> > "version": "3797'23629"
>> >
>> > },
>> >
>> > {
>> >
>> > "object":
>> > "14:ff0a8f8f:::10000001bea.00000202:head",
>> >
>> > "version": "3653'13526"
>> >
>> > }
>> >
>> > ]
>> >
>> > },
>> >
>> > "peer_backfill_info": [
>> >
>> > "5(0)",
>> >
>> > {
>> >
>> > "begin": "MAX",
>> >
>> > "end": "MAX",
>> >
>> > "objects": []
>> >
>> > },
>> >
>> > "18(4)",
>> >
>> > {
>> >
>> > "begin": "MAX",
>> >
>> > "end": "MAX",
>> >
>> > "objects": []
>> >
>> > },
>> >
>> > "24(5)",
>> >
>> > {
>> >
>> > "begin": "MAX",
>> >
>> > "end": "MAX",
>> >
>> > "objects": []
>> >
>> > },
>> >
>> > "42(3)",
>> >
>> > {
>> >
>> > "begin": "MAX",
>> >
>> > "end": "MAX",
>> >
>> > "objects": []
>> >
>> > },
>> >
>> > "58(2)",
>> >
>> > {
>> >
>> > "begin": "MAX",
>> >
>> > "end": "MAX",
>> >
>> > "objects": []
>> >
>> > }
>> >
>> > ],
>> >
>> > "backfills_in_flight": [
>> >
>> > "14:ff09b0d9:::100000033c2.00000146:head",
>> >
>> > "14:ff09b6e5:::10000004b37.00000161:head",
>> >
>> > "14:ff09b7b4:::1000000278f.00000374:head",
>> >
>> > "14:ff09b920:::10000005377.00000209:head"
>> >
>> > ],
>> >
>> > "recovering": [
>> >
>> > "14:ff09b0d9:::100000033c2.00000146:head",
>> >
>> > "14:ff09b6e5:::10000004b37.00000161:head",
>> >
>> > "14:ff09b7b4:::1000000278f.00000374:head",
>> >
>> > "14:ff09b920:::10000005377.00000209:head"
>> >
>> > ],
>> >
>> > "pg_backend": {
>> >
>> > "recovery_ops": [
>> >
>> > {
>> >
>> > "hoid":
>> > "14:ff09b0d9:::100000033c2.00000146:head",
>> >
>> > "v": "3677'15440",
>> >
>> > "missing_on":
>> "5(0),18(4),24(5),42(3),58(2)",
>> >
>> > "missing_on_shards": "0,2,3,4,5",
>> >
>> > "recovery_info":
>> > "ObjectRecoveryInfo(14:ff09b0d9:::100000033c2.00000146:head@3677'15440,
>> > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
>> > object_exist: 1)",
>> >
>> > "recovery_progress":
>> > "ObjectRecoveryProgress(!first, data_recovered_to:4194304,
>> > data_complete:true, omap_recovered_to:, omap_complete:true,
>> error:false)",
>> >
>> > "state": "WRITING",
>> >
>> > "waiting_on_pushes": "5(0)",
>> >
>> > "extent_requested": "0,8388608"
>> >
>> > },
>> >
>> > {
>> >
>> > "hoid":
>> > "14:ff09b6e5:::10000004b37.00000161:head",
>> >
>> > "v": "3549'10993",
>> >
>> > "missing_on":
>> "5(0),18(4),24(5),42(3),58(2)",
>> >
>> > "missing_on_shards": "0,2,3,4,5",
>> >
>> > "recovery_info":
>> > "ObjectRecoveryInfo(14:ff09b6e5:::10000004b37.00000161:head@3549'10993,
>> > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
>> > object_exist: 1)",
>> >
>> > "recovery_progress":
>> > "ObjectRecoveryProgress(!first, data_recovered_to:4194304,
>> > data_complete:true, omap_recovered_to:, omap_complete:true,
>> error:false)",
>> >
>> > "state": "WRITING",
>> >
>> > "waiting_on_pushes": "5(0)",
>> >
>> > "extent_requested": "0,8388608"
>> >
>> > },
>> >
>> > {
>> >
>> > "hoid":
>> > "14:ff09b7b4:::1000000278f.00000374:head",
>> >
>> > "v": "13272'31836",
>> >
>> > "missing_on":
>> "5(0),18(4),24(5),42(3),58(2)",
>> >
>> > "missing_on_shards": "0,2,3,4,5",
>> >
>> > "recovery_info":
>> > "ObjectRecoveryInfo(14:ff09b7b4:::1000000278f.00000374:head@13272
>> '31836,
>> > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
>> > object_exist: 1)",
>> >
>> > "recovery_progress":
>> > "ObjectRecoveryProgress(!first, data_recovered_to:4194304,
>> > data_complete:true, omap_recovered_to:, omap_complete:true,
>> error:false)",
>> >
>> > "state": "WRITING",
>> >
>> > "waiting_on_pushes": "5(0)",
>> >
>> > "extent_requested": "0,8388608"
>> >
>> > },
>> >
>> > {
>> >
>> > "hoid":
>> > "14:ff09b920:::10000005377.00000209:head",
>> >
>> > "v": "10440'29880",
>> >
>> > "missing_on":
>> "5(0),18(4),24(5),42(3),58(2)",
>> >
>> > "missing_on_shards": "0,2,3,4,5",
>> >
>> > "recovery_info":
>> > "ObjectRecoveryInfo(14:ff09b920:::10000005377.00000209:head@10440
>> '29880,
>> > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
>> > object_exist: 1)",
>> >
>> > "recovery_progress":
>> > "ObjectRecoveryProgress(!first, data_recovered_to:4194304,
>> > data_complete:true, omap_recovered_to:, omap_complete:true,
>> error:false)",
>> >
>> > "state": "WRITING",
>> >
>> > "waiting_on_pushes": "5(0)",
>> >
>> > "extent_requested": "0,8388608"
>> >
>> > }
>> >
>> > ],
>> >
>> > "read_ops": []
>> >
>> > }
>> >
>> > }
>> >
>> > },
>> >
>> > {
>> >
>> > "name": "Started",
>> >
>> > "enter_time": "2022-12-26T21:27:48.908226+0000"
>> >
>> > }
>> >
>> > ],
>> >
>> > "scrubber": {
>> >
>> > "active": false,
>> >
>> > "must_scrub": true,
>> >
>> > "must_deep_scrub": true,
>> >
>> > "must_repair": false,
>> >
>> > "need_auto": false,
>> >
>> > "scrub_reg_stamp": "1.000000",
>> >
>> > "schedule": "queued for deep scrub"
>> >
>> > },
>> >
>> > "agent_state": {}
>> >
>> > }
>> >
>> >
>> >
>> > On Wed, Dec 28, 2022 at 6:46 AM Pavin Joseph <me@xxxxxxxxxxxxxxx>
>> wrote:
>> >
>> >> 1. This is a guess, but check /var/[lib|run]/ceph for any lock files.
>> >> 2. This is more straightforward to fix, add faster WAL/Block device/LV
>> >> for each OSD or create a fast storage pool just for metadata. Also,
>> >> experiment with MDS cache size/trim [0] settings.
>> >>
>> >> [0]: https://docs.ceph.com/en/latest/cephfs/cache-configuration/
>> >>
>> >> On 28-Dec-22 7:23 AM, Deep Dish wrote:
>> >>> Got logging enabled as per
>> >>> https://ceph.io/en/news/blog/2022/centralized_logging/. My embedded
>> >>> grafana doesn't come up in the dashboard, but at least I have log
>> (files)
>> >>> on my nodes. Interesting.
>> >>>
>> >>> Two issues plaguing my cluster:
>> >>>
>> >>> 1 - RGWs not manageable
>> >>> 2 - MDS_SLOW_METADATA_IO warning (impact to cephfs)
>> >>>
>> >>> Issue 1:
>> >>>
>> >>> I have 4x RGWs deployed. All started / processes running. They all
>> >>> report similar log entries:
>> >>>
>> >>> 7fcc32b6a5c0 0 deferred set uid:gid to 167:167 (ceph:ceph)
>> >>>
>> >>> 7fcc32b6a5c0 0 ceph version 17.2.5
>> >>> (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
>> >>> radosgw, pid 2
>> >>>
>> >>> 7fcc32b6a5c0 0 framework: beast
>> >>>
>> >>> 7fcc32b6a5c0 0 framework conf key: port, val: 80
>> >>>
>> >>> 7fcc32b6a5c0 1 radosgw_Main not setting numa affinity
>> >>>
>> >>> 7fcc32b6a5c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0
>> >>>
>> >>> 7fcc32b6a5c0 1 D3N datacache enabled: 0
>> >>>
>> >>> 7fcc0869a700 0 INFO: RGWReshardLock::lock found lock on
>> >> reshard.0000000011
>> >>> to be held by another RGW process; skipping for now
>> >>>
>> >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >> lc.1,
>> >>> sleep 5, try again
>> >>>
>> >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >> lc.3,
>> >>> sleep 5, try again
>> >>>
>> >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>>
>> >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>>
>> >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>>
>> >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>>
>> >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>>
>> >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>>
>> >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>>
>> >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>>
>> >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
>> >>> lc.16, sleep 5, try again
>> >>> (repeating)
>> >>>
>> >>> Seems like a stale lock, not previously cleaned up when the cluster
>> was
>> >>> busy recovering and rebalancing.
>> >>>
>> >>> Issue 2:
>> >>>
>> >>> ceph health detail:
>> >>>
>> >>> [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
>> >>>
>> >>> mds.fs01.ceph02mon03.rjcxat(mds.0): 8 slow metadata IOs are
>> blocked
>> >>>
>> >>> 30 secs, oldest blocked for 39485 secs
>> >>>
>> >>> Log entries from ceph02mon03 MDS host:
>> >>>
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131271 from mon.4
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131272 from mon.4
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131273 from mon.4
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131274 from mon.4
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131275 from mon.4
>> >>> 7fe36c6b8700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1
>> >>> included below; oldest blocked for > 33.126589 secs
>> >>> 7fe36c6b8700 0 log_channel(cluster) log [WRN] : slow request
>> 33.126588
>> >>> seconds old, received at 2022-12-27T19:45:45.952225+0000:
>> >>> client_request(client.55009:99980 create
>> >>> #0x10000000bc2/vzdump-qemu-30003-2022_12_27-14_43_43.log
>> >>> 2022-12-27T19:45:45.948045+0000 caller_uid=0, caller_gid=0{})
>> currently
>> >>> submit entry: journal_and_reply
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131276 from mon.4
>> >>> 7fe36c6b8700 0 log_channel(cluster) log [WRN] : 1 slow requests, 0
>> >>> included below; oldest blocked for > 38.126737 secs
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131277 from mon.4
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131278 from mon.4
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131279 from mon.4
>> >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
>> version
>> >>> 131280 from mon.4
>> >>>
>> >>>
>> >>> I suspect that the file in the log above int's the culprit. How can
>> I
>> >> get
>> >>> to the root cause of MDS slowdowns?
>> >>>
>> >>>
>> >>> On Tue, Dec 27, 2022 at 3:32 PM Pavin Joseph <me@xxxxxxxxxxxxxxx>
>> wrote:
>> >>>
>> >>>> Interesting, the logs show the crash module [0] itself has crashed.
>> >>>> Something sent it a SIGINT or SIGTERM and the module didn't handle it
>> >>>> correctly due to what seems like a bug in the code.
>> >>>>
>> >>>> I haven't experienced the crash module itself crashing yet (in
>> Quincy)
>> >>>> because nothing sent a SIG[INT|TERM] to it yet.
>> >>>>
>> >>>> So I'd continue investigating into why these signals were sent to the
>> >>>> crash module.
>> >>>>
>> >>>> To fix the crash module from crashing, go to "/usr/bin/ceph-crash"
>> and
>> >>>> edit the handler function on line 82 like so:
>> >>>>
>> >>>> def handler(signum, frame):
>> >>>> print('*** Interrupted with signal %d ***' % signum)
>> >>>> signame = signal.Signals(signum).name
>> >>>> print(f'Signal handler called with signal {signame} ({signum})')
>> >>>> print(frame)
>> >>>> sys.exit(0)
>> >>>>
>> >>>> ---
>> >>>>
>> >>>> Once the crash module is working, perhaps you could run a "ceph crash
>> >> ls"
>> >>>>
>> >>>> Regarding podman logs, perhaps try this [1].
>> >>>>
>> >>>> [0]: https://docs.ceph.com/en/quincy/mgr/crash/
>> >>>> [1]: https://docs.podman.io/en/latest/markdown/podman-logs.1.html
>> >>>>
>> >>>> On 27-Dec-22 11:59 PM, Deep Dish wrote:
>> >>>>> HI Pavin,
>> >>>>>
>> >>>>> Thanks for the reply. I'm a bit at a loss honestly as this worked
>> >>>>> perfectly without any issue up until the rebalance of the cluster.
>> >>>>> Orchestrator is great. Aside from this (which I suspect is not
>> >>>>> orchestrator related), I haven't had any issues.
>> >>>>>
>> >>>>> In terms of logs, I'm not sure where to start looking in this new
>> >>>>> containerized environment as they pertain to individual ceph
>> processes
>> >>>> -- I
>> >>>>> assumed everything would be centrally collected within orch.
>> >>>>>
>> >>>>> Connecting into the podman container of a RGW, there are no logs in
>> >>>>> /var/log/ceph aside from ceph-volume. My ceph.conf is minimal with
>> >> only
>> >>>>> monitors defined. The only log I'm able to pull is as follows:
>> >>>>>
>> >>>>> # podman logs 35d4ac5445ca
>> >>>>>
>> >>>>> INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
>> >>>>>
>> >>>>> Traceback (most recent call last):
>> >>>>>
>> >>>>> File "/usr/bin/ceph-crash", line 113, in <module>
>> >>>>>
>> >>>>> main()
>> >>>>>
>> >>>>> File "/usr/bin/ceph-crash", line 109, in main
>> >>>>>
>> >>>>> time.sleep(args.delay * 60)
>> >>>>>
>> >>>>> TypeError: handler() takes 1 positional argument but 2 were given
>> >>>>>
>> >>>>> INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Looks like the RGW daemon is crashing. How do I get logs to
>> persist?
>> >>>> I
>> >>>>> suspect I won't be able to use orchestrator to push down the config,
>> >> and
>> >>>>> would have to manipulate within the container image itself.
>> >>>>>
>> >>>>> I also attempted to redeply the RGW containers without success.
>> >>>>>
>> >>>>> On Tue, Dec 27, 2022 at 10:39 AM Pavin Joseph <me@xxxxxxxxxxxxxxx>
>> >>>> wrote:
>> >>>>>
>> >>>>>> Here's the first things I'd check in your situation:
>> >>>>>>
>> >>>>>> 1. Logs
>> >>>>>> 2. Is the RGW HTTP server running on its port?
>> >>>>>> 3. Re-check config including authentication.
>> >>>>>>
>> >>>>>> ceph orch is too new and didn't pass muster in our own internal
>> >> testing.
>> >>>>>> You're braver than most for using it in production.
>> >>>>>>
>> >>>>>> Pavin.
>> >>>>>>
>> >>>>>> On 27-Dec-22 8:48 PM, Deep Dish wrote:
>> >>>>>>> Quick update:
>> >>>>>>>
>> >>>>>>> - I followed documentation, and ran the following:
>> >>>>>>>
>> >>>>>>> # ceph dashboard set-rgw-credentials
>> >>>>>>>
>> >>>>>>> Error EINVAL: No RGW credentials found, please consult the
>> >>>> documentation
>> >>>>>> on
>> >>>>>>> how to enable RGW for the dashboard.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> - I see dashboard credentials configured (all this was working
>> fine
>> >>>>>> before):
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> # ceph dashboard get-rgw-api-access-key
>> >>>>>>>
>> >>>>>>> P?????????????????G (? commented out)
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Seems to me like my RGW config is non-existent / corrupted for
>> some
>> >>>>>>> reason. When trying to curl a RGW directly I get a "connection
>> >>>> refused".
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, Dec 27, 2022 at 9:41 AM Deep Dish <deeepdish@xxxxxxxxx>
>> >> wrote:
>> >>>>>>>
>> >>>>>>>> I built a net-new Quincy cluster (17.2.5) using ceph orch as
>> >> follows:
>> >>>>>>>>
>> >>>>>>>> 2x mgrs
>> >>>>>>>> 4x rgw
>> >>>>>>>> 5x mon
>> >>>>>>>> 4x rgw
>> >>>>>>>> 5x mds
>> >>>>>>>> 6x osd hosts w/ 10 drives each --> will be growing to 7 osd
>> hosts in
>> >>>> the
>> >>>>>>>> coming days.
>> >>>>>>>>
>> >>>>>>>> I migrated all data from my legacy nautilus cluster (via
>> rbd-mirror,
>> >>>>>>>> rclone for s3 buckets, etc.). All moved over successfully
>> without
>> >>>>>> issue.
>> >>>>>>>>
>> >>>>>>>> The cluster went through a series of rebalancing events (adding
>> >>>>>> capacity,
>> >>>>>>>> osd nodes, changing fault domain for EC volumes).
>> >>>>>>>>
>> >>>>>>>> It's settled now, however throughout the process all of my RGW
>> nodes
>> >>>> are
>> >>>>>>>> no longer part of the cluster -- meaning ceph doesn't recognize /
>> >>>> detect
>> >>>>>>>> them, despite containers, networking, etc. all being setup
>> >> correctly.
>> >>>>>>>> This also means I'm unable to manage any RGW functions (via the
>> >>>>>> dashboard
>> >>>>>>>> or cli). As an example via cli (within Cephadm shell):
>> >>>>>>>>
>> >>>>>>>> # radosgw-admin pools list
>> >>>>>>>>
>> >>>>>>>> could not list placement set: (2) No such file or directory
>> >>>>>>>>
>> >>>>>>>> I have data in buckets, how can I get my RGWs to return online?
>> >>>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>> >>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >>>>>>
>> >>>>> _______________________________________________
>> >>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>> >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >>>>
>> >>> _______________________________________________
>> >>> ceph-users mailing list -- ceph-users@xxxxxxx
>> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >>
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
