Re: Urgent help! RGW Disappeared on Quincy

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Pavin,

Happy New Year!

Many thanks for the commands.  I managed to get the cluster into green
status with the repeer command.

Still had some slow MDS ops, I decided to purge a few older backups in a
repository that's backed by one of the cephfs volumes.   This resolved all
slow OPS -- there must have been an inconsistent file causing an issue, not
a big deal as it's a backup (likely manifested itself when cluster was
rebuilding / rebalancing).

I'm still having an issue in getting RGWs to be manageable / detected by
the cluster (any ideas appreciated):
- Increased debug logging to 5/5 across debug_rgw,debug_rgw_datacache,
debug_rgw_sync
- Redeployed RGW service (removed and recreated service);
- All GWs show similar log entries -- There's a failed init realm ID error
(could that be it?):

2022-12-31T20:02:55.570+0000 7fea6d76a5c0  0 deferred set uid:gid to
167:167 (ceph:ceph)

2022-12-31T20:02:55.570+0000 7fea6d76a5c0  0 ceph version 17.2.5
(98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
radosgw, pid 2

2022-12-31T20:02:55.570+0000 7fea6d76a5c0  0 framework: beast

2022-12-31T20:02:55.570+0000 7fea6d76a5c0  0 framework conf key: port, val:
80

2022-12-31T20:02:55.570+0000 7fea6d76a5c0  1 radosgw_Main not setting numa
affinity

2022-12-31T20:02:55.574+0000 7fea6d76a5c0  1 rgw_d3n:
rgw_d3n_l1_local_datacache_enabled=0

2022-12-31T20:02:55.574+0000 7fea6d76a5c0  1 D3N datacache enabled: 0

2022-12-31T20:02:55.590+0000 7fea6d76a5c0  4 rgw main: RGWPeriod::init
failed to init realm  id  : (2) No such file or directory

2022-12-31T20:02:55.686+0000 7fea6d76a5c0  4 rgw main: Realm:
            ()

2022-12-31T20:02:55.686+0000 7fea6d76a5c0  4 rgw main: ZoneGroup: default
            (45518452-8aa6-41b4-99f0-059b255c31cd)

2022-12-31T20:02:55.686+0000 7fea6d76a5c0  4 rgw main: Zone:      default
            (743ea532-f5bc-4cca-891b-c27a586d5129)

2022-12-31T20:02:56.010+0000 7fea6d76a5c0  2 all 8 watchers are set,
enabling cache

2022-12-31T20:02:56.026+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:02:57.170+0000 7fea4eab1700  2 garbage collection: garbage
collection: start

2022-12-31T20:02:57.170+0000 7fea4e2b0700  2 rgw object expirer Worker
thread: object expiration: start

2022-12-31T20:02:57.170+0000 7fea48aa5700  5 lifecycle: schedule life cycle
next start time: Sun Jan  1 00:00:00 2023

2022-12-31T20:02:57.170+0000 7fea46aa1700  5 lifecycle: schedule life cycle
next start time: Sun Jan  1 00:00:00 2023

2022-12-31T20:02:57.170+0000 7fea44a9d700  5 lifecycle: schedule life cycle
next start time: Sun Jan  1 00:00:00 2023

2022-12-31T20:02:59.186+0000 7fea4eab1700  2 garbage collection: garbage
collection: stop

2022-12-31T20:03:18.029+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:03:40.029+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:04:02.028+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:04:24.027+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:04:46.031+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:05:08.030+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:05:30.030+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:05:52.029+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:06:14.029+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:06:36.028+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:06:58.032+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:07:20.031+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:07:42.031+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:08:04.030+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:08:26.029+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:08:48.029+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:09:10.032+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:09:32.028+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start

2022-12-31T20:09:54.027+0000 7fea52ab9700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start



On Sat, Dec 31, 2022 at 1:22 AM Pavin Joseph <me@xxxxxxxxxxxxxxx> wrote:

> Hey there,
>
> Sorry for the late reply.
> If the pg issue isn't solved yet, could you run these:
>
> ceph pg repeer <pgid>
> ceph pg repair <pgid>
>
> Pavin.
>
> On 29-Dec-22 4:08 AM, Deep Dish wrote:
> > Hi Pavin,
> >
> > The following are additional developments..  There's one PG that's
> > stuck and unable to recover.   I've attached relevant ceph -s / health
> > detail and pg stat outputs below.
> >
> > - There were some remaining lock files as suggested in /var/run/ceph/
> > pertaining to rgw.   I removed the service, deleted any stale lock files
> > and redeployed the RGWs.   All started with the common log entries across
> > all:
> >
> > 7ff5d9aaf5c0  0 deferred set uid:gid to 167:167 (ceph:ceph)
> >
> > 7ff5d9aaf5c0  0 ceph version 17.2.5
> > (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
> > radosgw, pid 2
> >
> > 7ff5d9aaf5c0  0 framework: beast
> >
> > 7ff5d9aaf5c0  0 framework conf key: port, val: 80
> >
> > 7ff5d9aaf5c0  1 radosgw_Main not setting numa affinity
> >
> > 7ff5d9aaf5c0  1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0
> >
> > 7ff5d9aaf5c0  1 D3N datacache enabled: 0
> >
> > No additional log entries are recorded since starting them post
> > re-deployment as per above.
> >
> > The cluster settled, there is no recovery activity.  There is one pg
> that's
> > stuck and I have a hunch that it's impacting MDS and RGW processes as
> > stated in the thread.   PG is stuck as as active+remapped+backfilling:
> >
> >
> >
> >    data:
> >
> >      volumes: 2/2 healthy
> >
> >      pools:   16 pools, 1504 pgs
> >
> >      objects: 24.49M objects, 79 TiB
> >
> >      usage:   119 TiB used, 390 TiB / 508 TiB avail
> >
> >      pgs:     65210/146755179 objects misplaced (0.044%)
> >
> >               1503 active+clean
> >
> >               1    active+remapped+backfilling
> >
> >
> >
> >    progress:
> >
> >      Global Recovery Event (6h)
> >
> >        [===========================.] (remaining: 73s)
> >
> > # ceph health detail
> >
> > HEALTH_WARN 1 MDSs report slow metadata IOs; 1 pgs not deep-scrubbed in
> > time; 1 pgs not scrubbed in time
> >
> > [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
> >
> >      mds.fs01.ceph02mon02.wicrdz(mds.0): 5 slow metadata IOs are blocked
> >
> > 30 secs, oldest blocked for 74436 secs
> >
> > [WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
> >
> >      pg 14.ff not deep-scrubbed since 2022-12-14T19:35:51.893008+0000
> >
> > [WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
> >
> >      pg 14.ff not scrubbed since 2022-12-17T06:33:40.577932+0000
> >
> >
> >
> >  From the following pg query:
> >
> > - "pgid": "14.ffs0" is stuck as peering (osd 5)
> > - "pgid": "14.ffs4" is stuck as unknown (osd 18)
> > - "pgid": "14.ffs5" is stuck as unknown (osd 24)
> > - "pgid": "14.ffs3" is stuck as unknown (osd 42)
> > - "pgid": "14.ffs2" is stick as unknown (osd 58)
> > - "pgid": "14.ffs1" is marked as active+clean (osd 36)
> >
> > # ceph pg 14.ff query
> >
> > {
> >
> >      "snap_trimq": "[]",
> >
> >      "snap_trimq_len": 0,
> >
> >      "state": "active+remapped+backfilling",
> >
> >      "epoch": 19594,
> >
> >      "up": [
> >
> >          5,
> >
> >          36,
> >
> >          58,
> >
> >          42,
> >
> >          18,
> >
> >          24
> >
> >      ],
> >
> >      "acting": [
> >
> >          50,
> >
> >          36,
> >
> >          5,
> >
> >          26,
> >
> >          15,
> >
> >          46
> >
> >      ],
> >
> >      "backfill_targets": [
> >
> >          "5(0)",
> >
> >          "18(4)",
> >
> >          "24(5)",
> >
> >          "42(3)",
> >
> >          "58(2)"
> >
> >      ],
> >
> >      "acting_recovery_backfill": [
> >
> >          "5(0)",
> >
> >          "5(2)",
> >
> >          "15(4)",
> >
> >          "18(4)",
> >
> >          "24(5)",
> >
> >          "26(3)",
> >
> >          "36(1)",
> >
> >          "42(3)",
> >
> >          "46(5)",
> >
> >          "50(0)",
> >
> >          "58(2)"
> >
> >      ],
> >
> >      "info": {
> >
> >          "pgid": "14.ffs0",
> >
> >          "last_update": "19550'35077",
> >
> >          "last_complete": "19550'35077",
> >
> >          "log_tail": "13761'32157",
> >
> >          "last_user_version": 35077,
> >
> >          "last_backfill": "MAX",
> >
> >          "purged_snaps": [],
> >
> >          "history": {
> >
> >              "epoch_created": 4537,
> >
> >              "epoch_pool_created": 2032,
> >
> >              "last_epoch_started": 16616,
> >
> >              "last_interval_started": 16615,
> >
> >              "last_epoch_clean": 14655,
> >
> >              "last_interval_clean": 14654,
> >
> >              "last_epoch_split": 4537,
> >
> >              "last_epoch_marked_full": 0,
> >
> >              "same_up_since": 16613,
> >
> >              "same_interval_since": 16615,
> >
> >              "same_primary_since": 16615,
> >
> >              "last_scrub": "3817'25569",
> >
> >              "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >              "last_deep_scrub": "3756'21592",
> >
> >              "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
> >
> >              "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >              "prior_readable_until_ub": 0
> >
> >          },
> >
> >          "stats": {
> >
> >              "version": "19550'35077",
> >
> >              "reported_seq": 396919,
> >
> >              "reported_epoch": 19594,
> >
> >              "state": "active+remapped+backfilling",
> >
> >              "last_fresh": "2022-12-28T22:03:20.278478+0000",
> >
> >              "last_change": "2022-12-26T21:27:51.600940+0000",
> >
> >              "last_active": "2022-12-28T22:03:20.278478+0000",
> >
> >              "last_peered": "2022-12-28T22:03:20.278478+0000",
> >
> >              "last_clean": "2022-12-26T21:27:45.471954+0000",
> >
> >              "last_became_active": "2022-12-26T21:27:51.085966+0000",
> >
> >              "last_became_peered": "2022-12-26T21:27:51.085966+0000",
> >
> >              "last_unstale": "2022-12-28T22:03:20.278478+0000",
> >
> >              "last_undegraded": "2022-12-28T22:03:20.278478+0000",
> >
> >              "last_fullsized": "2022-12-28T22:03:20.278478+0000",
> >
> >              "mapping_epoch": 16615,
> >
> >              "log_start": "13761'32157",
> >
> >              "ondisk_log_start": "13761'32157",
> >
> >              "created": 4537,
> >
> >              "last_epoch_clean": 14655,
> >
> >              "parent": "0.0",
> >
> >              "parent_split_bits": 8,
> >
> >              "last_scrub": "3817'25569",
> >
> >              "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >              "last_deep_scrub": "3756'21592",
> >
> >              "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
> >
> >              "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >              "objects_scrubbed": 16227,
> >
> >              "log_size": 2920,
> >
> >              "ondisk_log_size": 2920,
> >
> >              "stats_invalid": true,
> >
> >              "dirty_stats_invalid": false,
> >
> >              "omap_stats_invalid": false,
> >
> >              "hitset_stats_invalid": false,
> >
> >              "hitset_bytes_stats_invalid": false,
> >
> >              "pin_stats_invalid": false,
> >
> >              "manifest_stats_invalid": false,
> >
> >              "snaptrimq_len": 0,
> >
> >              "last_scrub_duration": 14,
> >
> >              "scrub_schedule": "queued for deep scrub",
> >
> >              "scrub_duration": 13.320415128,
> >
> >              "objects_trimmed": 0,
> >
> >              "snaptrim_duration": 0,
> >
> >              "stat_sum": {
> >
> >                  "num_bytes": 56709530650,
> >
> >                  "num_objects": 13548,
> >
> >                  "num_object_clones": 0,
> >
> >                  "num_object_copies": 81288,
> >
> >                  "num_objects_missing_on_primary": 0,
> >
> >                  "num_objects_missing": 0,
> >
> >                  "num_objects_degraded": 0,
> >
> >                  "num_objects_misplaced": 65210,
> >
> >                  "num_objects_unfound": 0,
> >
> >                  "num_objects_dirty": 13548,
> >
> >                  "num_whiteouts": 0,
> >
> >                  "num_read": 67760,
> >
> >                  "num_read_kb": 177798674,
> >
> >                  "num_write": 21231,
> >
> >                  "num_write_kb": 70024901,
> >
> >                  "num_scrub_errors": 0,
> >
> >                  "num_shallow_scrub_errors": 0,
> >
> >                  "num_deep_scrub_errors": 0,
> >
> >                  "num_objects_recovered": 34481,
> >
> >                  "num_bytes_recovered": 144364295675,
> >
> >                  "num_keys_recovered": 0,
> >
> >                  "num_objects_omap": 0,
> >
> >                  "num_objects_hit_set_archive": 0,
> >
> >                  "num_bytes_hit_set_archive": 0,
> >
> >                  "num_flush": 0,
> >
> >                  "num_flush_kb": 0,
> >
> >                  "num_evict": 0,
> >
> >                  "num_evict_kb": 0,
> >
> >                  "num_promote": 0,
> >
> >                  "num_flush_mode_high": 0,
> >
> >                  "num_flush_mode_low": 0,
> >
> >                  "num_evict_mode_some": 0,
> >
> >                  "num_evict_mode_full": 0,
> >
> >                  "num_objects_pinned": 0,
> >
> >                  "num_legacy_snapsets": 0,
> >
> >                  "num_large_omap_objects": 0,
> >
> >                  "num_objects_manifest": 0,
> >
> >                  "num_omap_bytes": 0,
> >
> >                  "num_omap_keys": 0,
> >
> >                  "num_objects_repaired": 0
> >
> >              },
> >
> >              "up": [
> >
> >                  5,
> >
> >                  36,
> >
> >                  58,
> >
> >                  42,
> >
> >                  18,
> >
> >                  24
> >
> >              ],
> >
> >              "acting": [
> >
> >                  50,
> >
> >                  36,
> >
> >                  5,
> >
> >                  26,
> >
> >                  15,
> >
> >                  46
> >
> >              ],
> >
> >              "avail_no_missing": [
> >
> >                  "50(0)",
> >
> >                  "5(2)",
> >
> >                  "15(4)",
> >
> >                  "26(3)",
> >
> >                  "36(1)",
> >
> >                  "46(5)"
> >
> >              ],
> >
> >              "object_location_counts": [
> >
> >                  {
> >
> >                      "shards": "5(2),15(4),26(3),36(1),46(5),50(0)",
> >
> >                      "objects": 13548
> >
> >                  }
> >
> >              ],
> >
> >              "blocked_by": [],
> >
> >              "up_primary": 5,
> >
> >              "acting_primary": 50,
> >
> >              "purged_snaps": []
> >
> >          },
> >
> >          "empty": 0,
> >
> >          "dne": 0,
> >
> >          "incomplete": 0,
> >
> >          "last_epoch_started": 16616,
> >
> >          "hit_set_history": {
> >
> >              "current_last_update": "0'0",
> >
> >              "history": []
> >
> >          }
> >
> >      },
> >
> >      "peer_info": [
> >
> >          {
> >
> >              "peer": "5(0)",
> >
> >              "pgid": "14.ffs0",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 0,
> >
> >              "last_backfill": "14:ff09a915:::10000001249.00000353:head",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "0'0",
> >
> >                  "reported_seq": 2,
> >
> >                  "reported_epoch": 16614,
> >
> >                  "state": "peering",
> >
> >                  "last_fresh": "2022-12-26T21:27:47.993646+0000",
> >
> >                  "last_change": "2022-12-26T21:27:47.769141+0000",
> >
> >                  "last_active": "0.000000",
> >
> >                  "last_peered": "0.000000",
> >
> >                  "last_clean": "0.000000",
> >
> >                  "last_became_active": "0.000000",
> >
> >                  "last_became_peered": "0.000000",
> >
> >                  "last_unstale": "2022-12-26T21:27:47.993646+0000",
> >
> >                  "last_undegraded": "2022-12-26T21:27:47.993646+0000",
> >
> >                  "last_fullsized": "2022-12-26T21:27:47.993646+0000",
> >
> >                  "mapping_epoch": 16615,
> >
> >                  "log_start": "0'0",
> >
> >                  "ondisk_log_start": "0'0",
> >
> >                  "created": 4537,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 0,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "objects_scrubbed": 0,
> >
> >                  "log_size": 0,
> >
> >                  "ondisk_log_size": 0,
> >
> >                  "stats_invalid": false,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 0,
> >
> >                  "scrub_schedule": "queued for deep scrub",
> >
> >                  "scrub_duration": 0,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 2115497390,
> >
> >                      "num_objects": 506,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 0,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 13042,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 506,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 0,
> >
> >                      "num_read_kb": 0,
> >
> >                      "num_write": 42,
> >
> >                      "num_write_kb": 172032,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 0,
> >
> >                      "num_bytes_recovered": 0,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [
> >
> >                      5,
> >
> >                      36,
> >
> >                      58,
> >
> >                      42,
> >
> >                      18,
> >
> >                      24
> >
> >                  ],
> >
> >                  "acting": [
> >
> >                      50,
> >
> >                      36,
> >
> >                      5,
> >
> >                      26,
> >
> >                      15,
> >
> >                      46
> >
> >                  ],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [
> >
> >                      24
> >
> >                  ],
> >
> >                  "up_primary": 5,
> >
> >                  "acting_primary": 50,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 1,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "5(2)",
> >
> >              "pgid": "14.ffs2",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 33714,
> >
> >              "last_backfill": "MAX",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "16465'33714",
> >
> >                  "reported_seq": 388679,
> >
> >                  "reported_epoch": 16465,
> >
> >                  "state": "active+clean",
> >
> >                  "last_fresh": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_change": "2022-12-26T02:28:03.455705+0000",
> >
> >                  "last_active": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_peered": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_clean": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_became_active": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_became_peered": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_unstale": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_undegraded": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_fullsized": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "mapping_epoch": 16615,
> >
> >                  "log_start": "12957'31152",
> >
> >                  "ondisk_log_start": "12957'31152",
> >
> >                  "created": 4537,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 8,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "objects_scrubbed": 16227,
> >
> >                  "log_size": 2562,
> >
> >                  "ondisk_log_size": 2562,
> >
> >                  "stats_invalid": true,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 14,
> >
> >                  "scrub_schedule": "queued for deep scrub",
> >
> >                  "scrub_duration": 13.320415128,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 51491812456,
> >
> >                      "num_objects": 12301,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 73806,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 0,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 12301,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 66867,
> >
> >                      "num_read_kb": 177079789,
> >
> >                      "num_write": 19946,
> >
> >                      "num_write_kb": 64929471,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 34017,
> >
> >                      "num_bytes_recovered": 142424959053,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [
> >
> >                      5,
> >
> >                      36,
> >
> >                      58,
> >
> >                      42,
> >
> >                      18,
> >
> >                      24
> >
> >                  ],
> >
> >                  "acting": [
> >
> >                      50,
> >
> >                      36,
> >
> >                      5,
> >
> >                      26,
> >
> >                      15,
> >
> >                      46
> >
> >                  ],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": 5,
> >
> >                  "acting_primary": 50,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 0,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "15(4)",
> >
> >              "pgid": "14.ffs4",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 33714,
> >
> >              "last_backfill": "MAX",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "16465'33714",
> >
> >                  "reported_seq": 388679,
> >
> >                  "reported_epoch": 16465,
> >
> >                  "state": "active+clean",
> >
> >                  "last_fresh": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_change": "2022-12-26T02:28:03.455705+0000",
> >
> >                  "last_active": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_peered": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_clean": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_became_active": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_became_peered": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_unstale": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_undegraded": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_fullsized": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "mapping_epoch": 16615,
> >
> >                  "log_start": "12957'31152",
> >
> >                  "ondisk_log_start": "12957'31152",
> >
> >                  "created": 4537,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 8,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "objects_scrubbed": 16227,
> >
> >                  "log_size": 2562,
> >
> >                  "ondisk_log_size": 2562,
> >
> >                  "stats_invalid": true,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 14,
> >
> >                  "scrub_schedule": "queued for deep scrub",
> >
> >                  "scrub_duration": 13.320415128,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 51491812456,
> >
> >                      "num_objects": 12301,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 73806,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 0,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 12301,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 66867,
> >
> >                      "num_read_kb": 177079789,
> >
> >                      "num_write": 19946,
> >
> >                      "num_write_kb": 64929471,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 34017,
> >
> >                      "num_bytes_recovered": 142424959053,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [
> >
> >                      5,
> >
> >                      36,
> >
> >                      58,
> >
> >                      42,
> >
> >                      18,
> >
> >                      24
> >
> >                  ],
> >
> >                  "acting": [
> >
> >                      50,
> >
> >                      36,
> >
> >                      5,
> >
> >                      26,
> >
> >                      15,
> >
> >                      46
> >
> >                  ],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": 5,
> >
> >                  "acting_primary": 50,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 0,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "18(4)",
> >
> >              "pgid": "14.ffs4",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 0,
> >
> >              "last_backfill": "14:ff09a915:::10000001249.00000353:head",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "0'0",
> >
> >                  "reported_seq": 0,
> >
> >                  "reported_epoch": 0,
> >
> >                  "state": "unknown",
> >
> >                  "last_fresh": "0.000000",
> >
> >                  "last_change": "0.000000",
> >
> >                  "last_active": "0.000000",
> >
> >                  "last_peered": "0.000000",
> >
> >                  "last_clean": "0.000000",
> >
> >                  "last_became_active": "0.000000",
> >
> >                  "last_became_peered": "0.000000",
> >
> >                  "last_unstale": "0.000000",
> >
> >                  "last_undegraded": "0.000000",
> >
> >                  "last_fullsized": "0.000000",
> >
> >                  "mapping_epoch": 0,
> >
> >                  "log_start": "0'0",
> >
> >                  "ondisk_log_start": "0'0",
> >
> >                  "created": 0,
> >
> >                  "last_epoch_clean": 0,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 0,
> >
> >                  "last_scrub": "0'0",
> >
> >                  "last_scrub_stamp": "0.000000",
> >
> >                  "last_deep_scrub": "0'0",
> >
> >                  "last_deep_scrub_stamp": "0.000000",
> >
> >                  "last_clean_scrub_stamp": "0.000000",
> >
> >                  "objects_scrubbed": 0,
> >
> >                  "log_size": 0,
> >
> >                  "ondisk_log_size": 0,
> >
> >                  "stats_invalid": false,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 0,
> >
> >                  "scrub_schedule": "--",
> >
> >                  "scrub_duration": 0,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 2115497390,
> >
> >                      "num_objects": 506,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 0,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 13042,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 506,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 0,
> >
> >                      "num_read_kb": 0,
> >
> >                      "num_write": 42,
> >
> >                      "num_write_kb": 172032,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 0,
> >
> >                      "num_bytes_recovered": 0,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [],
> >
> >                  "acting": [],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": -1,
> >
> >                  "acting_primary": -1,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 1,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "24(5)",
> >
> >              "pgid": "14.ffs5",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 0,
> >
> >              "last_backfill": "14:ff09a915:::10000001249.00000353:head",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "0'0",
> >
> >                  "reported_seq": 0,
> >
> >                  "reported_epoch": 0,
> >
> >                  "state": "unknown",
> >
> >                  "last_fresh": "0.000000",
> >
> >                  "last_change": "0.000000",
> >
> >                  "last_active": "0.000000",
> >
> >                  "last_peered": "0.000000",
> >
> >                  "last_clean": "0.000000",
> >
> >                  "last_became_active": "0.000000",
> >
> >                  "last_became_peered": "0.000000",
> >
> >                  "last_unstale": "0.000000",
> >
> >                  "last_undegraded": "0.000000",
> >
> >                  "last_fullsized": "0.000000",
> >
> >                  "mapping_epoch": 0,
> >
> >                  "log_start": "0'0",
> >
> >                  "ondisk_log_start": "0'0",
> >
> >                  "created": 0,
> >
> >                  "last_epoch_clean": 0,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 0,
> >
> >                  "last_scrub": "0'0",
> >
> >                  "last_scrub_stamp": "0.000000",
> >
> >                  "last_deep_scrub": "0'0",
> >
> >                  "last_deep_scrub_stamp": "0.000000",
> >
> >                  "last_clean_scrub_stamp": "0.000000",
> >
> >                  "objects_scrubbed": 0,
> >
> >                  "log_size": 0,
> >
> >                  "ondisk_log_size": 0,
> >
> >                  "stats_invalid": false,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 0,
> >
> >                  "scrub_schedule": "--",
> >
> >                  "scrub_duration": 0,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 2115497390,
> >
> >                      "num_objects": 506,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 0,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 13042,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 506,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 0,
> >
> >                      "num_read_kb": 0,
> >
> >                      "num_write": 42,
> >
> >                      "num_write_kb": 172032,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 0,
> >
> >                      "num_bytes_recovered": 0,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [],
> >
> >                  "acting": [],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": -1,
> >
> >                  "acting_primary": -1,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 1,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "26(3)",
> >
> >              "pgid": "14.ffs3",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 33714,
> >
> >              "last_backfill": "MAX",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "16465'33714",
> >
> >                  "reported_seq": 388679,
> >
> >                  "reported_epoch": 16465,
> >
> >                  "state": "active+clean",
> >
> >                  "last_fresh": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_change": "2022-12-26T02:28:03.455705+0000",
> >
> >                  "last_active": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_peered": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_clean": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_became_active": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_became_peered": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_unstale": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_undegraded": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_fullsized": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "mapping_epoch": 16615,
> >
> >                  "log_start": "12957'31152",
> >
> >                  "ondisk_log_start": "12957'31152",
> >
> >                  "created": 4537,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 8,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "objects_scrubbed": 16227,
> >
> >                  "log_size": 2562,
> >
> >                  "ondisk_log_size": 2562,
> >
> >                  "stats_invalid": true,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 14,
> >
> >                  "scrub_schedule": "queued for deep scrub",
> >
> >                  "scrub_duration": 13.320415128,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 51491812456,
> >
> >                      "num_objects": 12301,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 73806,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 0,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 12301,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 66867,
> >
> >                      "num_read_kb": 177079789,
> >
> >                      "num_write": 19946,
> >
> >                      "num_write_kb": 64929471,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 34017,
> >
> >                      "num_bytes_recovered": 142424959053,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [
> >
> >                      5,
> >
> >                      36,
> >
> >                      58,
> >
> >                      42,
> >
> >                      18,
> >
> >                      24
> >
> >                  ],
> >
> >                  "acting": [
> >
> >                      50,
> >
> >                      36,
> >
> >                      5,
> >
> >                      26,
> >
> >                      15,
> >
> >                      46
> >
> >                  ],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": 5,
> >
> >                  "acting_primary": 50,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 0,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "36(1)",
> >
> >              "pgid": "14.ffs1",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 33714,
> >
> >              "last_backfill": "MAX",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "16465'33714",
> >
> >                  "reported_seq": 388679,
> >
> >                  "reported_epoch": 16465,
> >
> >                  "state": "active+clean",
> >
> >                  "last_fresh": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_change": "2022-12-26T02:28:03.455705+0000",
> >
> >                  "last_active": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_peered": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_clean": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_became_active": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_became_peered": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_unstale": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_undegraded": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_fullsized": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "mapping_epoch": 16615,
> >
> >                  "log_start": "12957'31152",
> >
> >                  "ondisk_log_start": "12957'31152",
> >
> >                  "created": 4537,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 8,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "objects_scrubbed": 16227,
> >
> >                  "log_size": 2562,
> >
> >                  "ondisk_log_size": 2562,
> >
> >                  "stats_invalid": true,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 14,
> >
> >                  "scrub_schedule": "queued for deep scrub",
> >
> >                  "scrub_duration": 13.320415128,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 51491812456,
> >
> >                      "num_objects": 12301,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 73806,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 0,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 12301,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 66867,
> >
> >                      "num_read_kb": 177079789,
> >
> >                      "num_write": 19946,
> >
> >                      "num_write_kb": 64929471,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 34017,
> >
> >                      "num_bytes_recovered": 142424959053,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [
> >
> >                      5,
> >
> >                      36,
> >
> >                      58,
> >
> >                      42,
> >
> >                      18,
> >
> >                      24
> >
> >                  ],
> >
> >                  "acting": [
> >
> >                      50,
> >
> >                      36,
> >
> >                      5,
> >
> >                      26,
> >
> >                      15,
> >
> >                      46
> >
> >                  ],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": 5,
> >
> >                  "acting_primary": 50,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 0,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "42(3)",
> >
> >              "pgid": "14.ffs3",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 0,
> >
> >              "last_backfill": "14:ff09a915:::10000001249.00000353:head",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "0'0",
> >
> >                  "reported_seq": 0,
> >
> >                  "reported_epoch": 0,
> >
> >                  "state": "unknown",
> >
> >                  "last_fresh": "0.000000",
> >
> >                  "last_change": "0.000000",
> >
> >                  "last_active": "0.000000",
> >
> >                  "last_peered": "0.000000",
> >
> >                  "last_clean": "0.000000",
> >
> >                  "last_became_active": "0.000000",
> >
> >                  "last_became_peered": "0.000000",
> >
> >                  "last_unstale": "0.000000",
> >
> >                  "last_undegraded": "0.000000",
> >
> >                  "last_fullsized": "0.000000",
> >
> >                  "mapping_epoch": 0,
> >
> >                  "log_start": "0'0",
> >
> >                  "ondisk_log_start": "0'0",
> >
> >                  "created": 0,
> >
> >                  "last_epoch_clean": 0,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 0,
> >
> >                  "last_scrub": "0'0",
> >
> >                  "last_scrub_stamp": "0.000000",
> >
> >                  "last_deep_scrub": "0'0",
> >
> >                  "last_deep_scrub_stamp": "0.000000",
> >
> >                  "last_clean_scrub_stamp": "0.000000",
> >
> >                  "objects_scrubbed": 0,
> >
> >                  "log_size": 0,
> >
> >                  "ondisk_log_size": 0,
> >
> >                  "stats_invalid": false,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 0,
> >
> >                  "scrub_schedule": "--",
> >
> >                  "scrub_duration": 0,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 2115497390,
> >
> >                      "num_objects": 506,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 0,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 13042,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 506,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 0,
> >
> >                      "num_read_kb": 0,
> >
> >                      "num_write": 42,
> >
> >                      "num_write_kb": 172032,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 0,
> >
> >                      "num_bytes_recovered": 0,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [],
> >
> >                  "acting": [],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": -1,
> >
> >                  "acting_primary": -1,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 1,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "46(5)",
> >
> >              "pgid": "14.ffs5",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 33714,
> >
> >              "last_backfill": "MAX",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "16465'33714",
> >
> >                  "reported_seq": 388679,
> >
> >                  "reported_epoch": 16465,
> >
> >                  "state": "active+clean",
> >
> >                  "last_fresh": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_change": "2022-12-26T02:28:03.455705+0000",
> >
> >                  "last_active": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_peered": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_clean": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_became_active": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_became_peered": "2022-12-26T02:28:03.455192+0000",
> >
> >                  "last_unstale": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_undegraded": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "last_fullsized": "2022-12-26T16:10:12.977587+0000",
> >
> >                  "mapping_epoch": 16615,
> >
> >                  "log_start": "12957'31152",
> >
> >                  "ondisk_log_start": "12957'31152",
> >
> >                  "created": 4537,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 8,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "objects_scrubbed": 16227,
> >
> >                  "log_size": 2562,
> >
> >                  "ondisk_log_size": 2562,
> >
> >                  "stats_invalid": true,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 14,
> >
> >                  "scrub_schedule": "queued for deep scrub",
> >
> >                  "scrub_duration": 13.320415128,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 51491812456,
> >
> >                      "num_objects": 12301,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 73806,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 0,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 12301,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 66867,
> >
> >                      "num_read_kb": 177079789,
> >
> >                      "num_write": 19946,
> >
> >                      "num_write_kb": 64929471,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 34017,
> >
> >                      "num_bytes_recovered": 142424959053,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [
> >
> >                      5,
> >
> >                      36,
> >
> >                      58,
> >
> >                      42,
> >
> >                      18,
> >
> >                      24
> >
> >                  ],
> >
> >                  "acting": [
> >
> >                      50,
> >
> >                      36,
> >
> >                      5,
> >
> >                      26,
> >
> >                      15,
> >
> >                      46
> >
> >                  ],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": 5,
> >
> >                  "acting_primary": 50,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 0,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "peer": "58(2)",
> >
> >              "pgid": "14.ffs2",
> >
> >              "last_update": "19550'35077",
> >
> >              "last_complete": "17223'34381",
> >
> >              "log_tail": "12957'31152",
> >
> >              "last_user_version": 0,
> >
> >              "last_backfill": "14:ff09a915:::10000001249.00000353:head",
> >
> >              "purged_snaps": [],
> >
> >              "history": {
> >
> >                  "epoch_created": 4537,
> >
> >                  "epoch_pool_created": 2032,
> >
> >                  "last_epoch_started": 16616,
> >
> >                  "last_interval_started": 16615,
> >
> >                  "last_epoch_clean": 14655,
> >
> >                  "last_interval_clean": 14654,
> >
> >                  "last_epoch_split": 4537,
> >
> >                  "last_epoch_marked_full": 0,
> >
> >                  "same_up_since": 16613,
> >
> >                  "same_interval_since": 16615,
> >
> >                  "same_primary_since": 16615,
> >
> >                  "last_scrub": "3817'25569",
> >
> >                  "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
> >
> >                  "last_deep_scrub": "3756'21592",
> >
> >                  "last_deep_scrub_stamp":
> "2022-12-14T19:35:51.893008+0000",
> >
> >                  "last_clean_scrub_stamp":
> "2022-12-17T06:33:40.577932+0000",
> >
> >                  "prior_readable_until_ub": 0
> >
> >              },
> >
> >              "stats": {
> >
> >                  "version": "0'0",
> >
> >                  "reported_seq": 0,
> >
> >                  "reported_epoch": 0,
> >
> >                  "state": "unknown",
> >
> >                  "last_fresh": "0.000000",
> >
> >                  "last_change": "0.000000",
> >
> >                  "last_active": "0.000000",
> >
> >                  "last_peered": "0.000000",
> >
> >                  "last_clean": "0.000000",
> >
> >                  "last_became_active": "0.000000",
> >
> >                  "last_became_peered": "0.000000",
> >
> >                  "last_unstale": "0.000000",
> >
> >                  "last_undegraded": "0.000000",
> >
> >                  "last_fullsized": "0.000000",
> >
> >                  "mapping_epoch": 0,
> >
> >                  "log_start": "0'0",
> >
> >                  "ondisk_log_start": "0'0",
> >
> >                  "created": 0,
> >
> >                  "last_epoch_clean": 0,
> >
> >                  "parent": "0.0",
> >
> >                  "parent_split_bits": 0,
> >
> >                  "last_scrub": "0'0",
> >
> >                  "last_scrub_stamp": "0.000000",
> >
> >                  "last_deep_scrub": "0'0",
> >
> >                  "last_deep_scrub_stamp": "0.000000",
> >
> >                  "last_clean_scrub_stamp": "0.000000",
> >
> >                  "objects_scrubbed": 0,
> >
> >                  "log_size": 0,
> >
> >                  "ondisk_log_size": 0,
> >
> >                  "stats_invalid": false,
> >
> >                  "dirty_stats_invalid": false,
> >
> >                  "omap_stats_invalid": false,
> >
> >                  "hitset_stats_invalid": false,
> >
> >                  "hitset_bytes_stats_invalid": false,
> >
> >                  "pin_stats_invalid": false,
> >
> >                  "manifest_stats_invalid": false,
> >
> >                  "snaptrimq_len": 0,
> >
> >                  "last_scrub_duration": 0,
> >
> >                  "scrub_schedule": "--",
> >
> >                  "scrub_duration": 0,
> >
> >                  "objects_trimmed": 0,
> >
> >                  "snaptrim_duration": 0,
> >
> >                  "stat_sum": {
> >
> >                      "num_bytes": 2115497390,
> >
> >                      "num_objects": 506,
> >
> >                      "num_object_clones": 0,
> >
> >                      "num_object_copies": 0,
> >
> >                      "num_objects_missing_on_primary": 0,
> >
> >                      "num_objects_missing": 13042,
> >
> >                      "num_objects_degraded": 0,
> >
> >                      "num_objects_misplaced": 0,
> >
> >                      "num_objects_unfound": 0,
> >
> >                      "num_objects_dirty": 506,
> >
> >                      "num_whiteouts": 0,
> >
> >                      "num_read": 0,
> >
> >                      "num_read_kb": 0,
> >
> >                      "num_write": 42,
> >
> >                      "num_write_kb": 172032,
> >
> >                      "num_scrub_errors": 0,
> >
> >                      "num_shallow_scrub_errors": 0,
> >
> >                      "num_deep_scrub_errors": 0,
> >
> >                      "num_objects_recovered": 0,
> >
> >                      "num_bytes_recovered": 0,
> >
> >                      "num_keys_recovered": 0,
> >
> >                      "num_objects_omap": 0,
> >
> >                      "num_objects_hit_set_archive": 0,
> >
> >                      "num_bytes_hit_set_archive": 0,
> >
> >                      "num_flush": 0,
> >
> >                      "num_flush_kb": 0,
> >
> >                      "num_evict": 0,
> >
> >                      "num_evict_kb": 0,
> >
> >                      "num_promote": 0,
> >
> >                      "num_flush_mode_high": 0,
> >
> >                      "num_flush_mode_low": 0,
> >
> >                      "num_evict_mode_some": 0,
> >
> >                      "num_evict_mode_full": 0,
> >
> >                      "num_objects_pinned": 0,
> >
> >                      "num_legacy_snapsets": 0,
> >
> >                      "num_large_omap_objects": 0,
> >
> >                      "num_objects_manifest": 0,
> >
> >                      "num_omap_bytes": 0,
> >
> >                      "num_omap_keys": 0,
> >
> >                      "num_objects_repaired": 0
> >
> >                  },
> >
> >                  "up": [],
> >
> >                  "acting": [],
> >
> >                  "avail_no_missing": [],
> >
> >                  "object_location_counts": [],
> >
> >                  "blocked_by": [],
> >
> >                  "up_primary": -1,
> >
> >                  "acting_primary": -1,
> >
> >                  "purged_snaps": []
> >
> >              },
> >
> >              "empty": 0,
> >
> >              "dne": 0,
> >
> >              "incomplete": 1,
> >
> >              "last_epoch_started": 16616,
> >
> >              "hit_set_history": {
> >
> >                  "current_last_update": "0'0",
> >
> >                  "history": []
> >
> >              }
> >
> >          }
> >
> >      ],
> >
> >      "recovery_state": [
> >
> >          {
> >
> >              "name": "Started/Primary/Active",
> >
> >              "enter_time": "2022-12-26T21:27:49.970477+0000",
> >
> >              "might_have_unfound": [],
> >
> >              "recovery_progress": {
> >
> >                  "backfill_targets": [
> >
> >                      "5(0)",
> >
> >                      "18(4)",
> >
> >                      "24(5)",
> >
> >                      "42(3)",
> >
> >                      "58(2)"
> >
> >                  ],
> >
> >                  "waiting_on_backfill": [],
> >
> >                  "last_backfill_started":
> > "14:ff09b920:::10000005377.00000209:head",
> >
> >                  "backfill_info": {
> >
> >                      "begin": "14:ff09bed4:::10000001a7f.0000182d:head",
> >
> >                      "end": "14:ff0a8fd7:::1000000072c.000001a0:head",
> >
> >                      "objects": [
> >
> >                          {
> >
> >                              "object":
> > "14:ff09bed4:::10000001a7f.0000182d:head",
> >
> >                              "version": "3801'24969"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09c414:::10000004daf.00001063:head",
> >
> >                              "version": "6812'28471"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09c530:::10000003957.00000189:head",
> >
> >                              "version": "9167'29571"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09c607:::100000011db.00000977:head",
> >
> >                              "version": "3703'16620"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09c922:::10000001230.000001e7:head",
> >
> >                              "version": "3728'18321"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09cb55:::1000000454e.000009a3:head",
> >
> >                              "version": "3706'17783"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09ce4d:::10000001316.00000154:head",
> >
> >                              "version": "3798'23751"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09d097:::10000003c4f.00000085:head",
> >
> >                              "version": "3822'26172"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09d449:::1000000188a.0000034a:head",
> >
> >                              "version": "4627'27150"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09d6b4:::10000003c4a.00000b0f:head",
> >
> >                              "version": "3822'26170"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09de4e:::10000001b63.000004c5:head",
> >
> >                              "version": "3608'11396"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09e1e3:::10000002a02.000016cb:head",
> >
> >                              "version": "12960'31316"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09f059:::10000002c09.00000095:head",
> >
> >                              "version": "13226'31609"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff09f069:::1000000279e.0000012f:head",
> >
> >                              "version": "13274'31844"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a0029:::10000001cd0.000000a2:head",
> >
> >                              "version": "3702'16325"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a083b:::100000016c6.0000013e:head",
> >
> >                              "version": "3497'8495"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a0d58:::1000000468d.00001cba:head",
> >
> >                              "version": "9168'29618"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a0dfd:::10000000540.0000064a:head",
> >
> >                              "version": "3493'8171"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a1149:::10000005049.00000b52:head",
> >
> >                              "version": "3529'10013"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a1cdb:::100000009df.0000afec:head",
> >
> >                              "version": "2106'380"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a1e1a:::1000000390b.00000106:head",
> >
> >                              "version": "6812'28456"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a2180:::10000003868.000000b9:head",
> >
> >                              "version": "3706'17681"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a2438:::1000000133e.00000024:head",
> >
> >                              "version": "3798'24057"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a280e:::10000002ccc.000004af:head",
> >
> >                              "version": "15259'33150"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a2f43:::10000003d40.00000665:head",
> >
> >                              "version": "10508'29957"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a315f:::100000011db.00000caa:head",
> >
> >                              "version": "3703'16628"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a3347:::10000004576.00001fb9:head",
> >
> >                              "version": "3732'19654"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a3948:::100000032eb.0000011a:head",
> >
> >                              "version": "3612'11760"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a3af1:::1000000275d.0000027a:head",
> >
> >                              "version": "12960'31292"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a4073:::100000018fe.0000355d:head",
> >
> >                              "version": "3498'8949"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a44db:::10000001a11.00000203:head",
> >
> >                              "version": "3755'21336"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a4888:::1000000246e.0000360a:head",
> >
> >                              "version": "13794'32296"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a4e86:::100000054b1.000019c1:head",
> >
> >                              "version": "12890'30650"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a51c6:::10000003970.0000096e:head",
> >
> >                              "version": "10636'30139"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a5302:::10000001871.000018eb:head",
> >
> >                              "version": "3817'25516"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a56ba:::10000002848.00000266:head",
> >
> >                              "version": "13855'32461"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a61f3:::10000000759.00000030:head",
> >
> >                              "version": "3703'17434"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a68e3:::10000001c17.0000012c:head",
> >
> >                              "version": "3653'13629"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a722a:::10000003e74.00000091:head",
> >
> >                              "version": "3778'22970"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a7557:::100000044fe.00001664:head",
> >
> >                              "version": "3660'14671"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a8f78:::10000001821.00000844:head",
> >
> >                              "version": "3797'23629"
> >
> >                          },
> >
> >                          {
> >
> >                              "object":
> > "14:ff0a8f8f:::10000001bea.00000202:head",
> >
> >                              "version": "3653'13526"
> >
> >                          }
> >
> >                      ]
> >
> >                  },
> >
> >                  "peer_backfill_info": [
> >
> >                      "5(0)",
> >
> >                      {
> >
> >                          "begin": "MAX",
> >
> >                          "end": "MAX",
> >
> >                          "objects": []
> >
> >                      },
> >
> >                      "18(4)",
> >
> >                      {
> >
> >                          "begin": "MAX",
> >
> >                          "end": "MAX",
> >
> >                          "objects": []
> >
> >                      },
> >
> >                      "24(5)",
> >
> >                      {
> >
> >                          "begin": "MAX",
> >
> >                          "end": "MAX",
> >
> >                          "objects": []
> >
> >                      },
> >
> >                      "42(3)",
> >
> >                      {
> >
> >                          "begin": "MAX",
> >
> >                          "end": "MAX",
> >
> >                          "objects": []
> >
> >                      },
> >
> >                      "58(2)",
> >
> >                      {
> >
> >                          "begin": "MAX",
> >
> >                          "end": "MAX",
> >
> >                          "objects": []
> >
> >                      }
> >
> >                  ],
> >
> >                  "backfills_in_flight": [
> >
> >                      "14:ff09b0d9:::100000033c2.00000146:head",
> >
> >                      "14:ff09b6e5:::10000004b37.00000161:head",
> >
> >                      "14:ff09b7b4:::1000000278f.00000374:head",
> >
> >                      "14:ff09b920:::10000005377.00000209:head"
> >
> >                  ],
> >
> >                  "recovering": [
> >
> >                      "14:ff09b0d9:::100000033c2.00000146:head",
> >
> >                      "14:ff09b6e5:::10000004b37.00000161:head",
> >
> >                      "14:ff09b7b4:::1000000278f.00000374:head",
> >
> >                      "14:ff09b920:::10000005377.00000209:head"
> >
> >                  ],
> >
> >                  "pg_backend": {
> >
> >                      "recovery_ops": [
> >
> >                          {
> >
> >                              "hoid":
> > "14:ff09b0d9:::100000033c2.00000146:head",
> >
> >                              "v": "3677'15440",
> >
> >                              "missing_on":
> "5(0),18(4),24(5),42(3),58(2)",
> >
> >                              "missing_on_shards": "0,2,3,4,5",
> >
> >                              "recovery_info":
> > "ObjectRecoveryInfo(14:ff09b0d9:::100000033c2.00000146:head@3677'15440,
> > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
> > object_exist: 1)",
> >
> >                              "recovery_progress":
> > "ObjectRecoveryProgress(!first, data_recovered_to:4194304,
> > data_complete:true, omap_recovered_to:, omap_complete:true,
> error:false)",
> >
> >                              "state": "WRITING",
> >
> >                              "waiting_on_pushes": "5(0)",
> >
> >                              "extent_requested": "0,8388608"
> >
> >                          },
> >
> >                          {
> >
> >                              "hoid":
> > "14:ff09b6e5:::10000004b37.00000161:head",
> >
> >                              "v": "3549'10993",
> >
> >                              "missing_on":
> "5(0),18(4),24(5),42(3),58(2)",
> >
> >                              "missing_on_shards": "0,2,3,4,5",
> >
> >                              "recovery_info":
> > "ObjectRecoveryInfo(14:ff09b6e5:::10000004b37.00000161:head@3549'10993,
> > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
> > object_exist: 1)",
> >
> >                              "recovery_progress":
> > "ObjectRecoveryProgress(!first, data_recovered_to:4194304,
> > data_complete:true, omap_recovered_to:, omap_complete:true,
> error:false)",
> >
> >                              "state": "WRITING",
> >
> >                              "waiting_on_pushes": "5(0)",
> >
> >                              "extent_requested": "0,8388608"
> >
> >                          },
> >
> >                          {
> >
> >                              "hoid":
> > "14:ff09b7b4:::1000000278f.00000374:head",
> >
> >                              "v": "13272'31836",
> >
> >                              "missing_on":
> "5(0),18(4),24(5),42(3),58(2)",
> >
> >                              "missing_on_shards": "0,2,3,4,5",
> >
> >                              "recovery_info":
> > "ObjectRecoveryInfo(14:ff09b7b4:::1000000278f.00000374:head@13272'31836,
> > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
> > object_exist: 1)",
> >
> >                              "recovery_progress":
> > "ObjectRecoveryProgress(!first, data_recovered_to:4194304,
> > data_complete:true, omap_recovered_to:, omap_complete:true,
> error:false)",
> >
> >                              "state": "WRITING",
> >
> >                              "waiting_on_pushes": "5(0)",
> >
> >                              "extent_requested": "0,8388608"
> >
> >                          },
> >
> >                          {
> >
> >                              "hoid":
> > "14:ff09b920:::10000005377.00000209:head",
> >
> >                              "v": "10440'29880",
> >
> >                              "missing_on":
> "5(0),18(4),24(5),42(3),58(2)",
> >
> >                              "missing_on_shards": "0,2,3,4,5",
> >
> >                              "recovery_info":
> > "ObjectRecoveryInfo(14:ff09b920:::10000005377.00000209:head@10440'29880,
> > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
> > object_exist: 1)",
> >
> >                              "recovery_progress":
> > "ObjectRecoveryProgress(!first, data_recovered_to:4194304,
> > data_complete:true, omap_recovered_to:, omap_complete:true,
> error:false)",
> >
> >                              "state": "WRITING",
> >
> >                              "waiting_on_pushes": "5(0)",
> >
> >                              "extent_requested": "0,8388608"
> >
> >                          }
> >
> >                      ],
> >
> >                      "read_ops": []
> >
> >                  }
> >
> >              }
> >
> >          },
> >
> >          {
> >
> >              "name": "Started",
> >
> >              "enter_time": "2022-12-26T21:27:48.908226+0000"
> >
> >          }
> >
> >      ],
> >
> >      "scrubber": {
> >
> >          "active": false,
> >
> >          "must_scrub": true,
> >
> >          "must_deep_scrub": true,
> >
> >          "must_repair": false,
> >
> >          "need_auto": false,
> >
> >          "scrub_reg_stamp": "1.000000",
> >
> >          "schedule": "queued for deep scrub"
> >
> >      },
> >
> >      "agent_state": {}
> >
> > }
> >
> >
> >
> > On Wed, Dec 28, 2022 at 6:46 AM Pavin Joseph <me@xxxxxxxxxxxxxxx> wrote:
> >
> >> 1. This is a guess, but check /var/[lib|run]/ceph for any lock files.
> >> 2. This is more straightforward to fix, add faster WAL/Block device/LV
> >> for each OSD or create a fast storage pool just for metadata. Also,
> >> experiment with MDS cache size/trim [0] settings.
> >>
> >> [0]: https://docs.ceph.com/en/latest/cephfs/cache-configuration/
> >>
> >> On 28-Dec-22 7:23 AM, Deep Dish wrote:
> >>> Got logging enabled as per
> >>> https://ceph.io/en/news/blog/2022/centralized_logging/.   My embedded
> >>> grafana doesn't come up in the dashboard, but at least I have log
> (files)
> >>> on my nodes.   Interesting.
> >>>
> >>> Two issues plaguing my cluster:
> >>>
> >>> 1 - RGWs not manageable
> >>> 2 - MDS_SLOW_METADATA_IO warning (impact to cephfs)
> >>>
> >>> Issue 1:
> >>>
> >>> I have 4x RGWs deployed.   All started / processes running.  They all
> >>> report similar log entries:
> >>>
> >>> 7fcc32b6a5c0  0 deferred set uid:gid to 167:167 (ceph:ceph)
> >>>
> >>> 7fcc32b6a5c0  0 ceph version 17.2.5
> >>> (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
> >>> radosgw, pid 2
> >>>
> >>> 7fcc32b6a5c0  0 framework: beast
> >>>
> >>> 7fcc32b6a5c0  0 framework conf key: port, val: 80
> >>>
> >>> 7fcc32b6a5c0  1 radosgw_Main not setting numa affinity
> >>>
> >>> 7fcc32b6a5c0  1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0
> >>>
> >>> 7fcc32b6a5c0  1 D3N datacache enabled: 0
> >>>
> >>> 7fcc0869a700  0 INFO: RGWReshardLock::lock found lock on
> >> reshard.0000000011
> >>> to be held by another RGW process; skipping for now
> >>>
> >>> 7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >> lc.1,
> >>> sleep 5, try again
> >>>
> >>> 7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >> lc.3,
> >>> sleep 5, try again
> >>>
> >>> 7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>>
> >>> 7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>>
> >>> 7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>>
> >>> 7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>>
> >>> 7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>>
> >>> 7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>>
> >>> 7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>>
> >>> 7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>>
> >>> 7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
> >>> lc.16, sleep 5, try again
> >>> (repeating)
> >>>
> >>> Seems like a stale lock, not previously cleaned up when the cluster was
> >>> busy recovering and rebalancing.
> >>>
> >>> Issue 2:
> >>>
> >>> ceph health detail:
> >>>
> >>> [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
> >>>
> >>>       mds.fs01.ceph02mon03.rjcxat(mds.0): 8 slow metadata IOs are
> blocked
> >>>
> >>> 30 secs, oldest blocked for 39485 secs
> >>>
> >>> Log entries from ceph02mon03 MDS host:
> >>>
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131271 from mon.4
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131272 from mon.4
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131273 from mon.4
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131274 from mon.4
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131275 from mon.4
> >>>    7fe36c6b8700  0 log_channel(cluster) log [WRN] : 1 slow requests, 1
> >>> included below; oldest blocked for > 33.126589 secs
> >>>    7fe36c6b8700  0 log_channel(cluster) log [WRN] : slow request
> 33.126588
> >>> seconds old, received at 2022-12-27T19:45:45.952225+0000:
> >>> client_request(client.55009:99980 create
> >>> #0x10000000bc2/vzdump-qemu-30003-2022_12_27-14_43_43.log
> >>> 2022-12-27T19:45:45.948045+0000 caller_uid=0, caller_gid=0{}) currently
> >>> submit entry: journal_and_reply
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131276 from mon.4
> >>>    7fe36c6b8700  0 log_channel(cluster) log [WRN] : 1 slow requests, 0
> >>> included below; oldest blocked for > 38.126737 secs
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131277 from mon.4
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131278 from mon.4
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131279 from mon.4
> >>>    7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to
> version
> >>> 131280 from mon.4
> >>>
> >>>
> >>> I suspect that the file in the log above int's the culprit.   How can I
> >> get
> >>> to the root cause of MDS slowdowns?
> >>>
> >>>
> >>> On Tue, Dec 27, 2022 at 3:32 PM Pavin Joseph <me@xxxxxxxxxxxxxxx>
> wrote:
> >>>
> >>>> Interesting, the logs show the crash module [0] itself has crashed.
> >>>> Something sent it a SIGINT or SIGTERM and the module didn't handle it
> >>>> correctly due to what seems like a bug in the code.
> >>>>
> >>>> I haven't experienced the crash module itself crashing yet (in Quincy)
> >>>> because nothing sent a SIG[INT|TERM] to it yet.
> >>>>
> >>>> So I'd continue investigating into why these signals were sent to the
> >>>> crash module.
> >>>>
> >>>> To fix the crash module from crashing, go to "/usr/bin/ceph-crash" and
> >>>> edit the handler function on line 82 like so:
> >>>>
> >>>> def handler(signum, frame):
> >>>>      print('*** Interrupted with signal %d ***' % signum)
> >>>>      signame = signal.Signals(signum).name
> >>>>      print(f'Signal handler called with signal {signame} ({signum})')
> >>>>      print(frame)
> >>>>      sys.exit(0)
> >>>>
> >>>> ---
> >>>>
> >>>> Once the crash module is working, perhaps you could run a "ceph crash
> >> ls"
> >>>>
> >>>> Regarding podman logs, perhaps try this [1].
> >>>>
> >>>> [0]: https://docs.ceph.com/en/quincy/mgr/crash/
> >>>> [1]: https://docs.podman.io/en/latest/markdown/podman-logs.1.html
> >>>>
> >>>> On 27-Dec-22 11:59 PM, Deep Dish wrote:
> >>>>> HI Pavin,
> >>>>>
> >>>>> Thanks for the reply.   I'm a bit at a loss honestly as this worked
> >>>>> perfectly without any issue up until the rebalance of the cluster.
> >>>>> Orchestrator is great.   Aside from this (which I suspect is not
> >>>>> orchestrator related), I haven't had any issues.
> >>>>>
> >>>>> In terms of logs, I'm not sure where to start looking in this new
> >>>>> containerized environment as they pertain to individual ceph
> processes
> >>>> -- I
> >>>>> assumed everything would be centrally collected within orch.
> >>>>>
> >>>>> Connecting into the podman container of a RGW, there are no logs in
> >>>>> /var/log/ceph aside from ceph-volume.   My ceph.conf is minimal with
> >> only
> >>>>> monitors defined.  The only log I'm able to pull is as follows:
> >>>>>
> >>>>> # podman logs 35d4ac5445ca
> >>>>>
> >>>>> INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
> >>>>>
> >>>>> Traceback (most recent call last):
> >>>>>
> >>>>>      File "/usr/bin/ceph-crash", line 113, in <module>
> >>>>>
> >>>>>        main()
> >>>>>
> >>>>>      File "/usr/bin/ceph-crash", line 109, in main
> >>>>>
> >>>>>        time.sleep(args.delay * 60)
> >>>>>
> >>>>> TypeError: handler() takes 1 positional argument but 2 were given
> >>>>>
> >>>>> INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
> >>>>>
> >>>>>
> >>>>>
> >>>>> Looks like the RGW daemon is crashing.   How do I get logs to
> persist?
> >>>>    I
> >>>>> suspect I won't be able to use orchestrator to push down the config,
> >> and
> >>>>> would have to manipulate within the container image itself.
> >>>>>
> >>>>> I also attempted to redeply the RGW containers without success.
> >>>>>
> >>>>> On Tue, Dec 27, 2022 at 10:39 AM Pavin Joseph <me@xxxxxxxxxxxxxxx>
> >>>> wrote:
> >>>>>
> >>>>>> Here's the first things I'd check in your situation:
> >>>>>>
> >>>>>> 1. Logs
> >>>>>> 2. Is the RGW HTTP server running on its port?
> >>>>>> 3. Re-check config including authentication.
> >>>>>>
> >>>>>> ceph orch is too new and didn't pass muster in our own internal
> >> testing.
> >>>>>> You're braver than most for using it in production.
> >>>>>>
> >>>>>> Pavin.
> >>>>>>
> >>>>>> On 27-Dec-22 8:48 PM, Deep Dish wrote:
> >>>>>>> Quick update:
> >>>>>>>
> >>>>>>> - I followed documentation, and ran the following:
> >>>>>>>
> >>>>>>> # ceph dashboard set-rgw-credentials
> >>>>>>>
> >>>>>>> Error EINVAL: No RGW credentials found, please consult the
> >>>> documentation
> >>>>>> on
> >>>>>>> how to enable RGW for the dashboard.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> - I see dashboard credentials configured (all this was working fine
> >>>>>> before):
> >>>>>>>
> >>>>>>>
> >>>>>>> # ceph dashboard get-rgw-api-access-key
> >>>>>>>
> >>>>>>> P?????????????????G  (? commented out)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Seems to me like my RGW config is non-existent / corrupted for some
> >>>>>>> reason.  When trying to curl a RGW directly I get a "connection
> >>>> refused".
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Dec 27, 2022 at 9:41 AM Deep Dish <deeepdish@xxxxxxxxx>
> >> wrote:
> >>>>>>>
> >>>>>>>> I built a net-new Quincy cluster (17.2.5) using ceph orch as
> >> follows:
> >>>>>>>>
> >>>>>>>> 2x mgrs
> >>>>>>>> 4x rgw
> >>>>>>>> 5x mon
> >>>>>>>> 4x rgw
> >>>>>>>> 5x mds
> >>>>>>>> 6x osd hosts w/ 10 drives each --> will be growing to 7 osd hosts
> in
> >>>> the
> >>>>>>>> coming days.
> >>>>>>>>
> >>>>>>>> I migrated all data from my legacy nautilus cluster (via
> rbd-mirror,
> >>>>>>>> rclone for s3 buckets, etc.).  All moved over successfully without
> >>>>>> issue.
> >>>>>>>>
> >>>>>>>> The cluster went through a series of rebalancing events (adding
> >>>>>> capacity,
> >>>>>>>> osd nodes, changing fault domain for EC volumes).
> >>>>>>>>
> >>>>>>>> It's settled now, however throughout the process all of my RGW
> nodes
> >>>> are
> >>>>>>>> no longer part of the cluster -- meaning ceph doesn't recognize /
> >>>> detect
> >>>>>>>> them, despite containers, networking, etc. all being setup
> >> correctly.
> >>>>>>>> This also means I'm unable to manage any RGW functions (via the
> >>>>>> dashboard
> >>>>>>>> or cli).   As an example via cli (within Cephadm shell):
> >>>>>>>>
> >>>>>>>> # radosgw-admin pools list
> >>>>>>>>
> >>>>>>>> could not list placement set: (2) No such file or directory
> >>>>>>>>
> >>>>>>>> I have data in buckets, how can I get my RGWs to return online?
> >>>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>>>>
> >>>>> _______________________________________________
> >>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>>
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux