Re: Urgent help! RGW Disappeared on Quincy

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hey there,

Sorry for the late reply.
If the pg issue isn't solved yet, could you run these:

ceph pg repeer <pgid>
ceph pg repair <pgid>

Pavin.

On 29-Dec-22 4:08 AM, Deep Dish wrote:
Hi Pavin,

The following are additional developments..  There's one PG that's
stuck and unable to recover.   I've attached relevant ceph -s / health
detail and pg stat outputs below.

- There were some remaining lock files as suggested in /var/run/ceph/
pertaining to rgw.   I removed the service, deleted any stale lock files
and redeployed the RGWs.   All started with the common log entries across
all:

7ff5d9aaf5c0  0 deferred set uid:gid to 167:167 (ceph:ceph)

7ff5d9aaf5c0  0 ceph version 17.2.5
(98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
radosgw, pid 2

7ff5d9aaf5c0  0 framework: beast

7ff5d9aaf5c0  0 framework conf key: port, val: 80

7ff5d9aaf5c0  1 radosgw_Main not setting numa affinity

7ff5d9aaf5c0  1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0

7ff5d9aaf5c0  1 D3N datacache enabled: 0

No additional log entries are recorded since starting them post
re-deployment as per above.

The cluster settled, there is no recovery activity.  There is one pg that's
stuck and I have a hunch that it's impacting MDS and RGW processes as
stated in the thread.   PG is stuck as as active+remapped+backfilling:



   data:

     volumes: 2/2 healthy

     pools:   16 pools, 1504 pgs

     objects: 24.49M objects, 79 TiB

     usage:   119 TiB used, 390 TiB / 508 TiB avail

     pgs:     65210/146755179 objects misplaced (0.044%)

              1503 active+clean

              1    active+remapped+backfilling



   progress:

     Global Recovery Event (6h)

       [===========================.] (remaining: 73s)

# ceph health detail

HEALTH_WARN 1 MDSs report slow metadata IOs; 1 pgs not deep-scrubbed in
time; 1 pgs not scrubbed in time

[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs

     mds.fs01.ceph02mon02.wicrdz(mds.0): 5 slow metadata IOs are blocked >
30 secs, oldest blocked for 74436 secs

[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time

     pg 14.ff not deep-scrubbed since 2022-12-14T19:35:51.893008+0000

[WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time

     pg 14.ff not scrubbed since 2022-12-17T06:33:40.577932+0000



 From the following pg query:

- "pgid": "14.ffs0" is stuck as peering (osd 5)
- "pgid": "14.ffs4" is stuck as unknown (osd 18)
- "pgid": "14.ffs5" is stuck as unknown (osd 24)
- "pgid": "14.ffs3" is stuck as unknown (osd 42)
- "pgid": "14.ffs2" is stick as unknown (osd 58)
- "pgid": "14.ffs1" is marked as active+clean (osd 36)

# ceph pg 14.ff query

{

     "snap_trimq": "[]",

     "snap_trimq_len": 0,

     "state": "active+remapped+backfilling",

     "epoch": 19594,

     "up": [

         5,

         36,

         58,

         42,

         18,

         24

     ],

     "acting": [

         50,

         36,

         5,

         26,

         15,

         46

     ],

     "backfill_targets": [

         "5(0)",

         "18(4)",

         "24(5)",

         "42(3)",

         "58(2)"

     ],

     "acting_recovery_backfill": [

         "5(0)",

         "5(2)",

         "15(4)",

         "18(4)",

         "24(5)",

         "26(3)",

         "36(1)",

         "42(3)",

         "46(5)",

         "50(0)",

         "58(2)"

     ],

     "info": {

         "pgid": "14.ffs0",

         "last_update": "19550'35077",

         "last_complete": "19550'35077",

         "log_tail": "13761'32157",

         "last_user_version": 35077,

         "last_backfill": "MAX",

         "purged_snaps": [],

         "history": {

             "epoch_created": 4537,

             "epoch_pool_created": 2032,

             "last_epoch_started": 16616,

             "last_interval_started": 16615,

             "last_epoch_clean": 14655,

             "last_interval_clean": 14654,

             "last_epoch_split": 4537,

             "last_epoch_marked_full": 0,

             "same_up_since": 16613,

             "same_interval_since": 16615,

             "same_primary_since": 16615,

             "last_scrub": "3817'25569",

             "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

             "last_deep_scrub": "3756'21592",

             "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

             "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

             "prior_readable_until_ub": 0

         },

         "stats": {

             "version": "19550'35077",

             "reported_seq": 396919,

             "reported_epoch": 19594,

             "state": "active+remapped+backfilling",

             "last_fresh": "2022-12-28T22:03:20.278478+0000",

             "last_change": "2022-12-26T21:27:51.600940+0000",

             "last_active": "2022-12-28T22:03:20.278478+0000",

             "last_peered": "2022-12-28T22:03:20.278478+0000",

             "last_clean": "2022-12-26T21:27:45.471954+0000",

             "last_became_active": "2022-12-26T21:27:51.085966+0000",

             "last_became_peered": "2022-12-26T21:27:51.085966+0000",

             "last_unstale": "2022-12-28T22:03:20.278478+0000",

             "last_undegraded": "2022-12-28T22:03:20.278478+0000",

             "last_fullsized": "2022-12-28T22:03:20.278478+0000",

             "mapping_epoch": 16615,

             "log_start": "13761'32157",

             "ondisk_log_start": "13761'32157",

             "created": 4537,

             "last_epoch_clean": 14655,

             "parent": "0.0",

             "parent_split_bits": 8,

             "last_scrub": "3817'25569",

             "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

             "last_deep_scrub": "3756'21592",

             "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

             "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

             "objects_scrubbed": 16227,

             "log_size": 2920,

             "ondisk_log_size": 2920,

             "stats_invalid": true,

             "dirty_stats_invalid": false,

             "omap_stats_invalid": false,

             "hitset_stats_invalid": false,

             "hitset_bytes_stats_invalid": false,

             "pin_stats_invalid": false,

             "manifest_stats_invalid": false,

             "snaptrimq_len": 0,

             "last_scrub_duration": 14,

             "scrub_schedule": "queued for deep scrub",

             "scrub_duration": 13.320415128,

             "objects_trimmed": 0,

             "snaptrim_duration": 0,

             "stat_sum": {

                 "num_bytes": 56709530650,

                 "num_objects": 13548,

                 "num_object_clones": 0,

                 "num_object_copies": 81288,

                 "num_objects_missing_on_primary": 0,

                 "num_objects_missing": 0,

                 "num_objects_degraded": 0,

                 "num_objects_misplaced": 65210,

                 "num_objects_unfound": 0,

                 "num_objects_dirty": 13548,

                 "num_whiteouts": 0,

                 "num_read": 67760,

                 "num_read_kb": 177798674,

                 "num_write": 21231,

                 "num_write_kb": 70024901,

                 "num_scrub_errors": 0,

                 "num_shallow_scrub_errors": 0,

                 "num_deep_scrub_errors": 0,

                 "num_objects_recovered": 34481,

                 "num_bytes_recovered": 144364295675,

                 "num_keys_recovered": 0,

                 "num_objects_omap": 0,

                 "num_objects_hit_set_archive": 0,

                 "num_bytes_hit_set_archive": 0,

                 "num_flush": 0,

                 "num_flush_kb": 0,

                 "num_evict": 0,

                 "num_evict_kb": 0,

                 "num_promote": 0,

                 "num_flush_mode_high": 0,

                 "num_flush_mode_low": 0,

                 "num_evict_mode_some": 0,

                 "num_evict_mode_full": 0,

                 "num_objects_pinned": 0,

                 "num_legacy_snapsets": 0,

                 "num_large_omap_objects": 0,

                 "num_objects_manifest": 0,

                 "num_omap_bytes": 0,

                 "num_omap_keys": 0,

                 "num_objects_repaired": 0

             },

             "up": [

                 5,

                 36,

                 58,

                 42,

                 18,

                 24

             ],

             "acting": [

                 50,

                 36,

                 5,

                 26,

                 15,

                 46

             ],

             "avail_no_missing": [

                 "50(0)",

                 "5(2)",

                 "15(4)",

                 "26(3)",

                 "36(1)",

                 "46(5)"

             ],

             "object_location_counts": [

                 {

                     "shards": "5(2),15(4),26(3),36(1),46(5),50(0)",

                     "objects": 13548

                 }

             ],

             "blocked_by": [],

             "up_primary": 5,

             "acting_primary": 50,

             "purged_snaps": []

         },

         "empty": 0,

         "dne": 0,

         "incomplete": 0,

         "last_epoch_started": 16616,

         "hit_set_history": {

             "current_last_update": "0'0",

             "history": []

         }

     },

     "peer_info": [

         {

             "peer": "5(0)",

             "pgid": "14.ffs0",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 0,

             "last_backfill": "14:ff09a915:::10000001249.00000353:head",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "0'0",

                 "reported_seq": 2,

                 "reported_epoch": 16614,

                 "state": "peering",

                 "last_fresh": "2022-12-26T21:27:47.993646+0000",

                 "last_change": "2022-12-26T21:27:47.769141+0000",

                 "last_active": "0.000000",

                 "last_peered": "0.000000",

                 "last_clean": "0.000000",

                 "last_became_active": "0.000000",

                 "last_became_peered": "0.000000",

                 "last_unstale": "2022-12-26T21:27:47.993646+0000",

                 "last_undegraded": "2022-12-26T21:27:47.993646+0000",

                 "last_fullsized": "2022-12-26T21:27:47.993646+0000",

                 "mapping_epoch": 16615,

                 "log_start": "0'0",

                 "ondisk_log_start": "0'0",

                 "created": 4537,

                 "last_epoch_clean": 14655,

                 "parent": "0.0",

                 "parent_split_bits": 0,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "objects_scrubbed": 0,

                 "log_size": 0,

                 "ondisk_log_size": 0,

                 "stats_invalid": false,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 0,

                 "scrub_schedule": "queued for deep scrub",

                 "scrub_duration": 0,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 2115497390,

                     "num_objects": 506,

                     "num_object_clones": 0,

                     "num_object_copies": 0,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 13042,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 506,

                     "num_whiteouts": 0,

                     "num_read": 0,

                     "num_read_kb": 0,

                     "num_write": 42,

                     "num_write_kb": 172032,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 0,

                     "num_bytes_recovered": 0,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [

                     5,

                     36,

                     58,

                     42,

                     18,

                     24

                 ],

                 "acting": [

                     50,

                     36,

                     5,

                     26,

                     15,

                     46

                 ],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [

                     24

                 ],

                 "up_primary": 5,

                 "acting_primary": 50,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 1,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "5(2)",

             "pgid": "14.ffs2",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 33714,

             "last_backfill": "MAX",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "16465'33714",

                 "reported_seq": 388679,

                 "reported_epoch": 16465,

                 "state": "active+clean",

                 "last_fresh": "2022-12-26T16:10:12.977587+0000",

                 "last_change": "2022-12-26T02:28:03.455705+0000",

                 "last_active": "2022-12-26T16:10:12.977587+0000",

                 "last_peered": "2022-12-26T16:10:12.977587+0000",

                 "last_clean": "2022-12-26T16:10:12.977587+0000",

                 "last_became_active": "2022-12-26T02:28:03.455192+0000",

                 "last_became_peered": "2022-12-26T02:28:03.455192+0000",

                 "last_unstale": "2022-12-26T16:10:12.977587+0000",

                 "last_undegraded": "2022-12-26T16:10:12.977587+0000",

                 "last_fullsized": "2022-12-26T16:10:12.977587+0000",

                 "mapping_epoch": 16615,

                 "log_start": "12957'31152",

                 "ondisk_log_start": "12957'31152",

                 "created": 4537,

                 "last_epoch_clean": 14655,

                 "parent": "0.0",

                 "parent_split_bits": 8,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "objects_scrubbed": 16227,

                 "log_size": 2562,

                 "ondisk_log_size": 2562,

                 "stats_invalid": true,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 14,

                 "scrub_schedule": "queued for deep scrub",

                 "scrub_duration": 13.320415128,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 51491812456,

                     "num_objects": 12301,

                     "num_object_clones": 0,

                     "num_object_copies": 73806,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 0,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 12301,

                     "num_whiteouts": 0,

                     "num_read": 66867,

                     "num_read_kb": 177079789,

                     "num_write": 19946,

                     "num_write_kb": 64929471,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 34017,

                     "num_bytes_recovered": 142424959053,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [

                     5,

                     36,

                     58,

                     42,

                     18,

                     24

                 ],

                 "acting": [

                     50,

                     36,

                     5,

                     26,

                     15,

                     46

                 ],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": 5,

                 "acting_primary": 50,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 0,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "15(4)",

             "pgid": "14.ffs4",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 33714,

             "last_backfill": "MAX",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "16465'33714",

                 "reported_seq": 388679,

                 "reported_epoch": 16465,

                 "state": "active+clean",

                 "last_fresh": "2022-12-26T16:10:12.977587+0000",

                 "last_change": "2022-12-26T02:28:03.455705+0000",

                 "last_active": "2022-12-26T16:10:12.977587+0000",

                 "last_peered": "2022-12-26T16:10:12.977587+0000",

                 "last_clean": "2022-12-26T16:10:12.977587+0000",

                 "last_became_active": "2022-12-26T02:28:03.455192+0000",

                 "last_became_peered": "2022-12-26T02:28:03.455192+0000",

                 "last_unstale": "2022-12-26T16:10:12.977587+0000",

                 "last_undegraded": "2022-12-26T16:10:12.977587+0000",

                 "last_fullsized": "2022-12-26T16:10:12.977587+0000",

                 "mapping_epoch": 16615,

                 "log_start": "12957'31152",

                 "ondisk_log_start": "12957'31152",

                 "created": 4537,

                 "last_epoch_clean": 14655,

                 "parent": "0.0",

                 "parent_split_bits": 8,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "objects_scrubbed": 16227,

                 "log_size": 2562,

                 "ondisk_log_size": 2562,

                 "stats_invalid": true,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 14,

                 "scrub_schedule": "queued for deep scrub",

                 "scrub_duration": 13.320415128,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 51491812456,

                     "num_objects": 12301,

                     "num_object_clones": 0,

                     "num_object_copies": 73806,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 0,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 12301,

                     "num_whiteouts": 0,

                     "num_read": 66867,

                     "num_read_kb": 177079789,

                     "num_write": 19946,

                     "num_write_kb": 64929471,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 34017,

                     "num_bytes_recovered": 142424959053,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [

                     5,

                     36,

                     58,

                     42,

                     18,

                     24

                 ],

                 "acting": [

                     50,

                     36,

                     5,

                     26,

                     15,

                     46

                 ],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": 5,

                 "acting_primary": 50,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 0,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "18(4)",

             "pgid": "14.ffs4",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 0,

             "last_backfill": "14:ff09a915:::10000001249.00000353:head",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "0'0",

                 "reported_seq": 0,

                 "reported_epoch": 0,

                 "state": "unknown",

                 "last_fresh": "0.000000",

                 "last_change": "0.000000",

                 "last_active": "0.000000",

                 "last_peered": "0.000000",

                 "last_clean": "0.000000",

                 "last_became_active": "0.000000",

                 "last_became_peered": "0.000000",

                 "last_unstale": "0.000000",

                 "last_undegraded": "0.000000",

                 "last_fullsized": "0.000000",

                 "mapping_epoch": 0,

                 "log_start": "0'0",

                 "ondisk_log_start": "0'0",

                 "created": 0,

                 "last_epoch_clean": 0,

                 "parent": "0.0",

                 "parent_split_bits": 0,

                 "last_scrub": "0'0",

                 "last_scrub_stamp": "0.000000",

                 "last_deep_scrub": "0'0",

                 "last_deep_scrub_stamp": "0.000000",

                 "last_clean_scrub_stamp": "0.000000",

                 "objects_scrubbed": 0,

                 "log_size": 0,

                 "ondisk_log_size": 0,

                 "stats_invalid": false,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 0,

                 "scrub_schedule": "--",

                 "scrub_duration": 0,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 2115497390,

                     "num_objects": 506,

                     "num_object_clones": 0,

                     "num_object_copies": 0,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 13042,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 506,

                     "num_whiteouts": 0,

                     "num_read": 0,

                     "num_read_kb": 0,

                     "num_write": 42,

                     "num_write_kb": 172032,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 0,

                     "num_bytes_recovered": 0,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [],

                 "acting": [],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": -1,

                 "acting_primary": -1,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 1,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "24(5)",

             "pgid": "14.ffs5",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 0,

             "last_backfill": "14:ff09a915:::10000001249.00000353:head",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "0'0",

                 "reported_seq": 0,

                 "reported_epoch": 0,

                 "state": "unknown",

                 "last_fresh": "0.000000",

                 "last_change": "0.000000",

                 "last_active": "0.000000",

                 "last_peered": "0.000000",

                 "last_clean": "0.000000",

                 "last_became_active": "0.000000",

                 "last_became_peered": "0.000000",

                 "last_unstale": "0.000000",

                 "last_undegraded": "0.000000",

                 "last_fullsized": "0.000000",

                 "mapping_epoch": 0,

                 "log_start": "0'0",

                 "ondisk_log_start": "0'0",

                 "created": 0,

                 "last_epoch_clean": 0,

                 "parent": "0.0",

                 "parent_split_bits": 0,

                 "last_scrub": "0'0",

                 "last_scrub_stamp": "0.000000",

                 "last_deep_scrub": "0'0",

                 "last_deep_scrub_stamp": "0.000000",

                 "last_clean_scrub_stamp": "0.000000",

                 "objects_scrubbed": 0,

                 "log_size": 0,

                 "ondisk_log_size": 0,

                 "stats_invalid": false,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 0,

                 "scrub_schedule": "--",

                 "scrub_duration": 0,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 2115497390,

                     "num_objects": 506,

                     "num_object_clones": 0,

                     "num_object_copies": 0,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 13042,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 506,

                     "num_whiteouts": 0,

                     "num_read": 0,

                     "num_read_kb": 0,

                     "num_write": 42,

                     "num_write_kb": 172032,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 0,

                     "num_bytes_recovered": 0,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [],

                 "acting": [],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": -1,

                 "acting_primary": -1,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 1,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "26(3)",

             "pgid": "14.ffs3",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 33714,

             "last_backfill": "MAX",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "16465'33714",

                 "reported_seq": 388679,

                 "reported_epoch": 16465,

                 "state": "active+clean",

                 "last_fresh": "2022-12-26T16:10:12.977587+0000",

                 "last_change": "2022-12-26T02:28:03.455705+0000",

                 "last_active": "2022-12-26T16:10:12.977587+0000",

                 "last_peered": "2022-12-26T16:10:12.977587+0000",

                 "last_clean": "2022-12-26T16:10:12.977587+0000",

                 "last_became_active": "2022-12-26T02:28:03.455192+0000",

                 "last_became_peered": "2022-12-26T02:28:03.455192+0000",

                 "last_unstale": "2022-12-26T16:10:12.977587+0000",

                 "last_undegraded": "2022-12-26T16:10:12.977587+0000",

                 "last_fullsized": "2022-12-26T16:10:12.977587+0000",

                 "mapping_epoch": 16615,

                 "log_start": "12957'31152",

                 "ondisk_log_start": "12957'31152",

                 "created": 4537,

                 "last_epoch_clean": 14655,

                 "parent": "0.0",

                 "parent_split_bits": 8,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "objects_scrubbed": 16227,

                 "log_size": 2562,

                 "ondisk_log_size": 2562,

                 "stats_invalid": true,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 14,

                 "scrub_schedule": "queued for deep scrub",

                 "scrub_duration": 13.320415128,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 51491812456,

                     "num_objects": 12301,

                     "num_object_clones": 0,

                     "num_object_copies": 73806,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 0,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 12301,

                     "num_whiteouts": 0,

                     "num_read": 66867,

                     "num_read_kb": 177079789,

                     "num_write": 19946,

                     "num_write_kb": 64929471,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 34017,

                     "num_bytes_recovered": 142424959053,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [

                     5,

                     36,

                     58,

                     42,

                     18,

                     24

                 ],

                 "acting": [

                     50,

                     36,

                     5,

                     26,

                     15,

                     46

                 ],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": 5,

                 "acting_primary": 50,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 0,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "36(1)",

             "pgid": "14.ffs1",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 33714,

             "last_backfill": "MAX",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "16465'33714",

                 "reported_seq": 388679,

                 "reported_epoch": 16465,

                 "state": "active+clean",

                 "last_fresh": "2022-12-26T16:10:12.977587+0000",

                 "last_change": "2022-12-26T02:28:03.455705+0000",

                 "last_active": "2022-12-26T16:10:12.977587+0000",

                 "last_peered": "2022-12-26T16:10:12.977587+0000",

                 "last_clean": "2022-12-26T16:10:12.977587+0000",

                 "last_became_active": "2022-12-26T02:28:03.455192+0000",

                 "last_became_peered": "2022-12-26T02:28:03.455192+0000",

                 "last_unstale": "2022-12-26T16:10:12.977587+0000",

                 "last_undegraded": "2022-12-26T16:10:12.977587+0000",

                 "last_fullsized": "2022-12-26T16:10:12.977587+0000",

                 "mapping_epoch": 16615,

                 "log_start": "12957'31152",

                 "ondisk_log_start": "12957'31152",

                 "created": 4537,

                 "last_epoch_clean": 14655,

                 "parent": "0.0",

                 "parent_split_bits": 8,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "objects_scrubbed": 16227,

                 "log_size": 2562,

                 "ondisk_log_size": 2562,

                 "stats_invalid": true,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 14,

                 "scrub_schedule": "queued for deep scrub",

                 "scrub_duration": 13.320415128,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 51491812456,

                     "num_objects": 12301,

                     "num_object_clones": 0,

                     "num_object_copies": 73806,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 0,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 12301,

                     "num_whiteouts": 0,

                     "num_read": 66867,

                     "num_read_kb": 177079789,

                     "num_write": 19946,

                     "num_write_kb": 64929471,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 34017,

                     "num_bytes_recovered": 142424959053,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [

                     5,

                     36,

                     58,

                     42,

                     18,

                     24

                 ],

                 "acting": [

                     50,

                     36,

                     5,

                     26,

                     15,

                     46

                 ],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": 5,

                 "acting_primary": 50,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 0,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "42(3)",

             "pgid": "14.ffs3",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 0,

             "last_backfill": "14:ff09a915:::10000001249.00000353:head",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "0'0",

                 "reported_seq": 0,

                 "reported_epoch": 0,

                 "state": "unknown",

                 "last_fresh": "0.000000",

                 "last_change": "0.000000",

                 "last_active": "0.000000",

                 "last_peered": "0.000000",

                 "last_clean": "0.000000",

                 "last_became_active": "0.000000",

                 "last_became_peered": "0.000000",

                 "last_unstale": "0.000000",

                 "last_undegraded": "0.000000",

                 "last_fullsized": "0.000000",

                 "mapping_epoch": 0,

                 "log_start": "0'0",

                 "ondisk_log_start": "0'0",

                 "created": 0,

                 "last_epoch_clean": 0,

                 "parent": "0.0",

                 "parent_split_bits": 0,

                 "last_scrub": "0'0",

                 "last_scrub_stamp": "0.000000",

                 "last_deep_scrub": "0'0",

                 "last_deep_scrub_stamp": "0.000000",

                 "last_clean_scrub_stamp": "0.000000",

                 "objects_scrubbed": 0,

                 "log_size": 0,

                 "ondisk_log_size": 0,

                 "stats_invalid": false,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 0,

                 "scrub_schedule": "--",

                 "scrub_duration": 0,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 2115497390,

                     "num_objects": 506,

                     "num_object_clones": 0,

                     "num_object_copies": 0,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 13042,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 506,

                     "num_whiteouts": 0,

                     "num_read": 0,

                     "num_read_kb": 0,

                     "num_write": 42,

                     "num_write_kb": 172032,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 0,

                     "num_bytes_recovered": 0,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [],

                 "acting": [],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": -1,

                 "acting_primary": -1,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 1,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "46(5)",

             "pgid": "14.ffs5",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 33714,

             "last_backfill": "MAX",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "16465'33714",

                 "reported_seq": 388679,

                 "reported_epoch": 16465,

                 "state": "active+clean",

                 "last_fresh": "2022-12-26T16:10:12.977587+0000",

                 "last_change": "2022-12-26T02:28:03.455705+0000",

                 "last_active": "2022-12-26T16:10:12.977587+0000",

                 "last_peered": "2022-12-26T16:10:12.977587+0000",

                 "last_clean": "2022-12-26T16:10:12.977587+0000",

                 "last_became_active": "2022-12-26T02:28:03.455192+0000",

                 "last_became_peered": "2022-12-26T02:28:03.455192+0000",

                 "last_unstale": "2022-12-26T16:10:12.977587+0000",

                 "last_undegraded": "2022-12-26T16:10:12.977587+0000",

                 "last_fullsized": "2022-12-26T16:10:12.977587+0000",

                 "mapping_epoch": 16615,

                 "log_start": "12957'31152",

                 "ondisk_log_start": "12957'31152",

                 "created": 4537,

                 "last_epoch_clean": 14655,

                 "parent": "0.0",

                 "parent_split_bits": 8,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "objects_scrubbed": 16227,

                 "log_size": 2562,

                 "ondisk_log_size": 2562,

                 "stats_invalid": true,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 14,

                 "scrub_schedule": "queued for deep scrub",

                 "scrub_duration": 13.320415128,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 51491812456,

                     "num_objects": 12301,

                     "num_object_clones": 0,

                     "num_object_copies": 73806,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 0,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 12301,

                     "num_whiteouts": 0,

                     "num_read": 66867,

                     "num_read_kb": 177079789,

                     "num_write": 19946,

                     "num_write_kb": 64929471,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 34017,

                     "num_bytes_recovered": 142424959053,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [

                     5,

                     36,

                     58,

                     42,

                     18,

                     24

                 ],

                 "acting": [

                     50,

                     36,

                     5,

                     26,

                     15,

                     46

                 ],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": 5,

                 "acting_primary": 50,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 0,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         },

         {

             "peer": "58(2)",

             "pgid": "14.ffs2",

             "last_update": "19550'35077",

             "last_complete": "17223'34381",

             "log_tail": "12957'31152",

             "last_user_version": 0,

             "last_backfill": "14:ff09a915:::10000001249.00000353:head",

             "purged_snaps": [],

             "history": {

                 "epoch_created": 4537,

                 "epoch_pool_created": 2032,

                 "last_epoch_started": 16616,

                 "last_interval_started": 16615,

                 "last_epoch_clean": 14655,

                 "last_interval_clean": 14654,

                 "last_epoch_split": 4537,

                 "last_epoch_marked_full": 0,

                 "same_up_since": 16613,

                 "same_interval_since": 16615,

                 "same_primary_since": 16615,

                 "last_scrub": "3817'25569",

                 "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "last_deep_scrub": "3756'21592",

                 "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",

                 "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",

                 "prior_readable_until_ub": 0

             },

             "stats": {

                 "version": "0'0",

                 "reported_seq": 0,

                 "reported_epoch": 0,

                 "state": "unknown",

                 "last_fresh": "0.000000",

                 "last_change": "0.000000",

                 "last_active": "0.000000",

                 "last_peered": "0.000000",

                 "last_clean": "0.000000",

                 "last_became_active": "0.000000",

                 "last_became_peered": "0.000000",

                 "last_unstale": "0.000000",

                 "last_undegraded": "0.000000",

                 "last_fullsized": "0.000000",

                 "mapping_epoch": 0,

                 "log_start": "0'0",

                 "ondisk_log_start": "0'0",

                 "created": 0,

                 "last_epoch_clean": 0,

                 "parent": "0.0",

                 "parent_split_bits": 0,

                 "last_scrub": "0'0",

                 "last_scrub_stamp": "0.000000",

                 "last_deep_scrub": "0'0",

                 "last_deep_scrub_stamp": "0.000000",

                 "last_clean_scrub_stamp": "0.000000",

                 "objects_scrubbed": 0,

                 "log_size": 0,

                 "ondisk_log_size": 0,

                 "stats_invalid": false,

                 "dirty_stats_invalid": false,

                 "omap_stats_invalid": false,

                 "hitset_stats_invalid": false,

                 "hitset_bytes_stats_invalid": false,

                 "pin_stats_invalid": false,

                 "manifest_stats_invalid": false,

                 "snaptrimq_len": 0,

                 "last_scrub_duration": 0,

                 "scrub_schedule": "--",

                 "scrub_duration": 0,

                 "objects_trimmed": 0,

                 "snaptrim_duration": 0,

                 "stat_sum": {

                     "num_bytes": 2115497390,

                     "num_objects": 506,

                     "num_object_clones": 0,

                     "num_object_copies": 0,

                     "num_objects_missing_on_primary": 0,

                     "num_objects_missing": 13042,

                     "num_objects_degraded": 0,

                     "num_objects_misplaced": 0,

                     "num_objects_unfound": 0,

                     "num_objects_dirty": 506,

                     "num_whiteouts": 0,

                     "num_read": 0,

                     "num_read_kb": 0,

                     "num_write": 42,

                     "num_write_kb": 172032,

                     "num_scrub_errors": 0,

                     "num_shallow_scrub_errors": 0,

                     "num_deep_scrub_errors": 0,

                     "num_objects_recovered": 0,

                     "num_bytes_recovered": 0,

                     "num_keys_recovered": 0,

                     "num_objects_omap": 0,

                     "num_objects_hit_set_archive": 0,

                     "num_bytes_hit_set_archive": 0,

                     "num_flush": 0,

                     "num_flush_kb": 0,

                     "num_evict": 0,

                     "num_evict_kb": 0,

                     "num_promote": 0,

                     "num_flush_mode_high": 0,

                     "num_flush_mode_low": 0,

                     "num_evict_mode_some": 0,

                     "num_evict_mode_full": 0,

                     "num_objects_pinned": 0,

                     "num_legacy_snapsets": 0,

                     "num_large_omap_objects": 0,

                     "num_objects_manifest": 0,

                     "num_omap_bytes": 0,

                     "num_omap_keys": 0,

                     "num_objects_repaired": 0

                 },

                 "up": [],

                 "acting": [],

                 "avail_no_missing": [],

                 "object_location_counts": [],

                 "blocked_by": [],

                 "up_primary": -1,

                 "acting_primary": -1,

                 "purged_snaps": []

             },

             "empty": 0,

             "dne": 0,

             "incomplete": 1,

             "last_epoch_started": 16616,

             "hit_set_history": {

                 "current_last_update": "0'0",

                 "history": []

             }

         }

     ],

     "recovery_state": [

         {

             "name": "Started/Primary/Active",

             "enter_time": "2022-12-26T21:27:49.970477+0000",

             "might_have_unfound": [],

             "recovery_progress": {

                 "backfill_targets": [

                     "5(0)",

                     "18(4)",

                     "24(5)",

                     "42(3)",

                     "58(2)"

                 ],

                 "waiting_on_backfill": [],

                 "last_backfill_started":
"14:ff09b920:::10000005377.00000209:head",

                 "backfill_info": {

                     "begin": "14:ff09bed4:::10000001a7f.0000182d:head",

                     "end": "14:ff0a8fd7:::1000000072c.000001a0:head",

                     "objects": [

                         {

                             "object":
"14:ff09bed4:::10000001a7f.0000182d:head",

                             "version": "3801'24969"

                         },

                         {

                             "object":
"14:ff09c414:::10000004daf.00001063:head",

                             "version": "6812'28471"

                         },

                         {

                             "object":
"14:ff09c530:::10000003957.00000189:head",

                             "version": "9167'29571"

                         },

                         {

                             "object":
"14:ff09c607:::100000011db.00000977:head",

                             "version": "3703'16620"

                         },

                         {

                             "object":
"14:ff09c922:::10000001230.000001e7:head",

                             "version": "3728'18321"

                         },

                         {

                             "object":
"14:ff09cb55:::1000000454e.000009a3:head",

                             "version": "3706'17783"

                         },

                         {

                             "object":
"14:ff09ce4d:::10000001316.00000154:head",

                             "version": "3798'23751"

                         },

                         {

                             "object":
"14:ff09d097:::10000003c4f.00000085:head",

                             "version": "3822'26172"

                         },

                         {

                             "object":
"14:ff09d449:::1000000188a.0000034a:head",

                             "version": "4627'27150"

                         },

                         {

                             "object":
"14:ff09d6b4:::10000003c4a.00000b0f:head",

                             "version": "3822'26170"

                         },

                         {

                             "object":
"14:ff09de4e:::10000001b63.000004c5:head",

                             "version": "3608'11396"

                         },

                         {

                             "object":
"14:ff09e1e3:::10000002a02.000016cb:head",

                             "version": "12960'31316"

                         },

                         {

                             "object":
"14:ff09f059:::10000002c09.00000095:head",

                             "version": "13226'31609"

                         },

                         {

                             "object":
"14:ff09f069:::1000000279e.0000012f:head",

                             "version": "13274'31844"

                         },

                         {

                             "object":
"14:ff0a0029:::10000001cd0.000000a2:head",

                             "version": "3702'16325"

                         },

                         {

                             "object":
"14:ff0a083b:::100000016c6.0000013e:head",

                             "version": "3497'8495"

                         },

                         {

                             "object":
"14:ff0a0d58:::1000000468d.00001cba:head",

                             "version": "9168'29618"

                         },

                         {

                             "object":
"14:ff0a0dfd:::10000000540.0000064a:head",

                             "version": "3493'8171"

                         },

                         {

                             "object":
"14:ff0a1149:::10000005049.00000b52:head",

                             "version": "3529'10013"

                         },

                         {

                             "object":
"14:ff0a1cdb:::100000009df.0000afec:head",

                             "version": "2106'380"

                         },

                         {

                             "object":
"14:ff0a1e1a:::1000000390b.00000106:head",

                             "version": "6812'28456"

                         },

                         {

                             "object":
"14:ff0a2180:::10000003868.000000b9:head",

                             "version": "3706'17681"

                         },

                         {

                             "object":
"14:ff0a2438:::1000000133e.00000024:head",

                             "version": "3798'24057"

                         },

                         {

                             "object":
"14:ff0a280e:::10000002ccc.000004af:head",

                             "version": "15259'33150"

                         },

                         {

                             "object":
"14:ff0a2f43:::10000003d40.00000665:head",

                             "version": "10508'29957"

                         },

                         {

                             "object":
"14:ff0a315f:::100000011db.00000caa:head",

                             "version": "3703'16628"

                         },

                         {

                             "object":
"14:ff0a3347:::10000004576.00001fb9:head",

                             "version": "3732'19654"

                         },

                         {

                             "object":
"14:ff0a3948:::100000032eb.0000011a:head",

                             "version": "3612'11760"

                         },

                         {

                             "object":
"14:ff0a3af1:::1000000275d.0000027a:head",

                             "version": "12960'31292"

                         },

                         {

                             "object":
"14:ff0a4073:::100000018fe.0000355d:head",

                             "version": "3498'8949"

                         },

                         {

                             "object":
"14:ff0a44db:::10000001a11.00000203:head",

                             "version": "3755'21336"

                         },

                         {

                             "object":
"14:ff0a4888:::1000000246e.0000360a:head",

                             "version": "13794'32296"

                         },

                         {

                             "object":
"14:ff0a4e86:::100000054b1.000019c1:head",

                             "version": "12890'30650"

                         },

                         {

                             "object":
"14:ff0a51c6:::10000003970.0000096e:head",

                             "version": "10636'30139"

                         },

                         {

                             "object":
"14:ff0a5302:::10000001871.000018eb:head",

                             "version": "3817'25516"

                         },

                         {

                             "object":
"14:ff0a56ba:::10000002848.00000266:head",

                             "version": "13855'32461"

                         },

                         {

                             "object":
"14:ff0a61f3:::10000000759.00000030:head",

                             "version": "3703'17434"

                         },

                         {

                             "object":
"14:ff0a68e3:::10000001c17.0000012c:head",

                             "version": "3653'13629"

                         },

                         {

                             "object":
"14:ff0a722a:::10000003e74.00000091:head",

                             "version": "3778'22970"

                         },

                         {

                             "object":
"14:ff0a7557:::100000044fe.00001664:head",

                             "version": "3660'14671"

                         },

                         {

                             "object":
"14:ff0a8f78:::10000001821.00000844:head",

                             "version": "3797'23629"

                         },

                         {

                             "object":
"14:ff0a8f8f:::10000001bea.00000202:head",

                             "version": "3653'13526"

                         }

                     ]

                 },

                 "peer_backfill_info": [

                     "5(0)",

                     {

                         "begin": "MAX",

                         "end": "MAX",

                         "objects": []

                     },

                     "18(4)",

                     {

                         "begin": "MAX",

                         "end": "MAX",

                         "objects": []

                     },

                     "24(5)",

                     {

                         "begin": "MAX",

                         "end": "MAX",

                         "objects": []

                     },

                     "42(3)",

                     {

                         "begin": "MAX",

                         "end": "MAX",

                         "objects": []

                     },

                     "58(2)",

                     {

                         "begin": "MAX",

                         "end": "MAX",

                         "objects": []

                     }

                 ],

                 "backfills_in_flight": [

                     "14:ff09b0d9:::100000033c2.00000146:head",

                     "14:ff09b6e5:::10000004b37.00000161:head",

                     "14:ff09b7b4:::1000000278f.00000374:head",

                     "14:ff09b920:::10000005377.00000209:head"

                 ],

                 "recovering": [

                     "14:ff09b0d9:::100000033c2.00000146:head",

                     "14:ff09b6e5:::10000004b37.00000161:head",

                     "14:ff09b7b4:::1000000278f.00000374:head",

                     "14:ff09b920:::10000005377.00000209:head"

                 ],

                 "pg_backend": {

                     "recovery_ops": [

                         {

                             "hoid":
"14:ff09b0d9:::100000033c2.00000146:head",

                             "v": "3677'15440",

                             "missing_on": "5(0),18(4),24(5),42(3),58(2)",

                             "missing_on_shards": "0,2,3,4,5",

                             "recovery_info":
"ObjectRecoveryInfo(14:ff09b0d9:::100000033c2.00000146:head@3677'15440,
size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
object_exist: 1)",

                             "recovery_progress":
"ObjectRecoveryProgress(!first, data_recovered_to:4194304,
data_complete:true, omap_recovered_to:, omap_complete:true, error:false)",

                             "state": "WRITING",

                             "waiting_on_pushes": "5(0)",

                             "extent_requested": "0,8388608"

                         },

                         {

                             "hoid":
"14:ff09b6e5:::10000004b37.00000161:head",

                             "v": "3549'10993",

                             "missing_on": "5(0),18(4),24(5),42(3),58(2)",

                             "missing_on_shards": "0,2,3,4,5",

                             "recovery_info":
"ObjectRecoveryInfo(14:ff09b6e5:::10000004b37.00000161:head@3549'10993,
size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
object_exist: 1)",

                             "recovery_progress":
"ObjectRecoveryProgress(!first, data_recovered_to:4194304,
data_complete:true, omap_recovered_to:, omap_complete:true, error:false)",

                             "state": "WRITING",

                             "waiting_on_pushes": "5(0)",

                             "extent_requested": "0,8388608"

                         },

                         {

                             "hoid":
"14:ff09b7b4:::1000000278f.00000374:head",

                             "v": "13272'31836",

                             "missing_on": "5(0),18(4),24(5),42(3),58(2)",

                             "missing_on_shards": "0,2,3,4,5",

                             "recovery_info":
"ObjectRecoveryInfo(14:ff09b7b4:::1000000278f.00000374:head@13272'31836,
size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
object_exist: 1)",

                             "recovery_progress":
"ObjectRecoveryProgress(!first, data_recovered_to:4194304,
data_complete:true, omap_recovered_to:, omap_complete:true, error:false)",

                             "state": "WRITING",

                             "waiting_on_pushes": "5(0)",

                             "extent_requested": "0,8388608"

                         },

                         {

                             "hoid":
"14:ff09b920:::10000005377.00000209:head",

                             "v": "10440'29880",

                             "missing_on": "5(0),18(4),24(5),42(3),58(2)",

                             "missing_on_shards": "0,2,3,4,5",

                             "recovery_info":
"ObjectRecoveryInfo(14:ff09b920:::10000005377.00000209:head@10440'29880,
size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
object_exist: 1)",

                             "recovery_progress":
"ObjectRecoveryProgress(!first, data_recovered_to:4194304,
data_complete:true, omap_recovered_to:, omap_complete:true, error:false)",

                             "state": "WRITING",

                             "waiting_on_pushes": "5(0)",

                             "extent_requested": "0,8388608"

                         }

                     ],

                     "read_ops": []

                 }

             }

         },

         {

             "name": "Started",

             "enter_time": "2022-12-26T21:27:48.908226+0000"

         }

     ],

     "scrubber": {

         "active": false,

         "must_scrub": true,

         "must_deep_scrub": true,

         "must_repair": false,

         "need_auto": false,

         "scrub_reg_stamp": "1.000000",

         "schedule": "queued for deep scrub"

     },

     "agent_state": {}

}



On Wed, Dec 28, 2022 at 6:46 AM Pavin Joseph <me@xxxxxxxxxxxxxxx> wrote:

1. This is a guess, but check /var/[lib|run]/ceph for any lock files.
2. This is more straightforward to fix, add faster WAL/Block device/LV
for each OSD or create a fast storage pool just for metadata. Also,
experiment with MDS cache size/trim [0] settings.

[0]: https://docs.ceph.com/en/latest/cephfs/cache-configuration/

On 28-Dec-22 7:23 AM, Deep Dish wrote:
Got logging enabled as per
https://ceph.io/en/news/blog/2022/centralized_logging/.   My embedded
grafana doesn't come up in the dashboard, but at least I have log (files)
on my nodes.   Interesting.

Two issues plaguing my cluster:

1 - RGWs not manageable
2 - MDS_SLOW_METADATA_IO warning (impact to cephfs)

Issue 1:

I have 4x RGWs deployed.   All started / processes running.  They all
report similar log entries:

7fcc32b6a5c0  0 deferred set uid:gid to 167:167 (ceph:ceph)

7fcc32b6a5c0  0 ceph version 17.2.5
(98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
radosgw, pid 2

7fcc32b6a5c0  0 framework: beast

7fcc32b6a5c0  0 framework conf key: port, val: 80

7fcc32b6a5c0  1 radosgw_Main not setting numa affinity

7fcc32b6a5c0  1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0

7fcc32b6a5c0  1 D3N datacache enabled: 0

7fcc0869a700  0 INFO: RGWReshardLock::lock found lock on
reshard.0000000011
to be held by another RGW process; skipping for now

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.1,
sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.3,
sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again
(repeating)

Seems like a stale lock, not previously cleaned up when the cluster was
busy recovering and rebalancing.

Issue 2:

ceph health detail:

[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs

      mds.fs01.ceph02mon03.rjcxat(mds.0): 8 slow metadata IOs are blocked

30 secs, oldest blocked for 39485 secs

Log entries from ceph02mon03 MDS host:

   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131271 from mon.4
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131272 from mon.4
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131273 from mon.4
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131274 from mon.4
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131275 from mon.4
   7fe36c6b8700  0 log_channel(cluster) log [WRN] : 1 slow requests, 1
included below; oldest blocked for > 33.126589 secs
   7fe36c6b8700  0 log_channel(cluster) log [WRN] : slow request 33.126588
seconds old, received at 2022-12-27T19:45:45.952225+0000:
client_request(client.55009:99980 create
#0x10000000bc2/vzdump-qemu-30003-2022_12_27-14_43_43.log
2022-12-27T19:45:45.948045+0000 caller_uid=0, caller_gid=0{}) currently
submit entry: journal_and_reply
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131276 from mon.4
   7fe36c6b8700  0 log_channel(cluster) log [WRN] : 1 slow requests, 0
included below; oldest blocked for > 38.126737 secs
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131277 from mon.4
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131278 from mon.4
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131279 from mon.4
   7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131280 from mon.4


I suspect that the file in the log above int's the culprit.   How can I
get
to the root cause of MDS slowdowns?


On Tue, Dec 27, 2022 at 3:32 PM Pavin Joseph <me@xxxxxxxxxxxxxxx> wrote:

Interesting, the logs show the crash module [0] itself has crashed.
Something sent it a SIGINT or SIGTERM and the module didn't handle it
correctly due to what seems like a bug in the code.

I haven't experienced the crash module itself crashing yet (in Quincy)
because nothing sent a SIG[INT|TERM] to it yet.

So I'd continue investigating into why these signals were sent to the
crash module.

To fix the crash module from crashing, go to "/usr/bin/ceph-crash" and
edit the handler function on line 82 like so:

def handler(signum, frame):
     print('*** Interrupted with signal %d ***' % signum)
     signame = signal.Signals(signum).name
     print(f'Signal handler called with signal {signame} ({signum})')
     print(frame)
     sys.exit(0)

---

Once the crash module is working, perhaps you could run a "ceph crash
ls"

Regarding podman logs, perhaps try this [1].

[0]: https://docs.ceph.com/en/quincy/mgr/crash/
[1]: https://docs.podman.io/en/latest/markdown/podman-logs.1.html

On 27-Dec-22 11:59 PM, Deep Dish wrote:
HI Pavin,

Thanks for the reply.   I'm a bit at a loss honestly as this worked
perfectly without any issue up until the rebalance of the cluster.
Orchestrator is great.   Aside from this (which I suspect is not
orchestrator related), I haven't had any issues.

In terms of logs, I'm not sure where to start looking in this new
containerized environment as they pertain to individual ceph processes
-- I
assumed everything would be centrally collected within orch.

Connecting into the podman container of a RGW, there are no logs in
/var/log/ceph aside from ceph-volume.   My ceph.conf is minimal with
only
monitors defined.  The only log I'm able to pull is as follows:

# podman logs 35d4ac5445ca

INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s

Traceback (most recent call last):

     File "/usr/bin/ceph-crash", line 113, in <module>

       main()

     File "/usr/bin/ceph-crash", line 109, in main

       time.sleep(args.delay * 60)

TypeError: handler() takes 1 positional argument but 2 were given

INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s



Looks like the RGW daemon is crashing.   How do I get logs to persist?
   I
suspect I won't be able to use orchestrator to push down the config,
and
would have to manipulate within the container image itself.

I also attempted to redeply the RGW containers without success.

On Tue, Dec 27, 2022 at 10:39 AM Pavin Joseph <me@xxxxxxxxxxxxxxx>
wrote:

Here's the first things I'd check in your situation:

1. Logs
2. Is the RGW HTTP server running on its port?
3. Re-check config including authentication.

ceph orch is too new and didn't pass muster in our own internal
testing.
You're braver than most for using it in production.

Pavin.

On 27-Dec-22 8:48 PM, Deep Dish wrote:
Quick update:

- I followed documentation, and ran the following:

# ceph dashboard set-rgw-credentials

Error EINVAL: No RGW credentials found, please consult the
documentation
on
how to enable RGW for the dashboard.



- I see dashboard credentials configured (all this was working fine
before):


# ceph dashboard get-rgw-api-access-key

P?????????????????G  (? commented out)



Seems to me like my RGW config is non-existent / corrupted for some
reason.  When trying to curl a RGW directly I get a "connection
refused".



On Tue, Dec 27, 2022 at 9:41 AM Deep Dish <deeepdish@xxxxxxxxx>
wrote:

I built a net-new Quincy cluster (17.2.5) using ceph orch as
follows:

2x mgrs
4x rgw
5x mon
4x rgw
5x mds
6x osd hosts w/ 10 drives each --> will be growing to 7 osd hosts in
the
coming days.

I migrated all data from my legacy nautilus cluster (via rbd-mirror,
rclone for s3 buckets, etc.).  All moved over successfully without
issue.

The cluster went through a series of rebalancing events (adding
capacity,
osd nodes, changing fault domain for EC volumes).

It's settled now, however throughout the process all of my RGW nodes
are
no longer part of the cluster -- meaning ceph doesn't recognize /
detect
them, despite containers, networking, etc. all being setup
correctly.
This also means I'm unable to manage any RGW functions (via the
dashboard
or cli).   As an example via cli (within Cephadm shell):

# radosgw-admin pools list

could not list placement set: (2) No such file or directory

I have data in buckets, how can I get my RGWs to return online?

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux