Hi Pavin,
The following are additional developments.. There's one PG that's
stuck and unable to recover. I've attached relevant ceph -s / health
detail and pg stat outputs below.
- There were some remaining lock files as suggested in /var/run/ceph/
pertaining to rgw. I removed the service, deleted any stale lock files
and redeployed the RGWs. All started with the common log entries across
all:
7ff5d9aaf5c0 0 deferred set uid:gid to 167:167 (ceph:ceph)
7ff5d9aaf5c0 0 ceph version 17.2.5
(98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
radosgw, pid 2
7ff5d9aaf5c0 0 framework: beast
7ff5d9aaf5c0 0 framework conf key: port, val: 80
7ff5d9aaf5c0 1 radosgw_Main not setting numa affinity
7ff5d9aaf5c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0
7ff5d9aaf5c0 1 D3N datacache enabled: 0
No additional log entries are recorded since starting them post
re-deployment as per above.
The cluster settled, there is no recovery activity. There is one pg that's
stuck and I have a hunch that it's impacting MDS and RGW processes as
stated in the thread. PG is stuck as as active+remapped+backfilling:
data:
volumes: 2/2 healthy
pools: 16 pools, 1504 pgs
objects: 24.49M objects, 79 TiB
usage: 119 TiB used, 390 TiB / 508 TiB avail
pgs: 65210/146755179 objects misplaced (0.044%)
1503 active+clean
1 active+remapped+backfilling
progress:
Global Recovery Event (6h)
[===========================.] (remaining: 73s)
# ceph health detail
HEALTH_WARN 1 MDSs report slow metadata IOs; 1 pgs not deep-scrubbed in
time; 1 pgs not scrubbed in time
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
mds.fs01.ceph02mon02.wicrdz(mds.0): 5 slow metadata IOs are blocked >
30 secs, oldest blocked for 74436 secs
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 14.ff not deep-scrubbed since 2022-12-14T19:35:51.893008+0000
[WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
pg 14.ff not scrubbed since 2022-12-17T06:33:40.577932+0000
>From the following pg query:
- "pgid": "14.ffs0" is stuck as peering (osd 5)
- "pgid": "14.ffs4" is stuck as unknown (osd 18)
- "pgid": "14.ffs5" is stuck as unknown (osd 24)
- "pgid": "14.ffs3" is stuck as unknown (osd 42)
- "pgid": "14.ffs2" is stick as unknown (osd 58)
- "pgid": "14.ffs1" is marked as active+clean (osd 36)
# ceph pg 14.ff query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+remapped+backfilling",
"epoch": 19594,
"up": [
5,
36,
58,
42,
18,
24
],
"acting": [
50,
36,
5,
26,
15,
46
],
"backfill_targets": [
"5(0)",
"18(4)",
"24(5)",
"42(3)",
"58(2)"
],
"acting_recovery_backfill": [
"5(0)",
"5(2)",
"15(4)",
"18(4)",
"24(5)",
"26(3)",
"36(1)",
"42(3)",
"46(5)",
"50(0)",
"58(2)"
],
"info": {
"pgid": "14.ffs0",
"last_update": "19550'35077",
"last_complete": "19550'35077",
"log_tail": "13761'32157",
"last_user_version": 35077,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "19550'35077",
"reported_seq": 396919,
"reported_epoch": 19594,
"state": "active+remapped+backfilling",
"last_fresh": "2022-12-28T22:03:20.278478+0000",
"last_change": "2022-12-26T21:27:51.600940+0000",
"last_active": "2022-12-28T22:03:20.278478+0000",
"last_peered": "2022-12-28T22:03:20.278478+0000",
"last_clean": "2022-12-26T21:27:45.471954+0000",
"last_became_active": "2022-12-26T21:27:51.085966+0000",
"last_became_peered": "2022-12-26T21:27:51.085966+0000",
"last_unstale": "2022-12-28T22:03:20.278478+0000",
"last_undegraded": "2022-12-28T22:03:20.278478+0000",
"last_fullsized": "2022-12-28T22:03:20.278478+0000",
"mapping_epoch": 16615,
"log_start": "13761'32157",
"ondisk_log_start": "13761'32157",
"created": 4537,
"last_epoch_clean": 14655,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"objects_scrubbed": 16227,
"log_size": 2920,
"ondisk_log_size": 2920,
"stats_invalid": true,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.320415128,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 56709530650,
"num_objects": 13548,
"num_object_clones": 0,
"num_object_copies": 81288,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 65210,
"num_objects_unfound": 0,
"num_objects_dirty": 13548,
"num_whiteouts": 0,
"num_read": 67760,
"num_read_kb": 177798674,
"num_write": 21231,
"num_write_kb": 70024901,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34481,
"num_bytes_recovered": 144364295675,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
36,
58,
42,
18,
24
],
"acting": [
50,
36,
5,
26,
15,
46
],
"avail_no_missing": [
"50(0)",
"5(2)",
"15(4)",
"26(3)",
"36(1)",
"46(5)"
],
"object_location_counts": [
{
"shards": "5(2),15(4),26(3),36(1),46(5),50(0)",
"objects": 13548
}
],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 50,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "5(0)",
"pgid": "14.ffs0",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 0,
"last_backfill": "14:ff09a915:::10000001249.00000353:head",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 2,
"reported_epoch": 16614,
"state": "peering",
"last_fresh": "2022-12-26T21:27:47.993646+0000",
"last_change": "2022-12-26T21:27:47.769141+0000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2022-12-26T21:27:47.993646+0000",
"last_undegraded": "2022-12-26T21:27:47.993646+0000",
"last_fullsized": "2022-12-26T21:27:47.993646+0000",
"mapping_epoch": 16615,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 4537,
"last_epoch_clean": 14655,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"objects_scrubbed": 0,
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 2115497390,
"num_objects": 506,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 13042,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 506,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 42,
"num_write_kb": 172032,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
36,
58,
42,
18,
24
],
"acting": [
50,
36,
5,
26,
15,
46
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
24
],
"up_primary": 5,
"acting_primary": 50,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "5(2)",
"pgid": "14.ffs2",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 33714,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "16465'33714",
"reported_seq": 388679,
"reported_epoch": 16465,
"state": "active+clean",
"last_fresh": "2022-12-26T16:10:12.977587+0000",
"last_change": "2022-12-26T02:28:03.455705+0000",
"last_active": "2022-12-26T16:10:12.977587+0000",
"last_peered": "2022-12-26T16:10:12.977587+0000",
"last_clean": "2022-12-26T16:10:12.977587+0000",
"last_became_active": "2022-12-26T02:28:03.455192+0000",
"last_became_peered": "2022-12-26T02:28:03.455192+0000",
"last_unstale": "2022-12-26T16:10:12.977587+0000",
"last_undegraded": "2022-12-26T16:10:12.977587+0000",
"last_fullsized": "2022-12-26T16:10:12.977587+0000",
"mapping_epoch": 16615,
"log_start": "12957'31152",
"ondisk_log_start": "12957'31152",
"created": 4537,
"last_epoch_clean": 14655,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"objects_scrubbed": 16227,
"log_size": 2562,
"ondisk_log_size": 2562,
"stats_invalid": true,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.320415128,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 51491812456,
"num_objects": 12301,
"num_object_clones": 0,
"num_object_copies": 73806,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 12301,
"num_whiteouts": 0,
"num_read": 66867,
"num_read_kb": 177079789,
"num_write": 19946,
"num_write_kb": 64929471,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34017,
"num_bytes_recovered": 142424959053,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
36,
58,
42,
18,
24
],
"acting": [
50,
36,
5,
26,
15,
46
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 50,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "15(4)",
"pgid": "14.ffs4",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 33714,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "16465'33714",
"reported_seq": 388679,
"reported_epoch": 16465,
"state": "active+clean",
"last_fresh": "2022-12-26T16:10:12.977587+0000",
"last_change": "2022-12-26T02:28:03.455705+0000",
"last_active": "2022-12-26T16:10:12.977587+0000",
"last_peered": "2022-12-26T16:10:12.977587+0000",
"last_clean": "2022-12-26T16:10:12.977587+0000",
"last_became_active": "2022-12-26T02:28:03.455192+0000",
"last_became_peered": "2022-12-26T02:28:03.455192+0000",
"last_unstale": "2022-12-26T16:10:12.977587+0000",
"last_undegraded": "2022-12-26T16:10:12.977587+0000",
"last_fullsized": "2022-12-26T16:10:12.977587+0000",
"mapping_epoch": 16615,
"log_start": "12957'31152",
"ondisk_log_start": "12957'31152",
"created": 4537,
"last_epoch_clean": 14655,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"objects_scrubbed": 16227,
"log_size": 2562,
"ondisk_log_size": 2562,
"stats_invalid": true,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.320415128,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 51491812456,
"num_objects": 12301,
"num_object_clones": 0,
"num_object_copies": 73806,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 12301,
"num_whiteouts": 0,
"num_read": 66867,
"num_read_kb": 177079789,
"num_write": 19946,
"num_write_kb": 64929471,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34017,
"num_bytes_recovered": 142424959053,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
36,
58,
42,
18,
24
],
"acting": [
50,
36,
5,
26,
15,
46
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 50,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "18(4)",
"pgid": "14.ffs4",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 0,
"last_backfill": "14:ff09a915:::10000001249.00000353:head",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 2115497390,
"num_objects": 506,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 13042,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 506,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 42,
"num_write_kb": 172032,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "24(5)",
"pgid": "14.ffs5",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 0,
"last_backfill": "14:ff09a915:::10000001249.00000353:head",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 2115497390,
"num_objects": 506,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 13042,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 506,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 42,
"num_write_kb": 172032,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "26(3)",
"pgid": "14.ffs3",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 33714,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "16465'33714",
"reported_seq": 388679,
"reported_epoch": 16465,
"state": "active+clean",
"last_fresh": "2022-12-26T16:10:12.977587+0000",
"last_change": "2022-12-26T02:28:03.455705+0000",
"last_active": "2022-12-26T16:10:12.977587+0000",
"last_peered": "2022-12-26T16:10:12.977587+0000",
"last_clean": "2022-12-26T16:10:12.977587+0000",
"last_became_active": "2022-12-26T02:28:03.455192+0000",
"last_became_peered": "2022-12-26T02:28:03.455192+0000",
"last_unstale": "2022-12-26T16:10:12.977587+0000",
"last_undegraded": "2022-12-26T16:10:12.977587+0000",
"last_fullsized": "2022-12-26T16:10:12.977587+0000",
"mapping_epoch": 16615,
"log_start": "12957'31152",
"ondisk_log_start": "12957'31152",
"created": 4537,
"last_epoch_clean": 14655,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"objects_scrubbed": 16227,
"log_size": 2562,
"ondisk_log_size": 2562,
"stats_invalid": true,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.320415128,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 51491812456,
"num_objects": 12301,
"num_object_clones": 0,
"num_object_copies": 73806,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 12301,
"num_whiteouts": 0,
"num_read": 66867,
"num_read_kb": 177079789,
"num_write": 19946,
"num_write_kb": 64929471,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34017,
"num_bytes_recovered": 142424959053,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
36,
58,
42,
18,
24
],
"acting": [
50,
36,
5,
26,
15,
46
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 50,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "36(1)",
"pgid": "14.ffs1",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 33714,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "16465'33714",
"reported_seq": 388679,
"reported_epoch": 16465,
"state": "active+clean",
"last_fresh": "2022-12-26T16:10:12.977587+0000",
"last_change": "2022-12-26T02:28:03.455705+0000",
"last_active": "2022-12-26T16:10:12.977587+0000",
"last_peered": "2022-12-26T16:10:12.977587+0000",
"last_clean": "2022-12-26T16:10:12.977587+0000",
"last_became_active": "2022-12-26T02:28:03.455192+0000",
"last_became_peered": "2022-12-26T02:28:03.455192+0000",
"last_unstale": "2022-12-26T16:10:12.977587+0000",
"last_undegraded": "2022-12-26T16:10:12.977587+0000",
"last_fullsized": "2022-12-26T16:10:12.977587+0000",
"mapping_epoch": 16615,
"log_start": "12957'31152",
"ondisk_log_start": "12957'31152",
"created": 4537,
"last_epoch_clean": 14655,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"objects_scrubbed": 16227,
"log_size": 2562,
"ondisk_log_size": 2562,
"stats_invalid": true,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.320415128,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 51491812456,
"num_objects": 12301,
"num_object_clones": 0,
"num_object_copies": 73806,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 12301,
"num_whiteouts": 0,
"num_read": 66867,
"num_read_kb": 177079789,
"num_write": 19946,
"num_write_kb": 64929471,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34017,
"num_bytes_recovered": 142424959053,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
36,
58,
42,
18,
24
],
"acting": [
50,
36,
5,
26,
15,
46
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 50,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "42(3)",
"pgid": "14.ffs3",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 0,
"last_backfill": "14:ff09a915:::10000001249.00000353:head",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 2115497390,
"num_objects": 506,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 13042,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 506,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 42,
"num_write_kb": 172032,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "46(5)",
"pgid": "14.ffs5",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 33714,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "16465'33714",
"reported_seq": 388679,
"reported_epoch": 16465,
"state": "active+clean",
"last_fresh": "2022-12-26T16:10:12.977587+0000",
"last_change": "2022-12-26T02:28:03.455705+0000",
"last_active": "2022-12-26T16:10:12.977587+0000",
"last_peered": "2022-12-26T16:10:12.977587+0000",
"last_clean": "2022-12-26T16:10:12.977587+0000",
"last_became_active": "2022-12-26T02:28:03.455192+0000",
"last_became_peered": "2022-12-26T02:28:03.455192+0000",
"last_unstale": "2022-12-26T16:10:12.977587+0000",
"last_undegraded": "2022-12-26T16:10:12.977587+0000",
"last_fullsized": "2022-12-26T16:10:12.977587+0000",
"mapping_epoch": 16615,
"log_start": "12957'31152",
"ondisk_log_start": "12957'31152",
"created": 4537,
"last_epoch_clean": 14655,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"objects_scrubbed": 16227,
"log_size": 2562,
"ondisk_log_size": 2562,
"stats_invalid": true,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.320415128,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 51491812456,
"num_objects": 12301,
"num_object_clones": 0,
"num_object_copies": 73806,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 12301,
"num_whiteouts": 0,
"num_read": 66867,
"num_read_kb": 177079789,
"num_write": 19946,
"num_write_kb": 64929471,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34017,
"num_bytes_recovered": 142424959053,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
36,
58,
42,
18,
24
],
"acting": [
50,
36,
5,
26,
15,
46
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 50,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "58(2)",
"pgid": "14.ffs2",
"last_update": "19550'35077",
"last_complete": "17223'34381",
"log_tail": "12957'31152",
"last_user_version": 0,
"last_backfill": "14:ff09a915:::10000001249.00000353:head",
"purged_snaps": [],
"history": {
"epoch_created": 4537,
"epoch_pool_created": 2032,
"last_epoch_started": 16616,
"last_interval_started": 16615,
"last_epoch_clean": 14655,
"last_interval_clean": 14654,
"last_epoch_split": 4537,
"last_epoch_marked_full": 0,
"same_up_since": 16613,
"same_interval_since": 16615,
"same_primary_since": 16615,
"last_scrub": "3817'25569",
"last_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"last_deep_scrub": "3756'21592",
"last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000",
"last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 2115497390,
"num_objects": 506,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 13042,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 506,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 42,
"num_write_kb": 172032,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 16616,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2022-12-26T21:27:49.970477+0000",
"might_have_unfound": [],
"recovery_progress": {
"backfill_targets": [
"5(0)",
"18(4)",
"24(5)",
"42(3)",
"58(2)"
],
"waiting_on_backfill": [],
"last_backfill_started":
"14:ff09b920:::10000005377.00000209:head",
"backfill_info": {
"begin": "14:ff09bed4:::10000001a7f.0000182d:head",
"end": "14:ff0a8fd7:::1000000072c.000001a0:head",
"objects": [
{
"object":
"14:ff09bed4:::10000001a7f.0000182d:head",
"version": "3801'24969"
},
{
"object":
"14:ff09c414:::10000004daf.00001063:head",
"version": "6812'28471"
},
{
"object":
"14:ff09c530:::10000003957.00000189:head",
"version": "9167'29571"
},
{
"object":
"14:ff09c607:::100000011db.00000977:head",
"version": "3703'16620"
},
{
"object":
"14:ff09c922:::10000001230.000001e7:head",
"version": "3728'18321"
},
{
"object":
"14:ff09cb55:::1000000454e.000009a3:head",
"version": "3706'17783"
},
{
"object":
"14:ff09ce4d:::10000001316.00000154:head",
"version": "3798'23751"
},
{
"object":
"14:ff09d097:::10000003c4f.00000085:head",
"version": "3822'26172"
},
{
"object":
"14:ff09d449:::1000000188a.0000034a:head",
"version": "4627'27150"
},
{
"object":
"14:ff09d6b4:::10000003c4a.00000b0f:head",
"version": "3822'26170"
},
{
"object":
"14:ff09de4e:::10000001b63.000004c5:head",
"version": "3608'11396"
},
{
"object":
"14:ff09e1e3:::10000002a02.000016cb:head",
"version": "12960'31316"
},
{
"object":
"14:ff09f059:::10000002c09.00000095:head",
"version": "13226'31609"
},
{
"object":
"14:ff09f069:::1000000279e.0000012f:head",
"version": "13274'31844"
},
{
"object":
"14:ff0a0029:::10000001cd0.000000a2:head",
"version": "3702'16325"
},
{
"object":
"14:ff0a083b:::100000016c6.0000013e:head",
"version": "3497'8495"
},
{
"object":
"14:ff0a0d58:::1000000468d.00001cba:head",
"version": "9168'29618"
},
{
"object":
"14:ff0a0dfd:::10000000540.0000064a:head",
"version": "3493'8171"
},
{
"object":
"14:ff0a1149:::10000005049.00000b52:head",
"version": "3529'10013"
},
{
"object":
"14:ff0a1cdb:::100000009df.0000afec:head",
"version": "2106'380"
},
{
"object":
"14:ff0a1e1a:::1000000390b.00000106:head",
"version": "6812'28456"
},
{
"object":
"14:ff0a2180:::10000003868.000000b9:head",
"version": "3706'17681"
},
{
"object":
"14:ff0a2438:::1000000133e.00000024:head",
"version": "3798'24057"
},
{
"object":
"14:ff0a280e:::10000002ccc.000004af:head",
"version": "15259'33150"
},
{
"object":
"14:ff0a2f43:::10000003d40.00000665:head",
"version": "10508'29957"
},
{
"object":
"14:ff0a315f:::100000011db.00000caa:head",
"version": "3703'16628"
},
{
"object":
"14:ff0a3347:::10000004576.00001fb9:head",
"version": "3732'19654"
},
{
"object":
"14:ff0a3948:::100000032eb.0000011a:head",
"version": "3612'11760"
},
{
"object":
"14:ff0a3af1:::1000000275d.0000027a:head",
"version": "12960'31292"
},
{
"object":
"14:ff0a4073:::100000018fe.0000355d:head",
"version": "3498'8949"
},
{
"object":
"14:ff0a44db:::10000001a11.00000203:head",
"version": "3755'21336"
},
{
"object":
"14:ff0a4888:::1000000246e.0000360a:head",
"version": "13794'32296"
},
{
"object":
"14:ff0a4e86:::100000054b1.000019c1:head",
"version": "12890'30650"
},
{
"object":
"14:ff0a51c6:::10000003970.0000096e:head",
"version": "10636'30139"
},
{
"object":
"14:ff0a5302:::10000001871.000018eb:head",
"version": "3817'25516"
},
{
"object":
"14:ff0a56ba:::10000002848.00000266:head",
"version": "13855'32461"
},
{
"object":
"14:ff0a61f3:::10000000759.00000030:head",
"version": "3703'17434"
},
{
"object":
"14:ff0a68e3:::10000001c17.0000012c:head",
"version": "3653'13629"
},
{
"object":
"14:ff0a722a:::10000003e74.00000091:head",
"version": "3778'22970"
},
{
"object":
"14:ff0a7557:::100000044fe.00001664:head",
"version": "3660'14671"
},
{
"object":
"14:ff0a8f78:::10000001821.00000844:head",
"version": "3797'23629"
},
{
"object":
"14:ff0a8f8f:::10000001bea.00000202:head",
"version": "3653'13526"
}
]
},
"peer_backfill_info": [
"5(0)",
{
"begin": "MAX",
"end": "MAX",
"objects": []
},
"18(4)",
{
"begin": "MAX",
"end": "MAX",
"objects": []
},
"24(5)",
{
"begin": "MAX",
"end": "MAX",
"objects": []
},
"42(3)",
{
"begin": "MAX",
"end": "MAX",
"objects": []
},
"58(2)",
{
"begin": "MAX",
"end": "MAX",
"objects": []
}
],
"backfills_in_flight": [
"14:ff09b0d9:::100000033c2.00000146:head",
"14:ff09b6e5:::10000004b37.00000161:head",
"14:ff09b7b4:::1000000278f.00000374:head",
"14:ff09b920:::10000005377.00000209:head"
],
"recovering": [
"14:ff09b0d9:::100000033c2.00000146:head",
"14:ff09b6e5:::10000004b37.00000161:head",
"14:ff09b7b4:::1000000278f.00000374:head",
"14:ff09b920:::10000005377.00000209:head"
],
"pg_backend": {
"recovery_ops": [
{
"hoid":
"14:ff09b0d9:::100000033c2.00000146:head",
"v": "3677'15440",
"missing_on": "5(0),18(4),24(5),42(3),58(2)",
"missing_on_shards": "0,2,3,4,5",
"recovery_info":
"ObjectRecoveryInfo(14:ff09b0d9:::100000033c2.00000146:head@3677'15440,
size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
object_exist: 1)",
"recovery_progress":
"ObjectRecoveryProgress(!first, data_recovered_to:4194304,
data_complete:true, omap_recovered_to:, omap_complete:true, error:false)",
"state": "WRITING",
"waiting_on_pushes": "5(0)",
"extent_requested": "0,8388608"
},
{
"hoid":
"14:ff09b6e5:::10000004b37.00000161:head",
"v": "3549'10993",
"missing_on": "5(0),18(4),24(5),42(3),58(2)",
"missing_on_shards": "0,2,3,4,5",
"recovery_info":
"ObjectRecoveryInfo(14:ff09b6e5:::10000004b37.00000161:head@3549'10993,
size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
object_exist: 1)",
"recovery_progress":
"ObjectRecoveryProgress(!first, data_recovered_to:4194304,
data_complete:true, omap_recovered_to:, omap_complete:true, error:false)",
"state": "WRITING",
"waiting_on_pushes": "5(0)",
"extent_requested": "0,8388608"
},
{
"hoid":
"14:ff09b7b4:::1000000278f.00000374:head",
"v": "13272'31836",
"missing_on": "5(0),18(4),24(5),42(3),58(2)",
"missing_on_shards": "0,2,3,4,5",
"recovery_info":
"ObjectRecoveryInfo(14:ff09b7b4:::1000000278f.00000374:head@13272'31836,
size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
object_exist: 1)",
"recovery_progress":
"ObjectRecoveryProgress(!first, data_recovered_to:4194304,
data_complete:true, omap_recovered_to:, omap_complete:true, error:false)",
"state": "WRITING",
"waiting_on_pushes": "5(0)",
"extent_requested": "0,8388608"
},
{
"hoid":
"14:ff09b920:::10000005377.00000209:head",
"v": "10440'29880",
"missing_on": "5(0),18(4),24(5),42(3),58(2)",
"missing_on_shards": "0,2,3,4,5",
"recovery_info":
"ObjectRecoveryInfo(14:ff09b920:::10000005377.00000209:head@10440'29880,
size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{},
object_exist: 1)",
"recovery_progress":
"ObjectRecoveryProgress(!first, data_recovered_to:4194304,
data_complete:true, omap_recovered_to:, omap_complete:true, error:false)",
"state": "WRITING",
"waiting_on_pushes": "5(0)",
"extent_requested": "0,8388608"
}
],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2022-12-26T21:27:48.908226+0000"
}
],
"scrubber": {
"active": false,
"must_scrub": true,
"must_deep_scrub": true,
"must_repair": false,
"need_auto": false,
"scrub_reg_stamp": "1.000000",
"schedule": "queued for deep scrub"
},
"agent_state": {}
}
On Wed, Dec 28, 2022 at 6:46 AM Pavin Joseph <me@xxxxxxxxxxxxxxx> wrote:
> 1. This is a guess, but check /var/[lib|run]/ceph for any lock files.
> 2. This is more straightforward to fix, add faster WAL/Block device/LV
> for each OSD or create a fast storage pool just for metadata. Also,
> experiment with MDS cache size/trim [0] settings.
>
> [0]: https://docs.ceph.com/en/latest/cephfs/cache-configuration/
>
> On 28-Dec-22 7:23 AM, Deep Dish wrote:
> > Got logging enabled as per
> > https://ceph.io/en/news/blog/2022/centralized_logging/. My embedded
> > grafana doesn't come up in the dashboard, but at least I have log (files)
> > on my nodes. Interesting.
> >
> > Two issues plaguing my cluster:
> >
> > 1 - RGWs not manageable
> > 2 - MDS_SLOW_METADATA_IO warning (impact to cephfs)
> >
> > Issue 1:
> >
> > I have 4x RGWs deployed. All started / processes running. They all
> > report similar log entries:
> >
> > 7fcc32b6a5c0 0 deferred set uid:gid to 167:167 (ceph:ceph)
> >
> > 7fcc32b6a5c0 0 ceph version 17.2.5
> > (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
> > radosgw, pid 2
> >
> > 7fcc32b6a5c0 0 framework: beast
> >
> > 7fcc32b6a5c0 0 framework conf key: port, val: 80
> >
> > 7fcc32b6a5c0 1 radosgw_Main not setting numa affinity
> >
> > 7fcc32b6a5c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0
> >
> > 7fcc32b6a5c0 1 D3N datacache enabled: 0
> >
> > 7fcc0869a700 0 INFO: RGWReshardLock::lock found lock on
> reshard.0000000011
> > to be held by another RGW process; skipping for now
> >
> > 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
> lc.1,
> > sleep 5, try again
> >
> > 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
> lc.3,
> > sleep 5, try again
> >
> > 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> >
> > 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> >
> > 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> >
> > 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> >
> > 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> >
> > 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> >
> > 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> >
> > 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> >
> > 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on
> > lc.16, sleep 5, try again
> > (repeating)
> >
> > Seems like a stale lock, not previously cleaned up when the cluster was
> > busy recovering and rebalancing.
> >
> > Issue 2:
> >
> > ceph health detail:
> >
> > [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
> >
> > mds.fs01.ceph02mon03.rjcxat(mds.0): 8 slow metadata IOs are blocked
> >
> > 30 secs, oldest blocked for 39485 secs
> >
> > Log entries from ceph02mon03 MDS host:
> >
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131271 from mon.4
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131272 from mon.4
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131273 from mon.4
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131274 from mon.4
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131275 from mon.4
> > 7fe36c6b8700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1
> > included below; oldest blocked for > 33.126589 secs
> > 7fe36c6b8700 0 log_channel(cluster) log [WRN] : slow request 33.126588
> > seconds old, received at 2022-12-27T19:45:45.952225+0000:
> > client_request(client.55009:99980 create
> > #0x10000000bc2/vzdump-qemu-30003-2022_12_27-14_43_43.log
> > 2022-12-27T19:45:45.948045+0000 caller_uid=0, caller_gid=0{}) currently
> > submit entry: journal_and_reply
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131276 from mon.4
> > 7fe36c6b8700 0 log_channel(cluster) log [WRN] : 1 slow requests, 0
> > included below; oldest blocked for > 38.126737 secs
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131277 from mon.4
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131278 from mon.4
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131279 from mon.4
> > 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
> > 131280 from mon.4
> >
> >
> > I suspect that the file in the log above int's the culprit. How can I
> get
> > to the root cause of MDS slowdowns?
> >
> >
> > On Tue, Dec 27, 2022 at 3:32 PM Pavin Joseph <me@xxxxxxxxxxxxxxx> wrote:
> >
> >> Interesting, the logs show the crash module [0] itself has crashed.
> >> Something sent it a SIGINT or SIGTERM and the module didn't handle it
> >> correctly due to what seems like a bug in the code.
> >>
> >> I haven't experienced the crash module itself crashing yet (in Quincy)
> >> because nothing sent a SIG[INT|TERM] to it yet.
> >>
> >> So I'd continue investigating into why these signals were sent to the
> >> crash module.
> >>
> >> To fix the crash module from crashing, go to "/usr/bin/ceph-crash" and
> >> edit the handler function on line 82 like so:
> >>
> >> def handler(signum, frame):
> >> print('*** Interrupted with signal %d ***' % signum)
> >> signame = signal.Signals(signum).name
> >> print(f'Signal handler called with signal {signame} ({signum})')
> >> print(frame)
> >> sys.exit(0)
> >>
> >> ---
> >>
> >> Once the crash module is working, perhaps you could run a "ceph crash
> ls"
> >>
> >> Regarding podman logs, perhaps try this [1].
> >>
> >> [0]: https://docs.ceph.com/en/quincy/mgr/crash/
> >> [1]: https://docs.podman.io/en/latest/markdown/podman-logs.1.html
> >>
> >> On 27-Dec-22 11:59 PM, Deep Dish wrote:
> >>> HI Pavin,
> >>>
> >>> Thanks for the reply. I'm a bit at a loss honestly as this worked
> >>> perfectly without any issue up until the rebalance of the cluster.
> >>> Orchestrator is great. Aside from this (which I suspect is not
> >>> orchestrator related), I haven't had any issues.
> >>>
> >>> In terms of logs, I'm not sure where to start looking in this new
> >>> containerized environment as they pertain to individual ceph processes
> >> -- I
> >>> assumed everything would be centrally collected within orch.
> >>>
> >>> Connecting into the podman container of a RGW, there are no logs in
> >>> /var/log/ceph aside from ceph-volume. My ceph.conf is minimal with
> only
> >>> monitors defined. The only log I'm able to pull is as follows:
> >>>
> >>> # podman logs 35d4ac5445ca
> >>>
> >>> INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
> >>>
> >>> Traceback (most recent call last):
> >>>
> >>> File "/usr/bin/ceph-crash", line 113, in <module>
> >>>
> >>> main()
> >>>
> >>> File "/usr/bin/ceph-crash", line 109, in main
> >>>
> >>> time.sleep(args.delay * 60)
> >>>
> >>> TypeError: handler() takes 1 positional argument but 2 were given
> >>>
> >>> INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
> >>>
> >>>
> >>>
> >>> Looks like the RGW daemon is crashing. How do I get logs to persist?
> >> I
> >>> suspect I won't be able to use orchestrator to push down the config,
> and
> >>> would have to manipulate within the container image itself.
> >>>
> >>> I also attempted to redeply the RGW containers without success.
> >>>
> >>> On Tue, Dec 27, 2022 at 10:39 AM Pavin Joseph <me@xxxxxxxxxxxxxxx>
> >> wrote:
> >>>
> >>>> Here's the first things I'd check in your situation:
> >>>>
> >>>> 1. Logs
> >>>> 2. Is the RGW HTTP server running on its port?
> >>>> 3. Re-check config including authentication.
> >>>>
> >>>> ceph orch is too new and didn't pass muster in our own internal
> testing.
> >>>> You're braver than most for using it in production.
> >>>>
> >>>> Pavin.
> >>>>
> >>>> On 27-Dec-22 8:48 PM, Deep Dish wrote:
> >>>>> Quick update:
> >>>>>
> >>>>> - I followed documentation, and ran the following:
> >>>>>
> >>>>> # ceph dashboard set-rgw-credentials
> >>>>>
> >>>>> Error EINVAL: No RGW credentials found, please consult the
> >> documentation
> >>>> on
> >>>>> how to enable RGW for the dashboard.
> >>>>>
> >>>>>
> >>>>>
> >>>>> - I see dashboard credentials configured (all this was working fine
> >>>> before):
> >>>>>
> >>>>>
> >>>>> # ceph dashboard get-rgw-api-access-key
> >>>>>
> >>>>> P?????????????????G (? commented out)
> >>>>>
> >>>>>
> >>>>>
> >>>>> Seems to me like my RGW config is non-existent / corrupted for some
> >>>>> reason. When trying to curl a RGW directly I get a "connection
> >> refused".
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Dec 27, 2022 at 9:41 AM Deep Dish <deeepdish@xxxxxxxxx>
> wrote:
> >>>>>
> >>>>>> I built a net-new Quincy cluster (17.2.5) using ceph orch as
> follows:
> >>>>>>
> >>>>>> 2x mgrs
> >>>>>> 4x rgw
> >>>>>> 5x mon
> >>>>>> 4x rgw
> >>>>>> 5x mds
> >>>>>> 6x osd hosts w/ 10 drives each --> will be growing to 7 osd hosts in
> >> the
> >>>>>> coming days.
> >>>>>>
> >>>>>> I migrated all data from my legacy nautilus cluster (via rbd-mirror,
> >>>>>> rclone for s3 buckets, etc.). All moved over successfully without
> >>>> issue.
> >>>>>>
> >>>>>> The cluster went through a series of rebalancing events (adding
> >>>> capacity,
> >>>>>> osd nodes, changing fault domain for EC volumes).
> >>>>>>
> >>>>>> It's settled now, however throughout the process all of my RGW nodes
> >> are
> >>>>>> no longer part of the cluster -- meaning ceph doesn't recognize /
> >> detect
> >>>>>> them, despite containers, networking, etc. all being setup
> correctly.
> >>>>>> This also means I'm unable to manage any RGW functions (via the
> >>>> dashboard
> >>>>>> or cli). As an example via cli (within Cephadm shell):
> >>>>>>
> >>>>>> # radosgw-admin pools list
> >>>>>>
> >>>>>> could not list placement set: (2) No such file or directory
> >>>>>>
> >>>>>> I have data in buckets, how can I get my RGWs to return online?
> >>>>>>
> >>>>> _______________________________________________
> >>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>>
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
