Hello,
After a host reboot the cluster could not find an object. The cluster was in stable state with all osd active+clean, no OSD was out, no other OSD was restarted during host reboot. It was 1 month ago, we hoped that the cluster will find the object eventually, but it did not.
Cluster version: ceph version 16.2.9, ceph-deploy cluster, pool size 2.
Attached are ceph.log, osds logs, pg query and other logs
Cluster status:
cluster:
id: 2517da9e-af62-405e-b71f-1f2e145822f7
health: HEALTH_ERR
client is using insecure global_id reclaim
mons are allowing insecure global_id reclaim
1/606943089 objects unfound (0.000%)
Possible data damage: 1 pg recovery_unfound
Degraded data redundancy: 7252/1219946300 objects degraded (0.001%), 1 pg degraded, 1 pg undersized
1 pgs not deep-scrubbed in time
1 pgs not scrubbed in time
data:
volumes: 1/1 healthy
pools: 12 pools, 6560 pgs
objects: 606.94M objects, 85 TiB
usage: 169 TiB used, 268 TiB / 438 TiB avail
pgs: 7252/1219946300 objects degraded (0.001%)
7250/1219946300 objects misplaced (0.001%)
1/606943089 objects unfound (0.000%)
6554 active+clean
4 active+clean+scrubbing+deep
1 active+recovery_unfound+undersized+degraded+remapped
1 active+clean+scrubbing
io:
client: 1.2 GiB/s rd, 1.4 GiB/s wr, 40.87k op/s rd, 72.80k op/s wr
progress:
Global Recovery Event (2w)
[===========================.] (remaining: 4m)
Ceph health detail
HEALTH_ERR clients are using insecure global_id reclaim; mons are allowing insecure global_id reclaim; 1/606997573 objects unfound (0.000%); Possible data damage: 1 pg recovery_unfound; Degraded data redundancy: 7294/1220048932 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 1 pgs not deep-scrubbed in time; 1 pgs not scrubbed in time
...
[WRN] OBJECT_UNFOUND: 1/606997573 objects unfound (0.000%)
pg 16.1e has 1 unfound objects
[ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 7294/1220048932 objects degraded (0.001%), 1 pg degraded, 1 pg undersized
pg 16.1e is stuck undersized for 3h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
[WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
The PG is acting only on osd.131, even we move the PG to other OSD:
ceph pg map 16.1e
osdmap e723093 pg 16.1e (16.1e) -> up [41,141] acting [131]
On ceph osd dump the pg is mapped as a pg_temp:
ceph osd dump | grep -w 16.1e
pg_temp 16.1e [131]
What we did:
- restarted all osd and hosts involved
- force a deep-scrub on PG (the pg cannot be scrubed anymore)
- If we stop osd.131 the PG becomes inactive and down (like it is the only osd containing the objects): Reduced data availability: 1 pg inactive, 1 pg down
- If we take out the osd.131, the pg is not moving to the new osd, it remains the only object on osd.131
- ceph force recovery
- ceph force repeer
- ceph pg repair 16.1e
- Used ceph-objectstore-tool to search for the unfound object (rbd_data.ad5ab66b8b4567.0000000000011055) on all osd's involved, the object is present only on osd.41 and osd.131 even the PG is mapped to other OSD's.
- ceph-objectstore-tool
ceph pg remap: we tryed to remap the pg to others OSD's (ceph osd pg-upmap-items 16.1e 131 141) but the PG does not move to new OSD's, remain on osd.41 and osd.131 (ceph pg map 16.1e: osdmap e723093 pg 16.1e (16.1e) -> up [41,141] acting [131])
Why is this happening ?
How can we help the cluster to find the lost object?
Can we remove pg_temp 16.1e [131] from upmap (ceph osd dump) ?
Thank you,
Martin Culcea
ceph pg map 16.1e
osdmap e723050 pg 16.1e (16.1e) -> up [41,141] acting [131]
ceph pg 16.1e query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+recovery_unfound+undersized+degraded+remapped",
"epoch": 723050,
"up": [
41,
141
],
"acting": [
131
],
"backfill_targets": [
"41",
"141"
],
"acting_recovery_backfill": [
"41",
"131",
"141"
],
"info": {
"pgid": "16.1e",
"last_update": "723050'1044649126",
"last_complete": "0'0",
"log_tail": "723049'1044639041",
"last_user_version": 1044649126,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 6813,
"epoch_pool_created": 6813,
"last_epoch_started": 723046,
"last_interval_started": 723045,
"last_epoch_clean": 713865,
"last_interval_clean": 713864,
"last_epoch_split": 53947,
"last_epoch_marked_full": 76013,
"same_up_since": 723045,
"same_interval_since": 723045,
"same_primary_since": 722482,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"prior_readable_until_ub": 0
},
"stats": {
"version": "723050'1044649126",
"reported_seq": 1520851317,
"reported_epoch": 723050,
"state": "active+recovery_unfound+undersized+degraded+remapped",
"last_fresh": "2022-06-29T13:58:01.090917+0300",
"last_change": "2022-06-29T13:44:37.670390+0300",
"last_active": "2022-06-29T13:58:01.090917+0300",
"last_peered": "2022-06-29T13:58:01.090917+0300",
"last_clean": "2022-06-09T16:18:34.751609+0300",
"last_became_active": "2022-06-29T13:44:37.586107+0300",
"last_became_peered": "2022-06-29T13:44:37.586107+0300",
"last_unstale": "2022-06-29T13:58:01.090917+0300",
"last_undegraded": "2022-06-29T13:44:37.669704+0300",
"last_fullsized": "2022-06-29T13:44:37.072980+0300",
"mapping_epoch": 723045,
"log_start": "723049'1044639041",
"ondisk_log_start": "723049'1044639041",
"created": 6813,
"last_epoch_clean": 713865,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"log_size": 10085,
"ondisk_log_size": 10085,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 29269213184,
"num_objects": 7255,
"num_object_clones": 0,
"num_object_copies": 14510,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 1,
"num_objects_degraded": 7256,
"num_objects_misplaced": 7254,
"num_objects_unfound": 1,
"num_objects_dirty": 7255,
"num_whiteouts": 0,
"num_read": 449932920,
"num_read_kb": 8094044560,
"num_write": 1022291571,
"num_write_kb": 17372688024,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 82882,
"num_bytes_recovered": 328765462528,
"num_keys_recovered": 6,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
41,
141
],
"acting": [
131
],
"avail_no_missing": [
"42",
"210",
"348"
],
"object_location_counts": [
{
"shards": "131",
"objects": 7255
}
],
"blocked_by": [],
"up_primary": 41,
"acting_primary": 131,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 723046,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "0",
"pgid": "16.1e",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 7255,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "41",
"pgid": "16.1e",
"last_update": "723050'1044649126",
"last_complete": "723050'1044649126",
"log_tail": "723039'1044614668",
"last_user_version": 1044611140,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 6813,
"epoch_pool_created": 6813,
"last_epoch_started": 723046,
"last_interval_started": 723045,
"last_epoch_clean": 713865,
"last_interval_clean": 713864,
"last_epoch_split": 53947,
"last_epoch_marked_full": 76013,
"same_up_since": 723045,
"same_interval_since": 723045,
"same_primary_since": 722482,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"prior_readable_until_ub": 0
},
"stats": {
"version": "722254'1039600127",
"reported_seq": 1443357445,
"reported_epoch": 722260,
"state": "peering",
"last_fresh": "2022-06-27T15:22:20.422891+0300",
"last_change": "2022-06-27T15:22:20.422694+0300",
"last_active": "2022-06-09T16:24:10.485928+0300",
"last_peered": "2022-06-09T16:18:34.751609+0300",
"last_clean": "2022-06-09T16:18:34.751609+0300",
"last_became_active": "2022-06-09T14:46:08.142913+0300",
"last_became_peered": "2022-06-09T14:46:08.142913+0300",
"last_unstale": "2022-06-27T15:22:20.422891+0300",
"last_undegraded": "2022-06-27T15:22:20.422891+0300",
"last_fullsized": "2022-06-27T15:22:20.422891+0300",
"mapping_epoch": 723045,
"log_start": "722254'1039590041",
"ondisk_log_start": "722254'1039590041",
"created": 6813,
"last_epoch_clean": 713865,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"log_size": 10086,
"ondisk_log_size": 10086,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 46920675328,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 7255,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
41,
141
],
"acting": [
131
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
210,
348
],
"up_primary": 41,
"acting_primary": 131,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 723046,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "42",
"pgid": "16.1e",
"last_update": "723044'1044624668",
"last_complete": "723044'1044624668",
"log_tail": "723039'1044614641",
"last_user_version": 1044624668,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 6813,
"epoch_pool_created": 6813,
"last_epoch_started": 723039,
"last_interval_started": 723038,
"last_epoch_clean": 713865,
"last_interval_clean": 713864,
"last_epoch_split": 53947,
"last_epoch_marked_full": 76013,
"same_up_since": 723045,
"same_interval_since": 723045,
"same_primary_since": 722482,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"prior_readable_until_ub": 0
},
"stats": {
"version": "722477'1039743723",
"reported_seq": 17,
"reported_epoch": 722481,
"state": "peering",
"last_fresh": "2022-06-27T16:39:50.568020+0300",
"last_change": "2022-06-27T16:39:50.568020+0300",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2022-06-27T16:39:50.568020+0300",
"last_undegraded": "2022-06-27T16:39:50.568020+0300",
"last_fullsized": "2022-06-27T16:39:50.568020+0300",
"mapping_epoch": 723045,
"log_start": "722460'1039733641",
"ondisk_log_start": "722460'1039733641",
"created": 6813,
"last_epoch_clean": 713865,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"log_size": 10082,
"ondisk_log_size": 10082,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
41,
141
],
"acting": [
131
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
0,
41,
131,
210,
348
],
"up_primary": 41,
"acting_primary": 131,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 723039,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "141",
"pgid": "16.1e",
"last_update": "723050'1044649126",
"last_complete": "723050'1044649126",
"log_tail": "723039'1044614668",
"last_user_version": 1044565479,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 6813,
"epoch_pool_created": 6813,
"last_epoch_started": 723046,
"last_interval_started": 723045,
"last_epoch_clean": 713865,
"last_interval_clean": 713864,
"last_epoch_split": 53947,
"last_epoch_marked_full": 76013,
"same_up_since": 723045,
"same_interval_since": 723045,
"same_primary_since": 722482,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 723045,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 7255,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
41,
141
],
"acting": [
131
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 41,
"acting_primary": 131,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 723046,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "210",
"pgid": "16.1e",
"last_update": "720573'1025257351",
"last_complete": "720573'1025257351",
"log_tail": "720569'1025247341",
"last_user_version": 1025257351,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 6813,
"epoch_pool_created": 6813,
"last_epoch_started": 723039,
"last_interval_started": 723038,
"last_epoch_clean": 713865,
"last_interval_clean": 713864,
"last_epoch_split": 53947,
"last_epoch_marked_full": 76013,
"same_up_since": 723045,
"same_interval_since": 723045,
"same_primary_since": 722482,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"prior_readable_until_ub": 0
},
"stats": {
"version": "718146'1005652863",
"reported_seq": 92,
"reported_epoch": 718151,
"state": "peering",
"last_fresh": "2022-06-15T14:30:08.220366+0300",
"last_change": "2022-06-15T14:30:08.220331+0300",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2022-06-15T14:30:08.220366+0300",
"last_undegraded": "2022-06-15T14:30:08.220366+0300",
"last_fullsized": "2022-06-15T14:30:08.220366+0300",
"mapping_epoch": 723045,
"log_start": "718129'1005642841",
"ondisk_log_start": "718129'1005642841",
"created": 6813,
"last_epoch_clean": 713865,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"log_size": 10022,
"ondisk_log_size": 10022,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
41,
141
],
"acting": [
131
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
41,
348
],
"up_primary": 41,
"acting_primary": 131,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 720232,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "348",
"pgid": "16.1e",
"last_update": "720570'1025257287",
"last_complete": "720570'1025257287",
"log_tail": "720569'1025247241",
"last_user_version": 1025257287,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 6813,
"epoch_pool_created": 6813,
"last_epoch_started": 723039,
"last_interval_started": 723038,
"last_epoch_clean": 713865,
"last_interval_clean": 713864,
"last_epoch_split": 53947,
"last_epoch_marked_full": 76013,
"same_up_since": 723045,
"same_interval_since": 723045,
"same_primary_since": 722482,
"last_scrub": "710579'988167138",
"last_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"last_deep_scrub": "708883'971173140",
"last_deep_scrub_stamp": "2022-06-03T01:20:13.786232+0300",
"last_clean_scrub_stamp": "2022-06-09T03:27:36.771392+0300",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 723045,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
41,
141
],
"acting": [
131
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 41,
"acting_primary": 131,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 720291,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2022-06-29T13:44:37.073051+0300",
"might_have_unfound": [
{
"osd": "0",
"status": "not queried"
},
{
"osd": "41",
"status": "already probed"
},
{
"osd": "42",
"status": "already probed"
},
{
"osd": "141",
"status": "already probed"
},
{
"osd": "210",
"status": "already probed"
},
{
"osd": "348",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [
"41",
"141"
],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"pull_from_peer": [],
"pushing": []
}
}
},
{
"name": "Started",
"enter_time": "2022-06-29T13:44:36.027931+0300"
}
],
"scrubber": {
"epoch_start": "0",
"active": false
},
"agent_state": {}
}
grep 16.1e /var/log/ceph/ceph-osd.131.log
2022-06-29T12:33:49.213+0300 7f37cefad700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:49:29.272+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:50:46.629+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:54:00.683+0300 7f37cefad700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:04:05.758+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:12:52.468+0300 7f37cefad700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:13:13.188+0300 7f37cafa5700 1 osd.131 pg_epoch: 723033 pg[16.1e( v 723032'1044565479 lc 0'0 (723031'1044555441,723032'1044565479] local-lis/les=722726/722727 n=7259 ec=6813/6813 lis/c=722726/713864 les/c/f=722727/713865/76013 sis=723033 pruub=8.442323685s) [41,131]/[131] backfill=[41,141] r=0 lpr=723033 pi=[713864,723033)/4 crt=723032'1044565479 mlcod 0'0 active+remapped pruub 160422.625000000s@ MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB m=1 u=1 mbc={}] start_peering_interval up [41,141] -> [41,131], acting [131] -> [131], acting_primary 131 -> 131, up_primary 41 -> 41, role 0 -> 0, features acting 4540138297136906239 upacting 4540138297136906239
2022-06-29T13:13:13.192+0300 7f37cafa5700 1 osd.131 pg_epoch: 723033 pg[16.1e( v 723032'1044565479 lc 0'0 (723031'1044555441,723032'1044565479] local-lis/les=722726/722727 n=7259 ec=6813/6813 lis/c=722726/713864 les/c/f=722727/713865/76013 sis=723033 pruub=8.442323685s) [41,131]/[131] r=0 lpr=723033 pi=[713864,723033)/4 crt=723032'1044565479 mlcod 0'0 remapped pruub 160422.625000000s@ MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB m=1 mbc={}] state<Start>: transitioning to Primary
2022-06-29T13:13:14.712+0300 7f37cefad700 0 log_channel(cluster) log [INF] : 16.1e continuing backfill to osd.41 from (723031'1044555441,723032'1044565479] MIN to 723032'1044565479
2022-06-29T13:18:01.451+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:27:10.766+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:34:42.955+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:37:06.973+0300 7f37cefad700 1 osd.131 pg_epoch: 723038 pg[16.1e( v 723037'1044611140 lc 0'0 (723036'1044601041,723037'1044611140] local-lis/les=723033/723034 n=7254 ec=6813/6813 lis/c=723033/713864 les/c/f=723034/713865/76013 sis=723038 pruub=15.828340530s) [42,131]/[131] backfill=[41] r=0 lpr=723038 pi=[713864,723038)/4 crt=723037'1044611140 mlcod 0'0 active+remapped pruub 161863.812500000s@ MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB m=1 u=1 mbc={}] start_peering_interval up [41,131] -> [42,131], acting [131] -> [131], acting_primary 131 -> 131, up_primary 41 -> 42, role 0 -> 0, features acting 4540138297136906239 upacting 4540138297136906239
2022-06-29T13:37:06.973+0300 7f37cefad700 1 osd.131 pg_epoch: 723038 pg[16.1e( v 723037'1044611140 lc 0'0 (723036'1044601041,723037'1044611140] local-lis/les=723033/723034 n=7254 ec=6813/6813 lis/c=723033/713864 les/c/f=723034/713865/76013 sis=723038 pruub=15.828340530s) [42,131]/[131] r=0 lpr=723038 pi=[713864,723038)/4 crt=723037'1044611140 mlcod 0'0 remapped pruub 161863.812500000s@ MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB m=1 mbc={}] state<Start>: transitioning to Primary
2022-06-29T13:37:08.005+0300 7f37cefad700 0 log_channel(cluster) log [DBG] : 16.1e starting backfill to osd.42 from (722712'1041612441,722712'1041622482] MIN to 723037'1044611140
2022-06-29T13:40:12.043+0300 7f37cefad700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:41:09.416+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:42:27.073+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:42:34.633+0300 7f37cefad700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:42:39.753+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:44:36.027+0300 7f37cafa5700 1 osd.131 pg_epoch: 723045 pg[16.1e( v 723044'1044624668 lc 0'0 (723039'1044614641,723044'1044624668] local-lis/les=723038/723039 n=7265 ec=6813/6813 lis/c=723038/713864 les/c/f=723039/713865/76013 sis=723045 pruub=8.584972382s) [41,141]/[131] backfill=[42] r=0 lpr=723045 pi=[713864,723045)/4 crt=723044'1044624668 mlcod 0'0 active+remapped pruub 162305.609375000s@ MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB m=1 u=1 mbc={}] start_peering_interval up [42,131] -> [41,141], acting [131] -> [131], acting_primary 131 -> 131, up_primary 42 -> 41, role 0 -> 0, features acting 4540138297136906239 upacting 4540138297136906239
2022-06-29T13:44:36.027+0300 7f37cafa5700 1 osd.131 pg_epoch: 723045 pg[16.1e( v 723044'1044624668 lc 0'0 (723039'1044614641,723044'1044624668] local-lis/les=723038/723039 n=7265 ec=6813/6813 lis/c=723038/713864 les/c/f=723039/713865/76013 sis=723045 pruub=8.584972382s) [41,141]/[131] r=0 lpr=723045 pi=[713864,723045)/4 crt=723044'1044624668 mlcod 0'0 remapped pruub 162305.609375000s@ MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB m=1 mbc={}] state<Start>: transitioning to Primary
2022-06-29T13:44:37.071+0300 7f37cefad700 0 log_channel(cluster) log [DBG] : 16.1e starting backfill to osd.41 from (723036'1044601041,723037'1044611140] MIN to 723044'1044624668
2022-06-29T13:44:37.099+0300 7f37cefad700 0 log_channel(cluster) log [DBG] : 16.1e starting backfill to osd.141 from (723031'1044555441,723032'1044565479] MIN to 723044'1044624668
2022-06-29T13:46:59.696+0300 7f37cefad700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:47:01.196+0300 7f37cafa5700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:48:47.026+0300 7f37cefad700 -1 log_channel(cluster) log [ERR] : 16.1e has 1 objects unfound and apparently lost
grep 16.1e /var/log/ceph/ceph-osd.41.log
2022-06-29T13:13:13.191+0300 7f61c4c26700 1 osd.41 pg_epoch: 723033 pg[16.1e( v 723032'1044565479 (723031'1044555441,723032'1044565479] lb MIN local-lis/les=722713/722714 n=0 ec=6813/6813 lis/c=722713/713864 les/c/f=722714/713865/76013 sis=723033) [41,131]/[131] r=-1 lpr=723033 pi=[713864,723033)/4 luod=0'0 lua=722929'1043551464 crt=723032'1044565479 lcod 723032'1044565478 mlcod 0'0 active+remapped mbc={}] start_peering_interval up [41,141] -> [41,131], acting [131] -> [131], acting_primary 131 -> 131, up_primary 41 -> 41, role -1 -> -1, features acting 4540138297136906239 upacting 4540138297136906239
2022-06-29T13:13:13.195+0300 7f61c4c26700 1 osd.41 pg_epoch: 723033 pg[16.1e( v 723032'1044565479 (723031'1044555441,723032'1044565479] lb MIN local-lis/les=722713/722714 n=0 ec=6813/6813 lis/c=722713/713864 les/c/f=722714/713865/76013 sis=723033) [41,131]/[131] r=-1 lpr=723033 pi=[713864,723033)/4 crt=723032'1044565479 lcod 723032'1044565478 mlcod 0'0 remapped NOTIFY mbc={}] state<Start>: transitioning to Stray
2022-06-29T13:37:06.989+0300 7f61c4c26700 1 osd.41 pg_epoch: 723038 pg[16.1e( v 723037'1044611140 (723036'1044601041,723037'1044611140] lb MIN local-lis/les=722713/722714 n=0 ec=6813/6813 lis/c=722713/713864 les/c/f=722714/713865/76013 sis=723038) [42,131]/[131] r=-1 lpr=723038 pi=[713864,723038)/4 luod=0'0 lua=723032'1044565479 crt=723037'1044611140 lcod 723037'1044611139 mlcod 0'0 active+remapped mbc={}] start_peering_interval up [41,131] -> [42,131], acting [131] -> [131], acting_primary 131 -> 131, up_primary 41 -> 42, role -1 -> -1, features acting 4540138297136906239 upacting 4540138297136906239
2022-06-29T13:37:06.989+0300 7f61c4c26700 1 osd.41 pg_epoch: 723038 pg[16.1e( v 723037'1044611140 (723036'1044601041,723037'1044611140] lb MIN local-lis/les=722713/722714 n=0 ec=6813/6813 lis/c=722713/713864 les/c/f=722714/713865/76013 sis=723038) [42,131]/[131] r=-1 lpr=723038 pi=[713864,723038)/4 crt=723037'1044611140 lcod 723037'1044611139 mlcod 0'0 remapped NOTIFY mbc={}] state<Start>: transitioning to Stray
2022-06-29T13:44:36.023+0300 7f61c4c26700 1 osd.41 pg_epoch: 723045 pg[16.1e( v 723037'1044611140 (723036'1044601041,723037'1044611140] lb MIN local-lis/les=722713/722714 n=0 ec=6813/6813 lis/c=723033/713864 les/c/f=723034/713865/76013 sis=723045) [41,141]/[131] r=-1 lpr=723045 pi=[713864,723045)/4 crt=723037'1044611140 lcod 723037'1044611139 mlcod 0'0 remapped NOTIFY mbc={}] start_peering_interval up [42,131] -> [41,141], acting [131] -> [131], acting_primary 131 -> 131, up_primary 42 -> 41, role -1 -> -1, features acting 4540138297136906239 upacting 4540138297136906239
2022-06-29T13:44:36.023+0300 7f61c4c26700 1 osd.41 pg_epoch: 723045 pg[16.1e( v 723037'1044611140 (723036'1044601041,723037'1044611140] lb MIN local-lis/les=722713/722714 n=0 ec=6813/6813 lis/c=723033/713864 les/c/f=723034/713865/76013 sis=723045) [41,141]/[131] r=-1 lpr=723045 pi=[713864,723045)/4 crt=723037'1044611140 lcod 723037'1044611139 mlcod 0'0 remapped NOTIFY mbc={}] state<Start>: transitioning to Stray
grep -w 16.1e /var/log/ceph/ceph.log
2022-06-29T11:40:00.000887+0300 mon.ceph-nimbus-05 (mon.0) 1023041 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T11:40:00.000908+0300 mon.ceph-nimbus-05 (mon.0) 1023043 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T11:48:32.755122+0300 osd.131 (osd.131) 554 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T11:50:00.000336+0300 mon.ceph-nimbus-05 (mon.0) 1023350 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T11:50:00.000356+0300 mon.ceph-nimbus-05 (mon.0) 1023352 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T11:50:00.000383+0300 mon.ceph-nimbus-05 (mon.0) 1023354 : cluster [ERR] pg 16.1e is stuck undersized for 24h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T11:50:00.000405+0300 mon.ceph-nimbus-05 (mon.0) 1023356 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T11:50:00.000429+0300 mon.ceph-nimbus-05 (mon.0) 1023358 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T11:50:39.961350+0300 osd.131 (osd.131) 555 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T11:56:04.370136+0300 osd.131 (osd.131) 556 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:00:00.000467+0300 mon.ceph-nimbus-05 (mon.0) 1023653 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T12:00:00.000509+0300 mon.ceph-nimbus-05 (mon.0) 1023655 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T12:00:00.000537+0300 mon.ceph-nimbus-05 (mon.0) 1023657 : cluster [ERR] pg 16.1e is stuck undersized for 24h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T12:00:00.000663+0300 mon.ceph-nimbus-05 (mon.0) 1023659 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T12:00:00.000695+0300 mon.ceph-nimbus-05 (mon.0) 1023661 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T12:04:28.946537+0300 osd.131 (osd.131) 557 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:09:05.789427+0300 osd.131 (osd.131) 558 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:10:00.000406+0300 mon.ceph-nimbus-05 (mon.0) 1023955 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T12:10:00.000433+0300 mon.ceph-nimbus-05 (mon.0) 1023957 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T12:10:00.000454+0300 mon.ceph-nimbus-05 (mon.0) 1023959 : cluster [ERR] pg 16.1e is stuck undersized for 25h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T12:10:00.000478+0300 mon.ceph-nimbus-05 (mon.0) 1023961 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T12:10:00.000500+0300 mon.ceph-nimbus-05 (mon.0) 1023963 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T12:13:33.139030+0300 osd.131 (osd.131) 559 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:20:00.000376+0300 mon.ceph-nimbus-05 (mon.0) 1024251 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T12:20:00.000407+0300 mon.ceph-nimbus-05 (mon.0) 1024253 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T12:20:00.000432+0300 mon.ceph-nimbus-05 (mon.0) 1024255 : cluster [ERR] pg 16.1e is stuck undersized for 25h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T12:20:00.000460+0300 mon.ceph-nimbus-05 (mon.0) 1024257 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T12:20:00.000513+0300 mon.ceph-nimbus-05 (mon.0) 1024259 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T12:27:58.826058+0300 osd.131 (osd.131) 560 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:28:43.300823+0300 osd.131 (osd.131) 561 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:30:00.000288+0300 mon.ceph-nimbus-05 (mon.0) 1024561 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T12:30:00.000311+0300 mon.ceph-nimbus-05 (mon.0) 1024563 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T12:30:00.000334+0300 mon.ceph-nimbus-05 (mon.0) 1024565 : cluster [ERR] pg 16.1e is stuck undersized for 25h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T12:30:00.000355+0300 mon.ceph-nimbus-05 (mon.0) 1024567 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T12:30:00.000385+0300 mon.ceph-nimbus-05 (mon.0) 1024569 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T12:33:23.373298+0300 osd.131 (osd.131) 562 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:33:49.216370+0300 osd.131 (osd.131) 563 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:40:00.000336+0300 mon.ceph-nimbus-05 (mon.0) 1024869 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T12:40:00.000369+0300 mon.ceph-nimbus-05 (mon.0) 1024871 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T12:40:00.000399+0300 mon.ceph-nimbus-05 (mon.0) 1024873 : cluster [ERR] pg 16.1e is stuck undersized for 25h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T12:40:00.000429+0300 mon.ceph-nimbus-05 (mon.0) 1024875 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T12:40:00.000465+0300 mon.ceph-nimbus-05 (mon.0) 1024877 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T12:49:29.277633+0300 osd.131 (osd.131) 565 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:50:00.000402+0300 mon.ceph-nimbus-05 (mon.0) 1025177 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T12:50:00.000434+0300 mon.ceph-nimbus-05 (mon.0) 1025179 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T12:50:00.000464+0300 mon.ceph-nimbus-05 (mon.0) 1025181 : cluster [ERR] pg 16.1e is stuck undersized for 25h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T12:50:00.000496+0300 mon.ceph-nimbus-05 (mon.0) 1025183 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T12:50:00.000526+0300 mon.ceph-nimbus-05 (mon.0) 1025185 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T12:50:46.632326+0300 osd.131 (osd.131) 566 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T12:54:00.688822+0300 osd.131 (osd.131) 567 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:00:00.000350+0300 mon.ceph-nimbus-05 (mon.0) 1025477 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T13:00:00.000371+0300 mon.ceph-nimbus-05 (mon.0) 1025479 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T13:00:00.000401+0300 mon.ceph-nimbus-05 (mon.0) 1025481 : cluster [ERR] pg 16.1e is stuck undersized for 25h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T13:00:00.000435+0300 mon.ceph-nimbus-05 (mon.0) 1025483 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T13:00:00.000466+0300 mon.ceph-nimbus-05 (mon.0) 1025485 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T13:04:05.760884+0300 osd.131 (osd.131) 568 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:10:00.000364+0300 mon.ceph-nimbus-05 (mon.0) 1025789 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T13:10:00.000389+0300 mon.ceph-nimbus-05 (mon.0) 1025791 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T13:10:00.000414+0300 mon.ceph-nimbus-05 (mon.0) 1025793 : cluster [ERR] pg 16.1e is stuck undersized for 26h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T13:10:00.000435+0300 mon.ceph-nimbus-05 (mon.0) 1025795 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T13:10:00.000464+0300 mon.ceph-nimbus-05 (mon.0) 1025797 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T13:12:52.472457+0300 osd.131 (osd.131) 569 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:13:14.718778+0300 osd.131 (osd.131) 570 : cluster [INF] 16.1e continuing backfill to osd.41 from (723031'1044555441,723032'1044565479] MIN to 723032'1044565479
2022-06-29T13:18:01.458724+0300 osd.131 (osd.131) 571 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:20:00.000327+0300 mon.ceph-nimbus-05 (mon.0) 1026102 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T13:20:00.000358+0300 mon.ceph-nimbus-05 (mon.0) 1026104 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T13:20:00.000383+0300 mon.ceph-nimbus-05 (mon.0) 1026106 : cluster [ERR] pg 16.1e is stuck undersized for 6m, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T13:20:00.000405+0300 mon.ceph-nimbus-05 (mon.0) 1026108 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T13:20:00.000428+0300 mon.ceph-nimbus-05 (mon.0) 1026110 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T13:27:10.770987+0300 osd.131 (osd.131) 572 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:30:00.000300+0300 mon.ceph-nimbus-05 (mon.0) 1026410 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T13:30:00.000325+0300 mon.ceph-nimbus-05 (mon.0) 1026412 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T13:30:00.000349+0300 mon.ceph-nimbus-05 (mon.0) 1026414 : cluster [ERR] pg 16.1e is stuck undersized for 16m, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T13:30:00.000388+0300 mon.ceph-nimbus-05 (mon.0) 1026416 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T13:30:00.000410+0300 mon.ceph-nimbus-05 (mon.0) 1026418 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T13:34:42.961359+0300 osd.131 (osd.131) 573 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:37:08.007290+0300 osd.131 (osd.131) 574 : cluster [DBG] 16.1e starting backfill to osd.42 from (722712'1041612441,722712'1041622482] MIN to 723037'1044611140
2022-06-29T13:40:00.000306+0300 mon.ceph-nimbus-05 (mon.0) 1026733 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T13:40:00.000324+0300 mon.ceph-nimbus-05 (mon.0) 1026735 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T13:40:00.000342+0300 mon.ceph-nimbus-05 (mon.0) 1026737 : cluster [ERR] pg 16.1e is stuck undersized for 2m, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T13:40:00.000377+0300 mon.ceph-nimbus-05 (mon.0) 1026739 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T13:40:00.000400+0300 mon.ceph-nimbus-05 (mon.0) 1026741 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
2022-06-29T13:40:12.048711+0300 osd.131 (osd.131) 575 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:41:09.419580+0300 osd.131 (osd.131) 576 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:42:27.078924+0300 osd.131 (osd.131) 577 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:42:34.636260+0300 osd.131 (osd.131) 578 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:42:39.756345+0300 osd.131 (osd.131) 579 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:44:37.073096+0300 osd.131 (osd.131) 580 : cluster [DBG] 16.1e starting backfill to osd.41 from (723036'1044601041,723037'1044611140] MIN to 723044'1044624668
2022-06-29T13:44:37.103043+0300 osd.131 (osd.131) 581 : cluster [DBG] 16.1e starting backfill to osd.141 from (723031'1044555441,723032'1044565479] MIN to 723044'1044624668
2022-06-29T13:46:59.702124+0300 osd.131 (osd.131) 582 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:47:01.201601+0300 osd.131 (osd.131) 583 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:48:47.029613+0300 osd.131 (osd.131) 584 : cluster [ERR] 16.1e has 1 objects unfound and apparently lost
2022-06-29T13:50:00.000384+0300 mon.ceph-nimbus-05 (mon.0) 1027052 : cluster [ERR] pg 16.1e has 1 unfound objects
2022-06-29T13:50:00.000414+0300 mon.ceph-nimbus-05 (mon.0) 1027054 : cluster [ERR] pg 16.1e is active+recovery_unfound+undersized+degraded+remapped, acting [131], 1 unfound
2022-06-29T13:50:00.000443+0300 mon.ceph-nimbus-05 (mon.0) 1027056 : cluster [ERR] pg 16.1e is stuck undersized for 5m, current state active+recovery_unfound+undersized+degraded+remapped, last acting [131]
2022-06-29T13:50:00.000474+0300 mon.ceph-nimbus-05 (mon.0) 1027058 : cluster [ERR] pg 16.1e not deep-scrubbed since 2022-06-03T01:20:13.786232+0300
2022-06-29T13:50:00.000506+0300 mon.ceph-nimbus-05 (mon.0) 1027060 : cluster [ERR] pg 16.1e not scrubbed since 2022-06-09T03:27:36.771392+0300
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
