Hi
We are running Ceph 16.2.6 deployed with Cephadm.
Around Christmas OSD 245 and 327 had about 20 read error so I set them
to out.
Around new year another OSD 313 more or less died since is become so
slow that it triggered Linux default I/O-timeout of 30 seconds.
In this period the OSD crashed 8 times and was restartet by Systemd and
we ended up with
[WRN] OBJECT_UNFOUND: 1/416287126 objects unfound (0.000%)
pg 404.1ff has 1 unfound objects
[ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
pg 404.1ff is active+recovery_unfound+degraded+remapped, acting
[208,220,269,175,313,329], 1 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 5/2364745884 objects
degraded (0.000%), 1 pg degraded
pg 404.1ff is active+recovery_unfound+degraded+remapped, acting
[208,220,269,175,313,329], 1 unfound
The pool 404 is "default.rgw.buckets.data" and pool 404 is erasure
encoding 4+2.
I have search for a solution but with no luck, what I have tried is
- Restarted all 6 OSD for the PG one by one
- Running repair of 404.1ff
Output of following command
- ceph -s
- ceph health detail
- ceph pg ls | grep -e PG -e ^404.1ff
- ceph osd pool ls detail | grep 404
- ceph osd tree out
- ceph crash ls | grep -e ID -e osd.313
- ceph pg 404.1ff list_unfound
- ceph pg 404.1ff
Is appended below, can also be read here
https://gitlab.com/-/snippets/2479624
or cloned with "git clone https://gitlab.com/-/snippets/2479624";
Does anyone have any idea on how to resolv the problem?
Any help is much appreciated.
-
Kai Stian Olstad
::::::::::::::
ceph-s.txt
::::::::::::::
ceph -s
-------
cluster:
id: d13c6b81-51ee-4d22-84e9-456f9307296c
health: HEALTH_ERR
1/416287125 objects unfound (0.000%)
Possible data damage: 1 pg recovery_unfound
Degraded data redundancy: 5/2364745860 objects degraded
(0.000%), 1 pg degraded
services:
mon: 3 daemons, quorum ceph-mon-1,ceph-mon-2,ceph-mon-3 (age 2M)
mgr: ceph-mon-2.mfdanx(active, since 3w), standbys:
ceph-mon-1.ptrsea
mds: 1/1 daemons up, 1 standby
osd: 355 osds: 355 up (since 20h), 352 in (since 2d); 1 remapped pgs
rgw: 4 daemons active (4 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 14 pools, 2505 pgs
objects: 416.29M objects, 540 TiB
usage: 939 TiB used, 2.1 PiB / 3.0 PiB avail
pgs: 5/2364745860 objects degraded (0.000%)
137931/2364745860 objects misplaced (0.006%)
1/416287125 objects unfound (0.000%)
2489 active+clean
14 active+clean+scrubbing+deep
1 active+recovery_unfound+degraded+remapped
1 active+clean+scrubbing
io:
client: 38 MiB/s rd, 23 MiB/s wr, 2.58k op/s rd, 326 op/s wr
progress:
Global Recovery Event (6d)
[===========================.] (remaining: 3m)
::::::::::::::
ceph_health_detail.txt
::::::::::::::
ceph health detail
------------------
HEALTH_ERR 1/416287126 objects unfound (0.000%); Possible data damage: 1
pg recovery_unfound; Degraded data redundancy: 5/2364745884 objects
degraded (0.000%), 1 pg degraded
[WRN] OBJECT_UNFOUND: 1/416287126 objects unfound (0.000%)
pg 404.1ff has 1 unfound objects
[ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
pg 404.1ff is active+recovery_unfound+degraded+remapped, acting
[208,220,269,175,313,329], 1 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 5/2364745884 objects
degraded (0.000%), 1 pg degraded
pg 404.1ff is active+recovery_unfound+degraded+remapped, acting
[208,220,269,175,313,329], 1 unfound
::::::::::::::
ceph_pg_ls.txt
::::::::::::::
ceph pg ls | grep -e PG -e ^404.1ff
-----------------------------------
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES
OMAP_BYTES* OMAP_KEYS* LOG STATE
SINCE VERSION REPORTED UP
ACTING SCRUB_STAMP
DEEP_SCRUB_STAMP
404.1ff 137912 5 137908 1 282417561722
0 0 5528 active+recovery_unfound+degraded+remapped 19h
141748'724163 141748:3558203 [208,220,269,175,343,329]p208
[208,220,269,175,313,329]p208 2022-12-31T19:27:10.993286+0000
2022-12-31T19:27:10.993286+0000
::::::::::::::
ceph_osd_pool_ls_detail.txt
::::::::::::::
ceph osd pool ls detail | grep 404
----------------------------------
pool 404 'default.rgw.buckets.data' erasure profile
ec42-jerasure-blaum_roth-hdd size 6 min_size 5 crush_rule 2 object_hash
rjenkins pg_num 2048 pgp_num 2048 autoscale_mode on last_change 124077
lfor 0/52091/108555 flags hashpspool stripe_width 229376
target_size_bytes 1099511627776000 application rgw
::::::::::::::
ceph_osd_tree_out.txt
::::::::::::::
ceph osd tree out
-----------------
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 3112.43481 root default
-58 192.35847 host ceph-hd-010
245 hdd 12.82390 osd.245 up 0 1.00000
-79 192.35847 host ceph-hd-014
313 hdd 12.82390 osd.313 up 0 1.00000
327 hdd 12.82390 osd.327 up 0 1.00000
::::::::::::::
ceph_crash_ls.txt
::::::::::::::
ceph crash ls | grep -e ID -e osd.313
-------------------------------------
ID ENTITY
NEW
2022-12-31T01:19:06.859805Z_8e94201c-8817-4c4e-b4dd-67c5b1d3a73a
osd.313
2022-12-31T01:19:47.265925Z_551b2de6-d1b0-4cbc-9e62-27ede7a46803
osd.313
2022-12-31T01:20:18.648279Z_64f232ec-2cb5-4dcf-8686-b12987ed7e45
osd.313
2022-12-31T20:28:43.093203Z_9c9a590f-7490-4f74-affe-5bda27b87618
osd.313
2023-01-02T01:34:25.087309Z_b7a4b341-5f79-467b-a4e2-3df5456a2ff7
osd.313
2023-01-02T03:46:32.295053Z_d71b5b62-5974-4b3e-b1c5-cc878d376f06
osd.313
2023-01-02T10:58:58.350473Z_ca87b412-bae5-4873-9997-092587d7c699
osd.313
2023-01-02T11:42:13.433011Z_c0e9ab85-a162-4331-b690-68847c641bd0
osd.313
::::::::::::::
ceph_pg_404.1ff_list_unfound.json
::::::::::::::
{
"num_missing": 1,
"num_unfound": 1,
"objects": [
{
"oid": {
"oid":
"1f244892-a2e7-406b-aa62-1b13511333a2.902205.1__multipart_ingest/date=2022-12-30/2022-12-30.json.gz.2~pORMz8G-kyQx8weY8GhNt-pgAmQDufG.684",
"key": "",
"snapid": -2,
"hash": 1069593087,
"max": 0,
"pool": 404,
"namespace": ""
},
"need": "140483'722076",
"have": "0'0",
"flags": "none",
"clean_regions": "clean_offsets: [], clean_omap: 0,
new_object: 1",
"locations": [
"329(5)"
]
}
],
"state": "NotRecovering",
"available_might_have_unfound": true,
"might_have_unfound": [],
"more": false
}
::::::::::::::
ceph_pg_404.1ff.json
::::::::::::::
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+recovery_unfound+degraded+remapped",
"epoch": 141748,
"up": [
208,
220,
269,
175,
343,
329
],
"acting": [
208,
220,
269,
175,
313,
329
],
"backfill_targets": [
"343(4)"
],
"acting_recovery_backfill": [
"175(3)",
"208(0)",
"220(1)",
"269(2)",
"313(4)",
"329(5)",
"343(4)"
],
"info": {
"pgid": "404.1ffs0",
"last_update": "141747'724161",
"last_complete": "140483'722075",
"log_tail": "138455'718635",
"last_user_version": 724161,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 36041,
"epoch_pool_created": 27862,
"last_epoch_started": 141660,
"last_interval_started": 141659,
"last_epoch_clean": 140452,
"last_interval_clean": 140451,
"last_epoch_split": 108548,
"last_epoch_marked_full": 0,
"same_up_since": 141659,
"same_interval_since": 141659,
"same_primary_since": 141615,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "141747'724161",
"reported_seq": 3558199,
"reported_epoch": 141748,
"state": "active+recovery_unfound+degraded+remapped",
"last_fresh": "2023-01-04T07:48:25.201886+0000",
"last_change": "2023-01-03T12:15:50.155062+0000",
"last_active": "2023-01-04T07:48:25.201886+0000",
"last_peered": "2023-01-04T07:48:25.201886+0000",
"last_clean": "2022-12-31T19:25:42.758665+0000",
"last_became_active": "2023-01-03T12:15:49.875672+0000",
"last_became_peered": "2023-01-03T12:15:49.875672+0000",
"last_unstale": "2023-01-04T07:48:25.201886+0000",
"last_undegraded": "2023-01-03T12:15:49.758975+0000",
"last_fullsized": "2023-01-04T07:48:25.201886+0000",
"mapping_epoch": 141659,
"log_start": "138455'718635",
"ondisk_log_start": "138455'718635",
"created": 36041,
"last_epoch_clean": 140452,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"log_size": 5526,
"ondisk_log_size": 5526,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 282409304186,
"num_objects": 137910,
"num_object_clones": 0,
"num_object_copies": 827460,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 1,
"num_objects_degraded": 5,
"num_objects_misplaced": 137906,
"num_objects_unfound": 1,
"num_objects_dirty": 137910,
"num_whiteouts": 0,
"num_read": 732600,
"num_read_kb": 181255155,
"num_write": 471445,
"num_write_kb": 102310260,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 406583,
"num_bytes_recovered": 891273764570,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 3
},
"up": [
208,
220,
269,
175,
343,
329
],
"acting": [
208,
220,
269,
175,
313,
329
],
"avail_no_missing": [
"329(5)"
],
"object_location_counts": [
{
"shards":
"175(3),208(0),220(1),269(2),313(4),329(5)",
"objects": 137909
},
{
"shards": "329(5)",
"objects": 1
}
],
"blocked_by": [],
"up_primary": 208,
"acting_primary": 208,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 141660,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "175(3)",
"pgid": "404.1ffs3",
"last_update": "141747'724161",
"last_complete": "141668'723872",
"log_tail": "138378'718235",
"last_user_version": 723769,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 36041,
"epoch_pool_created": 27862,
"last_epoch_started": 141660,
"last_interval_started": 141659,
"last_epoch_clean": 140452,
"last_interval_clean": 140451,
"last_epoch_split": 108548,
"last_epoch_marked_full": 0,
"same_up_since": 141659,
"same_interval_since": 141659,
"same_primary_since": 141615,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "141624'723769",
"reported_seq": 3550912,
"reported_epoch": 141624,
"state": "active+recovery_unfound+degraded+remapped",
"last_fresh": "2023-01-03T11:57:15.480994+0000",
"last_change": "2023-01-03T11:31:07.335605+0000",
"last_active": "2023-01-03T11:57:15.480994+0000",
"last_peered": "2023-01-03T11:57:15.480994+0000",
"last_clean": "2022-12-31T19:25:42.758665+0000",
"last_became_active": "2023-01-03T11:31:07.116154+0000",
"last_became_peered": "2023-01-03T11:31:07.116154+0000",
"last_unstale": "2023-01-03T11:57:15.480994+0000",
"last_undegraded": "2023-01-03T11:31:07.099558+0000",
"last_fullsized": "2023-01-03T11:57:15.480994+0000",
"mapping_epoch": 141659,
"log_start": "138378'718235",
"ondisk_log_start": "138378'718235",
"created": 36041,
"last_epoch_clean": 140452,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"log_size": 5534,
"ondisk_log_size": 5534,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 281850993538,
"num_objects": 137711,
"num_object_clones": 0,
"num_object_copies": 826266,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 1,
"num_objects_degraded": 5,
"num_objects_misplaced": 137707,
"num_objects_unfound": 1,
"num_objects_dirty": 137711,
"num_whiteouts": 0,
"num_read": 720741,
"num_read_kb": 180044133,
"num_write": 469883,
"num_write_kb": 101652239,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 406583,
"num_bytes_recovered": 891273764570,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 3
},
"up": [
208,
220,
269,
175,
343,
329
],
"acting": [
208,
220,
269,
175,
313,
329
],
"avail_no_missing": [
"329(5)"
],
"object_location_counts": [
{
"shards":
"175(3),208(0),220(1),269(2),313(4),329(5)",
"objects": 137710
},
{
"shards": "329(5)",
"objects": 1
}
],
"blocked_by": [],
"up_primary": 208,
"acting_primary": 208,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 141660,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "220(1)",
"pgid": "404.1ffs1",
"last_update": "141747'724161",
"last_complete": "141668'723872",
"log_tail": "138378'718235",
"last_user_version": 723769,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 36041,
"epoch_pool_created": 27862,
"last_epoch_started": 141660,
"last_interval_started": 141659,
"last_epoch_clean": 140452,
"last_interval_clean": 140451,
"last_epoch_split": 108548,
"last_epoch_marked_full": 0,
"same_up_since": 141659,
"same_interval_since": 141659,
"same_primary_since": 141615,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "141624'723769",
"reported_seq": 3550912,
"reported_epoch": 141624,
"state": "active+recovery_unfound+degraded+remapped",
"last_fresh": "2023-01-03T11:57:15.480994+0000",
"last_change": "2023-01-03T11:31:07.335605+0000",
"last_active": "2023-01-03T11:57:15.480994+0000",
"last_peered": "2023-01-03T11:57:15.480994+0000",
"last_clean": "2022-12-31T19:25:42.758665+0000",
"last_became_active": "2023-01-03T11:31:07.116154+0000",
"last_became_peered": "2023-01-03T11:31:07.116154+0000",
"last_unstale": "2023-01-03T11:57:15.480994+0000",
"last_undegraded": "2023-01-03T11:31:07.099558+0000",
"last_fullsized": "2023-01-03T11:57:15.480994+0000",
"mapping_epoch": 141659,
"log_start": "138378'718235",
"ondisk_log_start": "138378'718235",
"created": 36041,
"last_epoch_clean": 140452,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"log_size": 5534,
"ondisk_log_size": 5534,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 281850993538,
"num_objects": 137711,
"num_object_clones": 0,
"num_object_copies": 826266,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 1,
"num_objects_degraded": 5,
"num_objects_misplaced": 137707,
"num_objects_unfound": 1,
"num_objects_dirty": 137711,
"num_whiteouts": 0,
"num_read": 720741,
"num_read_kb": 180044133,
"num_write": 469883,
"num_write_kb": 101652239,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 406583,
"num_bytes_recovered": 891273764570,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 3
},
"up": [
208,
220,
269,
175,
343,
329
],
"acting": [
208,
220,
269,
175,
313,
329
],
"avail_no_missing": [
"329(5)"
],
"object_location_counts": [
{
"shards":
"175(3),208(0),220(1),269(2),313(4),329(5)",
"objects": 137710
},
{
"shards": "329(5)",
"objects": 1
}
],
"blocked_by": [],
"up_primary": 208,
"acting_primary": 208,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 141660,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "269(2)",
"pgid": "404.1ffs2",
"last_update": "141747'724161",
"last_complete": "141668'723872",
"log_tail": "138378'718235",
"last_user_version": 723769,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 36041,
"epoch_pool_created": 27862,
"last_epoch_started": 141660,
"last_interval_started": 141659,
"last_epoch_clean": 140452,
"last_interval_clean": 140451,
"last_epoch_split": 108548,
"last_epoch_marked_full": 0,
"same_up_since": 141659,
"same_interval_since": 141659,
"same_primary_since": 141615,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "141624'723769",
"reported_seq": 3550912,
"reported_epoch": 141624,
"state": "active+recovery_unfound+degraded+remapped",
"last_fresh": "2023-01-03T11:57:15.480994+0000",
"last_change": "2023-01-03T11:31:07.335605+0000",
"last_active": "2023-01-03T11:57:15.480994+0000",
"last_peered": "2023-01-03T11:57:15.480994+0000",
"last_clean": "2022-12-31T19:25:42.758665+0000",
"last_became_active": "2023-01-03T11:31:07.116154+0000",
"last_became_peered": "2023-01-03T11:31:07.116154+0000",
"last_unstale": "2023-01-03T11:57:15.480994+0000",
"last_undegraded": "2023-01-03T11:31:07.099558+0000",
"last_fullsized": "2023-01-03T11:57:15.480994+0000",
"mapping_epoch": 141659,
"log_start": "138378'718235",
"ondisk_log_start": "138378'718235",
"created": 36041,
"last_epoch_clean": 140452,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"log_size": 5534,
"ondisk_log_size": 5534,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 281850993538,
"num_objects": 137711,
"num_object_clones": 0,
"num_object_copies": 826266,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 1,
"num_objects_degraded": 5,
"num_objects_misplaced": 137707,
"num_objects_unfound": 1,
"num_objects_dirty": 137711,
"num_whiteouts": 0,
"num_read": 720741,
"num_read_kb": 180044133,
"num_write": 469883,
"num_write_kb": 101652239,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 406583,
"num_bytes_recovered": 891273764570,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 3
},
"up": [
208,
220,
269,
175,
343,
329
],
"acting": [
208,
220,
269,
175,
313,
329
],
"avail_no_missing": [
"329(5)"
],
"object_location_counts": [
{
"shards":
"175(3),208(0),220(1),269(2),313(4),329(5)",
"objects": 137710
},
{
"shards": "329(5)",
"objects": 1
}
],
"blocked_by": [],
"up_primary": 208,
"acting_primary": 208,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 141660,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "313(4)",
"pgid": "404.1ffs4",
"last_update": "141747'724161",
"last_complete": "141668'723872",
"log_tail": "138228'717435",
"last_user_version": 723769,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 36041,
"epoch_pool_created": 27862,
"last_epoch_started": 141660,
"last_interval_started": 141659,
"last_epoch_clean": 140452,
"last_interval_clean": 140451,
"last_epoch_split": 108548,
"last_epoch_marked_full": 0,
"same_up_since": 141659,
"same_interval_since": 141659,
"same_primary_since": 141615,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "141624'723769",
"reported_seq": 3551314,
"reported_epoch": 141659,
"state": "active+recovery_unfound+degraded+remapped",
"last_fresh": "2023-01-03T11:57:15.480994+0000",
"last_change": "2023-01-03T11:31:07.335605+0000",
"last_active": "2023-01-03T11:57:15.480994+0000",
"last_peered": "2023-01-03T11:57:15.480994+0000",
"last_clean": "2022-12-31T19:25:42.758665+0000",
"last_became_active": "2023-01-03T11:31:07.116154+0000",
"last_became_peered": "2023-01-03T11:31:07.116154+0000",
"last_unstale": "2023-01-03T11:57:15.480994+0000",
"last_undegraded": "2023-01-03T11:31:07.099558+0000",
"last_fullsized": "2023-01-03T11:57:15.480994+0000",
"mapping_epoch": 141659,
"log_start": "138378'718235",
"ondisk_log_start": "138378'718235",
"created": 36041,
"last_epoch_clean": 140452,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"log_size": 5534,
"ondisk_log_size": 5534,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 281850993538,
"num_objects": 137711,
"num_object_clones": 0,
"num_object_copies": 826266,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 1,
"num_objects_degraded": 5,
"num_objects_misplaced": 137707,
"num_objects_unfound": 1,
"num_objects_dirty": 137711,
"num_whiteouts": 0,
"num_read": 720741,
"num_read_kb": 180044133,
"num_write": 469883,
"num_write_kb": 101652239,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 406583,
"num_bytes_recovered": 891273764570,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 3
},
"up": [
208,
220,
269,
175,
343,
329
],
"acting": [
208,
220,
269,
175,
313,
329
],
"avail_no_missing": [
"329(5)"
],
"object_location_counts": [
{
"shards":
"175(3),208(0),220(1),269(2),313(4),329(5)",
"objects": 137710
},
{
"shards": "329(5)",
"objects": 1
}
],
"blocked_by": [],
"up_primary": 208,
"acting_primary": 208,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 141660,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "329(5)",
"pgid": "404.1ffs5",
"last_update": "141747'724161",
"last_complete": "141668'723872",
"log_tail": "138378'718235",
"last_user_version": 723769,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 36041,
"epoch_pool_created": 27862,
"last_epoch_started": 141660,
"last_interval_started": 141659,
"last_epoch_clean": 140452,
"last_interval_clean": 140451,
"last_epoch_split": 108548,
"last_epoch_marked_full": 0,
"same_up_since": 141659,
"same_interval_since": 141659,
"same_primary_since": 141615,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "141624'723769",
"reported_seq": 3550912,
"reported_epoch": 141624,
"state": "active+recovery_unfound+degraded+remapped",
"last_fresh": "2023-01-03T11:57:15.480994+0000",
"last_change": "2023-01-03T11:31:07.335605+0000",
"last_active": "2023-01-03T11:57:15.480994+0000",
"last_peered": "2023-01-03T11:57:15.480994+0000",
"last_clean": "2022-12-31T19:25:42.758665+0000",
"last_became_active": "2023-01-03T11:31:07.116154+0000",
"last_became_peered": "2023-01-03T11:31:07.116154+0000",
"last_unstale": "2023-01-03T11:57:15.480994+0000",
"last_undegraded": "2023-01-03T11:31:07.099558+0000",
"last_fullsized": "2023-01-03T11:57:15.480994+0000",
"mapping_epoch": 141659,
"log_start": "138378'718235",
"ondisk_log_start": "138378'718235",
"created": 36041,
"last_epoch_clean": 140452,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"log_size": 5534,
"ondisk_log_size": 5534,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 281850993538,
"num_objects": 137711,
"num_object_clones": 0,
"num_object_copies": 826266,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 0,
"num_objects_degraded": 5,
"num_objects_misplaced": 137707,
"num_objects_unfound": 1,
"num_objects_dirty": 137711,
"num_whiteouts": 0,
"num_read": 720741,
"num_read_kb": 180044133,
"num_write": 469883,
"num_write_kb": 101652239,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 406583,
"num_bytes_recovered": 891273764570,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 3
},
"up": [
208,
220,
269,
175,
343,
329
],
"acting": [
208,
220,
269,
175,
313,
329
],
"avail_no_missing": [
"329(5)"
],
"object_location_counts": [
{
"shards":
"175(3),208(0),220(1),269(2),313(4),329(5)",
"objects": 137710
},
{
"shards": "329(5)",
"objects": 1
}
],
"blocked_by": [],
"up_primary": 208,
"acting_primary": 208,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 141660,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "343(4)",
"pgid": "404.1ffs4",
"last_update": "141747'724161",
"last_complete": "141668'723872",
"log_tail": "138378'718235",
"last_user_version": 723769,
"last_backfill":
"404:ff80001b:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-09T12%3a41%3a05,478440857+00%3a00.2~XwOh1_IhVzghlM-zi7DfwoUIZl5UxVt.1:head",
"purged_snaps": [],
"history": {
"epoch_created": 36041,
"epoch_pool_created": 27862,
"last_epoch_started": 141660,
"last_interval_started": 141659,
"last_epoch_clean": 140452,
"last_interval_clean": 140451,
"last_epoch_split": 108548,
"last_epoch_marked_full": 0,
"same_up_since": 141659,
"same_interval_since": 141659,
"same_primary_since": 141615,
"last_scrub": "140525'722168",
"last_scrub_stamp": "2022-12-31T19:27:10.993286+0000",
"last_deep_scrub": "140525'722168",
"last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"last_clean_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 141659,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 9404416,
"num_objects": 3,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 137907,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 3,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
208,
220,
269,
175,
343,
329
],
"acting": [
208,
220,
269,
175,
313,
329
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 208,
"acting_primary": 208,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 141660,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2023-01-03T12:15:49.758685+0000",
"might_have_unfound": [
{
"osd": "175(3)",
"status": "already probed"
},
{
"osd": "220(1)",
"status": "already probed"
},
{
"osd": "269(2)",
"status": "already probed"
},
{
"osd": "313(4)",
"status": "already probed"
},
{
"osd": "329(5)",
"status": "already probed"
},
{
"osd": "343(4)",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [
"343(4)"
],
"waiting_on_backfill": [],
"last_backfill_started":
"404:ff80001b:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-09T12%3a41%3a05,478440857+00%3a00.2~XwOh1_IhVzghlM-zi7DfwoUIZl5UxVt.1:head",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2023-01-03T12:15:48.766366+0000"
}
],
"scrubber": {
"epoch_start": "0",
"active": false
},
"agent_state": {}
}
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
