On 23-03-2024 21:19, Alexander E. Patrakov wrote:
Hi Torkil,
Hi Alexander
I have looked at the CRUSH rules, and the equivalent rules work on my
test cluster. So this cannot be the cause of the blockage.
Thank you for taking the time =)
What happens if you increase the osd_max_backfills setting temporarily?
We already had the mclock override option in place and I re-enabled our
babysitter script which sets osd_max_backfills pr OSD to 1-3 depending
on how full they are. Active backfills went from 16 to 53 which is
probably because default osd_max_backfills for mclock is 1.
I think 53 is still a low number of active backfills given the large
percentage misplaced.
It may be a good idea to investigate a few of the stalled PGs. Please
run commands similar to this one:
ceph pg 37.0 query > query.37.0.txt
ceph pg 37.1 query > query.37.1.txt
...
and the same for the other affected pools.
A few samples attached.
Still, I must say that some of your rules are actually unsafe.
The 4+2 rule as used by rbd_ec_data will not survive a
datacenter-offline incident. Namely, for each PG, it chooses OSDs from
two hosts in each datacenter, so 6 OSDs total. When a datacenter is
offline, you will, therefore, have only 4 OSDs up, which is exactly
the number of data chunks. However, the pool requires min_size 5, so
all PGs will be inactive (to prevent data corruption) and will stay
inactive until the datacenter comes up again. However, please don't
set min_size to 4 - then, any additional incident (like a defective
disk) will lead to data loss, and the shards in the datacenter which
went offline would be useless because they do not correspond to the
updated shards written by the clients.
Thanks for the explanation. This is an old pool predating the 3 DC setup
and we'll migrate the data to a 4+5 pool when we can.
The 4+5 rule as used by cephfs.hdd.data has min_size equal to the
number of data chunks. See above why it is bad. Please set min_size to
5.
Thanks, that was a leftover for getting the PGs to peer (stuck at
creating+incomplete) when we created the pool. It's back to 5 now.
The rbd.ssd.data pool seems to be OK - and, by the way, its PGs are
100% active+clean.
There is very little data in this pool, that is probably the main reason.
Regarding the mon_max_pg_per_osd setting, you have a few OSDs that
have 300+ PGs, the observed maximum is 347. Please set it to 400.
Copy that. Didn't seem to make a difference though, and we have
osd_max_pg_per_osd_hard_ratio set to 5.000000.
Mvh.
Torkil
On Sun, Mar 24, 2024 at 3:16 AM Torkil Svensgaard <torkil@xxxxxxxx> wrote:
On 23-03-2024 19:05, Alexander E. Patrakov wrote:
Sorry for replying to myself, but "ceph osd pool ls detail" by itself
is insufficient. For every erasure code profile mentioned in the
output, please also run something like this:
ceph osd erasure-code-profile get prf-for-ec-data
...where "prf-for-ec-data" is the name that appears after the words
"erasure profile" in the "ceph osd pool ls detail" output.
[root@lazy ~]# ceph osd pool ls detail | grep erasure
pool 11 'rbd_ec_data' erasure profile DRCMR_k4m2 size 6 min_size 5
crush_rule 0 object_hash rjenkins pg_num 4096 pgp_num 4096
autoscale_mode off last_change 2257933 lfor 0/1291190/1755101 flags
hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 16384
fast_read 1 compression_algorithm snappy compression_mode aggressive
application rbd
pool 37 'cephfs.hdd.data' erasure profile DRCMR_k4m5_datacenter_hdd size
9 min_size 4 crush_rule 7 object_hash rjenkins pg_num 2048 pgp_num 2048
autoscale_mode off last_change 2257933 lfor 0/0/2139486 flags
hashpspool,ec_overwrites,bulk stripe_width 16384 fast_read 1
compression_algorithm zstd compression_mode aggressive application cephfs
pool 38 'rbd.ssd.data' erasure profile DRCMR_k4m5_datacenter_ssd size 9
min_size 5 crush_rule 8 object_hash rjenkins pg_num 32 pgp_num 32
autoscale_mode warn last_change 2198930 lfor 0/2198930/2198928 flags
hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 16384
compression_algorithm zstd compression_mode aggressive application rbd
[root@lazy ~]# ceph osd erasure-code-profile get DRCMR_k4m2
crush-device-class=hdd
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=2
plugin=jerasure
technique=reed_sol_van
w=8
[root@lazy ~]# ceph osd erasure-code-profile get DRCMR_k4m5_datacenter_hdd
crush-device-class=hdd
crush-failure-domain=datacenter
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=5
plugin=jerasure
technique=reed_sol_van
w=8
[root@lazy ~]# ceph osd erasure-code-profile get DRCMR_k4m5_datacenter_ssd
crush-device-class=ssd
crush-failure-domain=datacenter
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=5
plugin=jerasure
technique=reed_sol_van
w=8
But as I understand it those profiles are only used to create the
initial crush rule for the pool, and we have manually edited those along
the way. Here are the 3 rules in use for the 3 EC pools:
rule rbd_ec_data {
id 0
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 0 type datacenter
step chooseleaf indep 2 type host
step emit
}
rule cephfs.hdd.data {
id 7
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 0 type datacenter
step chooseleaf indep 3 type host
step emit
}
rule rbd.ssd.data {
id 8
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class ssd
step choose indep 0 type datacenter
step chooseleaf indep 3 type host
step emit
}
Which should first pick all 3 datacenters in the choose step and then
either 2 or 3 hosts in the chooseleaf step, matching EC 4+2 and 4+5
respectively.
Mvh.
Torkil
On Sun, Mar 24, 2024 at 1:56 AM Alexander E. Patrakov
<patrakov@xxxxxxxxx> wrote:
Hi Torkil,
I take my previous response back.
You have an erasure-coded pool with nine shards but only three
datacenters. This, in general, cannot work. You need either nine
datacenters or a very custom CRUSH rule. The second option may not be
available if the current EC setup is already incompatible, as there is
no way to change the EC parameters.
It would help if you provided the output of "ceph osd pool ls detail".
On Sun, Mar 24, 2024 at 1:43 AM Alexander E. Patrakov
<patrakov@xxxxxxxxx> wrote:
Hi Torkil,
Unfortunately, your files contain nothing obviously bad or suspicious,
except for two things: more PGs than usual and bad balance.
What's your "mon max pg per osd" setting?
On Sun, Mar 24, 2024 at 1:08 AM Torkil Svensgaard <torkil@xxxxxxxx> wrote:
On 2024-03-23 17:54, Kai Stian Olstad wrote:
On Sat, Mar 23, 2024 at 12:09:29PM +0100, Torkil Svensgaard wrote:
The other output is too big for pastebin and I'm not familiar with
paste services, any suggestion for a preferred way to share such
output?
You can attached files to the mail here on the list.
Doh, for some reason I was sure attachments would be stripped. Thanks,
attached.
Mvh.
Torkil
--
Alexander E. Patrakov
--
Alexander E. Patrakov
--
Torkil Svensgaard
Systems Administrator
Danish Research Centre for Magnetic Resonance DRCMR, Section 714
Copenhagen University Hospital Amager and Hvidovre
Kettegaard Allé 30, 2650 Hvidovre, Denmark
--
Torkil Svensgaard
Systems Administrator
Danish Research Centre for Magnetic Resonance DRCMR, Section 714
Copenhagen University Hospital Amager and Hvidovre
Kettegaard Allé 30, 2650 Hvidovre, Denmark
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+clean",
"epoch": 2261510,
"up": [
162,
290,
454,
464,
278,
122
],
"acting": [
162,
290,
454,
464,
278,
122
],
"acting_recovery_backfill": [
"122(5)",
"162(0)",
"278(4)",
"290(1)",
"454(2)",
"464(3)"
],
"info": {
"pgid": "11.3dbs0",
"last_update": "2261372'3019845",
"last_complete": "2261372'3019845",
"log_tail": "2249580'3017057",
"last_user_version": 3019845,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1503254,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261037,
"last_interval_started": 2261036,
"last_epoch_clean": 2261038,
"last_interval_clean": 2261036,
"last_epoch_split": 1743904,
"last_epoch_marked_full": 0,
"same_up_since": 2261036,
"same_interval_since": 2261036,
"same_primary_since": 2259582,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2261372'3019845",
"reported_seq": 27180137,
"reported_epoch": 2261510,
"state": "active+clean",
"last_fresh": "2024-03-23T20:34:23.285368+0000",
"last_change": "2024-03-23T13:14:01.004398+0000",
"last_active": "2024-03-23T20:34:23.285368+0000",
"last_peered": "2024-03-23T20:34:23.285368+0000",
"last_clean": "2024-03-23T20:34:23.285368+0000",
"last_became_active": "2024-03-23T13:13:59.061204+0000",
"last_became_peered": "2024-03-23T13:13:59.061204+0000",
"last_unstale": "2024-03-23T20:34:23.285368+0000",
"last_undegraded": "2024-03-23T20:34:23.285368+0000",
"last_fullsized": "2024-03-23T20:34:23.285368+0000",
"mapping_epoch": 2261036,
"log_start": "2249580'3017057",
"ondisk_log_start": "2249580'3017057",
"created": 1503254,
"last_epoch_clean": 2261038,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"objects_scrubbed": 46427,
"log_size": 2788,
"log_dups_size": 3000,
"ondisk_log_size": 2788,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.8005263170000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 194394914816,
"num_objects": 46688,
"num_object_clones": 12,
"num_object_copies": 280128,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 46688,
"num_whiteouts": 0,
"num_read": 1716199,
"num_read_kb": 181264028,
"num_write": 182112,
"num_write_kb": 128528520,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 749305,
"num_bytes_recovered": 3128538976256,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
162,
290,
454,
464,
278,
122
],
"acting": [
162,
290,
454,
464,
278,
122
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 162,
"acting_primary": 162,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261037,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "122(5)",
"pgid": "11.3dbs5",
"last_update": "2261372'3019845",
"last_complete": "2261372'3019845",
"log_tail": "2255706'3018702",
"last_user_version": 3019793,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1503254,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261037,
"last_interval_started": 2261036,
"last_epoch_clean": 2261038,
"last_interval_clean": 2261036,
"last_epoch_split": 1743904,
"last_epoch_marked_full": 0,
"same_up_since": 2261036,
"same_interval_since": 2261036,
"same_primary_since": 2259582,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2261016'3019793",
"reported_seq": 27175928,
"reported_epoch": 2261016,
"state": "active+undersized+degraded",
"last_fresh": "2024-03-23T13:09:16.331624+0000",
"last_change": "2024-03-23T13:07:19.908277+0000",
"last_active": "2024-03-23T13:09:16.331624+0000",
"last_peered": "2024-03-23T13:09:16.331624+0000",
"last_clean": "2024-03-23T13:07:16.168828+0000",
"last_became_active": "2024-03-23T13:07:19.908277+0000",
"last_became_peered": "2024-03-23T13:07:19.908277+0000",
"last_unstale": "2024-03-23T13:09:16.331624+0000",
"last_undegraded": "2024-03-23T13:07:19.690742+0000",
"last_fullsized": "2024-03-23T13:07:19.690695+0000",
"mapping_epoch": 2261036,
"log_start": "2249576'3016957",
"ondisk_log_start": "2249576'3016957",
"created": 1503254,
"last_epoch_clean": 2260147,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"objects_scrubbed": 46427,
"log_size": 2836,
"log_dups_size": 3000,
"ondisk_log_size": 2836,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.8005263170000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 194386526208,
"num_objects": 46686,
"num_object_clones": 12,
"num_object_copies": 280116,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 46686,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 46686,
"num_whiteouts": 0,
"num_read": 1712732,
"num_read_kb": 181204224,
"num_write": 182060,
"num_write_kb": 128511384,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 749304,
"num_bytes_recovered": 3128534781952,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
162,
290,
454,
464,
278,
122
],
"acting": [
162,
290,
454,
464,
278,
122
],
"avail_no_missing": [
"162(0)",
"122(5)",
"278(4)",
"290(1)",
"464(3)"
],
"object_location_counts": [
{
"shards": "122(5),162(0),278(4),290(1),464(3)",
"objects": 46686
}
],
"blocked_by": [],
"up_primary": 162,
"acting_primary": 162,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261037,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "278(4)",
"pgid": "11.3dbs4",
"last_update": "2261372'3019845",
"last_complete": "2261372'3019845",
"log_tail": "2255706'3018702",
"last_user_version": 3019793,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1503254,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261037,
"last_interval_started": 2261036,
"last_epoch_clean": 2261038,
"last_interval_clean": 2261036,
"last_epoch_split": 1743904,
"last_epoch_marked_full": 0,
"same_up_since": 2261036,
"same_interval_since": 2261036,
"same_primary_since": 2259582,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2261016'3019793",
"reported_seq": 27175928,
"reported_epoch": 2261016,
"state": "active+undersized+degraded",
"last_fresh": "2024-03-23T13:09:16.331624+0000",
"last_change": "2024-03-23T13:07:19.908277+0000",
"last_active": "2024-03-23T13:09:16.331624+0000",
"last_peered": "2024-03-23T13:09:16.331624+0000",
"last_clean": "2024-03-23T13:07:16.168828+0000",
"last_became_active": "2024-03-23T13:07:19.908277+0000",
"last_became_peered": "2024-03-23T13:07:19.908277+0000",
"last_unstale": "2024-03-23T13:09:16.331624+0000",
"last_undegraded": "2024-03-23T13:07:19.690742+0000",
"last_fullsized": "2024-03-23T13:07:19.690695+0000",
"mapping_epoch": 2261036,
"log_start": "2249576'3016957",
"ondisk_log_start": "2249576'3016957",
"created": 1503254,
"last_epoch_clean": 2260147,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"objects_scrubbed": 46427,
"log_size": 2836,
"log_dups_size": 3000,
"ondisk_log_size": 2836,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.8005263170000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 194386526208,
"num_objects": 46686,
"num_object_clones": 12,
"num_object_copies": 280116,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 46686,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 46686,
"num_whiteouts": 0,
"num_read": 1712732,
"num_read_kb": 181204224,
"num_write": 182060,
"num_write_kb": 128511384,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 749304,
"num_bytes_recovered": 3128534781952,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
162,
290,
454,
464,
278,
122
],
"acting": [
162,
290,
454,
464,
278,
122
],
"avail_no_missing": [
"162(0)",
"122(5)",
"278(4)",
"290(1)",
"464(3)"
],
"object_location_counts": [
{
"shards": "122(5),162(0),278(4),290(1),464(3)",
"objects": 46686
}
],
"blocked_by": [],
"up_primary": 162,
"acting_primary": 162,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261037,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "290(1)",
"pgid": "11.3dbs1",
"last_update": "2261372'3019845",
"last_complete": "2261372'3019845",
"log_tail": "2255706'3018702",
"last_user_version": 3019793,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1503254,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261037,
"last_interval_started": 2261036,
"last_epoch_clean": 2261038,
"last_interval_clean": 2261036,
"last_epoch_split": 1743904,
"last_epoch_marked_full": 0,
"same_up_since": 2261036,
"same_interval_since": 2261036,
"same_primary_since": 2259582,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2261016'3019793",
"reported_seq": 27175928,
"reported_epoch": 2261016,
"state": "active+undersized+degraded",
"last_fresh": "2024-03-23T13:09:16.331624+0000",
"last_change": "2024-03-23T13:07:19.908277+0000",
"last_active": "2024-03-23T13:09:16.331624+0000",
"last_peered": "2024-03-23T13:09:16.331624+0000",
"last_clean": "2024-03-23T13:07:16.168828+0000",
"last_became_active": "2024-03-23T13:07:19.908277+0000",
"last_became_peered": "2024-03-23T13:07:19.908277+0000",
"last_unstale": "2024-03-23T13:09:16.331624+0000",
"last_undegraded": "2024-03-23T13:07:19.690742+0000",
"last_fullsized": "2024-03-23T13:07:19.690695+0000",
"mapping_epoch": 2261036,
"log_start": "2249576'3016957",
"ondisk_log_start": "2249576'3016957",
"created": 1503254,
"last_epoch_clean": 2260147,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"objects_scrubbed": 46427,
"log_size": 2836,
"log_dups_size": 3000,
"ondisk_log_size": 2836,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.8005263170000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 194386526208,
"num_objects": 46686,
"num_object_clones": 12,
"num_object_copies": 280116,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 46686,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 46686,
"num_whiteouts": 0,
"num_read": 1712732,
"num_read_kb": 181204224,
"num_write": 182060,
"num_write_kb": 128511384,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 749304,
"num_bytes_recovered": 3128534781952,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
162,
290,
454,
464,
278,
122
],
"acting": [
162,
290,
454,
464,
278,
122
],
"avail_no_missing": [
"162(0)",
"122(5)",
"278(4)",
"290(1)",
"464(3)"
],
"object_location_counts": [
{
"shards": "122(5),162(0),278(4),290(1),464(3)",
"objects": 46686
}
],
"blocked_by": [],
"up_primary": 162,
"acting_primary": 162,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261037,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "454(2)",
"pgid": "11.3dbs2",
"last_update": "2261372'3019845",
"last_complete": "2260998'3019788",
"log_tail": "2255706'3018702",
"last_user_version": 3019788,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1503254,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261037,
"last_interval_started": 2261036,
"last_epoch_clean": 2261038,
"last_interval_clean": 2261036,
"last_epoch_split": 1743904,
"last_epoch_marked_full": 0,
"same_up_since": 2261036,
"same_interval_since": 2261036,
"same_primary_since": 2259582,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260998'3019788",
"reported_seq": 27175781,
"reported_epoch": 2260998,
"state": "active+clean",
"last_fresh": "2024-03-23T13:02:06.580565+0000",
"last_change": "2024-03-23T08:05:40.566283+0000",
"last_active": "2024-03-23T13:02:06.580565+0000",
"last_peered": "2024-03-23T13:02:06.580565+0000",
"last_clean": "2024-03-23T13:02:06.580565+0000",
"last_became_active": "2024-03-23T08:05:39.190863+0000",
"last_became_peered": "2024-03-23T08:05:39.190863+0000",
"last_unstale": "2024-03-23T13:02:06.580565+0000",
"last_undegraded": "2024-03-23T13:02:06.580565+0000",
"last_fullsized": "2024-03-23T13:02:06.580565+0000",
"mapping_epoch": 2261036,
"log_start": "2249576'3016957",
"ondisk_log_start": "2249576'3016957",
"created": 1503254,
"last_epoch_clean": 2260147,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"objects_scrubbed": 46427,
"log_size": 2831,
"log_dups_size": 3000,
"ondisk_log_size": 2831,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.8005263170000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 194386526208,
"num_objects": 46686,
"num_object_clones": 12,
"num_object_copies": 280116,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 46686,
"num_whiteouts": 0,
"num_read": 1712619,
"num_read_kb": 181202460,
"num_write": 182055,
"num_write_kb": 128511288,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 749304,
"num_bytes_recovered": 3128534781952,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
162,
290,
454,
464,
278,
122
],
"acting": [
162,
290,
454,
464,
278,
122
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 162,
"acting_primary": 162,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261037,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "464(3)",
"pgid": "11.3dbs3",
"last_update": "2261372'3019845",
"last_complete": "2261372'3019845",
"log_tail": "2255706'3018702",
"last_user_version": 3019793,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1503254,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261037,
"last_interval_started": 2261036,
"last_epoch_clean": 2261038,
"last_interval_clean": 2261036,
"last_epoch_split": 1743904,
"last_epoch_marked_full": 0,
"same_up_since": 2261036,
"same_interval_since": 2261036,
"same_primary_since": 2259582,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2261016'3019793",
"reported_seq": 27175928,
"reported_epoch": 2261016,
"state": "active+undersized+degraded",
"last_fresh": "2024-03-23T13:09:16.331624+0000",
"last_change": "2024-03-23T13:07:19.908277+0000",
"last_active": "2024-03-23T13:09:16.331624+0000",
"last_peered": "2024-03-23T13:09:16.331624+0000",
"last_clean": "2024-03-23T13:07:16.168828+0000",
"last_became_active": "2024-03-23T13:07:19.908277+0000",
"last_became_peered": "2024-03-23T13:07:19.908277+0000",
"last_unstale": "2024-03-23T13:09:16.331624+0000",
"last_undegraded": "2024-03-23T13:07:19.690742+0000",
"last_fullsized": "2024-03-23T13:07:19.690695+0000",
"mapping_epoch": 2261036,
"log_start": "2249576'3016957",
"ondisk_log_start": "2249576'3016957",
"created": 1503254,
"last_epoch_clean": 2260147,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "2239903'3014686",
"last_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"last_deep_scrub": "2224328'3010417",
"last_deep_scrub_stamp": "2024-03-05T07:05:36.866264+0000",
"last_clean_scrub_stamp": "2024-03-15T06:15:26.964830+0000",
"objects_scrubbed": 46427,
"log_size": 2836,
"log_dups_size": 3000,
"ondisk_log_size": 2836,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.8005263170000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 194386526208,
"num_objects": 46686,
"num_object_clones": 12,
"num_object_copies": 280116,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 46686,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 46686,
"num_whiteouts": 0,
"num_read": 1712732,
"num_read_kb": 181204224,
"num_write": 182060,
"num_write_kb": 128511384,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 749304,
"num_bytes_recovered": 3128534781952,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
162,
290,
454,
464,
278,
122
],
"acting": [
162,
290,
454,
464,
278,
122
],
"avail_no_missing": [
"162(0)",
"122(5)",
"278(4)",
"290(1)",
"464(3)"
],
"object_location_counts": [
{
"shards": "122(5),162(0),278(4),290(1),464(3)",
"objects": 46686
}
],
"blocked_by": [],
"up_primary": 162,
"acting_primary": 162,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261037,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2024-03-23T13:13:58.974388+0000",
"might_have_unfound": [
{
"osd": "122(5)",
"status": "already probed"
},
{
"osd": "278(4)",
"status": "already probed"
},
{
"osd": "290(1)",
"status": "already probed"
},
{
"osd": "454(2)",
"status": "already probed"
},
{
"osd": "464(3)",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2024-03-23T13:13:57.395127+0000"
}
],
"scrubber": {
"active": false,
"must_scrub": false,
"must_deep_scrub": false,
"must_repair": false,
"need_auto": false,
"scrub_reg_stamp": "2024-03-16T06:15:26.964830+0000",
"schedule": "queued for scrub"
},
"agent_state": {}
}
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+remapped+backfill_wait",
"epoch": 2261510,
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"backfill_targets": [
"442(3)",
"458(5)",
"506(4)"
],
"acting_recovery_backfill": [
"269(2)",
"273(3)",
"275(1)",
"298(0)",
"409(5)",
"440(4)",
"442(3)",
"458(5)",
"506(4)"
],
"info": {
"pgid": "11.fffs0",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114248,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2261478'3114248",
"reported_seq": 27297615,
"reported_epoch": 2261510,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T20:32:46.089783+0000",
"last_change": "2024-03-23T13:38:04.290356+0000",
"last_active": "2024-03-23T20:32:46.089783+0000",
"last_peered": "2024-03-23T20:32:46.089783+0000",
"last_clean": "2024-03-19T12:59:01.132410+0000",
"last_became_active": "2024-03-23T13:20:41.305412+0000",
"last_became_peered": "2024-03-23T13:20:41.305412+0000",
"last_unstale": "2024-03-23T20:32:46.089783+0000",
"last_undegraded": "2024-03-23T20:32:46.089783+0000",
"last_fullsized": "2024-03-23T20:32:46.089783+0000",
"mapping_epoch": 2261083,
"log_start": "2257134'3113294",
"ondisk_log_start": "2257134'3113294",
"created": 1755101,
"last_epoch_clean": 2248989,
"parent": "0.0",
"parent_split_bits": 12,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"objects_scrubbed": 45979,
"log_size": 954,
"log_dups_size": 3000,
"ondisk_log_size": 954,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.3869062870000004,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 192684589056,
"num_objects": 46279,
"num_object_clones": 9,
"num_object_copies": 277674,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 138381,
"num_objects_unfound": 0,
"num_objects_dirty": 46279,
"num_whiteouts": 0,
"num_read": 1766709,
"num_read_kb": 177520716,
"num_write": 187542,
"num_write_kb": 124853392,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 931582,
"num_bytes_recovered": 3891444797440,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [
"298(0)",
"269(2)",
"273(3)",
"275(1)",
"409(5)",
"440(4)"
],
"object_location_counts": [
{
"shards": "269(2),273(3),275(1),298(0),409(5),440(4)",
"objects": 46279
}
],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "237(4)",
"pgid": "11.fffs4",
"last_update": "2250634'3112256",
"last_complete": "2250634'3112256",
"log_tail": "2244781'3111291",
"last_user_version": 3112256,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2260405,
"last_interval_started": 2260401,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261083,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 46279,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249821,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "269(2)",
"pgid": "11.fffs2",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114202,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260999'3114202",
"reported_seq": 27292399,
"reported_epoch": 2260999,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T13:02:54.011853+0000",
"last_change": "2024-03-23T09:38:45.752967+0000",
"last_active": "2024-03-23T13:02:54.011853+0000",
"last_peered": "2024-03-23T13:02:54.011853+0000",
"last_clean": "2024-03-19T12:59:01.132410+0000",
"last_became_active": "2024-03-23T09:38:26.381152+0000",
"last_became_peered": "2024-03-23T09:38:26.381152+0000",
"last_unstale": "2024-03-23T13:02:54.011853+0000",
"last_undegraded": "2024-03-23T13:02:54.011853+0000",
"last_fullsized": "2024-03-23T13:02:54.011853+0000",
"mapping_epoch": 2261083,
"log_start": "2257134'3113294",
"ondisk_log_start": "2257134'3113294",
"created": 1755101,
"last_epoch_clean": 2248989,
"parent": "0.0",
"parent_split_bits": 12,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"objects_scrubbed": 45979,
"log_size": 908,
"log_dups_size": 3000,
"ondisk_log_size": 908,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.3869062870000004,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 192684589056,
"num_objects": 46279,
"num_object_clones": 9,
"num_object_copies": 277674,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 138441,
"num_objects_unfound": 0,
"num_objects_dirty": 46279,
"num_whiteouts": 0,
"num_read": 1762361,
"num_read_kb": 177459148,
"num_write": 187496,
"num_write_kb": 124852592,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 931562,
"num_bytes_recovered": 3891366678528,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [
"298(0)",
"237(4)",
"269(2)",
"273(3)",
"275(1)",
"409(5)",
"440(4)"
],
"object_location_counts": [
{
"shards": "269(2),273(3),275(1),298(0),409(5),440(4)",
"objects": 46279
}
],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "273(3)",
"pgid": "11.fffs3",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114202,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260999'3114202",
"reported_seq": 27292399,
"reported_epoch": 2260999,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T13:02:54.011853+0000",
"last_change": "2024-03-23T09:38:45.752967+0000",
"last_active": "2024-03-23T13:02:54.011853+0000",
"last_peered": "2024-03-23T13:02:54.011853+0000",
"last_clean": "2024-03-19T12:59:01.132410+0000",
"last_became_active": "2024-03-23T09:38:26.381152+0000",
"last_became_peered": "2024-03-23T09:38:26.381152+0000",
"last_unstale": "2024-03-23T13:02:54.011853+0000",
"last_undegraded": "2024-03-23T13:02:54.011853+0000",
"last_fullsized": "2024-03-23T13:02:54.011853+0000",
"mapping_epoch": 2261083,
"log_start": "2257134'3113294",
"ondisk_log_start": "2257134'3113294",
"created": 1755101,
"last_epoch_clean": 2248989,
"parent": "0.0",
"parent_split_bits": 12,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"objects_scrubbed": 45979,
"log_size": 908,
"log_dups_size": 3000,
"ondisk_log_size": 908,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.3869062870000004,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 192684589056,
"num_objects": 46279,
"num_object_clones": 9,
"num_object_copies": 277674,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 138441,
"num_objects_unfound": 0,
"num_objects_dirty": 46279,
"num_whiteouts": 0,
"num_read": 1762361,
"num_read_kb": 177459148,
"num_write": 187496,
"num_write_kb": 124852592,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 931562,
"num_bytes_recovered": 3891366678528,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [
"298(0)",
"237(4)",
"269(2)",
"273(3)",
"275(1)",
"409(5)",
"440(4)"
],
"object_location_counts": [
{
"shards": "269(2),273(3),275(1),298(0),409(5),440(4)",
"objects": 46279
}
],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "275(1)",
"pgid": "11.fffs1",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114202,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260999'3114202",
"reported_seq": 27292399,
"reported_epoch": 2260999,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T13:02:54.011853+0000",
"last_change": "2024-03-23T09:38:45.752967+0000",
"last_active": "2024-03-23T13:02:54.011853+0000",
"last_peered": "2024-03-23T13:02:54.011853+0000",
"last_clean": "2024-03-19T12:59:01.132410+0000",
"last_became_active": "2024-03-23T09:38:26.381152+0000",
"last_became_peered": "2024-03-23T09:38:26.381152+0000",
"last_unstale": "2024-03-23T13:02:54.011853+0000",
"last_undegraded": "2024-03-23T13:02:54.011853+0000",
"last_fullsized": "2024-03-23T13:02:54.011853+0000",
"mapping_epoch": 2261083,
"log_start": "2257134'3113294",
"ondisk_log_start": "2257134'3113294",
"created": 1755101,
"last_epoch_clean": 2248989,
"parent": "0.0",
"parent_split_bits": 12,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"objects_scrubbed": 45979,
"log_size": 908,
"log_dups_size": 3000,
"ondisk_log_size": 908,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.3869062870000004,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 192684589056,
"num_objects": 46279,
"num_object_clones": 9,
"num_object_copies": 277674,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 138441,
"num_objects_unfound": 0,
"num_objects_dirty": 46279,
"num_whiteouts": 0,
"num_read": 1762361,
"num_read_kb": 177459148,
"num_write": 187496,
"num_write_kb": 124852592,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 931562,
"num_bytes_recovered": 3891366678528,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [
"298(0)",
"237(4)",
"269(2)",
"273(3)",
"275(1)",
"409(5)",
"440(4)"
],
"object_location_counts": [
{
"shards": "269(2),273(3),275(1),298(0),409(5),440(4)",
"objects": 46279
}
],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "409(5)",
"pgid": "11.fffs5",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114202,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260999'3114202",
"reported_seq": 27292399,
"reported_epoch": 2260999,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T13:02:54.011853+0000",
"last_change": "2024-03-23T09:38:45.752967+0000",
"last_active": "2024-03-23T13:02:54.011853+0000",
"last_peered": "2024-03-23T13:02:54.011853+0000",
"last_clean": "2024-03-19T12:59:01.132410+0000",
"last_became_active": "2024-03-23T09:38:26.381152+0000",
"last_became_peered": "2024-03-23T09:38:26.381152+0000",
"last_unstale": "2024-03-23T13:02:54.011853+0000",
"last_undegraded": "2024-03-23T13:02:54.011853+0000",
"last_fullsized": "2024-03-23T13:02:54.011853+0000",
"mapping_epoch": 2261083,
"log_start": "2257134'3113294",
"ondisk_log_start": "2257134'3113294",
"created": 1755101,
"last_epoch_clean": 2248989,
"parent": "0.0",
"parent_split_bits": 12,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"objects_scrubbed": 45979,
"log_size": 908,
"log_dups_size": 3000,
"ondisk_log_size": 908,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.3869062870000004,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 192684589056,
"num_objects": 46279,
"num_object_clones": 9,
"num_object_copies": 277674,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 138441,
"num_objects_unfound": 0,
"num_objects_dirty": 46279,
"num_whiteouts": 0,
"num_read": 1762361,
"num_read_kb": 177459148,
"num_write": 187496,
"num_write_kb": 124852592,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 931562,
"num_bytes_recovered": 3891366678528,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [
"298(0)",
"237(4)",
"269(2)",
"273(3)",
"275(1)",
"409(5)",
"440(4)"
],
"object_location_counts": [
{
"shards": "269(2),273(3),275(1),298(0),409(5),440(4)",
"objects": 46279
}
],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "440(4)",
"pgid": "11.fffs4",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114202,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260999'3114202",
"reported_seq": 27292399,
"reported_epoch": 2260999,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T13:02:54.011853+0000",
"last_change": "2024-03-23T09:38:45.752967+0000",
"last_active": "2024-03-23T13:02:54.011853+0000",
"last_peered": "2024-03-23T13:02:54.011853+0000",
"last_clean": "2024-03-19T12:59:01.132410+0000",
"last_became_active": "2024-03-23T09:38:26.381152+0000",
"last_became_peered": "2024-03-23T09:38:26.381152+0000",
"last_unstale": "2024-03-23T13:02:54.011853+0000",
"last_undegraded": "2024-03-23T13:02:54.011853+0000",
"last_fullsized": "2024-03-23T13:02:54.011853+0000",
"mapping_epoch": 2261083,
"log_start": "2257134'3113294",
"ondisk_log_start": "2257134'3113294",
"created": 1755101,
"last_epoch_clean": 2248989,
"parent": "0.0",
"parent_split_bits": 12,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"objects_scrubbed": 45979,
"log_size": 908,
"log_dups_size": 3000,
"ondisk_log_size": 908,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 6,
"scrub_schedule": "queued for scrub",
"scrub_duration": 5.3869062870000004,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 192684589056,
"num_objects": 46279,
"num_object_clones": 9,
"num_object_copies": 277674,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 138441,
"num_objects_unfound": 0,
"num_objects_dirty": 46279,
"num_whiteouts": 0,
"num_read": 1762361,
"num_read_kb": 177459148,
"num_write": 187496,
"num_write_kb": 124852592,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 931562,
"num_bytes_recovered": 3891366678528,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [
"298(0)",
"237(4)",
"269(2)",
"273(3)",
"275(1)",
"409(5)",
"440(4)"
],
"object_location_counts": [
{
"shards": "269(2),273(3),275(1),298(0),409(5),440(4)",
"objects": 46279
}
],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "442(3)",
"pgid": "11.fffs3",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114202,
"last_backfill": "11:fff00eff:::rbd_data.10.15a979db61dda7.0000000008d44bd8:head",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261083,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 623378432,
"num_objects": 152,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 46127,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 152,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "458(5)",
"pgid": "11.fffs5",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114202,
"last_backfill": "11:fff00eff:::rbd_data.10.15a979db61dda7.0000000008d44bd8:head",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261083,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 623378432,
"num_objects": 152,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 46127,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 152,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "506(4)",
"pgid": "11.fffs4",
"last_update": "2261478'3114248",
"last_complete": "2261478'3114248",
"log_tail": "2257134'3113294",
"last_user_version": 3114202,
"last_backfill": "11:fff00eff:::rbd_data.10.15a979db61dda7.0000000008d44bd8:head",
"purged_snaps": [],
"history": {
"epoch_created": 1755101,
"epoch_pool_created": 1183522,
"last_epoch_started": 2261084,
"last_interval_started": 2261083,
"last_epoch_clean": 2248989,
"last_interval_clean": 2248988,
"last_epoch_split": 1755101,
"last_epoch_marked_full": 0,
"same_up_since": 2260456,
"same_interval_since": 2261083,
"same_primary_since": 2260457,
"last_scrub": "2237175'3110177",
"last_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"last_deep_scrub": "2230303'3107531",
"last_deep_scrub_stamp": "2024-03-09T00:03:48.057602+0000",
"last_clean_scrub_stamp": "2024-03-14T02:44:04.637166+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261083,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 623378432,
"num_objects": 152,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 46127,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 152,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
298,
275,
269,
442,
506,
458
],
"acting": [
298,
275,
269,
273,
440,
409
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 298,
"acting_primary": 298,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261084,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2024-03-23T13:20:41.154503+0000",
"might_have_unfound": [
{
"osd": "237(4)",
"status": "not queried"
}
],
"recovery_progress": {
"backfill_targets": [
"442(3)",
"458(5)",
"506(4)"
],
"waiting_on_backfill": [],
"last_backfill_started": "11:fff00eff:::rbd_data.10.15a979db61dda7.0000000008d44bd8:head",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2024-03-23T13:20:39.494676+0000"
}
],
"scrubber": {
"active": false,
"must_scrub": false,
"must_deep_scrub": false,
"must_repair": false,
"need_auto": false,
"scrub_reg_stamp": "2024-03-15T02:44:04.637166+0000",
"schedule": "queued for scrub"
},
"agent_state": {}
}
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+remapped+backfill_wait",
"epoch": 2261510,
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"backfill_targets": [
"56(5)",
"78(4)",
"183(6)",
"256(2)",
"265(8)",
"293(7)",
"463(0)",
"491(1)"
],
"acting_recovery_backfill": [
"30(6)",
"33(7)",
"56(5)",
"56(8)",
"78(4)",
"183(0)",
"183(6)",
"256(2)",
"256(5)",
"265(2)",
"265(8)",
"293(1)",
"293(7)",
"424(4)",
"436(3)",
"463(0)",
"491(1)"
],
"info": {
"pgid": "37.1s0",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257923'162308",
"last_user_version": 163591,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2261507'163591",
"reported_seq": 717548,
"reported_epoch": 2261510,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T20:30:37.154379+0000",
"last_change": "2024-03-23T13:21:35.209393+0000",
"last_active": "2024-03-23T20:30:37.154379+0000",
"last_peered": "2024-03-23T20:30:37.154379+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T13:21:35.209252+0000",
"last_became_peered": "2024-03-23T13:21:35.209252+0000",
"last_unstale": "2024-03-23T20:30:37.154379+0000",
"last_undegraded": "2024-03-23T20:30:37.154379+0000",
"last_fullsized": "2024-03-23T20:30:37.154379+0000",
"mapping_epoch": 2261102,
"log_start": "2257923'162308",
"ondisk_log_start": "2257923'162308",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1283,
"log_dups_size": 3000,
"ondisk_log_size": 1283,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 93359062318,
"num_objects": 78813,
"num_object_clones": 0,
"num_object_copies": 709317,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 630504,
"num_objects_unfound": 0,
"num_objects_dirty": 78813,
"num_whiteouts": 0,
"num_read": 80686,
"num_read_kb": 112206672,
"num_write": 228767,
"num_write_kb": 109403261,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [
"183(0)",
"30(6)",
"33(7)",
"56(8)",
"256(5)",
"265(2)",
"293(1)",
"424(4)",
"436(3)"
],
"object_location_counts": [
{
"shards": "30(6),33(7),56(8),183(0),256(5),265(2),293(1),424(4),436(3)",
"objects": 78813
}
],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "30(3)",
"pgid": "37.1s3",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249712,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "30(6)",
"pgid": "37.1s6",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'163465",
"reported_seq": 713683,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:49.686570+0000",
"last_change": "2024-03-23T08:03:18.391295+0000",
"last_active": "2024-03-23T08:23:49.686570+0000",
"last_peered": "2024-03-23T08:23:49.686570+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T08:03:18.390854+0000",
"last_became_peered": "2024-03-23T08:03:18.390854+0000",
"last_unstale": "2024-03-23T08:23:49.686570+0000",
"last_undegraded": "2024-03-23T08:23:49.686570+0000",
"last_fullsized": "2024-03-23T08:23:49.686570+0000",
"mapping_epoch": 2261102,
"log_start": "2257907'162208",
"ondisk_log_start": "2257907'162208",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1257,
"log_dups_size": 3000,
"ondisk_log_size": 1257,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92981574958,
"num_objects": 78723,
"num_object_clones": 0,
"num_object_copies": 708507,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 629784,
"num_objects_unfound": 0,
"num_objects_dirty": 78723,
"num_whiteouts": 0,
"num_read": 78736,
"num_read_kb": 112024442,
"num_write": 228641,
"num_write_kb": 109034621,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [
"183(0)",
"30(6)",
"33(7)",
"56(8)",
"256(5)",
"265(2)",
"293(1)",
"424(4)",
"436(3)"
],
"object_location_counts": [
{
"shards": "30(6),33(7),56(8),183(0),256(5),265(2),293(1),424(4),436(3)",
"objects": 78723
}
],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "33(4)",
"pgid": "37.1s4",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249712,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "33(7)",
"pgid": "37.1s7",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'163465",
"reported_seq": 713683,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:49.686570+0000",
"last_change": "2024-03-23T08:03:18.391295+0000",
"last_active": "2024-03-23T08:23:49.686570+0000",
"last_peered": "2024-03-23T08:23:49.686570+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T08:03:18.390854+0000",
"last_became_peered": "2024-03-23T08:03:18.390854+0000",
"last_unstale": "2024-03-23T08:23:49.686570+0000",
"last_undegraded": "2024-03-23T08:23:49.686570+0000",
"last_fullsized": "2024-03-23T08:23:49.686570+0000",
"mapping_epoch": 2261102,
"log_start": "2257907'162208",
"ondisk_log_start": "2257907'162208",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1257,
"log_dups_size": 3000,
"ondisk_log_size": 1257,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92981574958,
"num_objects": 78723,
"num_object_clones": 0,
"num_object_copies": 708507,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 629784,
"num_objects_unfound": 0,
"num_objects_dirty": 78723,
"num_whiteouts": 0,
"num_read": 78736,
"num_read_kb": 112024442,
"num_write": 228641,
"num_write_kb": 109034621,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [
"183(0)",
"30(6)",
"33(7)",
"56(8)",
"256(5)",
"265(2)",
"293(1)",
"424(4)",
"436(3)"
],
"object_location_counts": [
{
"shards": "30(6),33(7),56(8),183(0),256(5),265(2),293(1),424(4),436(3)",
"objects": 78723
}
],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "56(5)",
"pgid": "37.1s5",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "56(8)",
"pgid": "37.1s8",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'163465",
"reported_seq": 713683,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:49.686570+0000",
"last_change": "2024-03-23T08:03:18.391295+0000",
"last_active": "2024-03-23T08:23:49.686570+0000",
"last_peered": "2024-03-23T08:23:49.686570+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T08:03:18.390854+0000",
"last_became_peered": "2024-03-23T08:03:18.390854+0000",
"last_unstale": "2024-03-23T08:23:49.686570+0000",
"last_undegraded": "2024-03-23T08:23:49.686570+0000",
"last_fullsized": "2024-03-23T08:23:49.686570+0000",
"mapping_epoch": 2261102,
"log_start": "2257907'162208",
"ondisk_log_start": "2257907'162208",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1257,
"log_dups_size": 3000,
"ondisk_log_size": 1257,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92981574958,
"num_objects": 78723,
"num_object_clones": 0,
"num_object_copies": 708507,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 629784,
"num_objects_unfound": 0,
"num_objects_dirty": 78723,
"num_whiteouts": 0,
"num_read": 78736,
"num_read_kb": 112024442,
"num_write": 228641,
"num_write_kb": 109034621,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [
"183(0)",
"30(6)",
"33(7)",
"56(8)",
"256(5)",
"265(2)",
"293(1)",
"424(4)",
"436(3)"
],
"object_location_counts": [
{
"shards": "30(6),33(7),56(8),183(0),256(5),265(2),293(1),424(4),436(3)",
"objects": 78723
}
],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "78(4)",
"pgid": "37.1s4",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "78(7)",
"pgid": "37.1s7",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249821,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "183(6)",
"pgid": "37.1s6",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "256(2)",
"pgid": "37.1s2",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "256(5)",
"pgid": "37.1s5",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'163465",
"reported_seq": 713683,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:49.686570+0000",
"last_change": "2024-03-23T08:03:18.391295+0000",
"last_active": "2024-03-23T08:23:49.686570+0000",
"last_peered": "2024-03-23T08:23:49.686570+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T08:03:18.390854+0000",
"last_became_peered": "2024-03-23T08:03:18.390854+0000",
"last_unstale": "2024-03-23T08:23:49.686570+0000",
"last_undegraded": "2024-03-23T08:23:49.686570+0000",
"last_fullsized": "2024-03-23T08:23:49.686570+0000",
"mapping_epoch": 2261102,
"log_start": "2257907'162208",
"ondisk_log_start": "2257907'162208",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1257,
"log_dups_size": 3000,
"ondisk_log_size": 1257,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92981574958,
"num_objects": 78723,
"num_object_clones": 0,
"num_object_copies": 708507,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 629784,
"num_objects_unfound": 0,
"num_objects_dirty": 78723,
"num_whiteouts": 0,
"num_read": 78736,
"num_read_kb": 112024442,
"num_write": 228641,
"num_write_kb": 109034621,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [
"183(0)",
"30(6)",
"33(7)",
"56(8)",
"256(5)",
"265(2)",
"293(1)",
"424(4)",
"436(3)"
],
"object_location_counts": [
{
"shards": "30(6),33(7),56(8),183(0),256(5),265(2),293(1),424(4),436(3)",
"objects": 78723
}
],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "265(2)",
"pgid": "37.1s2",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'163465",
"reported_seq": 713683,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:49.686570+0000",
"last_change": "2024-03-23T08:03:18.391295+0000",
"last_active": "2024-03-23T08:23:49.686570+0000",
"last_peered": "2024-03-23T08:23:49.686570+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T08:03:18.390854+0000",
"last_became_peered": "2024-03-23T08:03:18.390854+0000",
"last_unstale": "2024-03-23T08:23:49.686570+0000",
"last_undegraded": "2024-03-23T08:23:49.686570+0000",
"last_fullsized": "2024-03-23T08:23:49.686570+0000",
"mapping_epoch": 2261102,
"log_start": "2257907'162208",
"ondisk_log_start": "2257907'162208",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1257,
"log_dups_size": 3000,
"ondisk_log_size": 1257,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92981574958,
"num_objects": 78723,
"num_object_clones": 0,
"num_object_copies": 708507,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 629784,
"num_objects_unfound": 0,
"num_objects_dirty": 78723,
"num_whiteouts": 0,
"num_read": 78736,
"num_read_kb": 112024442,
"num_write": 228641,
"num_write_kb": 109034621,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [
"183(0)",
"30(6)",
"33(7)",
"56(8)",
"256(5)",
"265(2)",
"293(1)",
"424(4)",
"436(3)"
],
"object_location_counts": [
{
"shards": "30(6),33(7),56(8),183(0),256(5),265(2),293(1),424(4),436(3)",
"objects": 78723
}
],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "265(8)",
"pgid": "37.1s8",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "293(1)",
"pgid": "37.1s1",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'163465",
"reported_seq": 713804,
"reported_epoch": 2260315,
"state": "remapped+peering",
"last_fresh": "2024-03-23T09:00:41.322076+0000",
"last_change": "2024-03-23T09:00:41.322076+0000",
"last_active": "2024-03-23T09:00:41.322025+0000",
"last_peered": "2024-03-23T09:00:39.236240+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T08:51:11.086602+0000",
"last_became_peered": "2024-03-23T08:51:11.086602+0000",
"last_unstale": "2024-03-23T09:00:41.322076+0000",
"last_undegraded": "2024-03-23T09:00:41.322076+0000",
"last_fullsized": "2024-03-23T09:00:41.322076+0000",
"mapping_epoch": 2261102,
"log_start": "2257907'162208",
"ondisk_log_start": "2257907'162208",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1257,
"log_dups_size": 3000,
"ondisk_log_size": 1257,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92981574958,
"num_objects": 78723,
"num_object_clones": 0,
"num_object_copies": 708507,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 78723,
"num_whiteouts": 0,
"num_read": 78753,
"num_read_kb": 112026786,
"num_write": 228641,
"num_write_kb": 109034621,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
30,
33,
56,
78,
183,
256,
265,
293,
424,
436,
463,
491
],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "293(7)",
"pgid": "37.1s7",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "424(4)",
"pgid": "37.1s4",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'163465",
"reported_seq": 713683,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:49.686570+0000",
"last_change": "2024-03-23T08:03:18.391295+0000",
"last_active": "2024-03-23T08:23:49.686570+0000",
"last_peered": "2024-03-23T08:23:49.686570+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T08:03:18.390854+0000",
"last_became_peered": "2024-03-23T08:03:18.390854+0000",
"last_unstale": "2024-03-23T08:23:49.686570+0000",
"last_undegraded": "2024-03-23T08:23:49.686570+0000",
"last_fullsized": "2024-03-23T08:23:49.686570+0000",
"mapping_epoch": 2261102,
"log_start": "2257907'162208",
"ondisk_log_start": "2257907'162208",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1257,
"log_dups_size": 3000,
"ondisk_log_size": 1257,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92981574958,
"num_objects": 78723,
"num_object_clones": 0,
"num_object_copies": 708507,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 629784,
"num_objects_unfound": 0,
"num_objects_dirty": 78723,
"num_whiteouts": 0,
"num_read": 78736,
"num_read_kb": 112024442,
"num_write": 228641,
"num_write_kb": 109034621,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [
"183(0)",
"30(6)",
"33(7)",
"56(8)",
"256(5)",
"265(2)",
"293(1)",
"424(4)",
"436(3)"
],
"object_location_counts": [
{
"shards": "30(6),33(7),56(8),183(0),256(5),265(2),293(1),424(4),436(3)",
"objects": 78723
}
],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "436(0)",
"pgid": "37.1s0",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249712,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "436(3)",
"pgid": "37.1s3",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'163465",
"reported_seq": 713683,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:49.686570+0000",
"last_change": "2024-03-23T08:03:18.391295+0000",
"last_active": "2024-03-23T08:23:49.686570+0000",
"last_peered": "2024-03-23T08:23:49.686570+0000",
"last_clean": "2024-03-19T13:01:08.581589+0000",
"last_became_active": "2024-03-23T08:03:18.390854+0000",
"last_became_peered": "2024-03-23T08:03:18.390854+0000",
"last_unstale": "2024-03-23T08:23:49.686570+0000",
"last_undegraded": "2024-03-23T08:23:49.686570+0000",
"last_fullsized": "2024-03-23T08:23:49.686570+0000",
"mapping_epoch": 2261102,
"log_start": "2257907'162208",
"ondisk_log_start": "2257907'162208",
"created": 2139449,
"last_epoch_clean": 2249011,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"objects_scrubbed": 78567,
"log_size": 1257,
"log_dups_size": 3000,
"ondisk_log_size": 1257,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 14,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 13.170252527000001,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92981574958,
"num_objects": 78723,
"num_object_clones": 0,
"num_object_copies": 708507,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 629784,
"num_objects_unfound": 0,
"num_objects_dirty": 78723,
"num_whiteouts": 0,
"num_read": 78736,
"num_read_kb": 112024442,
"num_write": 228641,
"num_write_kb": 109034621,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 34565,
"num_bytes_recovered": 65940324388,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [
"183(0)",
"30(6)",
"33(7)",
"56(8)",
"256(5)",
"265(2)",
"293(1)",
"424(4)",
"436(3)"
],
"object_location_counts": [
{
"shards": "30(6),33(7),56(8),183(0),256(5),265(2),293(1),424(4),436(3)",
"objects": 78723
}
],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "436(6)",
"pgid": "37.1s6",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249821,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "463(0)",
"pgid": "37.1s0",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "463(3)",
"pgid": "37.1s3",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249821,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "471(1)",
"pgid": "37.1s1",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249712,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "471(4)",
"pgid": "37.1s4",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 156942,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249054,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "491(1)",
"pgid": "37.1s1",
"last_update": "2261507'163591",
"last_complete": "2261507'163591",
"log_tail": "2257907'162208",
"last_user_version": 163465,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261103,
"last_interval_started": 2261102,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261103,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "491(4)",
"pgid": "37.1s4",
"last_update": "2249296'156942",
"last_complete": "2249296'156942",
"log_tail": "2242483'155558",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139449,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259141,
"last_interval_started": 2259140,
"last_epoch_clean": 2249011,
"last_interval_clean": 2249010,
"last_epoch_split": 2139452,
"last_epoch_marked_full": 0,
"same_up_since": 2260999,
"same_interval_since": 2261102,
"same_primary_since": 2260316,
"last_scrub": "2235826'154521",
"last_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"last_deep_scrub": "2198936'124827",
"last_deep_scrub_stamp": "2024-02-16T04:20:58.564794+0000",
"last_clean_scrub_stamp": "2024-03-12T18:26:58.560460+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261102,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78813,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
463,
491,
256,
436,
78,
56,
183,
293,
265
],
"acting": [
183,
293,
265,
436,
424,
256,
30,
33,
56
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 463,
"acting_primary": 183,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249821,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2024-03-23T13:21:35.135388+0000",
"might_have_unfound": [],
"recovery_progress": {
"backfill_targets": [
"56(5)",
"78(4)",
"183(6)",
"256(2)",
"265(8)",
"293(7)",
"463(0)",
"491(1)"
],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2024-03-23T13:21:33.475653+0000"
}
],
"scrubber": {
"active": false,
"must_scrub": false,
"must_deep_scrub": false,
"must_repair": false,
"need_auto": false,
"scrub_reg_stamp": "2024-03-13T18:26:58.560460+0000",
"schedule": "queued for deep scrub"
},
"agent_state": {}
}
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+remapped+backfill_wait",
"epoch": 2261510,
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"backfill_targets": [
"81(3)",
"149(1)",
"203(7)",
"221(8)",
"288(6)",
"441(4)",
"474(0)",
"524(5)"
],
"acting_recovery_backfill": [
"9(8)",
"51(2)",
"54(7)",
"81(3)",
"81(6)",
"149(1)",
"203(4)",
"203(7)",
"221(5)",
"221(8)",
"288(3)",
"288(6)",
"408(0)",
"441(1)",
"441(4)",
"474(0)",
"524(5)"
],
"info": {
"pgid": "37.7dcs0",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258312'159756",
"last_user_version": 160457,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2261510'160457",
"reported_seq": 570807,
"reported_epoch": 2261510,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T20:34:09.066718+0000",
"last_change": "2024-03-23T13:33:55.254328+0000",
"last_active": "2024-03-23T20:34:09.066718+0000",
"last_peered": "2024-03-23T20:34:09.066718+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T13:33:55.254025+0000",
"last_became_peered": "2024-03-23T13:33:55.254025+0000",
"last_unstale": "2024-03-23T20:34:09.066718+0000",
"last_undegraded": "2024-03-23T20:34:09.066718+0000",
"last_fullsized": "2024-03-23T20:34:09.066718+0000",
"mapping_epoch": 2261149,
"log_start": "2258312'159756",
"ondisk_log_start": "2258312'159756",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 701,
"log_dups_size": 3000,
"ondisk_log_size": 701,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92853372351,
"num_objects": 78103,
"num_object_clones": 0,
"num_object_copies": 702927,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 623322,
"num_objects_unfound": 0,
"num_objects_dirty": 78103,
"num_whiteouts": 0,
"num_read": 81761,
"num_read_kb": 111785098,
"num_write": 222131,
"num_write_kb": 108212113,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"408(0)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)",
"441(1)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),408(0),441(1)",
"objects": 78103
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "9(8)",
"pgid": "37.7dcs8",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'160305",
"reported_seq": 567018,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:41.335971+0000",
"last_change": "2024-03-23T08:03:34.887737+0000",
"last_active": "2024-03-23T08:23:41.335971+0000",
"last_peered": "2024-03-23T08:23:41.335971+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T08:03:34.887640+0000",
"last_became_peered": "2024-03-23T08:03:34.887640+0000",
"last_unstale": "2024-03-23T08:23:41.335971+0000",
"last_undegraded": "2024-03-23T08:23:41.335971+0000",
"last_fullsized": "2024-03-23T08:23:41.335971+0000",
"mapping_epoch": 2261149,
"log_start": "2258026'159556",
"ondisk_log_start": "2258026'159556",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 749,
"log_dups_size": 3000,
"ondisk_log_size": 749,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92400387519,
"num_objects": 77995,
"num_object_clones": 0,
"num_object_copies": 701955,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 622466,
"num_objects_unfound": 0,
"num_objects_dirty": 77995,
"num_whiteouts": 0,
"num_read": 80501,
"num_read_kb": 111657864,
"num_write": 221979,
"num_write_kb": 107769745,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"408(0)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)",
"441(1)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),408(0),441(1)",
"objects": 77995
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "51(2)",
"pgid": "37.7dcs2",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'160305",
"reported_seq": 567018,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:41.335971+0000",
"last_change": "2024-03-23T08:03:34.887737+0000",
"last_active": "2024-03-23T08:23:41.335971+0000",
"last_peered": "2024-03-23T08:23:41.335971+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T08:03:34.887640+0000",
"last_became_peered": "2024-03-23T08:03:34.887640+0000",
"last_unstale": "2024-03-23T08:23:41.335971+0000",
"last_undegraded": "2024-03-23T08:23:41.335971+0000",
"last_fullsized": "2024-03-23T08:23:41.335971+0000",
"mapping_epoch": 2261149,
"log_start": "2258026'159556",
"ondisk_log_start": "2258026'159556",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 749,
"log_dups_size": 3000,
"ondisk_log_size": 749,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92400387519,
"num_objects": 77995,
"num_object_clones": 0,
"num_object_copies": 701955,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 622466,
"num_objects_unfound": 0,
"num_objects_dirty": 77995,
"num_whiteouts": 0,
"num_read": 80501,
"num_read_kb": 111657864,
"num_write": 221979,
"num_write_kb": 107769745,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"408(0)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)",
"441(1)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),408(0),441(1)",
"objects": 77995
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "54(7)",
"pgid": "37.7dcs7",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'160305",
"reported_seq": 567018,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:41.335971+0000",
"last_change": "2024-03-23T08:03:34.887737+0000",
"last_active": "2024-03-23T08:23:41.335971+0000",
"last_peered": "2024-03-23T08:23:41.335971+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T08:03:34.887640+0000",
"last_became_peered": "2024-03-23T08:03:34.887640+0000",
"last_unstale": "2024-03-23T08:23:41.335971+0000",
"last_undegraded": "2024-03-23T08:23:41.335971+0000",
"last_fullsized": "2024-03-23T08:23:41.335971+0000",
"mapping_epoch": 2261149,
"log_start": "2258026'159556",
"ondisk_log_start": "2258026'159556",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 749,
"log_dups_size": 3000,
"ondisk_log_size": 749,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92400387519,
"num_objects": 77995,
"num_object_clones": 0,
"num_object_copies": 701955,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 622466,
"num_objects_unfound": 0,
"num_objects_dirty": 77995,
"num_whiteouts": 0,
"num_read": 80501,
"num_read_kb": 111657864,
"num_write": 221979,
"num_write_kb": 107769745,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"408(0)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)",
"441(1)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),408(0),441(1)",
"objects": 77995
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "58(8)",
"pgid": "37.7dcs8",
"last_update": "2252592'155929",
"last_complete": "2252592'155929",
"log_tail": "2252130'155256",
"last_user_version": 155929,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259550,
"last_interval_started": 2259549,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78103,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249821,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "81(3)",
"pgid": "37.7dcs3",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "37:3be0144d:::100002efcf4.00000000:head",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 234651648,
"num_objects": 194,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 77909,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 194,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 4,
"num_write_kb": 8192,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "81(6)",
"pgid": "37.7dcs6",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'160305",
"reported_seq": 567018,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:41.335971+0000",
"last_change": "2024-03-23T08:03:34.887737+0000",
"last_active": "2024-03-23T08:23:41.335971+0000",
"last_peered": "2024-03-23T08:23:41.335971+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T08:03:34.887640+0000",
"last_became_peered": "2024-03-23T08:03:34.887640+0000",
"last_unstale": "2024-03-23T08:23:41.335971+0000",
"last_undegraded": "2024-03-23T08:23:41.335971+0000",
"last_fullsized": "2024-03-23T08:23:41.335971+0000",
"mapping_epoch": 2261149,
"log_start": "2258026'159556",
"ondisk_log_start": "2258026'159556",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 749,
"log_dups_size": 3000,
"ondisk_log_size": 749,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92400387519,
"num_objects": 77995,
"num_object_clones": 0,
"num_object_copies": 701955,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 622466,
"num_objects_unfound": 0,
"num_objects_dirty": 77995,
"num_whiteouts": 0,
"num_read": 80501,
"num_read_kb": 111657864,
"num_write": 221979,
"num_write_kb": 107769745,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"408(0)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)",
"441(1)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),408(0),441(1)",
"objects": 77995
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "149(1)",
"pgid": "37.7dcs1",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "37:3be0144d:::100002efcf4.00000000:head",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 234651648,
"num_objects": 194,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 77909,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 194,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 4,
"num_write_kb": 8192,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "203(4)",
"pgid": "37.7dcs4",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'160305",
"reported_seq": 567018,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:41.335971+0000",
"last_change": "2024-03-23T08:03:34.887737+0000",
"last_active": "2024-03-23T08:23:41.335971+0000",
"last_peered": "2024-03-23T08:23:41.335971+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T08:03:34.887640+0000",
"last_became_peered": "2024-03-23T08:03:34.887640+0000",
"last_unstale": "2024-03-23T08:23:41.335971+0000",
"last_undegraded": "2024-03-23T08:23:41.335971+0000",
"last_fullsized": "2024-03-23T08:23:41.335971+0000",
"mapping_epoch": 2261149,
"log_start": "2258026'159556",
"ondisk_log_start": "2258026'159556",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 749,
"log_dups_size": 3000,
"ondisk_log_size": 749,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92400387519,
"num_objects": 77995,
"num_object_clones": 0,
"num_object_copies": 701955,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 622466,
"num_objects_unfound": 0,
"num_objects_dirty": 77995,
"num_whiteouts": 0,
"num_read": 80501,
"num_read_kb": 111657864,
"num_write": 221979,
"num_write_kb": 107769745,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"408(0)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)",
"441(1)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),408(0),441(1)",
"objects": 77995
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "203(7)",
"pgid": "37.7dcs7",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "37:3be0144d:::100002efcf4.00000000:head",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 234651648,
"num_objects": 194,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 77909,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 194,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 4,
"num_write_kb": 8192,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "221(5)",
"pgid": "37.7dcs5",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'160305",
"reported_seq": 567018,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:41.335971+0000",
"last_change": "2024-03-23T08:03:34.887737+0000",
"last_active": "2024-03-23T08:23:41.335971+0000",
"last_peered": "2024-03-23T08:23:41.335971+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T08:03:34.887640+0000",
"last_became_peered": "2024-03-23T08:03:34.887640+0000",
"last_unstale": "2024-03-23T08:23:41.335971+0000",
"last_undegraded": "2024-03-23T08:23:41.335971+0000",
"last_fullsized": "2024-03-23T08:23:41.335971+0000",
"mapping_epoch": 2261149,
"log_start": "2258026'159556",
"ondisk_log_start": "2258026'159556",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 749,
"log_dups_size": 3000,
"ondisk_log_size": 749,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92400387519,
"num_objects": 77995,
"num_object_clones": 0,
"num_object_copies": 701955,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 622466,
"num_objects_unfound": 0,
"num_objects_dirty": 77995,
"num_whiteouts": 0,
"num_read": 80501,
"num_read_kb": 111657864,
"num_write": 221979,
"num_write_kb": 107769745,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"408(0)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)",
"441(1)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),408(0),441(1)",
"objects": 77995
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "221(8)",
"pgid": "37.7dcs8",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "37:3be0144d:::100002efcf4.00000000:head",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 234651648,
"num_objects": 194,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 77909,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 194,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 4,
"num_write_kb": 8192,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "288(3)",
"pgid": "37.7dcs3",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'160305",
"reported_seq": 567018,
"reported_epoch": 2260182,
"state": "active+remapped+backfill_wait",
"last_fresh": "2024-03-23T08:23:41.335971+0000",
"last_change": "2024-03-23T08:03:34.887737+0000",
"last_active": "2024-03-23T08:23:41.335971+0000",
"last_peered": "2024-03-23T08:23:41.335971+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T08:03:34.887640+0000",
"last_became_peered": "2024-03-23T08:03:34.887640+0000",
"last_unstale": "2024-03-23T08:23:41.335971+0000",
"last_undegraded": "2024-03-23T08:23:41.335971+0000",
"last_fullsized": "2024-03-23T08:23:41.335971+0000",
"mapping_epoch": 2261149,
"log_start": "2258026'159556",
"ondisk_log_start": "2258026'159556",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 749,
"log_dups_size": 3000,
"ondisk_log_size": 749,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92400387519,
"num_objects": 77995,
"num_object_clones": 0,
"num_object_copies": 701955,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 622466,
"num_objects_unfound": 0,
"num_objects_dirty": 77995,
"num_whiteouts": 0,
"num_read": 80501,
"num_read_kb": 111657864,
"num_write": 221979,
"num_write_kb": 107769745,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"408(0)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)",
"441(1)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),408(0),441(1)",
"objects": 77995
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "288(6)",
"pgid": "37.7dcs6",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "37:3be0144d:::100002efcf4.00000000:head",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 234651648,
"num_objects": 194,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 77909,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 194,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 4,
"num_write_kb": 8192,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "429(2)",
"pgid": "37.7dcs2",
"last_update": "2249473'154298",
"last_complete": "2249473'154298",
"log_tail": "2246259'153487",
"last_user_version": 154298,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261112,
"last_interval_started": 2261111,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 12.283888945999999
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78103,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2248955,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "441(1)",
"pgid": "37.7dcs1",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2260182'160305",
"reported_seq": 567142,
"reported_epoch": 2261072,
"state": "active+undersized+degraded+remapped+backfill_wait",
"last_fresh": "2024-03-23T13:20:16.597975+0000",
"last_change": "2024-03-23T13:07:19.906293+0000",
"last_active": "2024-03-23T13:20:16.597975+0000",
"last_peered": "2024-03-23T13:20:16.597975+0000",
"last_clean": "2024-03-19T12:57:39.093899+0000",
"last_became_active": "2024-03-23T13:07:19.906163+0000",
"last_became_peered": "2024-03-23T13:07:19.906163+0000",
"last_unstale": "2024-03-23T13:20:16.597975+0000",
"last_undegraded": "2024-03-23T13:07:19.698384+0000",
"last_fullsized": "2024-03-23T13:07:19.698249+0000",
"mapping_epoch": 2261149,
"log_start": "2258026'159556",
"ondisk_log_start": "2258026'159556",
"created": 2139486,
"last_epoch_clean": 2238064,
"parent": "0.0",
"parent_split_bits": 11,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"objects_scrubbed": 77852,
"log_size": 749,
"log_dups_size": 3000,
"ondisk_log_size": 749,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 13,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 12.784428346,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 92400387519,
"num_objects": 77995,
"num_object_clones": 0,
"num_object_copies": 701955,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 77802,
"num_objects_misplaced": 544664,
"num_objects_unfound": 0,
"num_objects_dirty": 77995,
"num_whiteouts": 0,
"num_read": 80540,
"num_read_kb": 111660510,
"num_write": 221979,
"num_write_kb": 107769745,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 26566,
"num_bytes_recovered": 52884763428,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [
"441(1)",
"9(8)",
"51(2)",
"54(7)",
"81(6)",
"203(4)",
"221(5)",
"288(3)"
],
"object_location_counts": [
{
"shards": "9(8),51(2),54(7),81(6),203(4),221(5),288(3),441(1)",
"objects": 77995
}
],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "441(4)",
"pgid": "37.7dcs4",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "37:3be00f4c:::10006d2cddd.00000000:head",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 158138368,
"num_objects": 144,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 77959,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 144,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 3,
"num_write_kb": 8192,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "441(7)",
"pgid": "37.7dcs7",
"last_update": "2252592'155929",
"last_complete": "2252592'155929",
"log_tail": "2252130'155256",
"last_user_version": 155929,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2259550,
"last_interval_started": 2259549,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 78103,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2249821,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "474(0)",
"pgid": "37.7dcs0",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "37:3be0144d:::100002efcf4.00000000:head",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 234651648,
"num_objects": 194,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 77909,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 194,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 4,
"num_write_kb": 8192,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "524(5)",
"pgid": "37.7dcs5",
"last_update": "2261510'160457",
"last_complete": "2261510'160457",
"log_tail": "2258026'159556",
"last_user_version": 160305,
"last_backfill": "37:3be0144d:::100002efcf4.00000000:head",
"purged_snaps": [],
"history": {
"epoch_created": 2139486,
"epoch_pool_created": 2139449,
"last_epoch_started": 2261150,
"last_interval_started": 2261149,
"last_epoch_clean": 2238064,
"last_interval_clean": 2238063,
"last_epoch_split": 2139486,
"last_epoch_marked_full": 0,
"same_up_since": 2261148,
"same_interval_since": 2261149,
"same_primary_since": 2261073,
"last_scrub": "2236051'152185",
"last_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"last_deep_scrub": "2192897'113065",
"last_deep_scrub_stamp": "2024-02-12T05:54:48.898543+0000",
"last_clean_scrub_stamp": "2024-03-13T02:41:17.440734+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 2261149,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 234651648,
"num_objects": 194,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 77909,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 194,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 4,
"num_write_kb": 8192,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
474,
149,
51,
81,
441,
524,
288,
203,
221
],
"acting": [
408,
441,
51,
288,
203,
221,
81,
54,
9
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 474,
"acting_primary": 408,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 2261150,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2024-03-23T13:33:55.142083+0000",
"might_have_unfound": [],
"recovery_progress": {
"backfill_targets": [
"81(3)",
"149(1)",
"203(7)",
"221(8)",
"288(6)",
"441(4)",
"474(0)",
"524(5)"
],
"waiting_on_backfill": [],
"last_backfill_started": "37:3be00f4c:::10006d2cddd.00000000:head",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2024-03-23T13:33:53.532161+0000"
}
],
"scrubber": {
"active": false,
"must_scrub": false,
"must_deep_scrub": false,
"must_repair": false,
"need_auto": false,
"scrub_reg_stamp": "2024-03-14T02:41:17.440734+0000",
"schedule": "queued for deep scrub"
},
"agent_state": {}
}
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
