On 24/03/2024 01:14, Torkil Svensgaard wrote:
On 24-03-2024 00:31, Alexander E. Patrakov wrote:
Hi Torkil,
Hi Alexander
Thanks for the update. Even though the improvement is small, it is
still an improvement, consistent with the osd_max_backfills value, and
it proves that there are still unsolved peering issues.
I have looked at both the old and the new state of the PG, but could
not find anything else interesting.
I also looked again at the state of PG 37.1. It is known what blocks
the backfill of this PG; please search for "blocked_by." However, this
is just one data point, which is insufficient for any conclusions. Try
looking at other PGs. Is there anything too common in the non-empty
"blocked_by" blocks?
I'll take a look at that tomorrow, perhaps we can script something
meaningful.
Hi Alexander
While working on a script querying all PGs and making a list of all OSDs
found in a blocked_by list, and how many times for each, I discovered
something odd about pool 38:
"
[root@lazy blocked_by]# sh blocked_by.sh 38 |tee pool38
OSDs blocking other OSDs:
OSD 425: 5 instance(s)
OSD 426: 6 instance(s)
OSD 34: 7 instance(s)
OSD 36: 5 instance(s)
OSD 146: 3 instance(s)
OSD 6: 2 instance(s)
OSD 5: 8 instance(s)
OSD 131: 7 instance(s)
OSD 4: 9 instance(s)
OSD 3: 5 instance(s)
OSD 2: 5 instance(s)
OSD 1: 2 instance(s)
OSD 0: 4 instance(s)
OSD 167: 1 instance(s)
OSD 168: 3 instance(s)
OSD 450: 2 instance(s)
OSD 46: 6 instance(s)
OSD 154: 3 instance(s)
OSD 156: 2 instance(s)
OSD 90: 2 instance(s)
OSD 227: 4 instance(s)
OSD 10: 4 instance(s)
OSD 15: 6 instance(s)
OSD 449: 4 instance(s)
OSD 192: 2 instance(s)
OSD 67: 3 instance(s)
"
All PGs in the pool are active+clean so why are there any blocked_by at
all? One example attached.
Mvh.
Torkil
I think we have to look for patterns in other ways, too. One tool that
produces good visualizations is TheJJ balancer. Although it is called
a "balancer," it can also visualize the ongoing backfills.
The tool is available at
https://raw.githubusercontent.com/TheJJ/ceph-balancer/master/placementoptimizer.py
Run it as follows:
./placementoptimizer.py showremapped --by-osd | tee remapped.txt
Output attached.
Thanks again.
Mvh.
Torkil
On Sun, Mar 24, 2024 at 5:50 AM Torkil Svensgaard <torkil@xxxxxxxx>
wrote:
Hi Alex
New query output attached after restarting both OSDs. OSD 237 is no
longer mentioned but it unfortunately made no difference for the number
of backfills which went 59->62->62.
Mvh.
Torkil
On 23-03-2024 22:26, Alexander E. Patrakov wrote:
Hi Torkil,
I have looked at the files that you attached. They were helpful: pool
11 is problematic, it complains about degraded objects for no obvious
reason. I think that is the blocker.
I also noted that you mentioned peering problems, and I suspect that
they are not completely resolved. As a somewhat-irrational move, to
confirm this theory, you can restart osd.237 (it is mentioned at the
end of query.11.fff.txt, although I don't understand why it is there)
and then osd.298 (it is the primary for that pg) and see if any
additional backfills are unblocked after that. Also, please re-query
that PG again after the OSD restart.
On Sun, Mar 24, 2024 at 4:56 AM Torkil Svensgaard <torkil@xxxxxxxx>
wrote:
On 23-03-2024 21:19, Alexander E. Patrakov wrote:
Hi Torkil,
Hi Alexander
I have looked at the CRUSH rules, and the equivalent rules work on my
test cluster. So this cannot be the cause of the blockage.
Thank you for taking the time =)
What happens if you increase the osd_max_backfills setting
temporarily?
We already had the mclock override option in place and I re-enabled
our
babysitter script which sets osd_max_backfills pr OSD to 1-3 depending
on how full they are. Active backfills went from 16 to 53 which is
probably because default osd_max_backfills for mclock is 1.
I think 53 is still a low number of active backfills given the large
percentage misplaced.
It may be a good idea to investigate a few of the stalled PGs. Please
run commands similar to this one:
ceph pg 37.0 query > query.37.0.txt
ceph pg 37.1 query > query.37.1.txt
...
and the same for the other affected pools.
A few samples attached.
Still, I must say that some of your rules are actually unsafe.
The 4+2 rule as used by rbd_ec_data will not survive a
datacenter-offline incident. Namely, for each PG, it chooses OSDs
from
two hosts in each datacenter, so 6 OSDs total. When a datacenter is
offline, you will, therefore, have only 4 OSDs up, which is exactly
the number of data chunks. However, the pool requires min_size 5, so
all PGs will be inactive (to prevent data corruption) and will stay
inactive until the datacenter comes up again. However, please don't
set min_size to 4 - then, any additional incident (like a defective
disk) will lead to data loss, and the shards in the datacenter which
went offline would be useless because they do not correspond to the
updated shards written by the clients.
Thanks for the explanation. This is an old pool predating the 3 DC
setup
and we'll migrate the data to a 4+5 pool when we can.
The 4+5 rule as used by cephfs.hdd.data has min_size equal to the
number of data chunks. See above why it is bad. Please set
min_size to
5.
Thanks, that was a leftover for getting the PGs to peer (stuck at
creating+incomplete) when we created the pool. It's back to 5 now.
The rbd.ssd.data pool seems to be OK - and, by the way, its PGs are
100% active+clean.
There is very little data in this pool, that is probably the main
reason.
Regarding the mon_max_pg_per_osd setting, you have a few OSDs that
have 300+ PGs, the observed maximum is 347. Please set it to 400.
Copy that. Didn't seem to make a difference though, and we have
osd_max_pg_per_osd_hard_ratio set to 5.000000.
Mvh.
Torkil
On Sun, Mar 24, 2024 at 3:16 AM Torkil Svensgaard
<torkil@xxxxxxxx> wrote:
On 23-03-2024 19:05, Alexander E. Patrakov wrote:
Sorry for replying to myself, but "ceph osd pool ls detail" by
itself
is insufficient. For every erasure code profile mentioned in the
output, please also run something like this:
ceph osd erasure-code-profile get prf-for-ec-data
...where "prf-for-ec-data" is the name that appears after the words
"erasure profile" in the "ceph osd pool ls detail" output.
[root@lazy ~]# ceph osd pool ls detail | grep erasure
pool 11 'rbd_ec_data' erasure profile DRCMR_k4m2 size 6 min_size 5
crush_rule 0 object_hash rjenkins pg_num 4096 pgp_num 4096
autoscale_mode off last_change 2257933 lfor 0/1291190/1755101 flags
hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 16384
fast_read 1 compression_algorithm snappy compression_mode aggressive
application rbd
pool 37 'cephfs.hdd.data' erasure profile
DRCMR_k4m5_datacenter_hdd size
9 min_size 4 crush_rule 7 object_hash rjenkins pg_num 2048
pgp_num 2048
autoscale_mode off last_change 2257933 lfor 0/0/2139486 flags
hashpspool,ec_overwrites,bulk stripe_width 16384 fast_read 1
compression_algorithm zstd compression_mode aggressive
application cephfs
pool 38 'rbd.ssd.data' erasure profile DRCMR_k4m5_datacenter_ssd
size 9
min_size 5 crush_rule 8 object_hash rjenkins pg_num 32 pgp_num 32
autoscale_mode warn last_change 2198930 lfor 0/2198930/2198928 flags
hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 16384
compression_algorithm zstd compression_mode aggressive
application rbd
[root@lazy ~]# ceph osd erasure-code-profile get DRCMR_k4m2
crush-device-class=hdd
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=2
plugin=jerasure
technique=reed_sol_van
w=8
[root@lazy ~]# ceph osd erasure-code-profile get
DRCMR_k4m5_datacenter_hdd
crush-device-class=hdd
crush-failure-domain=datacenter
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=5
plugin=jerasure
technique=reed_sol_van
w=8
[root@lazy ~]# ceph osd erasure-code-profile get
DRCMR_k4m5_datacenter_ssd
crush-device-class=ssd
crush-failure-domain=datacenter
crush-root=default
jerasure-per-chunk-alignment=false
k=4
m=5
plugin=jerasure
technique=reed_sol_van
w=8
But as I understand it those profiles are only used to create the
initial crush rule for the pool, and we have manually edited
those along
the way. Here are the 3 rules in use for the 3 EC pools:
rule rbd_ec_data {
id 0
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 0 type datacenter
step chooseleaf indep 2 type host
step emit
}
rule cephfs.hdd.data {
id 7
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 0 type datacenter
step chooseleaf indep 3 type host
step emit
}
rule rbd.ssd.data {
id 8
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class ssd
step choose indep 0 type datacenter
step chooseleaf indep 3 type host
step emit
}
Which should first pick all 3 datacenters in the choose step and
then
either 2 or 3 hosts in the chooseleaf step, matching EC 4+2 and 4+5
respectively.
Mvh.
Torkil
On Sun, Mar 24, 2024 at 1:56 AM Alexander E. Patrakov
<patrakov@xxxxxxxxx> wrote:
Hi Torkil,
I take my previous response back.
You have an erasure-coded pool with nine shards but only three
datacenters. This, in general, cannot work. You need either nine
datacenters or a very custom CRUSH rule. The second option may
not be
available if the current EC setup is already incompatible, as
there is
no way to change the EC parameters.
It would help if you provided the output of "ceph osd pool ls
detail".
On Sun, Mar 24, 2024 at 1:43 AM Alexander E. Patrakov
<patrakov@xxxxxxxxx> wrote:
Hi Torkil,
Unfortunately, your files contain nothing obviously bad or
suspicious,
except for two things: more PGs than usual and bad balance.
What's your "mon max pg per osd" setting?
On Sun, Mar 24, 2024 at 1:08 AM Torkil Svensgaard
<torkil@xxxxxxxx> wrote:
On 2024-03-23 17:54, Kai Stian Olstad wrote:
On Sat, Mar 23, 2024 at 12:09:29PM +0100, Torkil Svensgaard
wrote:
The other output is too big for pastebin and I'm not
familiar with
paste services, any suggestion for a preferred way to share
such
output?
You can attached files to the mail here on the list.
Doh, for some reason I was sure attachments would be
stripped. Thanks,
attached.
Mvh.
Torkil
--
Alexander E. Patrakov
--
Alexander E. Patrakov
--
Torkil Svensgaard
Systems Administrator
Danish Research Centre for Magnetic Resonance DRCMR, Section 714
Copenhagen University Hospital Amager and Hvidovre
Kettegaard Allé 30, 2650 Hvidovre, Denmark
--
Torkil Svensgaard
Systems Administrator
Danish Research Centre for Magnetic Resonance DRCMR, Section 714
Copenhagen University Hospital Amager and Hvidovre
Kettegaard Allé 30, 2650 Hvidovre, Denmark
--
Torkil Svensgaard
Systems Administrator
Danish Research Centre for Magnetic Resonance DRCMR, Section 714
Copenhagen University Hospital Amager and Hvidovre
Kettegaard Allé 30, 2650 Hvidovre, Denmark
--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: torkil@xxxxxxxx
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+clean",
"epoch": 2266023,
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting_recovery_backfill": [
"0(5)",
"4(3)",
"89(2)",
"168(1)",
"192(0)",
"425(4)",
"450(8)",
"457(7)",
"498(6)"
],
"info": {
"pgid": "38.4s0",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 110601,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 289821,
"reported_epoch": 2266023,
"state": "active+clean",
"last_fresh": "2024-03-25T11:27:37.838184+0000",
"last_change": "2024-03-24T22:20:31.452436+0000",
"last_active": "2024-03-25T11:27:37.838184+0000",
"last_peered": "2024-03-25T11:27:37.838184+0000",
"last_clean": "2024-03-25T11:27:37.838184+0000",
"last_became_active": "2024-03-23T13:14:16.547780+0000",
"last_became_peered": "2024-03-23T13:14:16.547780+0000",
"last_unstale": "2024-03-25T11:27:37.838184+0000",
"last_undegraded": "2024-03-25T11:27:37.838184+0000",
"last_fullsized": "2024-03-25T11:27:37.838184+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2261041,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "periodic scrub scheduled @ 2024-03-25T22:20:31.452316+0000",
"scrub_duration": 0.0040474739999999997,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "0(5)",
"pgid": "38.4s5",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 270784,
"reported_epoch": 2251492,
"state": "active+remapped+backfilling",
"last_fresh": "2024-03-19T20:35:36.985601+0000",
"last_change": "2024-03-19T20:35:36.985601+0000",
"last_active": "2024-03-19T20:35:36.985601+0000",
"last_peered": "2024-03-19T20:35:36.985601+0000",
"last_clean": "2024-03-19T15:48:58.688233+0000",
"last_became_active": "2024-03-19T17:56:18.229388+0000",
"last_became_peered": "2024-03-19T17:56:18.229388+0000",
"last_unstale": "2024-03-19T20:35:36.985601+0000",
"last_undegraded": "2024-03-19T20:35:36.985601+0000",
"last_fullsized": "2024-03-19T20:35:36.985601+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2249429,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "queued for scrub",
"scrub_duration": 0.0021863690000000001,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [
"192(0)",
"0(5)",
"0(8)",
"2(5)",
"4(3)",
"4(6)",
"15(7)",
"89(2)",
"168(1)",
"425(3)",
"425(4)",
"450(8)",
"457(4)",
"457(7)",
"477(6)",
"498(6)"
],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "4(3)",
"pgid": "38.4s3",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 270784,
"reported_epoch": 2251492,
"state": "active+remapped+backfilling",
"last_fresh": "2024-03-19T20:35:36.985601+0000",
"last_change": "2024-03-19T20:35:36.985601+0000",
"last_active": "2024-03-19T20:35:36.985601+0000",
"last_peered": "2024-03-19T20:35:36.985601+0000",
"last_clean": "2024-03-19T15:48:58.688233+0000",
"last_became_active": "2024-03-19T17:56:18.229388+0000",
"last_became_peered": "2024-03-19T17:56:18.229388+0000",
"last_unstale": "2024-03-19T20:35:36.985601+0000",
"last_undegraded": "2024-03-19T20:35:36.985601+0000",
"last_fullsized": "2024-03-19T20:35:36.985601+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2249429,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "queued for scrub",
"scrub_duration": 0.0021863690000000001,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [
"192(0)",
"0(5)",
"0(8)",
"2(5)",
"4(3)",
"4(6)",
"15(7)",
"89(2)",
"168(1)",
"425(3)",
"425(4)",
"450(8)",
"457(4)",
"457(7)",
"477(6)",
"498(6)"
],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "89(2)",
"pgid": "38.4s2",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 110601,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 187323,
"reported_epoch": 2226691,
"state": "peering",
"last_fresh": "2024-03-06T11:12:32.755929+0000",
"last_change": "2024-03-05T12:46:49.264773+0000",
"last_active": "2024-03-05T11:52:47.045641+0000",
"last_peered": "2024-02-19T11:05:21.874574+0000",
"last_clean": "2024-02-14T12:32:32.969009+0000",
"last_became_active": "2024-02-19T11:05:20.378882+0000",
"last_became_peered": "2024-02-19T11:05:20.378882+0000",
"last_unstale": "2024-03-06T11:12:32.755929+0000",
"last_undegraded": "2024-03-06T11:12:32.755929+0000",
"last_fullsized": "2024-03-06T11:12:32.755929+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2226681,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-05T18:53:27.822952+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-05T18:53:27.822952+0000",
"objects_scrubbed": 2909,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "periodic scrub scheduled @ 2024-03-06T18:53:27.822952+0000",
"scrub_duration": 0.42725274499999999,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
0,
4,
15,
46,
425,
449
],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "168(1)",
"pgid": "38.4s1",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 110601,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 187440,
"reported_epoch": 2258355,
"state": "active+undersized",
"last_fresh": "2024-03-22T09:34:02.091481+0000",
"last_change": "2024-03-22T09:26:56.737308+0000",
"last_active": "2024-03-22T09:34:02.091481+0000",
"last_peered": "2024-03-22T09:34:02.091481+0000",
"last_clean": "2024-02-14T12:32:32.969009+0000",
"last_became_active": "2024-03-22T09:26:56.737308+0000",
"last_became_peered": "2024-03-22T09:26:56.737308+0000",
"last_unstale": "2024-03-22T09:34:02.091481+0000",
"last_undegraded": "2024-03-22T09:34:02.091481+0000",
"last_fullsized": "2024-03-22T09:26:56.686043+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2258269,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-21T22:17:25.809681+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-21T22:17:25.809681+0000",
"objects_scrubbed": 2909,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "periodic scrub scheduled @ 2024-03-22T22:17:25.809681+0000",
"scrub_duration": 0.42725274499999999,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [
"168(1)",
"0(5)",
"4(3)",
"89(2)",
"425(4)",
"450(8)",
"457(7)",
"498(6)"
],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "425(4)",
"pgid": "38.4s4",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 270784,
"reported_epoch": 2251492,
"state": "active+remapped+backfilling",
"last_fresh": "2024-03-19T20:35:36.985601+0000",
"last_change": "2024-03-19T20:35:36.985601+0000",
"last_active": "2024-03-19T20:35:36.985601+0000",
"last_peered": "2024-03-19T20:35:36.985601+0000",
"last_clean": "2024-03-19T15:48:58.688233+0000",
"last_became_active": "2024-03-19T17:56:18.229388+0000",
"last_became_peered": "2024-03-19T17:56:18.229388+0000",
"last_unstale": "2024-03-19T20:35:36.985601+0000",
"last_undegraded": "2024-03-19T20:35:36.985601+0000",
"last_fullsized": "2024-03-19T20:35:36.985601+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2249429,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "queued for scrub",
"scrub_duration": 0.0021863690000000001,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [
"192(0)",
"0(5)",
"0(8)",
"2(5)",
"4(3)",
"4(6)",
"15(7)",
"89(2)",
"168(1)",
"425(3)",
"425(4)",
"450(8)",
"457(4)",
"457(7)",
"477(6)",
"498(6)"
],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "450(8)",
"pgid": "38.4s8",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 270784,
"reported_epoch": 2251492,
"state": "active+remapped+backfilling",
"last_fresh": "2024-03-19T20:35:36.985601+0000",
"last_change": "2024-03-19T20:35:36.985601+0000",
"last_active": "2024-03-19T20:35:36.985601+0000",
"last_peered": "2024-03-19T20:35:36.985601+0000",
"last_clean": "2024-03-19T15:48:58.688233+0000",
"last_became_active": "2024-03-19T17:56:18.229388+0000",
"last_became_peered": "2024-03-19T17:56:18.229388+0000",
"last_unstale": "2024-03-19T20:35:36.985601+0000",
"last_undegraded": "2024-03-19T20:35:36.985601+0000",
"last_fullsized": "2024-03-19T20:35:36.985601+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2249429,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "queued for scrub",
"scrub_duration": 0.0021863690000000001,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [
"192(0)",
"0(5)",
"0(8)",
"2(5)",
"4(3)",
"4(6)",
"15(7)",
"89(2)",
"168(1)",
"425(3)",
"425(4)",
"450(8)",
"457(4)",
"457(7)",
"477(6)",
"498(6)"
],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "457(7)",
"pgid": "38.4s7",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 270784,
"reported_epoch": 2251492,
"state": "active+remapped+backfilling",
"last_fresh": "2024-03-19T20:35:36.985601+0000",
"last_change": "2024-03-19T20:35:36.985601+0000",
"last_active": "2024-03-19T20:35:36.985601+0000",
"last_peered": "2024-03-19T20:35:36.985601+0000",
"last_clean": "2024-03-19T15:48:58.688233+0000",
"last_became_active": "2024-03-19T17:56:18.229388+0000",
"last_became_peered": "2024-03-19T17:56:18.229388+0000",
"last_unstale": "2024-03-19T20:35:36.985601+0000",
"last_undegraded": "2024-03-19T20:35:36.985601+0000",
"last_fullsized": "2024-03-19T20:35:36.985601+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2249429,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "queued for scrub",
"scrub_duration": 0.0021863690000000001,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [
"192(0)",
"0(5)",
"0(8)",
"2(5)",
"4(3)",
"4(6)",
"15(7)",
"89(2)",
"168(1)",
"425(3)",
"425(4)",
"450(8)",
"457(4)",
"457(7)",
"477(6)",
"498(6)"
],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "498(6)",
"pgid": "38.4s6",
"last_update": "2194945'113546",
"last_complete": "2194945'113546",
"log_tail": "2194945'113546",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 2174342,
"epoch_pool_created": 2174342,
"last_epoch_started": 2261041,
"last_interval_started": 2261040,
"last_epoch_clean": 2261041,
"last_interval_clean": 2261040,
"last_epoch_split": 2179030,
"last_epoch_marked_full": 0,
"same_up_since": 2261040,
"same_interval_since": 2261040,
"same_primary_since": 2258356,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-24T22:20:31.452316+0000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "2194945'113546",
"reported_seq": 270784,
"reported_epoch": 2251492,
"state": "active+remapped+backfilling",
"last_fresh": "2024-03-19T20:35:36.985601+0000",
"last_change": "2024-03-19T20:35:36.985601+0000",
"last_active": "2024-03-19T20:35:36.985601+0000",
"last_peered": "2024-03-19T20:35:36.985601+0000",
"last_clean": "2024-03-19T15:48:58.688233+0000",
"last_became_active": "2024-03-19T17:56:18.229388+0000",
"last_became_peered": "2024-03-19T17:56:18.229388+0000",
"last_unstale": "2024-03-19T20:35:36.985601+0000",
"last_undegraded": "2024-03-19T20:35:36.985601+0000",
"last_fullsized": "2024-03-19T20:35:36.985601+0000",
"mapping_epoch": 2261040,
"log_start": "2194945'113546",
"ondisk_log_start": "2194945'113546",
"created": 2174342,
"last_epoch_clean": 2249429,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2194945'113546",
"last_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"last_deep_scrub": "2194945'113546",
"last_deep_scrub_stamp": "2024-02-24T17:38:42.610320+0000",
"last_clean_scrub_stamp": "2024-03-18T18:55:52.159744+0000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 7456,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 1,
"scrub_schedule": "queued for scrub",
"scrub_duration": 0.0021863690000000001,
"objects_trimmed": 0,
"snaptrim_duration": 0.00034649600000000001,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 207830,
"num_read_kb": 2762204,
"num_write": 49638,
"num_write_kb": 10483858,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"acting": [
192,
168,
89,
4,
425,
0,
498,
457,
450
],
"avail_no_missing": [
"192(0)",
"0(5)",
"0(8)",
"2(5)",
"4(3)",
"4(6)",
"15(7)",
"89(2)",
"168(1)",
"425(3)",
"425(4)",
"450(8)",
"457(4)",
"457(7)",
"477(6)",
"498(6)"
],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 192,
"acting_primary": 192,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 2261041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2024-03-23T13:14:16.538748+0000",
"might_have_unfound": [],
"recovery_progress": {
"backfill_targets": [],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2024-03-23T13:14:14.921606+0000"
}
],
"scrubber": {
"active": false,
"must_scrub": false,
"must_deep_scrub": false,
"must_repair": false,
"need_auto": false,
"scrub_reg_stamp": "2024-03-25T22:20:31.452316+0000",
"schedule": "scrub scheduled @ 2024-03-25T22:20:31.452"
},
"agent_state": {}
}
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
