Thank you Mike!
This is honestly a way more detailed reply than I was expecting.
You've equipped me with new tools to work with. Thank you!
I don't actually have any unfound pgs... only "incomplete" ones, which
limits the usefulness of:
`grep recovery_unfound`
`ceph pg $pg list_unfound`
`ceph pg $pg mark_unfound_lost delete`
I don't seem to see equivalent commands for incomplete pgs, save for
grep of course.
This does make me slightly more hopeful that recovery might be
possible if the pgs are incomplete and stuck, but not unfound..? Not
going to get my hopes too high.
Going to attach a few items just to keep from bugging me, if anyone
can take a glance, it would be appreciated.
In the meantime, in the absence of the above commands, what's the best
way to clean this up under the assumption that the data is lost?
~Joshua
Joshua West
President
403-456-0072
CAYK.ca
On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas <wart@xxxxxxxxxxx> wrote:
>
> Hi Joshua,
>
> I have had a similar issue three different times on one of my cephfs
> pools (15.2.10). The first time this happened I had lost some OSDs. In
> all cases I ended up with degraded PGs with unfound objects that could
> not be recovered.
>
> Here's how I recovered from the situation. Note that this will
> permanently remove the affected files from ceph. Restoring them from
> backup is an excercise left to the reader.
>
> * Make a list of the affected PGs:
> ceph pg dump_stuck | grep recovery_unfound > pg.txt
>
> * Make a list of the affected objects (OIDs):
> cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg
> $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' >
> oid.txt
>
> * Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and
> put the results in a file called 'inum.txt'
>
> * On a ceph client, find the files that correspond to the affected inodes:
> cat inum.txt | while read inum ; do echo -n "${inum} " ; find
> /ceph/frames/O3/raw -inum ${inum} ; done > files.txt
>
> * It may be helpful to put this table of PG, OID, inum, and files into a
> spreadsheet to keep track of what's been done.
>
> * On the ceph client, use 'unlink' to remove the files from the
> filesystem. Do not use 'rm', as it will hang while calling 'stat()' on
> each file. Even unlink may hang when you first try it. If it does
> hang, do the following to get it unstuck:
> - Reboot the client
> - Restart each mon and the mgr. I rebooted each mon/mgr, but it may
> be sufficient to restart the services without a reboot.
> - Try using 'unlink' again
>
> * After all of the affected files have been removed, go through the list
> of PGs and remove the unfound OIDs:
> ceph pg $pgid mark_unfound_lost delete
>
> ...or if you're feeling brave, delete them all at once:
> cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg
> $pg mark_unfound_lost delete ; done
>
> * Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
>
> * Restore the deleted files from backup, or decide that you don't care
> about them and don't do anything
> This procedure lets you fix the problem without deleting the affected
> pool. To be honest, the first time it happened, my solution was to
> first copy all of the data off of the affected pool and onto a new pool.
> I later found this to be unnecessary. But if you want to pursue this,
> here's what I suggest:
>
> * Follow the steps above to get rid of the affected files. I feel this
> should still be done even though you don't care about saving the data,
> to prevent corruption in the cephfs metadata.
>
> * Go through the entire filesystem and look for:
> - files that are located on the pool (ceph.file.layout.pool = $pool_name)
> - directories that are set to write files to the pool
> (ceph.dir.layout.pool = $pool_name)
>
> * After you confirm that no files or directories are pointing at the
> pool anymore, run 'ceph df' and look at the number of objects in the
> pool. Ideally, it would be zero. But more than likely it isn't. This
> could be a simple mismatch in the object count in cephfs (harmless), or
> there could be clients with open filehandles on files that have been
> removed. such objects will still appear in the rados listing of the
> pool[1]:
> rados -p $pool_name ls
> for obj in $(rados -p $pool_name ls); do echo $obj; rados -p
> $pool_name getxattr parent | strings; done
>
> * To check for clients with access to these stray objects, dump the mds
> cache:
> ceph daemon mds.ceph1 dump cache /tmp/cache.txt
>
> * Look for lines that refer to the stray objects, like this:
> [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537
> s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863
> 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 |
> caps=1 authpin=0 0x563a7e52a000]
>
> * The 'caps' field in the output above contains the client session id
> (eg 9541437). Search the MDS for sessions that match to identify the
> client:
> ceph daemon mds.ceph1 session ls > session.txt
> Search through 'session.txt' for matching entries. This will give
> you the IP address of the client:
> "id": 9541437,
> "entity": {
> "name": {
> "type": "client",
> "num": 9541437
> },
> "addr": {
> "type": "v1",
> "addr": "10.13.5.48:0",
> "nonce": 2011077845
> }
> },
>
> * Restart the client's connection to ceph to get it to drop the cap. I
> did this by rebooting the client, but there may be gentler ways to do it.
>
> * Once you've done this clean up, it should be safe to remove the pool
> from cephfs:
> ceph fs rm_data_pool $fs_name $pool_name
>
> * Once the pool has been detached from cephfs, you can remove it from
> ceph altogether:
> ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it
>
> Hope this helps,
>
> --Mike
> [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html
>
>
>
> On 4/8/21 5:41 PM, Joshua West wrote:
> > Hey everyone.
> >
> > Inside of cephfs, I have a directory which I setup a directory layout
> > field to use an erasure coded (CLAY) pool, specific to the task. The
> > rest of my cephfs is using normal replication.
> >
> > Fast forward some time, and the EC directory has been used pretty
> > extensively, and through some bad luck and poor timing, ~200pgs are in
> > an incomplete state, and the OSDs are completely gone and
> > unrecoverable. (Specifically OSD 31 and 34, not that it matters at
> > this point)
> >
> > # ceph pg ls incomplete --> is attached for reference.
> >
> > Fortunately, it's primarily (only) my on-site backups, and other
> > replaceable data inside of
> >
> > I tried for a few days to recover the PGs:
> > - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs)
> > - Deep Scrub
> > - osd_find_best_info_ignore_history_les = true (`pg query` was
> > showing related error)
> > etc.
> >
> > I've finally just accepted this pool to be a lesson learned, and want
> > to get the rest of my cephfs back to normal.
> >
> > My questions:
> >
> > -- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs
> > with 0 objects
> > -- Deleting the pool seems like an appropriate step, but as I am
> > using an xattr within cephfs, which is otherwise on another pool, I am
> > not confident that this approach is safe?
> > -- cephfs currently blocks when attemping to impact every third file
> > in the EC directory. Once I delete the pool, how will I remove the
> > files if even `rm` is blocking?
> >
> > Thank you for your time,
> >
> > Joshua West
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
>
cluster:
id: 587ad752-283b-4359-8b77-f52d25f10c25
health: HEALTH_WARN
1 MDSs report slow metadata IOs
1 MDSs report slow requests
1 MDSs behind on trimming
noout flag(s) set
Reduced data availability: 218 pgs inactive, 218 pgs incomplete
429 pgs not deep-scrubbed in time
58 pgs not scrubbed in time
116 slow ops, oldest one blocked for 7735 sec, daemons [osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.18]... have slow ops.
too many PGs per OSD (317 > max 250)
services:
mon: 3 daemons, quorum rd240,server,dl380g7 (age 2h)
mgr: rd240(active, since 2h), standbys: server, dl380g7
mds: cephfs:1 {0=server=up:active} 2 up:standby
osd: 37 osds: 37 up (since 2h), 25 in (since 2h)
flags noout
data:
pools: 12 pools, 2649 pgs
objects: 87.62M objects, 45 TiB
usage: 100 TiB used, 84 TiB / 184 TiB avail
pgs: 8.230% pgs not active
2342 active+clean
218 incomplete
81 active+clean+snaptrim_wait
6 active+clean+scrubbing+deep
2 active+clean+snaptrim
io:
client: 1.2 KiB/s rd, 224 KiB/s wr, 17 op/s rd, 18 op/s wr
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "incomplete",
"epoch": 525943,
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"info": {
"pgid": "47.0s0",
"last_update": "519000'86682",
"last_complete": "519000'86682",
"log_tail": "518614'85590",
"last_user_version": 86682,
"last_backfill": "MAX",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2e"
},
{
"start": "5ce",
"length": "2"
},
{
"start": "5d1",
"length": "1"
},
{
"start": "5d3",
"length": "1"
},
{
"start": "5d5",
"length": "1"
},
{
"start": "5d7",
"length": "1"
},
{
"start": "5d9",
"length": "1"
},
{
"start": "5db",
"length": "1"
},
{
"start": "5dd",
"length": "1"
},
{
"start": "5df",
"length": "1"
},
{
"start": "5e1",
"length": "1"
},
{
"start": "5e3",
"length": "1"
},
{
"start": "5e5",
"length": "1"
},
{
"start": "5e7",
"length": "1"
},
{
"start": "5e9",
"length": "1"
},
{
"start": "5eb",
"length": "1"
},
{
"start": "5ed",
"length": "1"
},
{
"start": "5ef",
"length": "1"
},
{
"start": "5f1",
"length": "1"
},
{
"start": "5f3",
"length": "1"
}
],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "519000'86682",
"reported_seq": "346239",
"reported_epoch": "525943",
"state": "incomplete",
"last_fresh": "2021-04-08T14:54:57.285506-0600",
"last_change": "2021-04-08T14:38:31.054438-0600",
"last_active": "2021-04-03T10:17:20.578805-0600",
"last_peered": "2021-04-03T10:17:10.504682-0600",
"last_clean": "2021-04-03T09:23:47.953138-0600",
"last_became_active": "2021-04-03T10:17:08.445271-0600",
"last_became_peered": "2021-04-03T10:17:08.445271-0600",
"last_unstale": "2021-04-08T14:54:57.285506-0600",
"last_undegraded": "2021-04-08T14:54:57.285506-0600",
"last_fullsized": "2021-04-08T14:54:57.285506-0600",
"mapping_epoch": 525928,
"log_start": "518614'85590",
"ondisk_log_start": "518614'85590",
"created": 337872,
"last_epoch_clean": 518952,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"log_size": 1092,
"ondisk_log_size": 1092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 26759978094,
"num_objects": 20070,
"num_object_clones": 772,
"num_object_copies": 60210,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 20070,
"num_whiteouts": 652,
"num_read": 13967,
"num_read_kb": 2052182,
"num_write": 45875,
"num_write_kb": 2332074,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 37027,
"num_bytes_recovered": 59477202252,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 519062,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "4(2)",
"pgid": "47.0s2",
"last_update": "519000'86682",
"last_complete": "519000'86682",
"log_tail": "518614'85590",
"last_user_version": 86682,
"last_backfill": "47:00116113:::10001c85d5d.00000032:head",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2e"
},
{
"start": "5ce",
"length": "2"
},
{
"start": "5d1",
"length": "1"
},
{
"start": "5d3",
"length": "1"
},
{
"start": "5d5",
"length": "1"
},
{
"start": "5d7",
"length": "1"
},
{
"start": "5d9",
"length": "1"
},
{
"start": "5db",
"length": "1"
},
{
"start": "5dd",
"length": "1"
},
{
"start": "5df",
"length": "1"
},
{
"start": "5e1",
"length": "1"
},
{
"start": "5e3",
"length": "1"
},
{
"start": "5e5",
"length": "1"
},
{
"start": "5e7",
"length": "1"
},
{
"start": "5e9",
"length": "1"
},
{
"start": "5eb",
"length": "1"
},
{
"start": "5ed",
"length": "1"
},
{
"start": "5ef",
"length": "1"
},
{
"start": "5f1",
"length": "1"
},
{
"start": "5f3",
"length": "1"
}
],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "519000'86682",
"reported_seq": "22",
"reported_epoch": "525242",
"state": "down",
"last_fresh": "2021-04-07T21:23:43.829634-0600",
"last_change": "2021-04-07T21:23:43.829634-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-07T21:23:43.829634-0600",
"last_undegraded": "2021-04-07T21:23:43.829634-0600",
"last_fullsized": "2021-04-07T21:23:43.829634-0600",
"mapping_epoch": 525928,
"log_start": "518614'85590",
"ondisk_log_start": "518614'85590",
"created": 337872,
"last_epoch_clean": 518952,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"log_size": 1092,
"ondisk_log_size": 1092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 29770742894,
"num_objects": 5519,
"num_object_clones": 229,
"num_object_copies": 16557,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 5519,
"num_whiteouts": 193,
"num_read": 0,
"num_read_kb": 0,
"num_write": 3,
"num_write_kb": 32,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
5,
7,
9,
17,
34
],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 518961,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "7(2)",
"pgid": "47.0s2",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "9(1)",
"pgid": "47.0s1",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "484",
"reported_epoch": "525921",
"state": "down",
"last_fresh": "2021-04-08T14:38:13.207479-0600",
"last_change": "2021-04-08T14:38:13.207479-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-08T14:38:13.207479-0600",
"last_undegraded": "2021-04-08T14:38:13.207479-0600",
"last_fullsized": "2021-04-08T14:38:13.207479-0600",
"mapping_epoch": 525928,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 337872,
"last_epoch_clean": 518952,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
5
],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "17(0)",
"pgid": "47.0s0",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "397",
"reported_epoch": "525888",
"state": "incomplete",
"last_fresh": "2021-04-08T14:36:16.266214-0600",
"last_change": "2021-04-08T14:36:12.822429-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-08T14:36:16.266214-0600",
"last_undegraded": "2021-04-08T14:36:16.266214-0600",
"last_fullsized": "2021-04-08T14:36:16.266214-0600",
"mapping_epoch": 525928,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 337872,
"last_epoch_clean": 518952,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "18(1)",
"pgid": "47.0s1",
"last_update": "519000'86682",
"last_complete": "519000'86682",
"log_tail": "518614'85590",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2e"
},
{
"start": "5ce",
"length": "2"
},
{
"start": "5d1",
"length": "1"
},
{
"start": "5d3",
"length": "1"
},
{
"start": "5d5",
"length": "1"
},
{
"start": "5d7",
"length": "1"
},
{
"start": "5d9",
"length": "1"
},
{
"start": "5db",
"length": "1"
},
{
"start": "5dd",
"length": "1"
},
{
"start": "5df",
"length": "1"
},
{
"start": "5e1",
"length": "1"
},
{
"start": "5e3",
"length": "1"
},
{
"start": "5e5",
"length": "1"
},
{
"start": "5e7",
"length": "1"
},
{
"start": "5e9",
"length": "1"
},
{
"start": "5eb",
"length": "1"
},
{
"start": "5ed",
"length": "1"
},
{
"start": "5ef",
"length": "1"
},
{
"start": "5f1",
"length": "1"
},
{
"start": "5f3",
"length": "1"
}
],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 525928,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 519062,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "19(2)",
"pgid": "47.0s2",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "34(1)",
"pgid": "47.0s1",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Peering/Incomplete",
"enter_time": "2021-04-08T14:38:31.054423-0600",
"comment": "not enough complete instances of this PG"
},
{
"name": "Started/Primary/Peering",
"enter_time": "2021-04-08T14:38:30.989327-0600",
"past_intervals": [
{
"first": "518951",
"last": "525927",
"all_participants": [
{
"osd": 4,
"shard": 2
},
{
"osd": 5,
"shard": 0
},
{
"osd": 7,
"shard": 2
},
{
"osd": 9,
"shard": 1
},
{
"osd": 17,
"shard": 0
},
{
"osd": 19,
"shard": 2
},
{
"osd": 34,
"shard": 1
}
],
"intervals": [
{
"first": "519027",
"last": "519044",
"acting": "5(0),19(2)"
},
{
"first": "519061",
"last": "519064",
"acting": "5(0),19(2),34(1)"
},
{
"first": "520151",
"last": "520156",
"acting": "7(2),17(0)"
},
{
"first": "520194",
"last": "520261",
"acting": "7(2),9(1)"
},
{
"first": "520262",
"last": "520304",
"acting": "7(2),9(1),17(0)"
},
{
"first": "525419",
"last": "525420",
"acting": "5(0),9(1)"
},
{
"first": "525616",
"last": "525624",
"acting": "4(2),5(0)"
},
{
"first": "525638",
"last": "525641",
"acting": "4(2),9(1)"
},
{
"first": "525884",
"last": "525888",
"acting": "4(2),9(1),17(0)"
},
{
"first": "525922",
"last": "525926",
"acting": "4(2),5(0),9(1)"
}
]
}
],
"probing_osds": [
"4(2)",
"5(0)",
"7(2)",
"9(1)",
"17(0)",
"19(2)",
"34(1)"
],
"down_osds_we_would_probe": [],
"peering_blocked_by": []
},
{
"name": "Started",
"enter_time": "2021-04-08T14:38:30.989236-0600"
}
],
"agent_state": {}
}
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "incomplete",
"epoch": 525955,
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"info": {
"pgid": "47.3f0s0",
"last_update": "518995'78417",
"last_complete": "518995'78417",
"log_tail": "518483'77304",
"last_user_version": 78417,
"last_backfill": "MAX",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2b"
}
],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 525932,
"same_interval_since": 525932,
"same_primary_since": 525932,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "518995'78417",
"reported_seq": "617838",
"reported_epoch": "525955",
"state": "incomplete",
"last_fresh": "2021-04-09T06:23:29.149869-0600",
"last_change": "2021-04-08T14:38:24.001318-0600",
"last_active": "2021-04-03T10:17:20.577281-0600",
"last_peered": "2021-04-03T10:17:10.685305-0600",
"last_clean": "2021-04-03T10:17:07.001797-0600",
"last_became_active": "2021-04-03T10:17:10.644563-0600",
"last_became_peered": "2021-04-03T10:17:10.644563-0600",
"last_unstale": "2021-04-09T06:23:29.149869-0600",
"last_undegraded": "2021-04-09T06:23:29.149869-0600",
"last_fullsized": "2021-04-09T06:23:29.149869-0600",
"mapping_epoch": 525932,
"log_start": "518483'77304",
"ondisk_log_start": "518483'77304",
"created": 489203,
"last_epoch_clean": 519046,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"log_size": 1113,
"ondisk_log_size": 1113,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 26031954397,
"num_objects": 20197,
"num_object_clones": 779,
"num_object_copies": 60591,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 20197,
"num_whiteouts": 651,
"num_read": 13898,
"num_read_kb": 1525718,
"num_write": 15931,
"num_write_kb": 876102,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 44059,
"num_bytes_recovered": 75750824328,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 519063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "6(1)",
"pgid": "47.3f0s1",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 525932,
"same_interval_since": 525932,
"same_primary_since": 525932,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "368",
"reported_epoch": "525931",
"state": "down",
"last_fresh": "2021-04-08T14:38:22.846849-0600",
"last_change": "2021-04-08T14:38:22.846849-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-08T14:38:22.846849-0600",
"last_undegraded": "2021-04-08T14:38:22.846849-0600",
"last_fullsized": "2021-04-08T14:38:22.846849-0600",
"mapping_epoch": 525932,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 489203,
"last_epoch_clean": 519046,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
11
],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "13(0)",
"pgid": "47.3f0s0",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 525932,
"same_interval_since": 525932,
"same_primary_since": 525932,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "569",
"reported_epoch": "520301",
"state": "incomplete",
"last_fresh": "2021-04-04T10:19:59.460563-0600",
"last_change": "2021-04-04T10:19:59.460563-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-04T10:19:59.460563-0600",
"last_undegraded": "2021-04-04T10:19:59.460563-0600",
"last_fullsized": "2021-04-04T10:19:59.460563-0600",
"mapping_epoch": 525932,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 489203,
"last_epoch_clean": 519046,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
34
],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "14(2)",
"pgid": "47.3f0s2",
"last_update": "518995'78417",
"last_complete": "518995'78417",
"log_tail": "518483'77304",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2b"
}
],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 525932,
"same_interval_since": 525932,
"same_primary_since": 525932,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 525932,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 519063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "15(1)",
"pgid": "47.3f0s1",
"last_update": "518995'78417",
"last_complete": "518995'78417",
"log_tail": "518483'77304",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2b"
}
],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 525932,
"same_interval_since": 525932,
"same_primary_since": 525932,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 525932,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 519063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "34(2)",
"pgid": "47.3f0s2",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Peering/Incomplete",
"enter_time": "2021-04-08T14:38:24.001303-0600",
"comment": "not enough complete instances of this PG"
},
{
"name": "Started/Primary/Peering",
"enter_time": "2021-04-08T14:38:23.848565-0600",
"past_intervals": [
{
"first": "519045",
"last": "525931",
"all_participants": [
{
"osd": 6,
"shard": 1
},
{
"osd": 11,
"shard": 0
},
{
"osd": 13,
"shard": 0
},
{
"osd": 14,
"shard": 2
},
{
"osd": 15,
"shard": 1
},
{
"osd": 34,
"shard": 2
}
],
"intervals": [
{
"first": "520091",
"last": "520148",
"acting": "13(0),14(2)"
},
{
"first": "520247",
"last": "520254",
"acting": "6(1),13(0)"
},
{
"first": "520960",
"last": "521560",
"acting": "11(0),14(2)"
},
{
"first": "525126",
"last": "525135",
"acting": "6(1),11(0)"
},
{
"first": "525633",
"last": "525638",
"acting": "6(1),14(2)"
},
{
"first": "525821",
"last": "525904",
"acting": "6(1),11(0),14(2)"
}
]
}
],
"probing_osds": [
"6(1)",
"11(0)",
"13(0)",
"14(2)",
"15(1)",
"34(2)"
],
"down_osds_we_would_probe": [],
"peering_blocked_by": []
},
{
"name": "Started",
"enter_time": "2021-04-08T14:38:23.848480-0600"
}
],
"agent_state": {}
}
pool 21 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 525839 lfor 0/488420/488963 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive compression_required_ratio 0.9 pg_num_min 1024 application cephfs
removed_snaps_queue [42c~10b,538~63,59c~1,59e~2e,5cd~32,600~7,608~31,63a~23,65e~c,66b~1f,68b~1,68d~1,68f~1,691~1,693~1,695~1,697~1,699~1,69b~1,69e~2,6a1~1,6a3~1,6a5~1,6a7~1,6a9~1,6ab~1,6ad~1,6af~1,6b1~1,6b3~1,6b5~1,6b7~1,6b9~1,6bb~1]
pool 22 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 pg_num_target 64 pgp_num_target 64 autoscale_mode on last_change 519055 lfor 0/124427/506237 flags hashpspool,nodelete stripe_width 0 pg_autoscale_bias 4 pg_num_min 64 recovery_priority 5 application cephfs
pool 30 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 525278 lfor 0/7304/327594 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive application ecpool,rbd
pool 31 'device_health_metrics' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 525939 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth
pool 32 '.rgw.root' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 135358 lfor 0/135358/135356 flags hashpspool stripe_width 0 application rgw
pool 33 'default.rgw.log' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 139171 lfor 0/139171/139169 flags hashpspool stripe_width 0 application rgw
pool 34 'default.rgw.control' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 141079 lfor 0/141079/141077 flags hashpspool stripe_width 0 application rgw
pool 35 'default.rgw.meta' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 142067 lfor 0/142067/142065 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw
pool 36 'nfs-ganesha' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 133226 flags hashpspool stripe_width 0 application rbd
pool 47 'claypool' erasure profile claypool size 3 min_size 2 crush_rule 7 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 525874 lfor 0/460696/489203 flags hashpspool,ec_overwrites,selfmanaged_snaps,creating stripe_width 8192 fast_read 1 compression_algorithm lz4 compression_mode aggressive pg_num_min 1024 application cephfs
removed_snaps_queue [129~1,428~10f,538~63,59c~1,59e~2e,5cd~32,600~7,608~31,63a~23,65e~c,66b~1f,68b~1,68d~1,68f~1,691~1,693~1,695~1,697~1,699~1,69b~1,69e~2,6a1~1,6a3~1,6a5~1,6a7~1,6a9~1,6ab~1,6ad~1,6af~1,6b1~1,6b3~1,6b5~1,6b7~1,6b9~1,6bb~1]
pool 54 'fast' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 524810 lfor 0/509819/509817 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 55 'caykcloud' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 524049 lfor 0/514664/514662 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP
47.0 20070 0 0 0 26759978094 0 0 1092 incomplete 15h 519000'86682 525954:346250 [5,9,4]p5 [5,9,4]p5 2021-03-31T03:21:18.407682-0600 2021-03-27T19:18:36.384901-0600
47.3 20169 0 0 0 26371196538 0 0 1168 incomplete 15h 518999'70605 525954:371495 [33,7,30]p33 [33,7,30]p33 2021-03-31T22:19:10.471925-0600 2021-03-25T20:21:16.093783-0600
47.4 20280 0 0 0 26260431825 0 0 1118 incomplete 15h 518995'80378 525954:398766 [23,10,19]p23 [23,10,19]p23 2021-03-31T22:40:12.339909-0600 2021-03-31T22:40:12.339909-0600
47.d 20525 0 0 0 26263948794 0 0 985 incomplete 15h 518999'80034 525954:354241 [0,8,6]p0 [0,8,6]p0 2021-04-01T02:26:35.982459-0600 2021-03-29T01:55:08.194735-0600
47.12 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4891 [27,1,8]p27 [27,1,8]p27 2021-04-01T01:01:22.575010-0600 2021-04-01T01:01:22.575010-0600
47.13 22005 0 0 0 27281035835 0 0 1245 incomplete 15h 519007'103848 525954:405943 [3,27,10]p3 [3,27,10]p3 2021-04-01T03:00:37.908783-0600 2021-03-25T03:15:07.751993-0600
47.18 19925 0 0 0 26041486043 0 0 831 incomplete 15h 518995'89106 525954:355262 [0,9,30]p0 [0,9,30]p0 2021-04-01T02:36:26.883662-0600 2021-03-28T20:40:53.970613-0600
47.20 20051 0 0 0 26505000546 0 0 1119 incomplete 15h 519000'99769 525954:348782 [15,8,30]p15 [15,8,30]p15 2021-04-01T01:14:42.269934-0600 2021-04-01T01:14:42.269934-0600
47.24 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4985 [9,19,15]p9 [9,19,15]p9 2021-03-31T00:43:17.841228-0600 2021-03-28T03:08:13.364769-0600
47.27 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4993 [30,28,6]p30 [30,28,6]p30 2021-04-01T03:40:34.645668-0600 2021-03-29T02:45:22.939338-0600
47.29 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4841 [19,16,8]p19 [19,16,8]p19 2021-03-31T20:58:34.015774-0600 2021-03-31T20:58:34.015774-0600
47.32 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:10973 [9,0,1]p9 [9,0,1]p9 2021-03-31T03:29:19.321570-0600 2021-03-25T03:55:28.079403-0600
47.38 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4959 [6,9,2]p6 [6,9,2]p6 2021-03-28T21:52:56.209769-0600 2021-03-26T23:58:40.078225-0600
47.39 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4998 [6,12,30]p6 [6,12,30]p6 2021-03-28T21:02:51.726250-0600 2021-03-26T22:30:41.534234-0600
47.3f 20334 0 0 0 26798526707 0 0 1037 incomplete 15h 518997'86706 525954:358726 [4,7,19]p4 [4,7,19]p4 2021-03-30T21:00:20.680102-0600 2021-03-30T21:00:20.680102-0600
47.43 20270 0 0 0 26401100570 0 0 1105 incomplete 15h 518997'79678 525954:332269 [1,6,8]p1 [1,6,8]p1 2021-04-01T03:19:25.565020-0600 2021-03-26T21:21:02.298726-0600
47.45 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5611 [27,7,0]p27 [27,7,0]p27 2021-03-31T23:57:18.800169-0600 2021-03-27T03:30:15.326626-0600
47.4e 20279 0 0 0 25784621329 0 0 963 incomplete 15h 518995'94727 525954:354973 [3,19,27]p3 [3,19,27]p3 2021-03-31T00:49:22.102173-0600 2021-03-31T00:49:22.102173-0600
47.4f 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9529 [6,8,11]p6 [6,8,11]p6 2021-03-28T19:47:56.332962-0600 2021-03-25T02:52:00.646232-0600
47.55 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9510 [7,16,8]p7 [7,16,8]p7 2021-04-01T22:59:55.257326-0600 2021-03-27T21:21:27.496151-0600
47.5a 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5703 [8,10,0]p8 [8,10,0]p8 2021-04-01T22:19:53.918809-0600 2021-04-01T22:19:53.918809-0600
47.68 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4913 [19,10,5]p19 [19,10,5]p19 2021-03-31T19:46:58.086704-0600 2021-03-25T02:36:58.633685-0600
47.6b 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4967 [8,33,6]p8 [8,33,6]p8 2021-03-28T19:08:20.531682-0600 2021-03-26T22:58:48.551650-0600
47.74 20028 0 0 0 26352535351 0 0 1119 incomplete 15h 518995'84646 525954:353895 [1,7,30]p1 [1,7,30]p1 2021-03-30T21:25:52.131450-0600 2021-03-24T23:39:23.368483-0600
47.75 20321 0 0 0 26728108311 0 0 1206 incomplete 15h 519007'108028 525954:426709 [17,27,19]p17 [17,27,19]p17 2021-03-31T21:40:42.269394-0600 2021-03-31T21:40:42.269394-0600
47.78 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4998 [9,30,5]p9 [9,30,5]p9 2021-03-31T20:11:41.210011-0600 2021-03-27T22:36:10.661087-0600
47.7b 20026 0 0 0 25999494334 0 0 1058 incomplete 15h 519049'101874 525954:409956 [28,30,10]p28 [28,30,10]p28 2021-03-31T23:07:28.497014-0600 2021-03-27T23:06:40.522139-0600
47.7d 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4865 [10,7,18]p10 [10,7,18]p10 2021-03-29T00:47:33.754276-0600 2021-03-26T21:43:09.385219-0600
47.81 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5046 [6,10,13]p6 [6,10,13]p6 2021-03-29T03:10:09.500658-0600 2021-03-24T00:55:05.716605-0600
47.83 19943 0 0 0 26143034719 0 0 1116 incomplete 15h 519029'97443 525954:388487 [28,27,7]p28 [28,27,7]p28 2021-04-01T01:07:13.961376-0600 2021-04-01T01:07:13.961376-0600
47.85 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4956 [7,6,11]p7 [7,6,11]p7 2021-03-31T21:46:38.615216-0600 2021-03-31T21:46:38.615216-0600
47.88 19744 0 0 0 25741847210 0 0 1063 incomplete 15h 519000'106717 525954:390360 [16,9,27]p16 [16,9,27]p16 2021-04-01T00:36:40.605136-0600 2021-04-01T00:36:40.605136-0600
47.8d 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4920 [30,19,14]p30 [30,19,14]p30 2021-04-01T01:02:07.798881-0600 2021-03-25T21:55:37.181983-0600
47.91 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4876 [8,14,27]p8 [8,14,27]p8 2021-03-31T20:07:35.087035-0600 2021-03-24T23:02:00.170210-0600
47.a2 20284 0 0 0 26569134699 0 0 1161 incomplete 15h 519000'104297 525954:398639 [13,19,6]p13 [13,19,6]p13 2021-03-31T19:56:03.695108-0600 2021-03-31T19:56:03.695108-0600
47.a5 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4984 [8,27,3]p8 [8,27,3]p8 2021-04-01T21:45:11.476900-0600 2021-03-28T01:24:09.482384-0600
47.ab 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4801 [10,8,14]p10 [10,8,14]p10 2021-03-30T21:49:07.499027-0600 2021-03-30T21:49:07.499027-0600
47.ac 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4969 [7,27,15]p7 [7,27,15]p7 2021-03-28T03:01:26.639534-0600 2021-03-25T00:50:14.553902-0600
47.b2 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5027 [6,7,18]p6 [6,7,18]p6 2021-03-28T19:00:52.306016-0600 2021-03-25T00:55:20.601141-0600
47.ba 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4869 [27,9,14]p27 [27,9,14]p27 2021-03-30T21:23:31.275720-0600 2021-03-30T21:23:31.275720-0600
47.bd 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9291 [19,8,1]p19 [19,8,1]p19 2021-04-01T02:10:45.120565-0600 2021-04-01T02:10:45.120565-0600
47.be 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4924 [19,33,6]p19 [19,33,6]p19 2021-04-01T03:34:38.102589-0600 2021-04-01T03:34:38.102589-0600
47.bf 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4850 [19,8,13]p19 [19,8,13]p19 2021-03-30T21:35:54.230025-0600 2021-03-30T21:35:54.230025-0600
47.c1 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4950 [30,19,13]p30 [30,19,13]p30 2021-03-28T20:18:17.342293-0600 2021-03-28T20:18:17.342293-0600
47.c2 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9275 [10,11,30]p10 [10,11,30]p10 2021-03-30T20:02:35.909530-0600 2021-03-23T20:55:48.695067-0600
47.c4 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5037 [9,7,5]p9 [9,7,5]p9 2021-03-31T22:25:47.118619-0600 2021-03-24T23:21:34.267990-0600
47.c6 19751 0 0 0 26365340716 0 0 1093 incomplete 15h 519010'83703 525954:426904 [1,27,8]p1 [1,27,8]p1 2021-04-01T21:11:47.655056-0600 2021-04-01T21:11:47.655056-0600
47.d2 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5624 [30,9,28]p30 [30,9,28]p30 2021-03-28T19:01:14.191464-0600 2021-03-23T19:12:59.768249-0600
47.d5 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5039 [9,3,7]p9 [9,3,7]p9 2021-03-31T21:06:28.077311-0600 2021-03-25T22:47:26.773692-0600
47.d6 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9397 [7,19,8]p7 [7,19,8]p7 2021-03-31T19:44:39.521844-0600 2021-03-31T19:44:39.521844-0600
47.d9 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4906 [19,17,9]p19 [19,17,9]p19 2021-03-31T01:09:56.596992-0600 2021-03-27T02:36:54.285634-0600
47.db 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4886 [10,28,7]p10 [10,28,7]p10 2021-03-31T21:29:00.484238-0600 2021-03-28T01:45:50.800338-0600
47.de 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4994 [8,7,15]p8 [8,7,15]p8 2021-04-01T01:30:22.243669-0600 2021-03-26T01:33:06.527431-0600
47.e1 20088 0 0 0 26226582893 0 0 1141 incomplete 15h 519000'108471 525954:345696 [15,19,27]p15 [15,19,27]p15 2021-04-01T22:24:35.935867-0600 2021-03-29T19:16:20.203954-0600
47.e7 19849 0 0 0 26339566289 0 0 1032 incomplete 15h 518987'86817 525954:448281 [3,10,30]p3 [3,10,30]p3 2021-03-30T20:22:16.603628-0600 2021-03-23T22:49:34.674749-0600
47.f8 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5040 [6,15,8]p6 [6,15,8]p6 2021-03-28T00:24:32.390309-0600 2021-03-28T00:24:32.390309-0600
47.ff 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4837 [10,18,9]p10 [10,18,9]p10 2021-04-01T03:18:45.139873-0600 2021-03-25T21:28:04.827073-0600
47.106 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5020 [30,2,3]p30 [30,2,3]p30 2021-04-01T00:42:20.714987-0600 2021-03-27T19:13:11.636060-0600
47.10b 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4857 [19,30,1]p19 [19,30,1]p19 2021-04-01T00:54:51.595530-0600 2021-04-01T00:54:51.595530-0600
47.111 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4902 [9,33,8]p9 [9,33,8]p9 2021-03-31T19:54:04.738313-0600 2021-03-24T20:13:03.007939-0600
47.114 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9505 [9,10,17]p9 [9,10,17]p9 2021-04-01T19:56:00.916625-0600 2021-03-28T23:49:56.145249-0600
47.115 19967 0 0 0 26250102170 0 0 1000 incomplete 15h 519007'82049 525954:378759 [4,6,9]p4 [4,6,9]p4 2021-04-01T21:40:16.002686-0600 2021-04-01T21:40:16.002686-0600
47.116 19852 0 0 0 25884529821 0 0 1121 incomplete 15h 519000'74737 525954:459697 [33,6,30]p33 [33,6,30]p33 2021-04-01T00:22:40.644004-0600 2021-04-01T00:22:40.644004-0600
47.11c 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9331 [19,12,30]p19 [19,12,30]p19 2021-04-01T00:13:13.070183-0600 2021-03-25T19:18:01.757931-0600
47.11e 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4954 [7,28,30]p7 [7,28,30]p7 2021-03-29T02:51:23.634263-0600 2021-03-27T02:55:35.150008-0600
47.121 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5008 [30,19,4]p30 [30,19,4]p30 2021-03-28T03:21:15.443939-0600 2021-03-26T21:34:23.689399-0600
47.12b 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4832 [19,3,6]p19 [19,3,6]p19 2021-03-31T23:29:53.658826-0600 2021-03-31T23:29:53.658826-0600
47.12d 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5046 [9,28,8]p9 [9,28,8]p9 2021-03-31T23:39:03.266816-0600 2021-03-28T20:59:23.118803-0600
47.12e 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4849 [10,33,27]p10 [10,33,27]p10 2021-03-28T21:08:38.252809-0600 2021-03-23T21:36:01.003577-0600
47.12f 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4973 [27,33,9]p27 [27,33,9]p27 2021-03-31T23:26:16.234711-0600 2021-03-28T20:30:21.411969-0600
47.132 19934 0 0 0 26467844952 0 0 920 incomplete 15h 519007'83538 525954:409747 [2,19,7]p2 [2,19,7]p2 2021-04-01T03:28:35.691696-0600 2021-04-01T03:28:35.691696-0600
47.133 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9276 [10,28,9]p10 [10,28,9]p10 2021-04-02T03:49:39.216071-0600 2021-04-02T03:49:39.216071-0600
47.13b 0 0 0 0 0 0 0 915 incomplete 15h 518995'81210 525954:5166 [14,16,8]p14 [14,16,8]p14 2021-04-01T20:42:03.066302-0600 2021-03-27T01:43:56.229840-0600
47.13d 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4932 [9,30,28]p9 [9,30,28]p9 2021-04-01T19:46:45.285673-0600 2021-03-29T00:57:54.507752-0600
47.13e 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5061 [9,28,6]p9 [9,28,6]p9 2021-04-01T00:45:16.255848-0600 2021-03-29T00:34:55.455255-0600
47.148 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9304 [10,18,4]p10 [10,18,4]p10 2021-03-30T21:37:21.827042-0600 2021-03-30T21:37:21.827042-0600
47.14d 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5036 [27,3,8]p27 [27,3,8]p27 2021-03-30T21:04:55.204019-0600 2021-03-25T00:52:00.709998-0600
47.14e 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4948 [9,2,27]p9 [9,2,27]p9 2021-03-31T00:35:27.216322-0600 2021-03-25T20:11:04.278420-0600
47.153 20022 0 0 0 26289944955 0 0 1169 incomplete 15h 519022'106165 525954:381152 [12,27,8]p12 [12,27,8]p12 2021-03-31T19:23:47.708010-0600 2021-03-27T19:25:23.759889-0600
47.158 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4982 [6,13,19]p6 [6,13,19]p6 2021-03-30T19:13:13.576594-0600 2021-03-30T19:13:13.576594-0600
47.15a 19699 0 0 0 25511263757 0 0 1105 incomplete 15h 519031'105897 525954:427605 [13,19,10]p13 [13,19,10]p13 2021-03-31T23:13:41.188648-0600 2021-03-31T23:13:41.188648-0600
47.15b 19889 0 0 0 26272408882 0 0 1120 incomplete 15h 519007'103191 525954:471874 [16,30,27]p16 [16,30,27]p16 2021-03-31T23:04:31.104476-0600 2021-03-28T20:07:48.221437-0600
47.15d 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9199 [10,27,23]p10 [10,27,23]p10 2021-04-01T00:36:47.264513-0600 2021-03-25T00:42:47.196052-0600
47.166 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9542 [9,27,28]p9 [9,27,28]p9 2021-04-01T01:27:29.221950-0600 2021-04-01T01:27:29.221950-0600
47.16c 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9382 [7,16,19]p7 [7,16,19]p7 2021-03-31T23:54:09.693485-0600 2021-03-31T23:54:09.693485-0600
47.171 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5505 [19,0,6]p19 [19,0,6]p19 2021-04-01T00:26:19.211464-0600 2021-03-27T21:40:36.939559-0600
47.175 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9432 [7,27,4]p7 [7,27,4]p7 2021-03-30T22:17:18.528531-0600 2021-03-28T02:54:27.600714-0600
47.181 19969 0 0 0 26071125080 0 0 1108 incomplete 15h 519025'99249 525954:420303 [28,7,27]p28 [28,7,27]p28 2021-03-29T01:55:17.182313-0600 2021-03-29T01:55:17.182313-0600
47.183 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9388 [27,30,4]p27 [27,30,4]p27 2021-03-31T00:55:39.514956-0600 2021-03-31T00:55:39.514956-0600
47.18a 20183 0 0 0 26317047279 0 0 1105 incomplete 15h 518995'78580 525954:405908 [1,10,27]p1 [1,10,27]p1 2021-04-01T03:28:57.767008-0600 2021-04-01T03:28:57.767008-0600
47.193 20076 0 0 0 26545067723 0 0 999 incomplete 15h 519000'80492 525954:412929 [4,6,9]p4 [4,6,9]p4 2021-04-01T19:55:40.675099-0600 2021-03-29T00:03:32.626359-0600
47.1a5 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5065 [9,8,13]p9 [9,8,13]p9 2021-03-30T19:45:03.299521-0600 2021-03-30T19:45:03.299521-0600
47.1ae 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4885 [19,7,13]p19 [19,7,13]p19 2021-03-31T20:58:50.224568-0600 2021-03-26T00:39:11.884432-0600
47.1b1 19602 0 0 0 25642774871 0 0 1145 incomplete 15h 519049'98127 525954:447635 [14,7,9]p14 [14,7,9]p14 2021-03-30T19:28:37.473471-0600 2021-03-30T19:28:37.473471-0600
47.1b5 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4794 [10,2,27]p10 [10,2,27]p10 2021-04-01T21:28:37.616513-0600 2021-04-01T21:28:37.616513-0600
47.1bb 19937 0 0 0 26256947699 0 0 1167 incomplete 15h 519000'104777 525954:430237 [13,8,19]p13 [13,8,19]p13 2021-04-01T01:01:30.688372-0600 2021-03-28T19:19:21.517396-0600
47.1bc 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4928 [7,3,6]p7 [7,3,6]p7 2021-04-01T21:43:32.613447-0600 2021-03-28T02:28:26.988032-0600
47.1bd 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4941 [19,9,17]p19 [19,9,17]p19 2021-04-01T01:39:53.715270-0600 2021-03-27T21:01:21.857265-0600
47.1c5 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4962 [6,7,1]p6 [6,7,1]p6 2021-03-29T00:09:05.664124-0600 2021-03-29T00:09:05.664124-0600
47.1c8 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4938 [27,12,30]p27 [27,12,30]p27 2021-03-30T19:05:17.430464-0600 2021-03-30T19:05:17.430464-0600
47.1c9 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4988 [7,12,9]p7 [7,12,9]p7 2021-04-01T02:18:06.693708-0600 2021-04-01T02:18:06.693708-0600
47.1ca 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9612 [6,13,0]p6 [6,13,0]p6 2021-03-30T20:53:34.745151-0600 2021-03-30T20:53:34.745151-0600
47.1cb 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9575 [9,10,1]p9 [9,10,1]p9 2021-04-01T03:59:07.919592-0600 2021-03-27T19:06:05.459813-0600
47.1e0 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9454 [30,8,14]p30 [30,8,14]p30 2021-03-28T23:57:25.436078-0600 2021-03-27T01:38:05.517845-0600
47.1e1 20046 0 0 0 26082697600 0 0 1106 incomplete 15h 519007'84323 525954:482490 [23,19,8]p23 [23,19,8]p23 2021-03-31T19:44:50.391568-0600 2021-03-27T19:50:13.164921-0600
47.1f1 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9559 [30,23,9]p30 [30,23,9]p30 2021-03-30T20:53:47.462270-0600 2021-03-28T01:25:46.956359-0600
47.1f4 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9180 [10,23,27]p10 [10,23,27]p10 2021-03-31T01:43:42.240022-0600 2021-03-31T01:43:42.240022-0600
47.1f9 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4885 [10,7,12]p10 [10,7,12]p10 2021-03-29T02:19:18.367248-0600 2021-03-24T23:14:41.816716-0600
47.206 20330 0 0 0 26179646944 0 0 1115 incomplete 15h 518997'80093 525954:412654 [1,19,7]p1 [1,19,7]p1 2021-03-30T21:26:04.011341-0600 2021-03-24T03:59:12.427505-0600
47.20b 20053 0 0 0 26024476227 0 0 997 incomplete 15h 518997'78121 525954:363145 [4,6,12]p4 [4,6,12]p4 2021-04-01T00:45:09.219038-0600 2021-03-26T20:19:01.735932-0600
47.20c 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:10745 [7,0,8]p7 [7,0,8]p7 2021-03-31T20:22:59.759870-0600 2021-03-31T20:22:59.759870-0600
47.20e 20113 0 0 0 26526638542 0 0 990 incomplete 15h 518995'81408 525954:446875 [0,6,30]p0 [0,6,30]p0 2021-03-31T21:21:08.761007-0600 2021-03-26T02:50:03.752634-0600
47.217 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4871 [19,8,1]p19 [19,8,1]p19 2021-04-01T21:12:15.382870-0600 2021-03-27T23:26:09.896032-0600
47.21d 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4854 [19,2,27]p19 [19,2,27]p19 2021-04-01T01:22:15.964498-0600 2021-03-28T00:04:41.815145-0600
47.21f 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9459 [27,1,19]p27 [27,1,19]p27 2021-03-31T02:49:20.951543-0600 2021-03-26T19:50:30.551545-0600
47.222 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9450 [9,10,12]p9 [9,10,12]p9 2021-04-01T03:01:35.864905-0600 2021-03-26T21:23:24.045585-0600
47.227 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5017 [7,8,13]p7 [7,8,13]p7 2021-03-28T01:16:28.930572-0600 2021-03-25T03:39:00.406340-0600
47.22b 0 0 0 0 0 0 0 927 incomplete 15h 519000'94397 525954:4855 [3,9,19]p3 [3,9,19]p3 2021-03-31T03:23:17.906474-0600 2021-03-26T03:42:26.835052-0600
47.22d 19672 0 0 0 26120364889 0 0 1118 incomplete 15h 519044'103283 525954:352446 [28,10,27]p28 [28,10,27]p28 2021-04-01T01:08:06.408050-0600 2021-03-28T02:17:35.460478-0600
47.22e 20123 0 0 0 26688200644 0 0 1192 incomplete 15h 519007'106615 525954:419596 [18,19,30]p18 [18,19,30]p18 2021-03-30T22:44:24.037301-0600 2021-03-28T22:35:08.620692-0600
47.230 20257 0 0 0 25824570856 0 0 993 incomplete 15h 518987'84219 525954:337995 [4,30,19]p4 [4,30,19]p4 2021-04-01T21:56:05.621218-0600 2021-03-27T02:49:41.789992-0600
47.234 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4893 [10,17,19]p10 [10,17,19]p10 2021-03-28T21:24:34.834498-0600 2021-03-28T21:24:34.834498-0600
47.243 20157 0 0 0 26381686450 0 0 1209 incomplete 15h 519025'100366 525954:448725 [18,8,7]p18 [18,8,7]p18 2021-04-01T03:00:22.585387-0600 2021-03-28T21:10:11.662912-0600
47.244 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5641 [30,0,10]p30 [30,0,10]p30 2021-04-01T01:07:26.685071-0600 2021-03-29T02:44:15.370657-0600
47.24e 20127 0 0 0 26219190351 0 0 1206 incomplete 15h 518997'94954 525954:399308 [12,7,8]p12 [12,7,8]p12 2021-03-30T21:10:10.911135-0600 2021-03-30T21:10:10.911135-0600
47.254 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4957 [7,23,6]p7 [7,23,6]p7 2021-03-28T19:41:04.896055-0600 2021-03-28T19:41:04.896055-0600
47.255 20376 0 0 0 26742559275 0 0 1188 incomplete 15h 518995'77999 525954:431691 [33,30,8]p33 [33,30,8]p33 2021-03-31T19:17:21.537364-0600 2021-03-25T20:14:25.340283-0600
47.258 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5646 [6,0,30]p6 [6,0,30]p6 2021-04-01T00:00:49.579022-0600 2021-03-27T20:33:04.243319-0600
47.259 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5026 [8,30,5]p8 [8,30,5]p8 2021-03-27T22:39:37.697021-0600 2021-03-21T01:05:52.666420-0600
47.25d 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5631 [27,7,0]p27 [27,7,0]p27 2021-04-01T03:23:08.559922-0600 2021-04-01T03:23:08.559922-0600
47.25e 20336 0 0 0 26698183299 0 0 1061 incomplete 15h 518993'101057 525954:376295 [28,8,6]p28 [28,8,6]p28 2021-03-29T01:32:11.012945-0600 2021-03-27T00:12:19.948412-0600
47.261 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5059 [30,18,7]p30 [30,18,7]p30 2021-04-01T01:57:47.344046-0600 2021-03-26T20:41:03.399382-0600
47.266 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4973 [8,16,30]p8 [8,16,30]p8 2021-03-28T03:35:27.190995-0600 2021-03-24T23:14:49.566588-0600
47.26a 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4930 [7,16,19]p7 [7,16,19]p7 2021-03-31T23:15:16.369673-0600 2021-03-26T01:25:59.521139-0600
47.27a 20185 0 0 0 26574776111 0 0 1106 incomplete 15h 519007'107921 525954:417354 [16,9,19]p16 [16,9,19]p16 2021-03-31T03:16:02.910400-0600 2021-03-28T19:54:24.926806-0600
47.27f 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4946 [27,5,6]p27 [27,5,6]p27 2021-03-31T23:06:25.788178-0600 2021-03-24T23:39:08.357812-0600
47.280 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4948 [27,10,2]p27 [27,10,2]p27 2021-03-30T19:08:39.702485-0600 2021-03-27T21:20:30.399908-0600
47.281 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9284 [10,17,19]p10 [10,17,19]p10 2021-04-01T01:41:06.905989-0600 2021-03-29T01:33:29.605803-0600
47.284 20327 0 0 0 26605243228 0 0 1148 incomplete 15h 519049'98275 525954:529253 [28,8,30]p28 [28,8,30]p28 2021-03-28T21:43:34.141833-0600 2021-03-28T21:43:34.141833-0600
47.286 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4983 [30,7,4]p30 [30,7,4]p30 2021-03-31T19:17:35.407716-0600 2021-03-25T22:53:15.065688-0600
47.28e 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4879 [10,7,12]p10 [10,7,12]p10 2021-03-29T00:36:47.386296-0600 2021-03-27T03:44:01.214168-0600
47.290 19987 0 0 0 25866970367 0 0 1021 incomplete 15h 519000'87528 525954:409855 [3,8,9]p3 [3,8,9]p3 2021-04-01T01:42:34.676157-0600 2021-03-27T03:05:59.228923-0600
47.298 20066 0 0 0 26321626020 0 0 1221 incomplete 15h 519038'104666 525954:440865 [18,10,19]p18 [18,10,19]p18 2021-04-01T01:30:45.076227-0600 2021-03-29T01:17:32.665059-0600
47.299 20243 0 0 0 26434530287 0 0 1061 incomplete 15h 519007'76475 525954:450424 [11,10,6]p11 [11,10,6]p11 2021-04-01T22:21:20.156918-0600 2021-04-01T22:21:20.156918-0600
47.29a 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9378 [19,17,7]p19 [19,17,7]p19 2021-04-01T02:42:39.410210-0600 2021-03-30T19:35:55.569945-0600
47.2a5 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5730 [6,27,0]p6 [6,27,0]p6 2021-03-30T19:31:20.349618-0600 2021-03-24T01:36:27.398025-0600
47.2a7 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9446 [30,12,27]p30 [30,12,27]p30 2021-03-30T20:54:00.184154-0600 2021-03-27T21:32:01.793734-0600
47.2a9 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4987 [7,10,13]p7 [7,10,13]p7 2021-03-31T23:14:24.960039-0600 2021-03-25T20:58:47.775813-0600
47.2ac 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4969 [9,19,15]p9 [9,19,15]p9 2021-03-31T03:35:42.725346-0600 2021-03-28T02:48:55.814090-0600
47.2ae 181 0 0 0 281346048 0 0 1001 incomplete 15h 518999'82138 525954:4803 [3,4,27]p3 [3,4,27]p3 2021-04-01T01:44:46.662082-0600 2021-03-29T01:04:32.733476-0600
47.2af 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4950 [30,19,2]p30 [30,19,2]p30 2021-03-28T03:26:31.969098-0600 2021-03-28T03:26:31.969098-0600
47.2b0 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4994 [30,12,28]p30 [30,12,28]p30 2021-03-29T02:18:12.708308-0600 2021-03-29T02:18:12.708308-0600
47.2b3 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4921 [27,12,6]p27 [27,12,6]p27 2021-03-31T20:07:02.402478-0600 2021-03-31T20:07:02.402478-0600
47.2b8 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9368 [19,17,30]p19 [19,17,30]p19 2021-03-31T19:33:19.660717-0600 2021-03-31T19:33:19.660717-0600
47.2bc 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5035 [8,7,3]p8 [8,7,3]p8 2021-04-01T21:42:57.873950-0600 2021-03-28T01:00:02.780856-0600
47.2bf 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4951 [9,23,30]p9 [9,23,30]p9 2021-03-30T21:42:50.726301-0600 2021-03-30T21:42:50.726301-0600
47.2c6 20213 0 0 0 26175216381 0 0 1207 incomplete 15h 519041'99538 525954:440033 [18,8,9]p18 [18,8,9]p18 2021-03-30T21:16:24.525557-0600 2021-03-25T01:40:32.080408-0600
47.2d7 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4867 [10,28,7]p10 [10,28,7]p10 2021-03-29T01:04:27.951777-0600 2021-03-27T00:59:42.253682-0600
47.2db 20528 0 0 0 26356361520 0 0 1047 incomplete 15h 518995'81295 525954:448493 [4,6,8]p4 [4,6,8]p4 2021-03-31T23:09:23.616365-0600 2021-03-25T20:51:50.728325-0600
47.2e2 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5025 [8,7,16]p8 [8,7,16]p8 2021-03-31T19:23:39.409829-0600 2021-03-31T19:23:39.409829-0600
47.2e6 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4851 [19,10,15]p19 [19,10,15]p19 2021-03-30T20:02:05.088673-0600 2021-03-24T22:49:16.201562-0600
47.2e8 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4872 [19,10,28]p19 [19,10,28]p19 2021-03-31T22:45:53.015540-0600 2021-03-25T00:16:33.668694-0600
47.2ea 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4960 [8,30,16]p8 [8,30,16]p8 2021-03-28T20:45:30.210125-0600 2021-03-26T20:23:09.951864-0600
47.2ec 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4984 [27,28,8]p27 [27,28,8]p27 2021-03-30T21:35:59.692780-0600 2021-03-24T03:00:41.506393-0600
47.2f1 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9269 [7,9,5]p7 [7,9,5]p7 2021-04-01T21:01:19.688196-0600 2021-03-26T02:00:11.864717-0600
47.2f6 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4933 [27,18,8]p27 [27,18,8]p27 2021-03-31T20:48:35.821193-0600 2021-03-31T20:48:35.821193-0600
47.2fe 19879 0 0 0 26213307805 0 0 1156 incomplete 15h 519007'105685 525954:465852 [14,9,27]p14 [14,9,27]p14 2021-04-02T01:24:40.789845-0600 2021-03-29T00:14:39.375886-0600
47.300 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9306 [10,27,1]p10 [10,27,1]p10 2021-04-01T01:20:48.960507-0600 2021-04-01T01:20:48.960507-0600
47.301 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4966 [30,13,6]p30 [30,13,6]p30 2021-03-31T19:27:53.631542-0600 2021-03-31T19:27:53.631542-0600
47.303 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9309 [7,8,19]p7 [7,8,19]p7 2021-03-27T22:54:34.374295-0600 2021-03-27T22:54:34.374295-0600
47.313 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9312 [19,15,27]p19 [19,15,27]p19 2021-03-31T02:38:46.386103-0600 2021-03-28T22:53:06.056137-0600
47.319 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9329 [19,23,7]p19 [19,23,7]p19 2021-03-31T03:23:41.075244-0600 2021-03-29T02:30:42.413103-0600
47.31b 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9613 [8,6,9]p8 [8,6,9]p8 2021-04-01T03:32:12.992584-0600 2021-04-01T03:32:12.992584-0600
47.31f 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9553 [30,8,16]p30 [30,8,16]p30 2021-04-01T01:29:29.944359-0600 2021-03-27T03:45:30.257742-0600
47.321 919 0 0 0 1281605632 0 0 975 incomplete 15h 518995'79681 525954:9354 [3,10,11]p3 [3,10,11]p3 2021-04-01T19:28:50.975361-0600 2021-03-29T02:51:30.163971-0600
47.324 20085 0 0 0 26208362821 0 0 1053 incomplete 15h 518995'75760 525954:414870 [11,7,6]p11 [11,7,6]p11 2021-04-01T02:42:58.353856-0600 2021-03-28T02:03:43.936527-0600
47.331 20273 0 0 0 26509287480 0 0 926 incomplete 15h 518995'86971 525954:456172 [0,27,19]p0 [0,27,19]p0 2021-03-31T01:52:35.954752-0600 2021-03-31T01:52:35.954752-0600
47.333 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9369 [19,17,9]p19 [19,17,9]p19 2021-03-31T01:03:47.041056-0600 2021-03-27T23:24:12.558006-0600
47.33b 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9494 [30,6,5]p30 [30,6,5]p30 2021-04-01T19:55:32.787705-0600 2021-04-01T19:55:32.787705-0600
47.33e 19708 0 0 0 26064193992 0 0 1173 incomplete 15h 519031'102317 525954:471871 [12,19,7]p12 [12,19,7]p12 2021-04-01T02:59:13.329198-0600 2021-03-28T22:22:24.814456-0600
47.33f 20111 0 0 0 26332720024 0 0 1082 incomplete 15h 518993'80045 525954:495523 [11,10,27]p11 [11,10,27]p11 2021-04-01T22:08:56.745299-0600 2021-03-28T21:14:21.033329-0600
47.344 20379 0 0 0 26324431123 0 0 1039 incomplete 15h 519000'81486 525954:399504 [4,27,7]p4 [4,27,7]p4 2021-03-30T19:52:04.744052-0600 2021-03-27T23:41:09.245331-0600
47.34a 20414 0 0 0 26484594081 0 0 979 incomplete 15h 518997'84141 525954:493886 [0,9,30]p0 [0,9,30]p0 2021-03-30T23:19:17.569241-0600 2021-03-30T23:19:17.569241-0600
47.354 20158 0 0 0 26717629885 0 0 1044 incomplete 15h 519000'81790 525954:465269 [0,7,10]p0 [0,7,10]p0 2021-04-01T20:49:59.994389-0600 2021-03-27T00:48:18.566733-0600
47.359 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5012 [30,9,3]p30 [30,9,3]p30 2021-04-01T03:50:37.853429-0600 2021-03-29T01:36:21.909571-0600
47.35a 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4956 [30,27,28]p30 [30,27,28]p30 2021-03-28T03:19:21.985850-0600 2021-03-28T03:19:21.985850-0600
47.35c 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5680 [8,7,0]p8 [8,7,0]p8 2021-03-27T21:49:11.236313-0600 2021-03-22T03:06:57.805537-0600
47.361 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9417 [27,18,9]p27 [27,18,9]p27 2021-03-31T20:58:20.379750-0600 2021-03-31T20:58:20.379750-0600
47.363 19948 0 0 0 26453104637 0 0 1173 incomplete 15h 519007'100726 525954:583973 [18,19,27]p18 [18,19,27]p18 2021-04-01T03:02:28.548942-0600 2021-03-27T19:05:24.053031-0600
47.36b 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4908 [8,6,14]p8 [8,6,14]p8 2021-04-01T01:56:45.818815-0600 2021-04-01T01:56:45.818815-0600
47.374 19684 0 0 0 26272328522 0 0 1075 incomplete 15h 519007'99767 525954:455557 [13,6,19]p13 [13,6,19]p13 2021-04-01T20:16:11.852262-0600 2021-03-29T02:06:39.348197-0600
47.38f 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9437 [27,3,9]p27 [27,3,9]p27 2021-03-31T21:19:36.051615-0600 2021-03-31T21:19:36.051615-0600
47.391 20183 0 0 0 26174629240 0 0 968 incomplete 15h 519007'83479 525954:466254 [2,7,19]p2 [2,7,19]p2 2021-03-31T23:01:23.085327-0600 2021-03-25T03:13:24.843609-0600
47.395 20038 0 0 0 26205126126 0 0 1195 incomplete 15h 519007'105071 525954:415806 [18,9,6]p18 [18,9,6]p18 2021-04-01T00:30:15.873123-0600 2021-03-28T19:06:44.420250-0600
47.396 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4841 [10,27,11]p10 [10,27,11]p10 2021-04-01T03:08:00.801108-0600 2021-04-01T03:08:00.801108-0600
47.39d 20191 0 0 0 26785005771 0 0 1174 incomplete 15h 519000'97501 525954:435521 [18,8,30]p18 [18,8,30]p18 2021-04-01T03:19:33.193183-0600 2021-03-28T21:51:08.918743-0600
47.3a0 19704 0 0 0 25843237371 0 0 1232 incomplete 15h 519000'98109 525954:472226 [17,6,12]p17 [17,6,12]p17 2021-03-31T20:01:30.219341-0600 2021-03-31T20:01:30.219341-0600
47.3a2 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5781 [8,0,30]p8 [8,0,30]p8 2021-03-28T00:48:35.353609-0600 2021-03-23T21:36:48.891134-0600
47.3a5 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4811 [10,15,27]p10 [10,15,27]p10 2021-03-29T00:02:18.249404-0600 2021-03-22T01:49:34.329955-0600
47.3a6 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:9313 [10,8,1]p10 [10,8,1]p10 2021-04-01T00:34:19.181487-0600 2021-03-25T22:14:50.025036-0600
47.3a9 19781 0 0 0 26121293869 0 0 1059 incomplete 15h 518995'78798 525954:461410 [4,6,7]p4 [4,6,7]p4 2021-04-01T02:24:45.251229-0600 2021-03-26T21:16:30.559792-0600
47.3ac 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5023 [27,6,3]p27 [27,6,3]p27 2021-04-01T00:59:44.905040-0600 2021-03-28T22:17:46.812600-0600
47.3b9 20158 0 0 0 26309188882 0 0 1145 incomplete 15h 518991'97438 525954:446171 [23,27,19]p23 [23,27,19]p23 2021-04-01T22:45:24.442537-0600 2021-03-27T22:39:30.495374-0600
47.3bc 19965 0 0 0 26312334005 0 0 1073 incomplete 15h 519025'107166 525954:422797 [28,7,27]p28 [28,7,27]p28 2021-04-01T01:30:47.098431-0600 2021-03-28T02:54:00.932323-0600
47.3c4 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4925 [27,30,16]p27 [27,30,16]p27 2021-04-02T00:12:19.452532-0600 2021-03-29T22:03:48.403233-0600
47.3ca 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5513 [19,0,6]p19 [19,0,6]p19 2021-04-01T01:20:59.985905-0600 2021-04-01T01:20:59.985905-0600
47.3cc 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4870 [7,6,11]p7 [7,6,11]p7 2021-03-31T19:47:28.264857-0600 2021-03-25T20:46:06.383941-0600
47.3d0 19937 0 0 0 26049861620 0 0 1024 incomplete 15h 518993'84837 525954:452713 [0,8,9]p0 [0,8,9]p0 2021-03-31T19:55:26.225437-0600 2021-03-25T20:28:03.780821-0600
47.3d3 20339 0 0 0 25850714832 0 0 1095 incomplete 15h 518995'74885 525954:444834 [33,6,10]p33 [33,6,10]p33 2021-04-01T20:54:11.410821-0600 2021-03-26T21:28:26.184263-0600
47.3d4 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5723 [6,19,13]p6 [6,19,13]p6 2021-03-28T02:04:07.939796-0600 2021-03-25T19:59:12.362169-0600
47.3d7 20231 0 0 0 26678761009 0 0 1113 incomplete 15h 519000'92790 525954:513730 [1,9,30]p1 [1,9,30]p1 2021-03-30T23:25:38.089760-0600 2021-03-30T23:25:38.089760-0600
47.3dd 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4955 [7,5,6]p7 [7,5,6]p7 2021-03-29T02:45:38.444232-0600 2021-03-27T02:03:11.840141-0600
47.3de 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:5030 [7,17,6]p7 [7,17,6]p7 2021-03-30T19:51:55.985747-0600 2021-03-30T19:51:55.985747-0600
47.3e5 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4928 [9,2,8]p9 [9,2,8]p9 2021-03-30T20:23:36.528228-0600 2021-03-27T00:34:40.483972-0600
47.3f0 20197 0 0 0 26031954397 0 0 1113 incomplete 15h 518995'78417 525954:617837 [11,6,14]p11 [11,6,14]p11 2021-03-30T21:24:11.688564-0600 2021-03-28T00:18:24.404007-0600
47.3f2 20264 0 0 0 26822178361 0 0 1031 incomplete 15h 518997'84027 525954:561034 [1,10,30]p1 [1,10,30]p1 2021-03-30T21:30:04.470335-0600 2021-03-27T19:23:33.541545-0600
47.3fb 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4869 [10,1,7]p10 [10,1,7]p10 2021-03-31T23:40:26.909973-0600 2021-03-28T02:33:30.185498-0600
47.3ff 0 0 0 0 0 0 0 0 incomplete 15h 0'0 525954:4882 [7,9,2]p7 [7,9,2]p7 2021-03-31T23:15:02.476433-0600 2021-03-27T23:34:35.724944-0600
* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details.
HEALTH_ERR 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal'; noout flag(s) set; 1 scrub errors; Reduced data availability: 218 pgs inactive, 218 pgs incomplete; Possible data damage: 1 pg inconsistent; 267 pgs not deep-scrubbed in time; 37 pgs not scrubbed in time; 116 slow ops, oldest one blocked for 56681 sec, daemons [osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.18]... have slow ops.; too many PGs per OSD (317 > max 250)
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
mds.server(mds.0): 40 slow metadata IOs are blocked > 30 secs, oldest blocked for 56623 secs
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
mds.server(mds.0): 98 slow requests are blocked > 30 secs
[WRN] MDS_TRIM: 1 MDSs behind on trimming
mds.server(mds.0): Behind on trimming (593/128) max_segments: 128, num_segments: 593
[WRN] MGR_MODULE_DEPENDENCY: Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal'
Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal'
[WRN] OSDMAP_FLAGS: noout flag(s) set
[ERR] OSD_SCRUB_ERRORS: 1 scrub errors
[WRN] PG_AVAILABILITY: Reduced data availability: 218 pgs inactive, 218 pgs incomplete
pg 47.1c5 is incomplete, acting [6,7,1] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.1c8 is stuck inactive since forever, current state incomplete, last acting [27,12,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.1c9 is incomplete, acting [7,12,9] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.206 is incomplete, acting [1,19,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.20b is incomplete, acting [4,6,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.20c is incomplete, acting [7,0,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.20e is incomplete, acting [0,6,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.217 is incomplete, acting [19,8,1] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.21d is incomplete, acting [19,2,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.21f is incomplete, acting [27,1,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.222 is incomplete, acting [9,10,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.227 is incomplete, acting [7,8,13] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.22b is incomplete, acting [3,9,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.22d is incomplete, acting [28,10,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.22e is incomplete, acting [18,19,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.230 is incomplete, acting [4,30,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.234 is incomplete, acting [10,17,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.243 is incomplete, acting [18,8,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.244 is incomplete, acting [30,0,10] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.24e is incomplete, acting [12,7,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.254 is incomplete, acting [7,23,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.255 is incomplete, acting [33,30,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.258 is incomplete, acting [6,0,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.259 is incomplete, acting [8,30,5] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.25d is incomplete, acting [27,7,0] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.25e is incomplete, acting [28,8,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.261 is incomplete, acting [30,18,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.266 is incomplete, acting [8,16,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.26a is incomplete, acting [7,16,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.27a is incomplete, acting [16,9,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.27f is incomplete, acting [27,5,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.280 is incomplete, acting [27,10,2] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.281 is incomplete, acting [10,17,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.284 is incomplete, acting [28,8,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.286 is incomplete, acting [30,7,4] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.28e is incomplete, acting [10,7,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.290 is incomplete, acting [3,8,9] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.298 is incomplete, acting [18,10,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.299 is incomplete, acting [11,10,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.29a is incomplete, acting [19,17,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2a5 is incomplete, acting [6,27,0] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2a7 is incomplete, acting [30,12,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2a9 is incomplete, acting [7,10,13] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2ac is incomplete, acting [9,19,15] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2ae is incomplete, acting [3,4,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2af is incomplete, acting [30,19,2] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2b0 is incomplete, acting [30,12,28] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2b3 is incomplete, acting [27,12,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2b8 is incomplete, acting [19,17,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2bc is incomplete, acting [8,7,3] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2bf is incomplete, acting [9,23,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 21.280 is active+clean+inconsistent, acting [3,14,7]
[WRN] PG_NOT_DEEP_SCRUBBED: 267 pgs not deep-scrubbed in time
pg 47.286 not deep-scrubbed since 2021-03-25T22:53:15.065688-0600
pg 21.2bb not deep-scrubbed since 2021-03-26T20:24:27.616173-0600
pg 47.280 not deep-scrubbed since 2021-03-27T21:20:30.399908-0600
pg 47.28e not deep-scrubbed since 2021-03-27T03:44:01.214168-0600
pg 47.28b not deep-scrubbed since 2021-03-27T22:45:21.260059-0600
pg 47.290 not deep-scrubbed since 2021-03-27T03:05:59.228923-0600
pg 47.2a5 not deep-scrubbed since 2021-03-24T01:36:27.398025-0600
pg 47.2a7 not deep-scrubbed since 2021-03-27T21:32:01.793734-0600
pg 21.29c not deep-scrubbed since 2021-03-27T03:08:53.853499-0600
pg 47.2ad not deep-scrubbed since 2021-03-27T22:29:02.738458-0600
pg 47.2a9 not deep-scrubbed since 2021-03-25T20:58:47.775813-0600
pg 47.24d not deep-scrubbed since 2021-03-27T21:15:10.218831-0600
pg 21.274 not deep-scrubbed since 2021-03-27T02:20:16.591959-0600
pg 21.271 not deep-scrubbed since 2021-03-27T23:09:24.527503-0600
pg 47.255 not deep-scrubbed since 2021-03-25T20:14:25.340283-0600
pg 47.25e not deep-scrubbed since 2021-03-27T00:12:19.948412-0600
pg 47.259 not deep-scrubbed since 2021-03-21T01:05:52.666420-0600
pg 47.258 not deep-scrubbed since 2021-03-27T20:33:04.243319-0600
pg 21.262 not deep-scrubbed since 2021-03-27T01:58:50.415874-0600
pg 47.266 not deep-scrubbed since 2021-03-24T23:14:49.566588-0600
pg 47.261 not deep-scrubbed since 2021-03-26T20:41:03.399382-0600
pg 47.269 not deep-scrubbed since 2021-03-26T02:15:44.559049-0600
pg 47.26a not deep-scrubbed since 2021-03-26T01:25:59.521139-0600
pg 47.276 not deep-scrubbed since 2021-03-27T21:33:07.907478-0600
pg 47.27f not deep-scrubbed since 2021-03-24T23:39:08.357812-0600
pg 21.245 not deep-scrubbed since 2021-03-26T19:17:26.180939-0600
pg 47.206 not deep-scrubbed since 2021-03-24T03:59:12.427505-0600
pg 47.20e not deep-scrubbed since 2021-03-26T02:50:03.752634-0600
pg 47.20b not deep-scrubbed since 2021-03-26T20:19:01.735932-0600
pg 47.215 not deep-scrubbed since 2021-03-27T20:17:49.992450-0600
pg 47.217 not deep-scrubbed since 2021-03-27T23:26:09.896032-0600
pg 47.21d not deep-scrubbed since 2021-03-28T00:04:41.815145-0600
pg 47.21f not deep-scrubbed since 2021-03-26T19:50:30.551545-0600
pg 47.21e not deep-scrubbed since 2021-03-26T01:59:49.637250-0600
pg 47.227 not deep-scrubbed since 2021-03-25T03:39:00.406340-0600
pg 47.222 not deep-scrubbed since 2021-03-26T21:23:24.045585-0600
pg 21.218 not deep-scrubbed since 2021-03-27T03:16:35.287831-0600
pg 47.22b not deep-scrubbed since 2021-03-26T03:42:26.835052-0600
pg 47.230 not deep-scrubbed since 2021-03-27T02:49:41.789992-0600
pg 47.1c4 not deep-scrubbed since 2021-03-27T23:07:14.725369-0600
pg 21.1fd not deep-scrubbed since 2021-03-27T01:31:13.297653-0600
pg 47.1cb not deep-scrubbed since 2021-03-27T19:06:05.459813-0600
pg 21.1ef not deep-scrubbed since 2021-03-26T20:29:09.801484-0600
pg 47.1d7 not deep-scrubbed since 2021-03-28T00:01:02.836931-0600
pg 47.1da not deep-scrubbed since 2021-03-26T23:50:21.899786-0600
pg 21.1df not deep-scrubbed since 2021-03-26T01:34:56.338349-0600
pg 47.1e1 not deep-scrubbed since 2021-03-27T19:50:13.164921-0600
pg 47.1e0 not deep-scrubbed since 2021-03-27T01:38:05.517845-0600
pg 47.1f7 not deep-scrubbed since 2021-03-26T23:42:36.370211-0600
pg 47.1f9 not deep-scrubbed since 2021-03-24T23:14:41.816716-0600
217 more pgs...
[WRN] PG_NOT_SCRUBBED: 37 pgs not scrubbed in time
pg 47.284 not scrubbed since 2021-03-28T21:43:34.141833-0600
pg 47.28e not scrubbed since 2021-03-29T00:36:47.386296-0600
pg 47.2af not scrubbed since 2021-03-28T03:26:31.969098-0600
pg 47.2b0 not scrubbed since 2021-03-29T02:18:12.708308-0600
pg 47.254 not scrubbed since 2021-03-28T19:41:04.896055-0600
pg 47.25e not scrubbed since 2021-03-29T01:32:11.012945-0600
pg 47.259 not scrubbed since 2021-03-27T22:39:37.697021-0600
pg 47.266 not scrubbed since 2021-03-28T03:35:27.190995-0600
pg 47.227 not scrubbed since 2021-03-28T01:16:28.930572-0600
pg 47.234 not scrubbed since 2021-03-28T21:24:34.834498-0600
pg 47.1c5 not scrubbed since 2021-03-29T00:09:05.664124-0600
pg 47.1e0 not scrubbed since 2021-03-28T23:57:25.436078-0600
pg 47.1f9 not scrubbed since 2021-03-29T02:19:18.367248-0600
pg 47.181 not scrubbed since 2021-03-29T01:55:17.182313-0600
pg 47.ac not scrubbed since 2021-03-28T03:01:26.639534-0600
pg 47.b2 not scrubbed since 2021-03-28T19:00:52.306016-0600
pg 47.4f not scrubbed since 2021-03-28T19:47:56.332962-0600
pg 47.6b not scrubbed since 2021-03-28T19:08:20.531682-0600
pg 47.7d not scrubbed since 2021-03-29T00:47:33.754276-0600
pg 47.39 not scrubbed since 2021-03-28T21:02:51.726250-0600
pg 47.38 not scrubbed since 2021-03-28T21:52:56.209769-0600
pg 47.81 not scrubbed since 2021-03-29T03:10:09.500658-0600
pg 47.f8 not scrubbed since 2021-03-28T00:24:32.390309-0600
pg 47.d2 not scrubbed since 2021-03-28T19:01:14.191464-0600
pg 47.c1 not scrubbed since 2021-03-28T20:18:17.342293-0600
pg 47.12e not scrubbed since 2021-03-28T21:08:38.252809-0600
pg 47.121 not scrubbed since 2021-03-28T03:21:15.443939-0600
pg 47.11e not scrubbed since 2021-03-29T02:51:23.634263-0600
pg 47.2ea not scrubbed since 2021-03-28T20:45:30.210125-0600
pg 47.2d7 not scrubbed since 2021-03-29T01:04:27.951777-0600
pg 47.303 not scrubbed since 2021-03-27T22:54:34.374295-0600
pg 47.35a not scrubbed since 2021-03-28T03:19:21.985850-0600
pg 47.35c not scrubbed since 2021-03-27T21:49:11.236313-0600
pg 47.3a2 not scrubbed since 2021-03-28T00:48:35.353609-0600
pg 47.3a5 not scrubbed since 2021-03-29T00:02:18.249404-0600
pg 47.3dd not scrubbed since 2021-03-29T02:45:38.444232-0600
pg 47.3d4 not scrubbed since 2021-03-28T02:04:07.939796-0600
[WRN] SLOW_OPS: 116 slow ops, oldest one blocked for 56681 sec, daemons [osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.18]... have slow ops.
[WRN] TOO_MANY_PGS: too many PGs per OSD (317 > max 250)
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
