Absolutely!
Attached the files, they're not duplicate, but revised (as I tidied up
what I could to make things easier)
> Correct me if I'm wrong, but you are willing to throw away all of the data on this pool?
Correct, if push comes to shove, I accept that data-loss is probable.
If I can manage to save the data, I would definitely be okay with that
too though.
Still learning to program, but know python quite well. I am going to
push off on a script to clean up per your previously noted steps in
the language I know! But will hold off on unlinking everything for the
moment.
Thank you again for your time, your help has already been invaluable to me.
Joshua
Joshua West
President
403-456-0072
CAYK.ca
On Fri, Apr 9, 2021 at 7:03 AM Michael Thomas <wart@xxxxxxxxxxx> wrote:
>
> Hi Joshua,
>
> I'll dig into this output a bit more later, but here are my thoughts
> right now. I'll preface this by saying that I've never had to clean up
> from unrecoverable incomplete PGs, so some of what I suggest may not
> work/apply or be the ideal fix in your case.
>
> Correct me if I'm wrong, but you are willing to throw away all of the
> data on this pool? This should make it easier because we don't have to
> worry about recovering any lost data.
>
> If this is the case, then I think the general strategy would be:
>
> 1) Identify and remove any files/directories in cephfs that are located
> on this pool (based on ceph.file.layout.pool=claypool and
> ceph.dir.layout.pool=claypool). Use 'unlink' instead of 'rm' to remove
> the files; it should be less prone to hanging.
>
> 2) Wait a bit for ceph to clean up any unreferenced objects. Watch the
> output of 'ceph df' to see how many objects are listed for the pool.
>
> 3) Use 'rados -p claypool ls' to identify the remaining objects. Use
> the OID identifier to calculate the inode number of each file, then
> search cephfs to identify which files these belong to. I would expect
> it would be none, as you already deleted the files in step 1.
>
> 4) With nothing in the cephfs metadata referring to the objects anymore,
> it should be safe to remove them with 'rados -p rm'.
>
> 5) Remove the now-empty pool from cephfs
>
> 6) Remove the now-empty pool from ceph
>
> Can you also include the output of 'ceph df'?
>
> --Mike
>
> On 4/9/21 7:31 AM, Joshua West wrote:
> > Thank you Mike!
> >
> > This is honestly a way more detailed reply than I was expecting.
> > You've equipped me with new tools to work with. Thank you!
> >
> > I don't actually have any unfound pgs... only "incomplete" ones, which
> > limits the usefulness of:
> > `grep recovery_unfound`
> > `ceph pg $pg list_unfound`
> > `ceph pg $pg mark_unfound_lost delete`
> >
> > I don't seem to see equivalent commands for incomplete pgs, save for
> > grep of course.
> >
> > This does make me slightly more hopeful that recovery might be
> > possible if the pgs are incomplete and stuck, but not unfound..? Not
> > going to get my hopes too high.
> >
> > Going to attach a few items just to keep from bugging me, if anyone
> > can take a glance, it would be appreciated.
> >
> > In the meantime, in the absence of the above commands, what's the best
> > way to clean this up under the assumption that the data is lost?
> >
> > ~Joshua
> >
> >
> > Joshua West
> > President
> > 403-456-0072
> > CAYK.ca
> >
> >
> > On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas <wart@xxxxxxxxxxx> wrote:
> >>
> >> Hi Joshua,
> >>
> >> I have had a similar issue three different times on one of my cephfs
> >> pools (15.2.10). The first time this happened I had lost some OSDs. In
> >> all cases I ended up with degraded PGs with unfound objects that could
> >> not be recovered.
> >>
> >> Here's how I recovered from the situation. Note that this will
> >> permanently remove the affected files from ceph. Restoring them from
> >> backup is an excercise left to the reader.
> >>
> >> * Make a list of the affected PGs:
> >> ceph pg dump_stuck | grep recovery_unfound > pg.txt
> >>
> >> * Make a list of the affected objects (OIDs):
> >> cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg
> >> $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' >
> >> oid.txt
> >>
> >> * Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and
> >> put the results in a file called 'inum.txt'
> >>
> >> * On a ceph client, find the files that correspond to the affected inodes:
> >> cat inum.txt | while read inum ; do echo -n "${inum} " ; find
> >> /ceph/frames/O3/raw -inum ${inum} ; done > files.txt
> >>
> >> * It may be helpful to put this table of PG, OID, inum, and files into a
> >> spreadsheet to keep track of what's been done.
> >>
> >> * On the ceph client, use 'unlink' to remove the files from the
> >> filesystem. Do not use 'rm', as it will hang while calling 'stat()' on
> >> each file. Even unlink may hang when you first try it. If it does
> >> hang, do the following to get it unstuck:
> >> - Reboot the client
> >> - Restart each mon and the mgr. I rebooted each mon/mgr, but it may
> >> be sufficient to restart the services without a reboot.
> >> - Try using 'unlink' again
> >>
> >> * After all of the affected files have been removed, go through the list
> >> of PGs and remove the unfound OIDs:
> >> ceph pg $pgid mark_unfound_lost delete
> >>
> >> ...or if you're feeling brave, delete them all at once:
> >> cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg
> >> $pg mark_unfound_lost delete ; done
> >>
> >> * Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
> >>
> >> * Restore the deleted files from backup, or decide that you don't care
> >> about them and don't do anything
> >> This procedure lets you fix the problem without deleting the affected
> >> pool. To be honest, the first time it happened, my solution was to
> >> first copy all of the data off of the affected pool and onto a new pool.
> >> I later found this to be unnecessary. But if you want to pursue this,
> >> here's what I suggest:
> >>
> >> * Follow the steps above to get rid of the affected files. I feel this
> >> should still be done even though you don't care about saving the data,
> >> to prevent corruption in the cephfs metadata.
> >>
> >> * Go through the entire filesystem and look for:
> >> - files that are located on the pool (ceph.file.layout.pool = $pool_name)
> >> - directories that are set to write files to the pool
> >> (ceph.dir.layout.pool = $pool_name)
> >>
> >> * After you confirm that no files or directories are pointing at the
> >> pool anymore, run 'ceph df' and look at the number of objects in the
> >> pool. Ideally, it would be zero. But more than likely it isn't. This
> >> could be a simple mismatch in the object count in cephfs (harmless), or
> >> there could be clients with open filehandles on files that have been
> >> removed. such objects will still appear in the rados listing of the
> >> pool[1]:
> >> rados -p $pool_name ls
> >> for obj in $(rados -p $pool_name ls); do echo $obj; rados -p
> >> $pool_name getxattr parent | strings; done
> >>
> >> * To check for clients with access to these stray objects, dump the mds
> >> cache:
> >> ceph daemon mds.ceph1 dump cache /tmp/cache.txt
> >>
> >> * Look for lines that refer to the stray objects, like this:
> >> [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537
> >> s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863
> >> 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 |
> >> caps=1 authpin=0 0x563a7e52a000]
> >>
> >> * The 'caps' field in the output above contains the client session id
> >> (eg 9541437). Search the MDS for sessions that match to identify the
> >> client:
> >> ceph daemon mds.ceph1 session ls > session.txt
> >> Search through 'session.txt' for matching entries. This will give
> >> you the IP address of the client:
> >> "id": 9541437,
> >> "entity": {
> >> "name": {
> >> "type": "client",
> >> "num": 9541437
> >> },
> >> "addr": {
> >> "type": "v1",
> >> "addr": "10.13.5.48:0",
> >> "nonce": 2011077845
> >> }
> >> },
> >>
> >> * Restart the client's connection to ceph to get it to drop the cap. I
> >> did this by rebooting the client, but there may be gentler ways to do it.
> >>
> >> * Once you've done this clean up, it should be safe to remove the pool
> >> from cephfs:
> >> ceph fs rm_data_pool $fs_name $pool_name
> >>
> >> * Once the pool has been detached from cephfs, you can remove it from
> >> ceph altogether:
> >> ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it
> >>
> >> Hope this helps,
> >>
> >> --Mike
> >> [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html
> >>
> >>
> >>
> >> On 4/8/21 5:41 PM, Joshua West wrote:
> >>> Hey everyone.
> >>>
> >>> Inside of cephfs, I have a directory which I setup a directory layout
> >>> field to use an erasure coded (CLAY) pool, specific to the task. The
> >>> rest of my cephfs is using normal replication.
> >>>
> >>> Fast forward some time, and the EC directory has been used pretty
> >>> extensively, and through some bad luck and poor timing, ~200pgs are in
> >>> an incomplete state, and the OSDs are completely gone and
> >>> unrecoverable. (Specifically OSD 31 and 34, not that it matters at
> >>> this point)
> >>>
> >>> # ceph pg ls incomplete --> is attached for reference.
> >>>
> >>> Fortunately, it's primarily (only) my on-site backups, and other
> >>> replaceable data inside of
> >>>
> >>> I tried for a few days to recover the PGs:
> >>> - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs)
> >>> - Deep Scrub
> >>> - osd_find_best_info_ignore_history_les = true (`pg query` was
> >>> showing related error)
> >>> etc.
> >>>
> >>> I've finally just accepted this pool to be a lesson learned, and want
> >>> to get the rest of my cephfs back to normal.
> >>>
> >>> My questions:
> >>>
> >>> -- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs
> >>> with 0 objects
> >>> -- Deleting the pool seems like an appropriate step, but as I am
> >>> using an xattr within cephfs, which is otherwise on another pool, I am
> >>> not confident that this approach is safe?
> >>> -- cephfs currently blocks when attemping to impact every third file
> >>> in the EC directory. Once I delete the pool, how will I remove the
> >>> files if even `rm` is blocking?
> >>>
> >>> Thank you for your time,
> >>>
> >>> Joshua West
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>
> >>
>
HEALTH_ERR 1 MDSs report slow metadata IOs; 1 MDSs behind on trimming; Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal'; noout flag(s) set; 1 scrub errors; Reduced data availability: 218 pgs inactive, 218 pgs incomplete; Possible data damage: 1 pg inconsistent; 245 pgs not deep-scrubbed in time; 37 pgs not scrubbed in time; 5 daemons have recently crashed; too many PGs per OSD (317 > max 250)
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
mds.dl380g7(mds.0): 40 slow metadata IOs are blocked > 30 secs, oldest blocked for 11581 secs
[WRN] MDS_TRIM: 1 MDSs behind on trimming
mds.dl380g7(mds.0): Behind on trimming (594/128) max_segments: 128, num_segments: 594
[WRN] MGR_MODULE_DEPENDENCY: Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal'
Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal'
[WRN] OSDMAP_FLAGS: noout flag(s) set
[ERR] OSD_SCRUB_ERRORS: 1 scrub errors
[WRN] PG_AVAILABILITY: Reduced data availability: 218 pgs inactive, 218 pgs incomplete
pg 47.1c5 is incomplete, acting [6,7,1] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.1c8 is stuck inactive since forever, current state incomplete, last acting [27,12,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.1c9 is incomplete, acting [7,12,9] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.206 is incomplete, acting [1,19,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.20b is incomplete, acting [4,6,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.20c is incomplete, acting [7,0,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.20e is incomplete, acting [0,6,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.217 is incomplete, acting [19,8,1] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.21d is incomplete, acting [19,2,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.21f is incomplete, acting [27,1,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.222 is incomplete, acting [9,10,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.227 is incomplete, acting [7,8,13] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.22b is incomplete, acting [3,9,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.22d is incomplete, acting [28,10,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.22e is incomplete, acting [18,19,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.230 is incomplete, acting [4,30,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.234 is incomplete, acting [10,17,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.243 is incomplete, acting [18,8,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.244 is incomplete, acting [30,0,10] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.24e is incomplete, acting [12,7,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.254 is incomplete, acting [7,23,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.255 is incomplete, acting [33,30,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.258 is incomplete, acting [6,0,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.259 is incomplete, acting [8,30,5] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.25d is incomplete, acting [27,7,0] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.25e is incomplete, acting [28,8,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.261 is incomplete, acting [30,18,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.266 is incomplete, acting [8,16,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.26a is incomplete, acting [7,16,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.27a is incomplete, acting [16,9,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.27f is incomplete, acting [27,5,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.280 is incomplete, acting [27,10,2] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.281 is incomplete, acting [10,17,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.284 is incomplete, acting [28,8,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.286 is incomplete, acting [30,7,4] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.28e is incomplete, acting [10,7,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.290 is incomplete, acting [3,8,9] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.298 is incomplete, acting [18,10,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.299 is incomplete, acting [11,10,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.29a is incomplete, acting [19,17,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2a5 is incomplete, acting [6,27,0] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2a7 is incomplete, acting [30,12,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2a9 is incomplete, acting [7,10,13] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2ac is incomplete, acting [9,19,15] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2ae is incomplete, acting [3,4,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2af is incomplete, acting [30,19,2] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2b0 is incomplete, acting [30,12,28] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2b3 is incomplete, acting [27,12,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2b8 is incomplete, acting [19,17,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2bc is incomplete, acting [8,7,3] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
pg 47.2bf is incomplete, acting [9,23,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete')
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 21.280 is active+clean+inconsistent, acting [3,14,7]
[WRN] PG_NOT_DEEP_SCRUBBED: 245 pgs not deep-scrubbed in time
pg 47.286 not deep-scrubbed since 2021-03-25T22:53:15.065688-0600
pg 21.2bb not deep-scrubbed since 2021-03-26T20:24:27.616173-0600
pg 47.280 not deep-scrubbed since 2021-03-27T21:20:30.399908-0600
pg 47.28e not deep-scrubbed since 2021-03-27T03:44:01.214168-0600
pg 47.28b not deep-scrubbed since 2021-03-27T22:45:21.260059-0600
pg 47.290 not deep-scrubbed since 2021-03-27T03:05:59.228923-0600
pg 47.2a5 not deep-scrubbed since 2021-03-24T01:36:27.398025-0600
pg 47.2a7 not deep-scrubbed since 2021-03-27T21:32:01.793734-0600
pg 21.29c not deep-scrubbed since 2021-03-27T03:08:53.853499-0600
pg 47.2ac not deep-scrubbed since 2021-03-28T02:48:55.814090-0600
pg 47.2af not deep-scrubbed since 2021-03-28T03:26:31.969098-0600
pg 47.2a9 not deep-scrubbed since 2021-03-25T20:58:47.775813-0600
pg 47.2bc not deep-scrubbed since 2021-03-28T01:00:02.780856-0600
pg 21.271 not deep-scrubbed since 2021-03-27T23:09:24.527503-0600
pg 47.255 not deep-scrubbed since 2021-03-25T20:14:25.340283-0600
pg 47.25e not deep-scrubbed since 2021-03-27T00:12:19.948412-0600
pg 47.259 not deep-scrubbed since 2021-03-21T01:05:52.666420-0600
pg 47.258 not deep-scrubbed since 2021-03-27T20:33:04.243319-0600
pg 47.266 not deep-scrubbed since 2021-03-24T23:14:49.566588-0600
pg 47.261 not deep-scrubbed since 2021-03-26T20:41:03.399382-0600
pg 47.26a not deep-scrubbed since 2021-03-26T01:25:59.521139-0600
pg 47.276 not deep-scrubbed since 2021-03-27T21:33:07.907478-0600
pg 47.27f not deep-scrubbed since 2021-03-24T23:39:08.357812-0600
pg 47.206 not deep-scrubbed since 2021-03-24T03:59:12.427505-0600
pg 47.20e not deep-scrubbed since 2021-03-26T02:50:03.752634-0600
pg 47.20b not deep-scrubbed since 2021-03-26T20:19:01.735932-0600
pg 47.217 not deep-scrubbed since 2021-03-27T23:26:09.896032-0600
pg 47.21d not deep-scrubbed since 2021-03-28T00:04:41.815145-0600
pg 47.21f not deep-scrubbed since 2021-03-26T19:50:30.551545-0600
pg 47.227 not deep-scrubbed since 2021-03-25T03:39:00.406340-0600
pg 47.222 not deep-scrubbed since 2021-03-26T21:23:24.045585-0600
pg 21.218 not deep-scrubbed since 2021-03-27T03:16:35.287831-0600
pg 47.22d not deep-scrubbed since 2021-03-28T02:17:35.460478-0600
pg 47.22b not deep-scrubbed since 2021-03-26T03:42:26.835052-0600
pg 47.230 not deep-scrubbed since 2021-03-27T02:49:41.789992-0600
pg 47.1c4 not deep-scrubbed since 2021-03-27T23:07:14.725369-0600
pg 21.1fd not deep-scrubbed since 2021-03-27T01:31:13.297653-0600
pg 47.1cb not deep-scrubbed since 2021-03-27T19:06:05.459813-0600
pg 21.1ef not deep-scrubbed since 2021-03-26T20:29:09.801484-0600
pg 47.1d4 not deep-scrubbed since 2021-03-28T01:27:47.958651-0600
pg 47.1da not deep-scrubbed since 2021-03-26T23:50:21.899786-0600
pg 47.1e6 not deep-scrubbed since 2021-03-28T02:28:18.846086-0600
pg 47.1e1 not deep-scrubbed since 2021-03-27T19:50:13.164921-0600
pg 47.1e0 not deep-scrubbed since 2021-03-27T01:38:05.517845-0600
pg 47.1f7 not deep-scrubbed since 2021-03-26T23:42:36.370211-0600
pg 47.1f1 not deep-scrubbed since 2021-03-28T01:25:46.956359-0600
pg 47.1f9 not deep-scrubbed since 2021-03-24T23:14:41.816716-0600
pg 21.1ab not deep-scrubbed since 2021-03-25T21:50:32.358023-0600
pg 21.199 not deep-scrubbed since 2021-03-26T01:31:52.140717-0600
pg 47.1ae not deep-scrubbed since 2021-03-26T00:39:11.884432-0600
195 more pgs...
[WRN] PG_NOT_SCRUBBED: 37 pgs not scrubbed in time
pg 47.284 not scrubbed since 2021-03-28T21:43:34.141833-0600
pg 47.28e not scrubbed since 2021-03-29T00:36:47.386296-0600
pg 47.2af not scrubbed since 2021-03-28T03:26:31.969098-0600
pg 47.2b0 not scrubbed since 2021-03-29T02:18:12.708308-0600
pg 47.254 not scrubbed since 2021-03-28T19:41:04.896055-0600
pg 47.25e not scrubbed since 2021-03-29T01:32:11.012945-0600
pg 47.259 not scrubbed since 2021-03-27T22:39:37.697021-0600
pg 47.266 not scrubbed since 2021-03-28T03:35:27.190995-0600
pg 47.227 not scrubbed since 2021-03-28T01:16:28.930572-0600
pg 47.234 not scrubbed since 2021-03-28T21:24:34.834498-0600
pg 47.1c5 not scrubbed since 2021-03-29T00:09:05.664124-0600
pg 47.1e0 not scrubbed since 2021-03-28T23:57:25.436078-0600
pg 47.1f9 not scrubbed since 2021-03-29T02:19:18.367248-0600
pg 47.181 not scrubbed since 2021-03-29T01:55:17.182313-0600
pg 47.ac not scrubbed since 2021-03-28T03:01:26.639534-0600
pg 47.b2 not scrubbed since 2021-03-28T19:00:52.306016-0600
pg 47.4f not scrubbed since 2021-03-28T19:47:56.332962-0600
pg 47.6b not scrubbed since 2021-03-28T19:08:20.531682-0600
pg 47.7d not scrubbed since 2021-03-29T00:47:33.754276-0600
pg 47.39 not scrubbed since 2021-03-28T21:02:51.726250-0600
pg 47.38 not scrubbed since 2021-03-28T21:52:56.209769-0600
pg 47.81 not scrubbed since 2021-03-29T03:10:09.500658-0600
pg 47.f8 not scrubbed since 2021-03-28T00:24:32.390309-0600
pg 47.d2 not scrubbed since 2021-03-28T19:01:14.191464-0600
pg 47.c1 not scrubbed since 2021-03-28T20:18:17.342293-0600
pg 47.12e not scrubbed since 2021-03-28T21:08:38.252809-0600
pg 47.121 not scrubbed since 2021-03-28T03:21:15.443939-0600
pg 47.11e not scrubbed since 2021-03-29T02:51:23.634263-0600
pg 47.2ea not scrubbed since 2021-03-28T20:45:30.210125-0600
pg 47.2d7 not scrubbed since 2021-03-29T01:04:27.951777-0600
pg 47.303 not scrubbed since 2021-03-27T22:54:34.374295-0600
pg 47.35a not scrubbed since 2021-03-28T03:19:21.985850-0600
pg 47.35c not scrubbed since 2021-03-27T21:49:11.236313-0600
pg 47.3a2 not scrubbed since 2021-03-28T00:48:35.353609-0600
pg 47.3a5 not scrubbed since 2021-03-29T00:02:18.249404-0600
pg 47.3dd not scrubbed since 2021-03-29T02:45:38.444232-0600
pg 47.3d4 not scrubbed since 2021-03-28T02:04:07.939796-0600
[WRN] RECENT_CRASH: 5 daemons have recently crashed
osd.0 crashed on host rd240 at 2021-04-09T12:56:55.565074Z
osd.28 crashed on host server at 2021-04-09T12:56:57.024061Z
osd.15 crashed on host server at 2021-04-09T12:56:57.024074Z
osd.5 crashed on host rd240 at 2021-04-09T12:56:57.024591Z
osd.14 crashed on host server at 2021-04-09T12:56:57.829343Z
[WRN] TOO_MANY_PGS: too many PGs per OSD (317 > max 250)
pool 21 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 526163 lfor 0/488420/488963 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive compression_required_ratio 0.9 pg_num_min 1024 application cephfs
removed_snaps_queue [44b~ec,538~65,59e~2e,5cd~32,600~7,608~31,63a~23,65e~c,66b~27,693~1,695~1,697~1,699~1,69b~1,69e~2,6a1~1,6a3~1,6a5~1,6a7~1,6a9~1,6ab~1,6ad~1,6af~1,6b1~1,6b3~1,6b5~1,6b7~1,6b9~1,6bb~1,6be~2,6c1~1,6c3~1,6c5~1]
pool 22 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 pg_num_target 64 pgp_num_target 64 autoscale_mode on last_change 519055 lfor 0/124427/506237 flags hashpspool,nodelete stripe_width 0 pg_autoscale_bias 4 pg_num_min 64 recovery_priority 5 application cephfs
pool 30 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 525278 lfor 0/7304/327594 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive application ecpool,rbd
pool 31 'device_health_metrics' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 526099 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth
pool 32 '.rgw.root' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 135358 lfor 0/135358/135356 flags hashpspool stripe_width 0 application rgw
pool 33 'default.rgw.log' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 139171 lfor 0/139171/139169 flags hashpspool stripe_width 0 application rgw
pool 34 'default.rgw.control' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 141079 lfor 0/141079/141077 flags hashpspool stripe_width 0 application rgw
pool 35 'default.rgw.meta' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 142067 lfor 0/142067/142065 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw
pool 36 'nfs-ganesha' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 133226 flags hashpspool stripe_width 0 application rbd
pool 47 'claypool' erasure profile claypool size 3 min_size 2 crush_rule 7 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 526163 lfor 0/460696/489203 flags hashpspool,ec_overwrites,selfmanaged_snaps,creating stripe_width 8192 fast_read 1 compression_algorithm lz4 compression_mode aggressive pg_num_min 1024 application cephfs
removed_snaps_queue [129~1,428~10f,538~65,59e~2e,5cd~32,600~7,608~31,63a~23,65e~c,66b~27,693~1,695~1,697~1,699~1,69b~1,69e~2,6a1~1,6a3~1,6a5~1,6a7~1,6a9~1,6ab~1,6ad~1,6af~1,6b1~1,6b3~1,6b5~1,6b7~1,6b9~1,6bb~1,6be~2,6c1~1,6c3~1,6c5~1]
pool 54 'fast' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 524810 lfor 0/509819/509817 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 55 'caykcloud' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 524049 lfor 0/514664/514662 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "incomplete",
"epoch": 525943,
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"info": {
"pgid": "47.0s0",
"last_update": "519000'86682",
"last_complete": "519000'86682",
"log_tail": "518614'85590",
"last_user_version": 86682,
"last_backfill": "MAX",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2e"
},
{
"start": "5ce",
"length": "2"
},
{
"start": "5d1",
"length": "1"
},
{
"start": "5d3",
"length": "1"
},
{
"start": "5d5",
"length": "1"
},
{
"start": "5d7",
"length": "1"
},
{
"start": "5d9",
"length": "1"
},
{
"start": "5db",
"length": "1"
},
{
"start": "5dd",
"length": "1"
},
{
"start": "5df",
"length": "1"
},
{
"start": "5e1",
"length": "1"
},
{
"start": "5e3",
"length": "1"
},
{
"start": "5e5",
"length": "1"
},
{
"start": "5e7",
"length": "1"
},
{
"start": "5e9",
"length": "1"
},
{
"start": "5eb",
"length": "1"
},
{
"start": "5ed",
"length": "1"
},
{
"start": "5ef",
"length": "1"
},
{
"start": "5f1",
"length": "1"
},
{
"start": "5f3",
"length": "1"
}
],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "519000'86682",
"reported_seq": "346239",
"reported_epoch": "525943",
"state": "incomplete",
"last_fresh": "2021-04-08T14:54:57.285506-0600",
"last_change": "2021-04-08T14:38:31.054438-0600",
"last_active": "2021-04-03T10:17:20.578805-0600",
"last_peered": "2021-04-03T10:17:10.504682-0600",
"last_clean": "2021-04-03T09:23:47.953138-0600",
"last_became_active": "2021-04-03T10:17:08.445271-0600",
"last_became_peered": "2021-04-03T10:17:08.445271-0600",
"last_unstale": "2021-04-08T14:54:57.285506-0600",
"last_undegraded": "2021-04-08T14:54:57.285506-0600",
"last_fullsized": "2021-04-08T14:54:57.285506-0600",
"mapping_epoch": 525928,
"log_start": "518614'85590",
"ondisk_log_start": "518614'85590",
"created": 337872,
"last_epoch_clean": 518952,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"log_size": 1092,
"ondisk_log_size": 1092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 26759978094,
"num_objects": 20070,
"num_object_clones": 772,
"num_object_copies": 60210,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 20070,
"num_whiteouts": 652,
"num_read": 13967,
"num_read_kb": 2052182,
"num_write": 45875,
"num_write_kb": 2332074,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 37027,
"num_bytes_recovered": 59477202252,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 519062,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "4(2)",
"pgid": "47.0s2",
"last_update": "519000'86682",
"last_complete": "519000'86682",
"log_tail": "518614'85590",
"last_user_version": 86682,
"last_backfill": "47:00116113:::10001c85d5d.00000032:head",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2e"
},
{
"start": "5ce",
"length": "2"
},
{
"start": "5d1",
"length": "1"
},
{
"start": "5d3",
"length": "1"
},
{
"start": "5d5",
"length": "1"
},
{
"start": "5d7",
"length": "1"
},
{
"start": "5d9",
"length": "1"
},
{
"start": "5db",
"length": "1"
},
{
"start": "5dd",
"length": "1"
},
{
"start": "5df",
"length": "1"
},
{
"start": "5e1",
"length": "1"
},
{
"start": "5e3",
"length": "1"
},
{
"start": "5e5",
"length": "1"
},
{
"start": "5e7",
"length": "1"
},
{
"start": "5e9",
"length": "1"
},
{
"start": "5eb",
"length": "1"
},
{
"start": "5ed",
"length": "1"
},
{
"start": "5ef",
"length": "1"
},
{
"start": "5f1",
"length": "1"
},
{
"start": "5f3",
"length": "1"
}
],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "519000'86682",
"reported_seq": "22",
"reported_epoch": "525242",
"state": "down",
"last_fresh": "2021-04-07T21:23:43.829634-0600",
"last_change": "2021-04-07T21:23:43.829634-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-07T21:23:43.829634-0600",
"last_undegraded": "2021-04-07T21:23:43.829634-0600",
"last_fullsized": "2021-04-07T21:23:43.829634-0600",
"mapping_epoch": 525928,
"log_start": "518614'85590",
"ondisk_log_start": "518614'85590",
"created": 337872,
"last_epoch_clean": 518952,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"log_size": 1092,
"ondisk_log_size": 1092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 29770742894,
"num_objects": 5519,
"num_object_clones": 229,
"num_object_copies": 16557,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 5519,
"num_whiteouts": 193,
"num_read": 0,
"num_read_kb": 0,
"num_write": 3,
"num_write_kb": 32,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
5,
7,
9,
17,
34
],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 518961,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "7(2)",
"pgid": "47.0s2",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "9(1)",
"pgid": "47.0s1",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "484",
"reported_epoch": "525921",
"state": "down",
"last_fresh": "2021-04-08T14:38:13.207479-0600",
"last_change": "2021-04-08T14:38:13.207479-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-08T14:38:13.207479-0600",
"last_undegraded": "2021-04-08T14:38:13.207479-0600",
"last_fullsized": "2021-04-08T14:38:13.207479-0600",
"mapping_epoch": 525928,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 337872,
"last_epoch_clean": 518952,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
5
],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "17(0)",
"pgid": "47.0s0",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "397",
"reported_epoch": "525888",
"state": "incomplete",
"last_fresh": "2021-04-08T14:36:16.266214-0600",
"last_change": "2021-04-08T14:36:12.822429-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-08T14:36:16.266214-0600",
"last_undegraded": "2021-04-08T14:36:16.266214-0600",
"last_fullsized": "2021-04-08T14:36:16.266214-0600",
"mapping_epoch": 525928,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 337872,
"last_epoch_clean": 518952,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "18(1)",
"pgid": "47.0s1",
"last_update": "519000'86682",
"last_complete": "519000'86682",
"log_tail": "518614'85590",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2e"
},
{
"start": "5ce",
"length": "2"
},
{
"start": "5d1",
"length": "1"
},
{
"start": "5d3",
"length": "1"
},
{
"start": "5d5",
"length": "1"
},
{
"start": "5d7",
"length": "1"
},
{
"start": "5d9",
"length": "1"
},
{
"start": "5db",
"length": "1"
},
{
"start": "5dd",
"length": "1"
},
{
"start": "5df",
"length": "1"
},
{
"start": "5e1",
"length": "1"
},
{
"start": "5e3",
"length": "1"
},
{
"start": "5e5",
"length": "1"
},
{
"start": "5e7",
"length": "1"
},
{
"start": "5e9",
"length": "1"
},
{
"start": "5eb",
"length": "1"
},
{
"start": "5ed",
"length": "1"
},
{
"start": "5ef",
"length": "1"
},
{
"start": "5f1",
"length": "1"
},
{
"start": "5f3",
"length": "1"
}
],
"history": {
"epoch_created": 337872,
"epoch_pool_created": 328088,
"last_epoch_started": 519062,
"last_interval_started": 519061,
"last_epoch_clean": 518952,
"last_interval_clean": 518951,
"last_epoch_split": 488979,
"last_epoch_marked_full": 0,
"same_up_since": 525928,
"same_interval_since": 525928,
"same_primary_since": 525922,
"last_scrub": "516068'74046",
"last_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"last_deep_scrub": "515384'67542",
"last_deep_scrub_stamp": "2021-03-27T19:18:36.384901-0600",
"last_clean_scrub_stamp": "2021-03-31T03:21:18.407682-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 525928,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
5,
9,
4
],
"acting": [
5,
9,
4
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 5,
"acting_primary": 5,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 519062,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "19(2)",
"pgid": "47.0s2",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "34(1)",
"pgid": "47.0s1",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Peering/Incomplete",
"enter_time": "2021-04-08T14:38:31.054423-0600",
"comment": "not enough complete instances of this PG"
},
{
"name": "Started/Primary/Peering",
"enter_time": "2021-04-08T14:38:30.989327-0600",
"past_intervals": [
{
"first": "518951",
"last": "525927",
"all_participants": [
{
"osd": 4,
"shard": 2
},
{
"osd": 5,
"shard": 0
},
{
"osd": 7,
"shard": 2
},
{
"osd": 9,
"shard": 1
},
{
"osd": 17,
"shard": 0
},
{
"osd": 19,
"shard": 2
},
{
"osd": 34,
"shard": 1
}
],
"intervals": [
{
"first": "519027",
"last": "519044",
"acting": "5(0),19(2)"
},
{
"first": "519061",
"last": "519064",
"acting": "5(0),19(2),34(1)"
},
{
"first": "520151",
"last": "520156",
"acting": "7(2),17(0)"
},
{
"first": "520194",
"last": "520261",
"acting": "7(2),9(1)"
},
{
"first": "520262",
"last": "520304",
"acting": "7(2),9(1),17(0)"
},
{
"first": "525419",
"last": "525420",
"acting": "5(0),9(1)"
},
{
"first": "525616",
"last": "525624",
"acting": "4(2),5(0)"
},
{
"first": "525638",
"last": "525641",
"acting": "4(2),9(1)"
},
{
"first": "525884",
"last": "525888",
"acting": "4(2),9(1),17(0)"
},
{
"first": "525922",
"last": "525926",
"acting": "4(2),5(0),9(1)"
}
]
}
],
"probing_osds": [
"4(2)",
"5(0)",
"7(2)",
"9(1)",
"17(0)",
"19(2)",
"34(1)"
],
"down_osds_we_would_probe": [],
"peering_blocked_by": []
},
{
"name": "Started",
"enter_time": "2021-04-08T14:38:30.989236-0600"
}
],
"agent_state": {}
}
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP
47.0 20070 0 0 0 26759978094 0 0 1092 incomplete 3h 519000'86682 526166:346363 [5,9,4]p5 [5,9,4]p5 2021-03-31T03:21:18.407682-0600 2021-03-27T19:18:36.384901-0600
47.3 20169 0 0 0 26371196538 0 0 1168 incomplete 3h 518999'70605 526166:371635 [33,7,30]p33 [33,7,30]p33 2021-03-31T22:19:10.471925-0600 2021-03-25T20:21:16.093783-0600
47.4 20280 0 0 0 26260431825 0 0 1118 incomplete 3h 518995'80378 526166:398877 [23,10,19]p23 [23,10,19]p23 2021-03-31T22:40:12.339909-0600 2021-03-31T22:40:12.339909-0600
47.d 20525 0 0 0 26263948794 0 0 985 incomplete 3h 518999'80034 526166:354391 [0,8,6]p0 [0,8,6]p0 2021-04-01T02:26:35.982459-0600 2021-03-29T01:55:08.194735-0600
47.12 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4996 [27,1,8]p27 [27,1,8]p27 2021-04-01T01:01:22.575010-0600 2021-04-01T01:01:22.575010-0600
47.13 22005 0 0 0 27281035835 0 0 1245 incomplete 3h 519007'103848 526166:406088 [3,27,10]p3 [3,27,10]p3 2021-04-01T03:00:37.908783-0600 2021-03-25T03:15:07.751993-0600
47.18 19925 0 0 0 26041486043 0 0 831 incomplete 3h 518995'89106 526166:355410 [0,9,30]p0 [0,9,30]p0 2021-04-01T02:36:26.883662-0600 2021-03-28T20:40:53.970613-0600
47.20 20051 0 0 0 26505000546 0 0 1119 incomplete 3h 519000'99769 526166:348900 [15,8,30]p15 [15,8,30]p15 2021-04-01T01:14:42.269934-0600 2021-04-01T01:14:42.269934-0600
47.24 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5103 [9,19,15]p9 [9,19,15]p9 2021-03-31T00:43:17.841228-0600 2021-03-28T03:08:13.364769-0600
47.27 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5088 [30,28,6]p30 [30,28,6]p30 2021-04-01T03:40:34.645668-0600 2021-03-29T02:45:22.939338-0600
47.29 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4939 [19,16,8]p19 [19,16,8]p19 2021-03-31T20:58:34.015774-0600 2021-03-31T20:58:34.015774-0600
47.32 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:11097 [9,0,1]p9 [9,0,1]p9 2021-03-31T03:29:19.321570-0600 2021-03-25T03:55:28.079403-0600
47.38 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5061 [6,9,2]p6 [6,9,2]p6 2021-03-28T21:52:56.209769-0600 2021-03-26T23:58:40.078225-0600
47.39 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5109 [6,12,30]p6 [6,12,30]p6 2021-03-28T21:02:51.726250-0600 2021-03-26T22:30:41.534234-0600
47.3f 20334 0 0 0 26798526707 0 0 1037 incomplete 3h 518997'86706 526166:358831 [4,7,19]p4 [4,7,19]p4 2021-03-30T21:00:20.680102-0600 2021-03-30T21:00:20.680102-0600
47.43 20270 0 0 0 26401100570 0 0 1105 incomplete 3h 518997'79678 526166:332389 [1,6,8]p1 [1,6,8]p1 2021-04-01T03:19:25.565020-0600 2021-03-26T21:21:02.298726-0600
47.45 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5725 [27,7,0]p27 [27,7,0]p27 2021-03-31T23:57:18.800169-0600 2021-03-27T03:30:15.326626-0600
47.4e 20279 0 0 0 25784621329 0 0 963 incomplete 3h 518995'94727 526166:355120 [3,19,27]p3 [3,19,27]p3 2021-03-31T00:49:22.102173-0600 2021-03-31T00:49:22.102173-0600
47.4f 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9602 [6,8,11]p6 [6,8,11]p6 2021-03-28T19:47:56.332962-0600 2021-03-25T02:52:00.646232-0600
47.55 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9620 [7,16,8]p7 [7,16,8]p7 2021-04-01T22:59:55.257326-0600 2021-03-27T21:21:27.496151-0600
47.5a 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5822 [8,10,0]p8 [8,10,0]p8 2021-04-01T22:19:53.918809-0600 2021-04-01T22:19:53.918809-0600
47.68 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5016 [19,10,5]p19 [19,10,5]p19 2021-03-31T19:46:58.086704-0600 2021-03-25T02:36:58.633685-0600
47.6b 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5058 [8,33,6]p8 [8,33,6]p8 2021-03-28T19:08:20.531682-0600 2021-03-26T22:58:48.551650-0600
47.74 20028 0 0 0 26352535351 0 0 1119 incomplete 3h 518995'84646 526166:354014 [1,7,30]p1 [1,7,30]p1 2021-03-30T21:25:52.131450-0600 2021-03-24T23:39:23.368483-0600
47.75 20321 0 0 0 26728108311 0 0 1206 incomplete 3h 519007'108028 526166:426772 [17,27,19]p17 [17,27,19]p17 2021-03-31T21:40:42.269394-0600 2021-03-31T21:40:42.269394-0600
47.78 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5122 [9,30,5]p9 [9,30,5]p9 2021-03-31T20:11:41.210011-0600 2021-03-27T22:36:10.661087-0600
47.7b 20026 0 0 0 25999494334 0 0 1058 incomplete 3h 519049'101874 526166:410071 [28,30,10]p28 [28,30,10]p28 2021-03-31T23:07:28.497014-0600 2021-03-27T23:06:40.522139-0600
47.7d 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4983 [10,7,18]p10 [10,7,18]p10 2021-03-29T00:47:33.754276-0600 2021-03-26T21:43:09.385219-0600
47.81 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5148 [6,10,13]p6 [6,10,13]p6 2021-03-29T03:10:09.500658-0600 2021-03-24T00:55:05.716605-0600
47.83 19943 0 0 0 26143034719 0 0 1116 incomplete 3h 519029'97443 526166:388595 [28,27,7]p28 [28,27,7]p28 2021-04-01T01:07:13.961376-0600 2021-04-01T01:07:13.961376-0600
47.85 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5045 [7,6,11]p7 [7,6,11]p7 2021-03-31T21:46:38.615216-0600 2021-03-31T21:46:38.615216-0600
47.88 19744 0 0 0 25741847210 0 0 1063 incomplete 3h 519000'106717 526166:390476 [16,9,27]p16 [16,9,27]p16 2021-04-01T00:36:40.605136-0600 2021-04-01T00:36:40.605136-0600
47.8d 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5025 [30,19,14]p30 [30,19,14]p30 2021-04-01T01:02:07.798881-0600 2021-03-25T21:55:37.181983-0600
47.91 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4998 [8,14,27]p8 [8,14,27]p8 2021-03-31T20:07:35.087035-0600 2021-03-24T23:02:00.170210-0600
47.a2 20284 0 0 0 26569134699 0 0 1161 incomplete 3h 519000'104297 526166:398751 [13,19,6]p13 [13,19,6]p13 2021-03-31T19:56:03.695108-0600 2021-03-31T19:56:03.695108-0600
47.a5 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5069 [8,27,3]p8 [8,27,3]p8 2021-04-01T21:45:11.476900-0600 2021-03-28T01:24:09.482384-0600
47.ab 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4921 [10,8,14]p10 [10,8,14]p10 2021-03-30T21:49:07.499027-0600 2021-03-30T21:49:07.499027-0600
47.ac 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5086 [7,27,15]p7 [7,27,15]p7 2021-03-28T03:01:26.639534-0600 2021-03-25T00:50:14.553902-0600
47.b2 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5135 [6,7,18]p6 [6,7,18]p6 2021-03-28T19:00:52.306016-0600 2021-03-25T00:55:20.601141-0600
47.ba 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4984 [27,9,14]p27 [27,9,14]p27 2021-03-30T21:23:31.275720-0600 2021-03-30T21:23:31.275720-0600
47.bd 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9388 [19,8,1]p19 [19,8,1]p19 2021-04-01T02:10:45.120565-0600 2021-04-01T02:10:45.120565-0600
47.be 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4998 [19,33,6]p19 [19,33,6]p19 2021-04-01T03:34:38.102589-0600 2021-04-01T03:34:38.102589-0600
47.bf 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4948 [19,8,13]p19 [19,8,13]p19 2021-03-30T21:35:54.230025-0600 2021-03-30T21:35:54.230025-0600
47.c1 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5048 [30,19,13]p30 [30,19,13]p30 2021-03-28T20:18:17.342293-0600 2021-03-28T20:18:17.342293-0600
47.c2 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9361 [10,11,30]p10 [10,11,30]p10 2021-03-30T20:02:35.909530-0600 2021-03-23T20:55:48.695067-0600
47.c4 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5160 [9,7,5]p9 [9,7,5]p9 2021-03-31T22:25:47.118619-0600 2021-03-24T23:21:34.267990-0600
47.c6 19751 0 0 0 26365340716 0 0 1093 incomplete 3h 519010'83703 526166:427023 [1,27,8]p1 [1,27,8]p1 2021-04-01T21:11:47.655056-0600 2021-04-01T21:11:47.655056-0600
47.d2 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5719 [30,9,28]p30 [30,9,28]p30 2021-03-28T19:01:14.191464-0600 2021-03-23T19:12:59.768249-0600
47.d5 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5129 [9,3,7]p9 [9,3,7]p9 2021-03-31T21:06:28.077311-0600 2021-03-25T22:47:26.773692-0600
47.d6 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9513 [7,19,8]p7 [7,19,8]p7 2021-03-31T19:44:39.521844-0600 2021-03-31T19:44:39.521844-0600
47.d9 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4983 [19,17,9]p19 [19,17,9]p19 2021-03-31T01:09:56.596992-0600 2021-03-27T02:36:54.285634-0600
47.db 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4998 [10,28,7]p10 [10,28,7]p10 2021-03-31T21:29:00.484238-0600 2021-03-28T01:45:50.800338-0600
47.de 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5108 [8,7,15]p8 [8,7,15]p8 2021-04-01T01:30:22.243669-0600 2021-03-26T01:33:06.527431-0600
47.e1 20088 0 0 0 26226582893 0 0 1141 incomplete 3h 519000'108471 526166:345813 [15,19,27]p15 [15,19,27]p15 2021-04-01T22:24:35.935867-0600 2021-03-29T19:16:20.203954-0600
47.e7 19849 0 0 0 26339566289 0 0 1032 incomplete 3h 518987'86817 526166:448427 [3,10,30]p3 [3,10,30]p3 2021-03-30T20:22:16.603628-0600 2021-03-23T22:49:34.674749-0600
47.f8 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5140 [6,15,8]p6 [6,15,8]p6 2021-03-28T00:24:32.390309-0600 2021-03-28T00:24:32.390309-0600
47.ff 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4955 [10,18,9]p10 [10,18,9]p10 2021-04-01T03:18:45.139873-0600 2021-03-25T21:28:04.827073-0600
47.106 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5090 [30,2,3]p30 [30,2,3]p30 2021-04-01T00:42:20.714987-0600 2021-03-27T19:13:11.636060-0600
47.10b 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4955 [19,30,1]p19 [19,30,1]p19 2021-04-01T00:54:51.595530-0600 2021-04-01T00:54:51.595530-0600
47.111 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4994 [9,33,8]p9 [9,33,8]p9 2021-03-31T19:54:04.738313-0600 2021-03-24T20:13:03.007939-0600
47.114 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9601 [9,10,17]p9 [9,10,17]p9 2021-04-01T19:56:00.916625-0600 2021-03-28T23:49:56.145249-0600
47.115 19967 0 0 0 26250102170 0 0 1000 incomplete 3h 519007'82049 526166:378864 [4,6,9]p4 [4,6,9]p4 2021-04-01T21:40:16.002686-0600 2021-04-01T21:40:16.002686-0600
47.116 19852 0 0 0 25884529821 0 0 1121 incomplete 3h 519000'74737 526166:459835 [33,6,30]p33 [33,6,30]p33 2021-04-01T00:22:40.644004-0600 2021-04-01T00:22:40.644004-0600
47.11c 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9437 [19,12,30]p19 [19,12,30]p19 2021-04-01T00:13:13.070183-0600 2021-03-25T19:18:01.757931-0600
47.11e 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5067 [7,28,30]p7 [7,28,30]p7 2021-03-29T02:51:23.634263-0600 2021-03-27T02:55:35.150008-0600
47.121 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5076 [30,19,4]p30 [30,19,4]p30 2021-03-28T03:21:15.443939-0600 2021-03-26T21:34:23.689399-0600
47.12b 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4901 [19,3,6]p19 [19,3,6]p19 2021-03-31T23:29:53.658826-0600 2021-03-31T23:29:53.658826-0600
47.12d 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5162 [9,28,8]p9 [9,28,8]p9 2021-03-31T23:39:03.266816-0600 2021-03-28T20:59:23.118803-0600
47.12e 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4939 [10,33,27]p10 [10,33,27]p10 2021-03-28T21:08:38.252809-0600 2021-03-23T21:36:01.003577-0600
47.12f 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5056 [27,33,9]p27 [27,33,9]p27 2021-03-31T23:26:16.234711-0600 2021-03-28T20:30:21.411969-0600
47.132 19934 0 0 0 26467844952 0 0 920 incomplete 3h 519007'83538 526166:409888 [2,19,7]p2 [2,19,7]p2 2021-04-01T03:28:35.691696-0600 2021-04-01T03:28:35.691696-0600
47.133 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9389 [10,28,9]p10 [10,28,9]p10 2021-04-02T03:49:39.216071-0600 2021-04-02T03:49:39.216071-0600
47.13b 0 0 0 0 0 0 0 915 incomplete 3h 518995'81210 526166:5270 [14,16,8]p14 [14,16,8]p14 2021-04-01T20:42:03.066302-0600 2021-03-27T01:43:56.229840-0600
47.13d 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5049 [9,30,28]p9 [9,30,28]p9 2021-04-01T19:46:45.285673-0600 2021-03-29T00:57:54.507752-0600
47.13e 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5175 [9,28,6]p9 [9,28,6]p9 2021-04-01T00:45:16.255848-0600 2021-03-29T00:34:55.455255-0600
47.148 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9386 [10,18,4]p10 [10,18,4]p10 2021-03-30T21:37:21.827042-0600 2021-03-30T21:37:21.827042-0600
47.14d 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5115 [27,3,8]p27 [27,3,8]p27 2021-03-30T21:04:55.204019-0600 2021-03-25T00:52:00.709998-0600
47.14e 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5069 [9,2,27]p9 [9,2,27]p9 2021-03-31T00:35:27.216322-0600 2021-03-25T20:11:04.278420-0600
47.153 20022 0 0 0 26289944955 0 0 1169 incomplete 3h 519022'106165 526166:381250 [12,27,8]p12 [12,27,8]p12 2021-03-31T19:23:47.708010-0600 2021-03-27T19:25:23.759889-0600
47.158 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5084 [6,13,19]p6 [6,13,19]p6 2021-03-30T19:13:13.576594-0600 2021-03-30T19:13:13.576594-0600
47.15a 19699 0 0 0 25511263757 0 0 1105 incomplete 3h 519031'105897 526166:427717 [13,19,10]p13 [13,19,10]p13 2021-03-31T23:13:41.188648-0600 2021-03-31T23:13:41.188648-0600
47.15b 19889 0 0 0 26272408882 0 0 1120 incomplete 3h 519007'103191 526166:471992 [16,30,27]p16 [16,30,27]p16 2021-03-31T23:04:31.104476-0600 2021-03-28T20:07:48.221437-0600
47.15d 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9314 [10,27,23]p10 [10,27,23]p10 2021-04-01T00:36:47.264513-0600 2021-03-25T00:42:47.196052-0600
47.166 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9658 [9,27,28]p9 [9,27,28]p9 2021-04-01T01:27:29.221950-0600 2021-04-01T01:27:29.221950-0600
47.16c 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9498 [7,16,19]p7 [7,16,19]p7 2021-03-31T23:54:09.693485-0600 2021-03-31T23:54:09.693485-0600
47.171 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5610 [19,0,6]p19 [19,0,6]p19 2021-04-01T00:26:19.211464-0600 2021-03-27T21:40:36.939559-0600
47.175 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9516 [7,27,4]p7 [7,27,4]p7 2021-03-30T22:17:18.528531-0600 2021-03-28T02:54:27.600714-0600
47.181 19969 0 0 0 26071125080 0 0 1108 incomplete 3h 519025'99249 526166:420410 [28,7,27]p28 [28,7,27]p28 2021-03-29T01:55:17.182313-0600 2021-03-29T01:55:17.182313-0600
47.183 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9467 [27,30,4]p27 [27,30,4]p27 2021-03-31T00:55:39.514956-0600 2021-03-31T00:55:39.514956-0600
47.18a 20183 0 0 0 26317047279 0 0 1105 incomplete 3h 518995'78580 526166:406032 [1,10,27]p1 [1,10,27]p1 2021-04-01T03:28:57.767008-0600 2021-04-01T03:28:57.767008-0600
47.193 20076 0 0 0 26545067723 0 0 999 incomplete 3h 519000'80492 526166:413035 [4,6,9]p4 [4,6,9]p4 2021-04-01T19:55:40.675099-0600 2021-03-29T00:03:32.626359-0600
47.1a5 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5181 [9,8,13]p9 [9,8,13]p9 2021-03-30T19:45:03.299521-0600 2021-03-30T19:45:03.299521-0600
47.1ae 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4983 [19,7,13]p19 [19,7,13]p19 2021-03-31T20:58:50.224568-0600 2021-03-26T00:39:11.884432-0600
47.1b1 19602 0 0 0 25642774871 0 0 1145 incomplete 3h 519049'98127 526166:447738 [14,7,9]p14 [14,7,9]p14 2021-03-30T19:28:37.473471-0600 2021-03-30T19:28:37.473471-0600
47.1b5 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4909 [10,2,27]p10 [10,2,27]p10 2021-04-01T21:28:37.616513-0600 2021-04-01T21:28:37.616513-0600
47.1bb 19937 0 0 0 26256947699 0 0 1167 incomplete 3h 519000'104777 526166:430348 [13,8,19]p13 [13,8,19]p13 2021-04-01T01:01:30.688372-0600 2021-03-28T19:19:21.517396-0600
47.1bc 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5014 [7,3,6]p7 [7,3,6]p7 2021-04-01T21:43:32.613447-0600 2021-03-28T02:28:26.988032-0600
47.1bd 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5017 [19,9,17]p19 [19,9,17]p19 2021-04-01T01:39:53.715270-0600 2021-03-27T21:01:21.857265-0600
47.1c5 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5062 [6,7,1]p6 [6,7,1]p6 2021-03-29T00:09:05.664124-0600 2021-03-29T00:09:05.664124-0600
47.1c8 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5056 [27,12,30]p27 [27,12,30]p27 2021-03-30T19:05:17.430464-0600 2021-03-30T19:05:17.430464-0600
47.1c9 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5108 [7,12,9]p7 [7,12,9]p7 2021-04-01T02:18:06.693708-0600 2021-04-01T02:18:06.693708-0600
47.1ca 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9717 [6,13,0]p6 [6,13,0]p6 2021-03-30T20:53:34.745151-0600 2021-03-30T20:53:34.745151-0600
47.1cb 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9697 [9,10,1]p9 [9,10,1]p9 2021-04-01T03:59:07.919592-0600 2021-03-27T19:06:05.459813-0600
47.1e0 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9559 [30,8,14]p30 [30,8,14]p30 2021-03-28T23:57:25.436078-0600 2021-03-27T01:38:05.517845-0600
47.1e1 20046 0 0 0 26082697600 0 0 1106 incomplete 3h 519007'84323 526166:482600 [23,19,8]p23 [23,19,8]p23 2021-03-31T19:44:50.391568-0600 2021-03-27T19:50:13.164921-0600
47.1f1 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9654 [30,23,9]p30 [30,23,9]p30 2021-03-30T20:53:47.462270-0600 2021-03-28T01:25:46.956359-0600
47.1f4 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9295 [10,23,27]p10 [10,23,27]p10 2021-03-31T01:43:42.240022-0600 2021-03-31T01:43:42.240022-0600
47.1f9 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5004 [10,7,12]p10 [10,7,12]p10 2021-03-29T02:19:18.367248-0600 2021-03-24T23:14:41.816716-0600
47.206 20330 0 0 0 26179646944 0 0 1115 incomplete 3h 518997'80093 526166:412776 [1,19,7]p1 [1,19,7]p1 2021-03-30T21:26:04.011341-0600 2021-03-24T03:59:12.427505-0600
47.20b 20053 0 0 0 26024476227 0 0 997 incomplete 3h 518997'78121 526166:363253 [4,6,12]p4 [4,6,12]p4 2021-04-01T00:45:09.219038-0600 2021-03-26T20:19:01.735932-0600
47.20c 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:10865 [7,0,8]p7 [7,0,8]p7 2021-03-31T20:22:59.759870-0600 2021-03-31T20:22:59.759870-0600
47.20e 20113 0 0 0 26526638542 0 0 990 incomplete 3h 518995'81408 526166:447025 [0,6,30]p0 [0,6,30]p0 2021-03-31T21:21:08.761007-0600 2021-03-26T02:50:03.752634-0600
47.217 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4968 [19,8,1]p19 [19,8,1]p19 2021-04-01T21:12:15.382870-0600 2021-03-27T23:26:09.896032-0600
47.21d 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4953 [19,2,27]p19 [19,2,27]p19 2021-04-01T01:22:15.964498-0600 2021-03-28T00:04:41.815145-0600
47.21f 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9569 [27,1,19]p27 [27,1,19]p27 2021-03-31T02:49:20.951543-0600 2021-03-26T19:50:30.551545-0600
47.222 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9575 [9,10,12]p9 [9,10,12]p9 2021-04-01T03:01:35.864905-0600 2021-03-26T21:23:24.045585-0600
47.227 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5127 [7,8,13]p7 [7,8,13]p7 2021-03-28T01:16:28.930572-0600 2021-03-25T03:39:00.406340-0600
47.22b 0 0 0 0 0 0 0 927 incomplete 3h 519000'94397 526166:5001 [3,9,19]p3 [3,9,19]p3 2021-03-31T03:23:17.906474-0600 2021-03-26T03:42:26.835052-0600
47.22d 19672 0 0 0 26120364889 0 0 1118 incomplete 3h 519044'103283 526166:352559 [28,10,27]p28 [28,10,27]p28 2021-04-01T01:08:06.408050-0600 2021-03-28T02:17:35.460478-0600
47.22e 20123 0 0 0 26688200644 0 0 1192 incomplete 3h 519007'106615 526166:419698 [18,19,30]p18 [18,19,30]p18 2021-03-30T22:44:24.037301-0600 2021-03-28T22:35:08.620692-0600
47.230 20257 0 0 0 25824570856 0 0 993 incomplete 3h 518987'84219 526166:338100 [4,30,19]p4 [4,30,19]p4 2021-04-01T21:56:05.621218-0600 2021-03-27T02:49:41.789992-0600
47.234 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4984 [10,17,19]p10 [10,17,19]p10 2021-03-28T21:24:34.834498-0600 2021-03-28T21:24:34.834498-0600
47.243 20157 0 0 0 26381686450 0 0 1209 incomplete 3h 519025'100366 526166:448826 [18,8,7]p18 [18,8,7]p18 2021-04-01T03:00:22.585387-0600 2021-03-28T21:10:11.662912-0600
47.244 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5744 [30,0,10]p30 [30,0,10]p30 2021-04-01T01:07:26.685071-0600 2021-03-29T02:44:15.370657-0600
47.24e 20127 0 0 0 26219190351 0 0 1206 incomplete 3h 518997'94954 526166:399406 [12,7,8]p12 [12,7,8]p12 2021-03-30T21:10:10.911135-0600 2021-03-30T21:10:10.911135-0600
47.254 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5070 [7,23,6]p7 [7,23,6]p7 2021-03-28T19:41:04.896055-0600 2021-03-28T19:41:04.896055-0600
47.255 20376 0 0 0 26742559275 0 0 1188 incomplete 3h 518995'77999 526166:431830 [33,30,8]p33 [33,30,8]p33 2021-03-31T19:17:21.537364-0600 2021-03-25T20:14:25.340283-0600
47.258 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5755 [6,0,30]p6 [6,0,30]p6 2021-04-01T00:00:49.579022-0600 2021-03-27T20:33:04.243319-0600
47.259 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5147 [8,30,5]p8 [8,30,5]p8 2021-03-27T22:39:37.697021-0600 2021-03-21T01:05:52.666420-0600
47.25d 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5743 [27,7,0]p27 [27,7,0]p27 2021-04-01T03:23:08.559922-0600 2021-04-01T03:23:08.559922-0600
47.25e 20336 0 0 0 26698183299 0 0 1061 incomplete 3h 518993'101057 526166:376401 [28,8,6]p28 [28,8,6]p28 2021-03-29T01:32:11.012945-0600 2021-03-27T00:12:19.948412-0600
47.261 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5165 [30,18,7]p30 [30,18,7]p30 2021-04-01T01:57:47.344046-0600 2021-03-26T20:41:03.399382-0600
47.266 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5084 [8,16,30]p8 [8,16,30]p8 2021-03-28T03:35:27.190995-0600 2021-03-24T23:14:49.566588-0600
47.26a 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5045 [7,16,19]p7 [7,16,19]p7 2021-03-31T23:15:16.369673-0600 2021-03-26T01:25:59.521139-0600
47.27a 20185 0 0 0 26574776111 0 0 1106 incomplete 3h 519007'107921 526166:417472 [16,9,19]p16 [16,9,19]p16 2021-03-31T03:16:02.910400-0600 2021-03-28T19:54:24.926806-0600
47.27f 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5061 [27,5,6]p27 [27,5,6]p27 2021-03-31T23:06:25.788178-0600 2021-03-24T23:39:08.357812-0600
47.280 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5059 [27,10,2]p27 [27,10,2]p27 2021-03-30T19:08:39.702485-0600 2021-03-27T21:20:30.399908-0600
47.281 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9375 [10,17,19]p10 [10,17,19]p10 2021-04-01T01:41:06.905989-0600 2021-03-29T01:33:29.605803-0600
47.284 20327 0 0 0 26605243228 0 0 1148 incomplete 3h 519049'98275 526166:529364 [28,8,30]p28 [28,8,30]p28 2021-03-28T21:43:34.141833-0600 2021-03-28T21:43:34.141833-0600
47.286 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5050 [30,7,4]p30 [30,7,4]p30 2021-03-31T19:17:35.407716-0600 2021-03-25T22:53:15.065688-0600
47.28e 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4998 [10,7,12]p10 [10,7,12]p10 2021-03-29T00:36:47.386296-0600 2021-03-27T03:44:01.214168-0600
47.290 19987 0 0 0 25866970367 0 0 1021 incomplete 3h 519000'87528 526166:410002 [3,8,9]p3 [3,8,9]p3 2021-04-01T01:42:34.676157-0600 2021-03-27T03:05:59.228923-0600
47.298 20066 0 0 0 26321626020 0 0 1221 incomplete 3h 519038'104666 526166:440966 [18,10,19]p18 [18,10,19]p18 2021-04-01T01:30:45.076227-0600 2021-03-29T01:17:32.665059-0600
47.299 20243 0 0 0 26434530287 0 0 1061 incomplete 3h 519007'76475 526166:450570 [11,10,6]p11 [11,10,6]p11 2021-04-01T22:21:20.156918-0600 2021-04-01T22:21:20.156918-0600
47.29a 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9454 [19,17,7]p19 [19,17,7]p19 2021-04-01T02:42:39.410210-0600 2021-03-30T19:35:55.569945-0600
47.2a5 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5837 [6,27,0]p6 [6,27,0]p6 2021-03-30T19:31:20.349618-0600 2021-03-24T01:36:27.398025-0600
47.2a7 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9552 [30,12,27]p30 [30,12,27]p30 2021-03-30T20:54:00.184154-0600 2021-03-27T21:32:01.793734-0600
47.2a9 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5103 [7,10,13]p7 [7,10,13]p7 2021-03-31T23:14:24.960039-0600 2021-03-25T20:58:47.775813-0600
47.2ac 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5088 [9,19,15]p9 [9,19,15]p9 2021-03-31T03:35:42.725346-0600 2021-03-28T02:48:55.814090-0600
47.2ae 181 0 0 0 281346048 0 0 1001 incomplete 3h 518999'82138 526166:4951 [3,4,27]p3 [3,4,27]p3 2021-04-01T01:44:46.662082-0600 2021-03-29T01:04:32.733476-0600
47.2af 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5051 [30,19,2]p30 [30,19,2]p30 2021-03-28T03:26:31.969098-0600 2021-03-28T03:26:31.969098-0600
47.2b0 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5100 [30,12,28]p30 [30,12,28]p30 2021-03-29T02:18:12.708308-0600 2021-03-29T02:18:12.708308-0600
47.2b3 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5038 [27,12,6]p27 [27,12,6]p27 2021-03-31T20:07:02.402478-0600 2021-03-31T20:07:02.402478-0600
47.2b8 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9444 [19,17,30]p19 [19,17,30]p19 2021-03-31T19:33:19.660717-0600 2021-03-31T19:33:19.660717-0600
47.2bc 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5119 [8,7,3]p8 [8,7,3]p8 2021-04-01T21:42:57.873950-0600 2021-03-28T01:00:02.780856-0600
47.2bf 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5069 [9,23,30]p9 [9,23,30]p9 2021-03-30T21:42:50.726301-0600 2021-03-30T21:42:50.726301-0600
47.2c6 20213 0 0 0 26175216381 0 0 1207 incomplete 3h 519041'99538 526166:440134 [18,8,9]p18 [18,8,9]p18 2021-03-30T21:16:24.525557-0600 2021-03-25T01:40:32.080408-0600
47.2d7 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4981 [10,28,7]p10 [10,28,7]p10 2021-03-29T01:04:27.951777-0600 2021-03-27T00:59:42.253682-0600
47.2db 20528 0 0 0 26356361520 0 0 1047 incomplete 3h 518995'81295 526166:448600 [4,6,8]p4 [4,6,8]p4 2021-03-31T23:09:23.616365-0600 2021-03-25T20:51:50.728325-0600
47.2e2 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5134 [8,7,16]p8 [8,7,16]p8 2021-03-31T19:23:39.409829-0600 2021-03-31T19:23:39.409829-0600
47.2e6 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4949 [19,10,15]p19 [19,10,15]p19 2021-03-30T20:02:05.088673-0600 2021-03-24T22:49:16.201562-0600
47.2e8 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4972 [19,10,28]p19 [19,10,28]p19 2021-03-31T22:45:53.015540-0600 2021-03-25T00:16:33.668694-0600
47.2ea 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5074 [8,30,16]p8 [8,30,16]p8 2021-03-28T20:45:30.210125-0600 2021-03-26T20:23:09.951864-0600
47.2ec 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5089 [27,28,8]p27 [27,28,8]p27 2021-03-30T21:35:59.692780-0600 2021-03-24T03:00:41.506393-0600
47.2f1 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9386 [7,9,5]p7 [7,9,5]p7 2021-04-01T21:01:19.688196-0600 2021-03-26T02:00:11.864717-0600
47.2f6 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5048 [27,18,8]p27 [27,18,8]p27 2021-03-31T20:48:35.821193-0600 2021-03-31T20:48:35.821193-0600
47.2fe 19879 0 0 0 26213307805 0 0 1156 incomplete 3h 519007'105685 526166:465957 [14,9,27]p14 [14,9,27]p14 2021-04-02T01:24:40.789845-0600 2021-03-29T00:14:39.375886-0600
47.300 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9420 [10,27,1]p10 [10,27,1]p10 2021-04-01T01:20:48.960507-0600 2021-04-01T01:20:48.960507-0600
47.301 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5060 [30,13,6]p30 [30,13,6]p30 2021-03-31T19:27:53.631542-0600 2021-03-31T19:27:53.631542-0600
47.303 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9426 [7,8,19]p7 [7,8,19]p7 2021-03-27T22:54:34.374295-0600 2021-03-27T22:54:34.374295-0600
47.313 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9409 [19,15,27]p19 [19,15,27]p19 2021-03-31T02:38:46.386103-0600 2021-03-28T22:53:06.056137-0600
47.319 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9426 [19,23,7]p19 [19,23,7]p19 2021-03-31T03:23:41.075244-0600 2021-03-29T02:30:42.413103-0600
47.31b 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9725 [8,6,9]p8 [8,6,9]p8 2021-04-01T03:32:12.992584-0600 2021-04-01T03:32:12.992584-0600
47.31f 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9648 [30,8,16]p30 [30,8,16]p30 2021-04-01T01:29:29.944359-0600 2021-03-27T03:45:30.257742-0600
47.321 919 0 0 0 1281605632 0 0 975 incomplete 3h 518995'79681 526166:9506 [3,10,11]p3 [3,10,11]p3 2021-04-01T19:28:50.975361-0600 2021-03-29T02:51:30.163971-0600
47.324 20085 0 0 0 26208362821 0 0 1053 incomplete 3h 518995'75760 526166:415015 [11,7,6]p11 [11,7,6]p11 2021-04-01T02:42:58.353856-0600 2021-03-28T02:03:43.936527-0600
47.331 20273 0 0 0 26509287480 0 0 926 incomplete 3h 518995'86971 526166:456321 [0,27,19]p0 [0,27,19]p0 2021-03-31T01:52:35.954752-0600 2021-03-31T01:52:35.954752-0600
47.333 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9445 [19,17,9]p19 [19,17,9]p19 2021-03-31T01:03:47.041056-0600 2021-03-27T23:24:12.558006-0600
47.33b 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9596 [30,6,5]p30 [30,6,5]p30 2021-04-01T19:55:32.787705-0600 2021-04-01T19:55:32.787705-0600
47.33e 19708 0 0 0 26064193992 0 0 1173 incomplete 3h 519031'102317 526166:471969 [12,19,7]p12 [12,19,7]p12 2021-04-01T02:59:13.329198-0600 2021-03-28T22:22:24.814456-0600
47.33f 20111 0 0 0 26332720024 0 0 1082 incomplete 3h 518993'80045 526166:495668 [11,10,27]p11 [11,10,27]p11 2021-04-01T22:08:56.745299-0600 2021-03-28T21:14:21.033329-0600
47.344 20379 0 0 0 26324431123 0 0 1039 incomplete 3h 519000'81486 526166:399610 [4,27,7]p4 [4,27,7]p4 2021-03-30T19:52:04.744052-0600 2021-03-27T23:41:09.245331-0600
47.34a 20414 0 0 0 26484594081 0 0 979 incomplete 3h 518997'84141 526166:494035 [0,9,30]p0 [0,9,30]p0 2021-03-30T23:19:17.569241-0600 2021-03-30T23:19:17.569241-0600
47.354 20158 0 0 0 26717629885 0 0 1044 incomplete 3h 519000'81790 526166:465417 [0,7,10]p0 [0,7,10]p0 2021-04-01T20:49:59.994389-0600 2021-03-27T00:48:18.566733-0600
47.359 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5082 [30,9,3]p30 [30,9,3]p30 2021-04-01T03:50:37.853429-0600 2021-03-29T01:36:21.909571-0600
47.35a 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5050 [30,27,28]p30 [30,27,28]p30 2021-03-28T03:19:21.985850-0600 2021-03-28T03:19:21.985850-0600
47.35c 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5798 [8,7,0]p8 [8,7,0]p8 2021-03-27T21:49:11.236313-0600 2021-03-22T03:06:57.805537-0600
47.361 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9532 [27,18,9]p27 [27,18,9]p27 2021-03-31T20:58:20.379750-0600 2021-03-31T20:58:20.379750-0600
47.363 19948 0 0 0 26453104637 0 0 1173 incomplete 3h 519007'100726 526166:584075 [18,19,27]p18 [18,19,27]p18 2021-04-01T03:02:28.548942-0600 2021-03-27T19:05:24.053031-0600
47.36b 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5030 [8,6,14]p8 [8,6,14]p8 2021-04-01T01:56:45.818815-0600 2021-04-01T01:56:45.818815-0600
47.374 19684 0 0 0 26272328522 0 0 1075 incomplete 3h 519007'99767 526166:455669 [13,6,19]p13 [13,6,19]p13 2021-04-01T20:16:11.852262-0600 2021-03-29T02:06:39.348197-0600
47.38f 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9516 [27,3,9]p27 [27,3,9]p27 2021-03-31T21:19:36.051615-0600 2021-03-31T21:19:36.051615-0600
47.391 20183 0 0 0 26174629240 0 0 968 incomplete 3h 519007'83479 526166:466393 [2,7,19]p2 [2,7,19]p2 2021-03-31T23:01:23.085327-0600 2021-03-25T03:13:24.843609-0600
47.395 20038 0 0 0 26205126126 0 0 1195 incomplete 3h 519007'105071 526166:415908 [18,9,6]p18 [18,9,6]p18 2021-04-01T00:30:15.873123-0600 2021-03-28T19:06:44.420250-0600
47.396 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4926 [10,27,11]p10 [10,27,11]p10 2021-04-01T03:08:00.801108-0600 2021-04-01T03:08:00.801108-0600
47.39d 20191 0 0 0 26785005771 0 0 1174 incomplete 3h 519000'97501 526166:435622 [18,8,30]p18 [18,8,30]p18 2021-04-01T03:19:33.193183-0600 2021-03-28T21:51:08.918743-0600
47.3a0 19704 0 0 0 25843237371 0 0 1232 incomplete 3h 519000'98109 526166:472290 [17,6,12]p17 [17,6,12]p17 2021-03-31T20:01:30.219341-0600 2021-03-31T20:01:30.219341-0600
47.3a2 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5900 [8,0,30]p8 [8,0,30]p8 2021-03-28T00:48:35.353609-0600 2021-03-23T21:36:48.891134-0600
47.3a5 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4926 [10,15,27]p10 [10,15,27]p10 2021-03-29T00:02:18.249404-0600 2021-03-22T01:49:34.329955-0600
47.3a6 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:9425 [10,8,1]p10 [10,8,1]p10 2021-04-01T00:34:19.181487-0600 2021-03-25T22:14:50.025036-0600
47.3a9 19781 0 0 0 26121293869 0 0 1059 incomplete 3h 518995'78798 526166:461515 [4,6,7]p4 [4,6,7]p4 2021-04-01T02:24:45.251229-0600 2021-03-26T21:16:30.559792-0600
47.3ac 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5105 [27,6,3]p27 [27,6,3]p27 2021-04-01T00:59:44.905040-0600 2021-03-28T22:17:46.812600-0600
47.3b9 20158 0 0 0 26309188882 0 0 1145 incomplete 3h 518991'97438 526166:446281 [23,27,19]p23 [23,27,19]p23 2021-04-01T22:45:24.442537-0600 2021-03-27T22:39:30.495374-0600
47.3bc 19965 0 0 0 26312334005 0 0 1073 incomplete 3h 519025'107166 526166:422905 [28,7,27]p28 [28,7,27]p28 2021-04-01T01:30:47.098431-0600 2021-03-28T02:54:00.932323-0600
47.3c4 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5033 [27,30,16]p27 [27,30,16]p27 2021-04-02T00:12:19.452532-0600 2021-03-29T22:03:48.403233-0600
47.3ca 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5616 [19,0,6]p19 [19,0,6]p19 2021-04-01T01:20:59.985905-0600 2021-04-01T01:20:59.985905-0600
47.3cc 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4957 [7,6,11]p7 [7,6,11]p7 2021-03-31T19:47:28.264857-0600 2021-03-25T20:46:06.383941-0600
47.3d0 19937 0 0 0 26049861620 0 0 1024 incomplete 3h 518993'84837 526166:452861 [0,8,9]p0 [0,8,9]p0 2021-03-31T19:55:26.225437-0600 2021-03-25T20:28:03.780821-0600
47.3d3 20339 0 0 0 25850714832 0 0 1095 incomplete 3h 518995'74885 526166:444973 [33,6,10]p33 [33,6,10]p33 2021-04-01T20:54:11.410821-0600 2021-03-26T21:28:26.184263-0600
47.3d4 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5825 [6,19,13]p6 [6,19,13]p6 2021-03-28T02:04:07.939796-0600 2021-03-25T19:59:12.362169-0600
47.3d7 20231 0 0 0 26678761009 0 0 1113 incomplete 3h 519000'92790 526166:513850 [1,9,30]p1 [1,9,30]p1 2021-03-30T23:25:38.089760-0600 2021-03-30T23:25:38.089760-0600
47.3dd 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5073 [7,5,6]p7 [7,5,6]p7 2021-03-29T02:45:38.444232-0600 2021-03-27T02:03:11.840141-0600
47.3de 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5123 [7,17,6]p7 [7,17,6]p7 2021-03-30T19:51:55.985747-0600 2021-03-30T19:51:55.985747-0600
47.3e5 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:5047 [9,2,8]p9 [9,2,8]p9 2021-03-30T20:23:36.528228-0600 2021-03-27T00:34:40.483972-0600
47.3f0 20197 0 0 0 26031954397 0 0 1113 incomplete 3h 518995'78417 526166:617984 [11,6,14]p11 [11,6,14]p11 2021-03-30T21:24:11.688564-0600 2021-03-28T00:18:24.404007-0600
47.3f2 20264 0 0 0 26822178361 0 0 1031 incomplete 3h 518997'84027 526166:561157 [1,10,30]p1 [1,10,30]p1 2021-03-30T21:30:04.470335-0600 2021-03-27T19:23:33.541545-0600
47.3fb 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4980 [10,1,7]p10 [10,1,7]p10 2021-03-31T23:40:26.909973-0600 2021-03-28T02:33:30.185498-0600
47.3ff 0 0 0 0 0 0 0 0 incomplete 3h 0'0 526166:4997 [7,9,2]p7 [7,9,2]p7 2021-03-31T23:15:02.476433-0600 2021-03-27T23:34:35.724944-0600
* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details.
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "incomplete",
"epoch": 526167,
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"info": {
"pgid": "47.3f0s0",
"last_update": "518995'78417",
"last_complete": "518995'78417",
"log_tail": "518483'77304",
"last_user_version": 78417,
"last_backfill": "MAX",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2b"
}
],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 526088,
"same_interval_since": 526088,
"same_primary_since": 526052,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "518995'78417",
"reported_seq": "617985",
"reported_epoch": "526167",
"state": "incomplete",
"last_fresh": "2021-04-09T12:07:10.969454-0600",
"last_change": "2021-04-09T08:55:03.931696-0600",
"last_active": "2021-04-03T10:17:20.577281-0600",
"last_peered": "2021-04-03T10:17:10.685305-0600",
"last_clean": "2021-04-03T10:17:07.001797-0600",
"last_became_active": "2021-04-03T10:17:10.644563-0600",
"last_became_peered": "2021-04-03T10:17:10.644563-0600",
"last_unstale": "2021-04-09T12:07:10.969454-0600",
"last_undegraded": "2021-04-09T12:07:10.969454-0600",
"last_fullsized": "2021-04-09T12:07:10.969454-0600",
"mapping_epoch": 526088,
"log_start": "518483'77304",
"ondisk_log_start": "518483'77304",
"created": 489203,
"last_epoch_clean": 519046,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"log_size": 1113,
"ondisk_log_size": 1113,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 26031954397,
"num_objects": 20197,
"num_object_clones": 779,
"num_object_copies": 60591,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 20197,
"num_whiteouts": 651,
"num_read": 13898,
"num_read_kb": 1525718,
"num_write": 15931,
"num_write_kb": 876102,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 44059,
"num_bytes_recovered": 75750824328,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 519063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "6(1)",
"pgid": "47.3f0s1",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 526088,
"same_interval_since": 526088,
"same_primary_since": 526052,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "368",
"reported_epoch": "525931",
"state": "down",
"last_fresh": "2021-04-08T14:38:22.846849-0600",
"last_change": "2021-04-08T14:38:22.846849-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-08T14:38:22.846849-0600",
"last_undegraded": "2021-04-08T14:38:22.846849-0600",
"last_fullsized": "2021-04-08T14:38:22.846849-0600",
"mapping_epoch": 526088,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 489203,
"last_epoch_clean": 519046,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
11
],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "13(0)",
"pgid": "47.3f0s0",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 526088,
"same_interval_since": 526088,
"same_primary_since": 526052,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "569",
"reported_epoch": "520301",
"state": "incomplete",
"last_fresh": "2021-04-04T10:19:59.460563-0600",
"last_change": "2021-04-04T10:19:59.460563-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-04T10:19:59.460563-0600",
"last_undegraded": "2021-04-04T10:19:59.460563-0600",
"last_fullsized": "2021-04-04T10:19:59.460563-0600",
"mapping_epoch": 526088,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 489203,
"last_epoch_clean": 519046,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
34
],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "14(2)",
"pgid": "47.3f0s2",
"last_update": "518995'78417",
"last_complete": "518995'78417",
"log_tail": "518483'77304",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2b"
}
],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 526088,
"same_interval_since": 526088,
"same_primary_since": 526052,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "518995'78417",
"reported_seq": "26",
"reported_epoch": "526051",
"state": "unknown",
"last_fresh": "2021-04-09T08:55:03.329450-0600",
"last_change": "2021-04-09T08:55:03.329450-0600",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2021-04-09T08:55:03.329450-0600",
"last_undegraded": "2021-04-09T08:55:03.329450-0600",
"last_fullsized": "2021-04-09T08:55:03.329450-0600",
"mapping_epoch": 526088,
"log_start": "518483'77304",
"ondisk_log_start": "518483'77304",
"created": 489203,
"last_epoch_clean": 519046,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"log_size": 1113,
"ondisk_log_size": 1113,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 519063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "15(1)",
"pgid": "47.3f0s1",
"last_update": "518995'78417",
"last_complete": "518995'78417",
"log_tail": "518483'77304",
"last_user_version": 0,
"last_backfill": "MIN",
"purged_snaps": [
{
"start": "428",
"length": "7b"
},
{
"start": "4a4",
"length": "32"
},
{
"start": "4d7",
"length": "2f"
},
{
"start": "507",
"length": "30"
},
{
"start": "539",
"length": "33"
},
{
"start": "56d",
"length": "2e"
},
{
"start": "59c",
"length": "1"
},
{
"start": "59e",
"length": "2b"
}
],
"history": {
"epoch_created": 489203,
"epoch_pool_created": 328088,
"last_epoch_started": 519063,
"last_interval_started": 519062,
"last_epoch_clean": 519046,
"last_interval_clean": 519045,
"last_epoch_split": 489203,
"last_epoch_marked_full": 0,
"same_up_since": 526088,
"same_interval_since": 526088,
"same_primary_since": 526052,
"last_scrub": "515994'73333",
"last_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"last_deep_scrub": "515407'70773",
"last_deep_scrub_stamp": "2021-03-28T00:18:24.404007-0600",
"last_clean_scrub_stamp": "2021-03-30T21:24:11.688564-0600",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 526088,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
11,
6,
14
],
"acting": [
11,
6,
14
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 11,
"acting_primary": 11,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 519063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "34(2)",
"pgid": "47.3f0s2",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Peering/Incomplete",
"enter_time": "2021-04-09T08:55:03.931670-0600",
"comment": "not enough complete instances of this PG"
},
{
"name": "Started/Primary/Peering",
"enter_time": "2021-04-09T08:55:03.832251-0600",
"past_intervals": [
{
"first": "519045",
"last": "526087",
"all_participants": [
{
"osd": 6,
"shard": 1
},
{
"osd": 11,
"shard": 0
},
{
"osd": 13,
"shard": 0
},
{
"osd": 14,
"shard": 2
},
{
"osd": 15,
"shard": 1
},
{
"osd": 34,
"shard": 2
}
],
"intervals": [
{
"first": "520091",
"last": "520148",
"acting": "13(0),14(2)"
},
{
"first": "520247",
"last": "520254",
"acting": "6(1),13(0)"
},
{
"first": "525633",
"last": "525638",
"acting": "6(1),14(2)"
},
{
"first": "526037",
"last": "526050",
"acting": "11(0),14(2)"
},
{
"first": "526074",
"last": "526087",
"acting": "6(1),11(0)"
}
]
}
],
"probing_osds": [
"6(1)",
"11(0)",
"13(0)",
"14(2)",
"15(1)",
"34(2)"
],
"down_osds_we_would_probe": [],
"peering_blocked_by": [],
"peering_blocked_by_detail": [
{
"detail": "peering_blocked_by_history_les_bound"
}
]
},
{
"name": "Started",
"enter_time": "2021-04-09T08:55:03.832195-0600"
}
],
"agent_state": {}
}
cluster:
id: 587ad752-283b-4359-8b77-f52d25f10c25
health: HEALTH_WARN
1 MDSs report slow metadata IOs
1 MDSs report slow requests
1 MDSs behind on trimming
noout flag(s) set
Reduced data availability: 218 pgs inactive, 218 pgs incomplete
429 pgs not deep-scrubbed in time
58 pgs not scrubbed in time
116 slow ops, oldest one blocked for 7735 sec, daemons [osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.18]... have slow ops.
too many PGs per OSD (317 > max 250)
services:
mon: 3 daemons, quorum rd240,server,dl380g7 (age 2h)
mgr: rd240(active, since 2h), standbys: server, dl380g7
mds: cephfs:1 {0=server=up:active} 2 up:standby
osd: 37 osds: 37 up (since 2h), 25 in (since 2h)
flags noout
data:
pools: 12 pools, 2649 pgs
objects: 87.62M objects, 45 TiB
usage: 100 TiB used, 84 TiB / 184 TiB avail
pgs: 8.230% pgs not active
2342 active+clean
218 incomplete
81 active+clean+snaptrim_wait
6 active+clean+scrubbing+deep
2 active+clean+snaptrim
io:
client: 1.2 KiB/s rd, 224 KiB/s wr, 17 op/s rd, 18 op/s wr
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 184 TiB 84 TiB 100 TiB 100 TiB 54.48
ssd 0 B 0 B 0 B 0 B 0
TOTAL 184 TiB 84 TiB 100 TiB 100 TiB 54.48
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
cephfs_data 21 1024 20 TiB 59.83M 55 TiB 51.39 17 TiB
cephfs_metadata 22 256 83 GiB 9.68M 223 GiB 0.42 17 TiB
rbd 30 128 4.0 TiB 1.16M 9.8 TiB 15.86 17 TiB
device_health_metrics 31 1 52 MiB 112 156 MiB 0 17 TiB
.rgw.root 32 32 5.8 KiB 4 782 KiB 0 17 TiB
default.rgw.log 33 32 3.4 KiB 32 6 MiB 0 17 TiB
default.rgw.control 34 32 0 B 8 0 B 0 17 TiB
default.rgw.meta 35 8 345 B 2 384 KiB 0 17 TiB
nfs-ganesha 36 16 0 B 1 0 B 0 26 TiB
claypool 47 1024 21 TiB 16.77M 32 TiB 37.99 35 TiB
fast 54 64 49 GiB 12.85k 45 GiB 0.08 17 TiB
caykcloud 55 32 563 GiB 145.52k 1.3 TiB 2.44 17 TiB
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
