Sage,
I changed that param and ran osd.21 again. This looks like the relevant output:==== 47+0+0 (1614986835 0 0) 0x7f11680009e0 con 0x7f12080145c0
-1> 2014-02-11 08:19:07.954574 7f1259ff3700 10 filestore(/var/lib/ceph/osd/ceph-21) FileStore::read(2.28b_head/c28f7a8b/rbd_data.a623c2ae8944a.00000000004ddda9/head//2) pread error: (5) Input/output error
I admittedly don't know how to decode the argument indicated for read() to look on the OSD filesystem to see if it exists and is readable. 'find' reveals a file with 'a623c2ae8944a' and '00000000004ddda9' in the name:
2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\udata.a623c2ae8944a.00000000004ddda9__head_C28F7A8B__2
That file does appear to have a problem:
riker current # stat "./2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\udata.a623c2ae8944a.00000000004ddda9__head_C28F7A8B__2"
File: './2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\\udata.a623c2ae8944a.00000000004ddda9__head_C28F7A8B__2'
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: fd06h/64774d Inode: 225631 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2014-02-04 01:42:29.173460625 -0800
Modify: 2014-02-04 01:42:33.006371867 -0800
Change: 2014-02-04 01:42:33.007371844 -0800
Birth: -
riker current # cat "./2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\udata.a623c2ae8944a.00000000004ddda9__head_C28F7A8B__2" > /dev/null
cat: ./2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\udata.a623c2ae8944a.00000000004ddda9__head_C28F7A8B__2: Input/output error
Kernel log examination leads me to believe that file is on a physically bad part of the disk:
Feb 11 08:29:20 riker kernel: [211279.501377] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
Feb 11 08:29:20 riker kernel: [211279.501379] ata4.00: irq_stat 0x40000008
Feb 11 08:29:20 riker kernel: [211279.501381] ata4.00: failed command: READ FPDMA QUEUED
Feb 11 08:29:20 riker kernel: [211279.501384] ata4.00: cmd 60/08:00:58:68:da/00:00:00:00:00/40 tag 0 ncq 4096 in
Feb 11 08:29:20 riker kernel: [211279.501384] res 41/40:08:5f:68:da/00:00:00:00:00/00 Emask 0x409 (media error) <F>
Feb 11 08:29:20 riker kernel: [211279.501386] ata4.00: status: { DRDY ERR }
Feb 11 08:29:20 riker kernel: [211279.501387] ata4.00: error: { UNC }
Feb 11 08:29:20 riker kernel: [211279.518847] xhci_hcd 0000:00:14.0: Waiting for status stage event
Feb 11 08:29:20 riker kernel: [211279.538535] xhci_hcd 0000:00:14.0: Waiting for status stage event
Feb 11 08:29:20 riker kernel: [211279.559359] xhci_hcd 0000:00:14.0: Waiting for status stage event
Feb 11 08:29:20 riker kernel: [211279.580231] xhci_hcd 0000:00:14.0: Waiting for status stage event
Feb 11 08:29:20 riker kernel: [211279.582045] ata4.00: configured for UDMA/133
Feb 11 08:29:20 riker kernel: [211279.582052] sd 3:0:0:0: [sdc] Unhandled sense code
Feb 11 08:29:20 riker kernel: [211279.582053] sd 3:0:0:0: [sdc]
Feb 11 08:29:20 riker kernel: [211279.582054] Result: hostbyte=0x00 driverbyte=0x08
Feb 11 08:29:20 riker kernel: [211279.582055] sd 3:0:0:0: [sdc]
Feb 11 08:29:20 riker kernel: [211279.582055] Sense Key : 0x3 [current] [descriptor]
Feb 11 08:29:20 riker kernel: [211279.582057] Descriptor sense data with sense descriptors (in hex):
Feb 11 08:29:20 riker kernel: [211279.582058] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Feb 11 08:29:20 riker kernel: [211279.582063] 00 da 68 5f
Feb 11 08:29:20 riker kernel: [211279.582065] sd 3:0:0:0: [sdc]
Feb 11 08:29:20 riker kernel: [211279.582065] ASC=0x11 ASCQ=0x4
Feb 11 08:29:20 riker kernel: [211279.582066] sd 3:0:0:0: [sdc] CDB:
Feb 11 08:29:20 riker kernel: [211279.582067] cdb[0]=0x28: 28 00 00 da 68 58 00 00 08 00
Feb 11 08:29:20 riker kernel: [211279.582071] end_request: I/O error, dev sdc, sector 14313567
Feb 11 08:29:20 riker kernel: [211279.582079] ata4: EH complete
Is the problem I'm having, essentially, that only OSD.21 had that file at the time it failed, so there is no other replica to repair with? Or could I try something to get Ceph to recover from another replica?
-Aaron
On Tue, Feb 11, 2014 at 8:15 AM, Sage Weil <sage@xxxxxxxxxxx> wrote:
You are getting EIO back from the disk. Try adding
debug filestore = 20
so that we can see which file(s) it is reading. If you're lucky, the
error is with a different PG than the one(s) that need to be recovered and
the data can be moved out of the way.
sage
On Mon, 10 Feb 2014, Aaron Ten Clay wrote:
> Sage,
>
> Thanks for the quick response!
>
> With these settings in the ceph.conf:
>
> --- snip ---
> [osd.21]
> host = riker
> devs =
> /dev/ceph-osd-21/data
>
> debug ms = 1/5
> debug osd = 1/5
> debug filestore = 1/5
> debug journal = 1
> debug monc = 5/20
> --- snip ---
>
> The log output is pasted at http://hastebin.com/tigimipofu to avoid bloating
> the listserv.
>
> -Aaron
>
>
>
>
>
>
> On Mon, Feb 10, 2014 at 8:19 PM, Sage Weil <sage@xxxxxxxxxxx> wrote:
> Hi Aaron,
>
> The key question is why osd.21 is crashing. Can you attach the
> last few
> hundred lines of the log after the crash (which should include a
> stack
> trace and a bit of context)?
>
> Thanks!
> sage
>
>
> On Mon, 10 Feb 2014, Aaron Ten Clay wrote:
>
> > Hi everyone,
> >
> > I've run into a problem with my cluster - 1 pg is incomplete,
> and that is
> > blocking reads for a 100TiB RBD volume. (The VM actually halts
> execution,
> > it's not like a normal I/O problem where the virtual
> controller times out
> > and tries to reset the bus, etc.)
> >
> > I've read several threads about the problem with incomplete
> pgs and have dug
> > around quite a bit, but I suspect I don't quite know where to
> look for the
> > information I need.
> >
> > The pool holding this volume has a size of 2, with min_size of
> 2. I suspect
> > the problem began when two osds, in separate hosts, failed
> within a short
> > time of each other. (osd.2 and osd.21 in this case.) The
> physical disk for
> > osd.2 is dead, but 21's disk seems okay and the XFS filesystem
> behind it
> > doesn't have any problems that xfs_repair can find.
> >
> > In attempt to resolve the issue, I've restarted the individual
> osds, the
> > entire cluster, rebooted all the cluster hosts, and upgraded
> to the latest
> > devel build to rule out having hit a known and fixed bug. Most
> recently, I
> > tried marking osd 8 'out', since I believe 5 has the data that
> is missing. I
> > can provide logfiles from the osds 5 and 8 with increased
> debugging if
> > that'll help.
> >
> > I can restart osd.21, and that makes the "incomplete" pg go
> away for a few
> > minutes, but osd.21 crashes within 30 seconds of being
> started, and the
> > incomplete pg comes back when that happens.
> >
> > Any suggestions on how to troubleshoot this further would be
> helpful. I
> > thought I could chronicle my attempts to date to avoid
> duplication of effort
> > but I have tried too many things to clearly reconstruct the
> path.
> >
> > Thanks in advance!
> >
> >
> > Here are some various stats that might help:
> >
> > aaron@seven ~ $ ceph -v
> > ceph version 0.76 (3b990136bfab74249f166dd742fd8e61637e63d9)
> >
> >
> > aaron@seven ~ $ ceph pg stat
> > v7286520: 2200 pgs: 2048 active+clean, 78
> active+remapped+wait_backfill, 72
> > active+remapped+backfilling, 1 incomplete, 1
> active+clean+inconsistent;
> > 21577 GB data, 41396 GB used, 27946 GB / 69343 GB avail;
> 461896/11663686
> > objects degraded (3.960%); 210 MB/s, 53 objects/s recovering
> >
> >
> > aaron@seven ~ $ ceph health detail
> > HEALTH_ERR 78 pgs backfill; 72 pgs backfilling; 1 pgs
> incomplete; 1 pgs
> > inconsistent; 1 pgs stuck inactive; 151 pgs stuck unclean;
> recovery
> > 459868/11663686 objects degraded (3.943%); 1 scrub errors; mds
> picard is
> > laggy
> > pg 2.28b is stuck inactive since forever, current state
> incomplete, last
> > acting [6,5]
> > pg 2.6b is stuck unclean for 1161.425227, current state
> > active+remapped+backfilling, last acting [19,8,4]
> > pg 3.1e4 is stuck unclean for 888.661689, current state
> > active+remapped+wait_backfill, last acting [8,20,22]
> > pg 2.1e5 is stuck unclean for 888.661578, current state
> > active+remapped+wait_backfill, last acting [8,20,22]
> > pg 2.127 is stuck unclean for 889.325675, current state
> > active+remapped+backfilling, last acting [5,8,11]
> > pg 2.63 is stuck unclean for 888.661689, current state
> > active+remapped+wait_backfill, last acting [8,16,3]
> > pg 3.62 is stuck unclean for 888.661682, current state
> > active+remapped+wait_backfill, last acting [8,16,3]
> > pg 2.1e0 is stuck unclean for 888.661675, current state
> > active+remapped+wait_backfill, last acting [8,19,1]
> > pg 2.420 is stuck unclean for 888.662537, current state
> > active+remapped+wait_backfill, last acting [8,17,1]
> > pg 3.1df is stuck unclean for 888.661671, current state
> > active+remapped+wait_backfill, last acting [8,19,1]
> > pg 2.118 is stuck unclean for 888.662593, current state
> > active+remapped+wait_backfill, last acting [8,19,3]
> > pg 2.29e is stuck unclean for 219581.715114, current state
> > active+remapped+backfilling, last acting [14,8,7]
> > pg 3.114 is stuck unclean for 888.662518, current state
> > active+remapped+wait_backfill, last acting [8,14,3]
> > pg 2.115 is stuck unclean for 888.662537, current state
> > active+remapped+backfilling, last acting [8,14,3]
> > pg 2.57 is stuck unclean for 159935.373946, current state
> > active+remapped+backfilling, last acting [6,8,0]
> > pg 3.56 is stuck unclean for 1643032.146070, current state
> > active+remapped+wait_backfill, last acting [6,8,0]
> > pg 2.1d4 is stuck unclean for 888.661671, current state
> > active+remapped+backfilling, last acting [8,13,3]
> > pg 3.117 is stuck unclean for 888.662503, current state
> > active+remapped+wait_backfill, last acting [8,19,3]
> > pg 2.294 is stuck unclean for 888.662490, current state
> > active+remapped+wait_backfill, last acting [8,17,20]
> > pg 2.41a is stuck unclean for 1161.455746, current state
> > active+remapped+wait_backfill, last acting [20,8,0]
> > pg 2.1d0 is stuck unclean for 888.661674, current state
> > active+remapped+wait_backfill, last acting [8,16,14]
> > pg 3.351 is stuck unclean for 888.661636, current state
> > active+remapped+wait_backfill, last acting [8,11,17]
> > pg 3.293 is stuck unclean for 888.662448, current state
> > active+remapped+wait_backfill, last acting [8,17,20]
> > pg 2.352 is stuck unclean for 888.661623, current state
> > active+remapped+wait_backfill, last acting [8,11,17]
> > pg 3.1cf is stuck unclean for 888.661638, current state
> > active+remapped+wait_backfill, last acting [8,16,14]
> > pg 2.40d is stuck unclean for 889.663590, current state
> > active+remapped+backfilling, last acting [16,8,3]
> > pg 2.34f is stuck unclean for 888.661581, current state
> > active+remapped+wait_backfill, last acting [8,18,9]
> > pg 3.34e is stuck unclean for 888.661644, current state
> > active+remapped+wait_backfill, last acting [8,18,9]
> > pg 2.4a is stuck unclean for 146084.212967, current state
> > active+remapped+backfilling, last acting [16,8,4]
> > pg 2.1cb is stuck unclean for 81410.959695, current state
> > active+remapped+backfilling, last acting [19,8,9]
> > pg 2.1ca is stuck unclean for 160231.487694, current state
> > active+remapped+backfilling, last acting [15,8,7]
> > pg 2.28b is stuck unclean since forever, current state
> incomplete, last
> > acting [6,5]
> > pg 2.409 is stuck unclean for 160095.566719, current state
> > active+remapped+backfilling, last acting [4,8,1]
> > pg 2.345 is stuck unclean for 219637.448262, current state
> > active+remapped+backfilling, last acting [6,8,15]
> > pg 2.405 is stuck unclean for 160095.566704, current state
> > active+remapped+backfilling, last acting [14,8,6]
> > pg 2.fc is stuck unclean for 888.662298, current state
> > active+remapped+wait_backfill, last acting [8,20,3]
> > pg 2.342 is stuck unclean for 219394.652780, current state
> > active+remapped+backfilling, last acting [6,8,22]
> > pg 2.1bf is stuck unclean for 889.325454, current state
> > active+remapped+backfilling, last acting [11,8,7]
> > pg 2.1b8 is stuck unclean for 889.326218, current state
> > active+remapped+backfilling, last acting [15,8,5]
> > pg 3.fb is stuck unclean for 888.662289, current state
> > active+remapped+wait_backfill, last acting [8,20,3]
> > pg 2.338 is stuck unclean for 1161.424960, current state
> > active+remapped+backfilling, last acting [19,8,13]
> > pg 2.3f9 is stuck unclean for 889.326078, current state
> > active+remapped+backfilling, last acting [14,8,10]
> > pg 3.1b6 is stuck unclean for 888.661612, current state
> > active+remapped+wait_backfill, last acting [8,17,0]
> > pg 2.1b7 is stuck unclean for 888.661625, current state
> > active+remapped+wait_backfill, last acting [8,17,0]
> > pg 2.3f7 is stuck unclean for 888.662259, current state
> > active+remapped+backfilling, last acting [8,11,0]
> > pg 2.270 is stuck unclean for 889.669800, current state
> > active+remapped+backfilling, last acting [6,8,7]
> > pg 2.2c is stuck unclean for 888.661599, current state
> > active+remapped+wait_backfill, last acting [8,15,9]
> > pg 3.2d is stuck unclean for 888.661593, current state
> > active+remapped+wait_backfill, last acting [8,15,3]
> > pg 2.273 is stuck unclean for 1161.773478, current state
> > active+remapped+backfilling, last acting [18,8,1]
> > pg 2.3f0 is stuck unclean for 85410.265493, current state
> > active+remapped+backfilling, last acting [5,8,10]
> > pg 2.2e is stuck unclean for 888.661595, current state
> > active+remapped+wait_backfill, last acting [8,15,3]
> > pg 3.2b is stuck unclean for 888.661580, current state
> > active+remapped+wait_backfill, last acting [8,15,9]
> > pg 3.1a9 is stuck unclean for 888.661574, current state
> > active+remapped+wait_backfill, last acting [8,18,7]
> > pg 2.3ef is stuck unclean for 1161.426416, current state
> > active+remapped+backfilling, last acting [19,8,7]
> > pg 2.1aa is stuck unclean for 888.661661, current state
> > active+remapped+wait_backfill, last acting [8,18,7]
> > pg 2.32b is stuck unclean for 156663.886748, current state
> > active+remapped+backfilling, last acting [14,8,17]
> > pg 2.21 is stuck unclean for 1161.769888, current state
> > active+remapped+backfilling, last acting [18,8,13]
> > pg 3.1c is stuck unclean for 888.661636, current state
> > active+remapped+wait_backfill, last acting [8,16,0]
> > pg 2.1d is stuck unclean for 888.661648, current state
> > active+remapped+wait_backfill, last acting [8,16,0]
> > pg 2.1c is stuck unclean for 888.661632, current state
> > active+remapped+wait_backfill, last acting [8,20,7]
> > pg 2.323 is stuck unclean for 302501.945292, current state
> > active+remapped+backfilling, last acting [13,8,10]
> > pg 2.4a0 is stuck unclean for 888.661565, current state
> > active+remapped+wait_backfill, last acting [8,16,11]
> > pg 3.19 is stuck unclean for 888.661636, current state
> > active+remapped+wait_backfill, last acting [8,13,0]
> > pg 3.1a is stuck unclean for 888.661619, current state
> > active+remapped+wait_backfill, last acting [8,10,5]
> > pg 2.1b is stuck unclean for 888.661633, current state
> > active+remapped+wait_backfill, last acting [8,10,5]
> > pg 2.25e is stuck unclean for 888.662278, current state
> > active+remapped+backfilling, last acting [8,14,22]
> > pg 3.1b is stuck unclean for 888.661612, current state
> > active+remapped+wait_backfill, last acting [8,20,7]
> > pg 2.1a is stuck unclean for 888.661626, current state
> > active+remapped+wait_backfill, last acting [8,13,0]
> > pg 2.3d9 is stuck unclean for 1161.773372, current state
> > active+remapped+backfilling, last acting [18,8,16]
> > pg 3.12 is stuck unclean for 888.661578, current state
> > active+remapped+wait_backfill, last acting [8,17,19]
> > pg 2.13 is stuck unclean for 888.661591, current state
> > active+remapped+wait_backfill, last acting [8,17,19]
> > pg 2.317 is stuck unclean for 159979.814667, current state
> > active+remapped+backfilling, last acting [4,8,1]
> > pg 2.3d7 is stuck unclean for 889.327661, current state
> > active+remapped+backfilling, last acting [11,8,0]
> > pg 2.30c is stuck unclean for 888.663342, current state
> > active+remapped+backfilling, last acting [8,15,0]
> > pg 2.492 is stuck unclean for 889.664908, current state
> > active+remapped+backfilling, last acting [16,8,22]
> > pg 2.b is stuck unclean for 888.663383, current state
> > active+remapped+backfilling, last acting [8,11,22]
> > pg 2.30f is stuck unclean for 889.327852, current state
> > active+remapped+backfilling, last acting [4,8,9]
> > pg 3.3cd is stuck unclean for 888.662282, current state
> > active+remapped+wait_backfill, last acting [8,11,14]
> > pg 2.3ce is stuck unclean for 888.662337, current state
> > active+remapped+wait_backfill, last acting [8,11,14]
> > pg 2.3c9 is stuck unclean for 888.662309, current state
> > active+remapped+wait_backfill, last acting [8,19,0]
> > pg 3.3c8 is stuck unclean for 888.662329, current state
> > active+remapped+wait_backfill, last acting [8,19,0]
> > pg 2.184 is stuck unclean for 209128.087843, current state
> > active+remapped+backfilling, last acting [13,8,9]
> > pg 2.3cb is stuck unclean for 888.662278, current state
> > active+remapped+wait_backfill, last acting [8,19,22]
> > pg 3.3ca is stuck unclean for 888.662310, current state
> > active+remapped+wait_backfill, last acting [8,19,22]
> > pg 2.3ca is stuck unclean for 1161.455302, current state
> > active+remapped+backfilling, last acting [20,8,15]
> > pg 2.c1 is stuck unclean for 888.662429, current state
> > active+remapped+wait_backfill, last acting [8,15,13]
> > pg 3.c0 is stuck unclean for 888.662398, current state
> > active+remapped+wait_backfill, last acting [8,15,13]
> > pg 2.2 is stuck unclean for 232941.977076, current state
> > active+remapped+backfilling, last acting [6,8,3]
> > pg 2.183 is stuck unclean for 160014.724114, current state
> > active+remapped+backfilling, last acting [5,8,1]
> > pg 3.241 is stuck unclean for 888.662292, current state
> > active+remapped+wait_backfill, last acting [8,18,9]
> > pg 3.301 is stuck unclean for 888.663258, current state
> > active+remapped+wait_backfill, last acting [8,19,9]
> > pg 2.242 is stuck unclean for 888.662284, current state
> > active+remapped+wait_backfill, last acting [8,18,9]
> > pg 2.302 is stuck unclean for 888.663248, current state
> > active+remapped+wait_backfill, last acting [8,19,9]
> > pg 2.23c is stuck unclean for 889.325420, current state
> > active+remapped+backfilling, last acting [14,8,0]
> > pg 2.ba is stuck unclean for 889.327554, current state
> > active+remapped+backfilling, last acting [12,8,1]
> > pg 2.175 is stuck unclean for 219770.886373, current state
> > active+remapped+backfilling, last acting [4,8,9]
> > pg 2.3b8 is stuck unclean for 888.662229, current state
> > active+remapped+wait_backfill, last acting [8,14,22]
> > pg 2.174 is stuck unclean for 888.663214, current state
> > active+remapped+wait_backfill, last acting [8,15,16]
> > pg 2.478 is stuck unclean for 1161.774401, current state
> > active+remapped+backfilling, last acting [18,8,1]
> > pg 2.2f5 is stuck unclean for 888.663146, current state
> > active+remapped+backfilling, last acting [8,12,7]
> > pg 2.176 is stuck unclean for 1161.427407, current state
> > active+remapped+backfilling, last acting [19,8,1]
> > pg 3.3b7 is stuck unclean for 888.662341, current state
> > active+remapped+wait_backfill, last acting [8,14,22]
> > pg 3.173 is stuck unclean for 888.663097, current state
> > active+remapped+wait_backfill, last acting [8,15,16]
> > pg 3.232 is stuck unclean for 888.662195, current state
> > active+remapped+wait_backfill, last acting [8,20,3]
> > pg 2.233 is stuck unclean for 888.662216, current state
> > active+remapped+wait_backfill, last acting [8,20,3]
> > pg 2.af is stuck unclean for 85410.264417, current state
> > active+remapped+backfilling, last acting [6,8,3]
> > pg 3.168 is stuck unclean for 888.663251, current state
> > active+remapped+wait_backfill, last acting [8,20,0]
> > pg 2.169 is stuck unclean for 888.663291, current state
> > active+remapped+wait_backfill, last acting [8,20,0]
> > pg 2.ab is stuck unclean for 888.662356, current state
> > active+remapped+wait_backfill, last acting [8,18,17]
> > pg 3.aa is stuck unclean for 888.662351, current state
> > active+remapped+wait_backfill, last acting [8,18,17]
> > pg 3.228 is stuck unclean for 888.662337, current state
> > active+remapped+wait_backfill, last acting [8,14,0]
> > pg 2.229 is stuck unclean for 888.662350, current state
> > active+remapped+wait_backfill, last acting [8,14,0]
> > pg 2.22b is stuck unclean for 1161.770522, current state
> > active+remapped+backfilling, last acting [18,8,0]
> > pg 2.161 is stuck unclean for 1161.839067, current state
> > active+remapped+backfilling, last acting [17,8,7]
> > pg 2.3a4 is stuck unclean for 160095.566258, current state
> > active+remapped+backfilling, last acting [18,8,0]
> > pg 2.3a7 is stuck unclean for 888.662256, current state
> > active+remapped+wait_backfill, last acting [8,18,22]
> > pg 3.3a6 is stuck unclean for 888.662272, current state
> > active+remapped+wait_backfill, last acting [8,18,22]
> > pg 2.a2 is stuck unclean for 889.669714, current state
> > active+remapped+backfilling, last acting [6,8,7]
> > pg 2.467 is stuck unclean for 889.327384, current state
> > active+remapped+backfilling, last acting [13,8,3]
> > pg 2.3a1 is stuck unclean for 159994.542551, current state
> > active+remapped+backfilling, last acting [19,8,5]
> > pg 3.3a3 is stuck unclean for 1643146.397267, current state
> > active+remapped+backfilling, last acting [18,8,0]
> > pg 2.158 is stuck unclean for 889.331607, current state
> > active+remapped+backfilling, last acting [15,8,9]
> > pg 2.218 is stuck unclean for 1161.455144, current state
> > active+remapped+backfilling, last acting [20,8,16]
> > pg 2.95 is stuck unclean for 888.662241, current state
> > active+remapped+wait_backfill, last acting [8,13,0]
> > pg 3.94 is stuck unclean for 888.662236, current state
> > active+remapped+wait_backfill, last acting [8,13,0]
> > pg 2.399 is stuck unclean for 1161.424873, current state
> > active+remapped+backfilling, last acting [19,8,11]
> > pg 2.2da is stuck unclean for 889.325694, current state
> > active+remapped+backfilling, last acting [10,8,22]
> > pg 2.458 is stuck unclean for 889.664588, current state
> > active+remapped+backfilling, last acting [16,8,19]
> > pg 2.455 is stuck unclean for 888.663123, current state
> > active+remapped+wait_backfill, last acting [8,18,17]
> > pg 2.397 is stuck unclean for 888.662229, current state
> > active+remapped+wait_backfill, last acting [8,17,1]
> > pg 3.396 is stuck unclean for 888.662244, current state
> > active+remapped+wait_backfill, last acting [8,17,1]
> > pg 2.152 is stuck unclean for 889.664670, current state
> > active+remapped+backfilling, last acting [16,8,3]
> > pg 2.20c is stuck unclean for 889.325056, current state
> > active+remapped+backfilling, last acting [14,8,20]
> > pg 2.44d is stuck unclean for 888.663004, current state
> > active+remapped+wait_backfill, last acting [8,11,13]
> > pg 2.2ce is stuck unclean for 1161.838375, current state
> > active+remapped+backfilling, last acting [17,8,13]
> > pg 3.20a is stuck unclean for 1161.425993, current state
> > active+remapped+wait_backfill, last acting [19,8,0]
> > pg 2.20b is stuck unclean for 1161.424832, current state
> > active+remapped+wait_backfill, last acting [19,8,0]
> > pg 2.2c8 is stuck unclean for 1161.427151, current state
> > active+remapped+backfilling, last acting [19,8,9]
> > pg 2.20a is stuck unclean for 889.326613, current state
> > active+remapped+backfilling, last acting [4,8,6]
> > pg 2.87 is stuck unclean for 1161.837786, current state
> > active+remapped+backfilling, last acting [17,8,15]
> > pg 2.384 is stuck unclean for 889.324992, current state
> > active+remapped+backfilling, last acting [14,8,5]
> > pg 2.2c0 is stuck unclean for 888.662914, current state
> > active+remapped+wait_backfill, last acting [8,11,18]
> > pg 2.446 is stuck unclean for 1161.774298, current state
> > active+remapped+backfilling, last acting [18,8,0]
> > pg 3.13c is stuck unclean for 888.662984, current state
> > active+remapped+wait_backfill, last acting [8,16,14]
> > pg 2.13d is stuck unclean for 888.663018, current state
> > active+remapped+backfilling, last acting [8,16,14]
> > pg 2.1fd is stuck unclean for 888.662197, current state
> > active+remapped+backfilling, last acting [8,12,22]
> > pg 2.440 is stuck unclean for 195955.637808, current state
> > active+remapped+backfilling, last acting [17,8,11]
> > pg 3.2bf is stuck unclean for 888.662780, current state
> > active+remapped+wait_backfill, last acting [8,11,18]
> > pg 2.1f8 is stuck unclean for 889.326608, current state
> > active+remapped+backfilling, last acting [11,8,22]
> > pg 2.135 is stuck unclean for 888.662853, current state
> > active+remapped+backfilling, last acting [8,11,0]
> > pg 2.2b5 is stuck unclean for 889.325574, current state
> > active+remapped+backfilling, last acting [10,8,7]
> > pg 2.436 is stuck unclean for 160015.918251, current state
> > active+remapped+backfilling, last acting [17,8,3]
> > pg 2.4a0 is active+remapped+wait_backfill, acting [8,16,11]
> > pg 2.492 is active+remapped+backfilling, acting [16,8,22]
> > pg 2.478 is active+remapped+backfilling, acting [18,8,1]
> > pg 2.467 is active+remapped+backfilling, acting [13,8,3]
> > pg 2.458 is active+remapped+backfilling, acting [16,8,19]
> > pg 2.455 is active+remapped+wait_backfill, acting [8,18,17]
> > pg 2.44d is active+remapped+wait_backfill, acting [8,11,13]
> > pg 2.446 is active+remapped+backfilling, acting [18,8,0]
> > pg 2.440 is active+remapped+backfilling, acting [17,8,11]
> > pg 2.436 is active+remapped+backfilling, acting [17,8,3]
> > pg 2.420 is active+remapped+wait_backfill, acting [8,17,1]
> > pg 2.41a is active+remapped+wait_backfill, acting [20,8,0]
> > pg 2.40d is active+remapped+backfilling, acting [16,8,3]
> > pg 2.409 is active+remapped+backfilling, acting [4,8,1]
> > pg 2.405 is active+remapped+backfilling, acting [14,8,6]
> > pg 2.3f9 is active+remapped+backfilling, acting [14,8,10]
> > pg 2.3f7 is active+remapped+backfilling, acting [8,11,0]
> > pg 2.3f0 is active+remapped+backfilling, acting [5,8,10]
> > pg 2.3ef is active+remapped+backfilling, acting [19,8,7]
> > pg 2.3d9 is active+remapped+backfilling, acting [18,8,16]
> > pg 2.3d7 is active+remapped+backfilling, acting [11,8,0]
> > pg 3.3cd is active+remapped+wait_backfill, acting [8,11,14]
> > pg 2.3ce is active+remapped+wait_backfill, acting [8,11,14]
> > pg 3.3c8 is active+remapped+wait_backfill, acting [8,19,0]
> > pg 2.3c9 is active+remapped+wait_backfill, acting [8,19,0]
> > pg 3.3ca is active+remapped+wait_backfill, acting [8,19,22]
> > pg 2.3cb is active+remapped+wait_backfill, acting [8,19,22]
> > pg 2.3ca is active+remapped+backfilling, acting [20,8,15]
> > pg 2.3b8 is active+remapped+wait_backfill, acting [8,14,22]
> > pg 3.3b7 is active+remapped+wait_backfill, acting [8,14,22]
> > pg 2.3a4 is active+remapped+backfilling, acting [18,8,0]
> > pg 3.3a6 is active+remapped+wait_backfill, acting [8,18,22]
> > pg 2.3a7 is active+remapped+wait_backfill, acting [8,18,22]
> > pg 2.3a1 is active+remapped+backfilling, acting [19,8,5]
> > pg 3.3a3 is active+remapped+backfilling, acting [18,8,0]
> > pg 2.399 is active+remapped+backfilling, acting [19,8,11]
> > pg 3.396 is active+remapped+wait_backfill, acting [8,17,1]
> > pg 2.397 is active+remapped+wait_backfill, acting [8,17,1]
> > pg 2.384 is active+remapped+backfilling, acting [14,8,5]
> > pg 3.351 is active+remapped+wait_backfill, acting [8,11,17]
> > pg 2.352 is active+remapped+wait_backfill, acting [8,11,17]
> > pg 3.34e is active+remapped+wait_backfill, acting [8,18,9]
> > pg 2.34f is active+remapped+wait_backfill, acting [8,18,9]
> > pg 2.345 is active+remapped+backfilling, acting [6,8,15]
> > pg 2.342 is active+remapped+backfilling, acting [6,8,22]
> > pg 2.338 is active+remapped+backfilling, acting [19,8,13]
> > pg 2.32b is active+remapped+backfilling, acting [14,8,17]
> > pg 2.323 is active+remapped+backfilling, acting [13,8,10]
> > pg 2.317 is active+remapped+backfilling, acting [4,8,1]
> > pg 2.30c is active+remapped+backfilling, acting [8,15,0]
> > pg 2.30f is active+remapped+backfilling, acting [4,8,9]
> > pg 3.301 is active+remapped+wait_backfill, acting [8,19,9]
> > pg 2.302 is active+remapped+wait_backfill, acting [8,19,9]
> > pg 2.2f5 is active+remapped+backfilling, acting [8,12,7]
> > pg 2.2da is active+remapped+backfilling, acting [10,8,22]
> > pg 2.2ce is active+remapped+backfilling, acting [17,8,13]
> > pg 2.2c8 is active+remapped+backfilling, acting [19,8,9]
> > pg 2.2c0 is active+remapped+wait_backfill, acting [8,11,18]
> > pg 3.2bf is active+remapped+wait_backfill, acting [8,11,18]
> > pg 2.2b5 is active+remapped+backfilling, acting [10,8,7]
> > pg 2.29e is active+remapped+backfilling, acting [14,8,7]
> > pg 2.294 is active+remapped+wait_backfill, acting [8,17,20]
> > pg 3.293 is active+remapped+wait_backfill, acting [8,17,20]
> > pg 2.28b is incomplete, acting [6,5]
> > pg 2.270 is active+remapped+backfilling, acting [6,8,7]
> > pg 2.273 is active+remapped+backfilling, acting [18,8,1]
> > pg 2.25e is active+remapped+backfilling, acting [8,14,22]
> > pg 3.241 is active+remapped+wait_backfill, acting [8,18,9]
> > pg 2.242 is active+remapped+wait_backfill, acting [8,18,9]
> > pg 2.23c is active+remapped+backfilling, acting [14,8,0]
> > pg 2.233 is active+remapped+wait_backfill, acting [8,20,3]
> > pg 3.232 is active+remapped+wait_backfill, acting [8,20,3]
> > pg 2.229 is active+remapped+wait_backfill, acting [8,14,0]
> > pg 3.228 is active+remapped+wait_backfill, acting [8,14,0]
> > pg 2.22b is active+remapped+backfilling, acting [18,8,0]
> > pg 2.218 is active+remapped+backfilling, acting [20,8,16]
> > pg 2.20c is active+remapped+backfilling, acting [14,8,20]
> > pg 2.20b is active+remapped+wait_backfill, acting [19,8,0]
> > pg 3.20a is active+remapped+wait_backfill, acting [19,8,0]
> > pg 2.20a is active+remapped+backfilling, acting [4,8,6]
> > pg 2.1fd is active+remapped+backfilling, acting [8,12,22]
> > pg 2.1f8 is active+remapped+backfilling, acting [11,8,22]
> > pg 2.1e5 is active+remapped+wait_backfill, acting [8,20,22]
> > pg 3.1e4 is active+remapped+wait_backfill, acting [8,20,22]
> > pg 2.1e0 is active+remapped+wait_backfill, acting [8,19,1]
> > pg 3.1df is active+remapped+wait_backfill, acting [8,19,1]
> > pg 2.1d4 is active+remapped+backfilling, acting [8,13,3]
> > pg 2.1d0 is active+remapped+wait_backfill, acting [8,16,14]
> > pg 3.1cf is active+remapped+wait_backfill, acting [8,16,14]
> > pg 2.1cb is active+remapped+backfilling, acting [19,8,9]
> > pg 2.1ca is active+remapped+backfilling, acting [15,8,7]
> > pg 2.1bf is active+remapped+backfilling, acting [11,8,7]
> > pg 2.1b8 is active+remapped+backfilling, acting [15,8,5]
> > pg 2.1b7 is active+remapped+wait_backfill, acting [8,17,0]
> > pg 3.1b6 is active+remapped+wait_backfill, acting [8,17,0]
> > pg 3.1a9 is active+remapped+wait_backfill, acting [8,18,7]
> > pg 2.1aa is active+remapped+wait_backfill, acting [8,18,7]
> > pg 2.184 is active+remapped+backfilling, acting [13,8,9]
> > pg 2.183 is active+remapped+backfilling, acting [5,8,1]
> > pg 2.175 is active+remapped+backfilling, acting [4,8,9]
> > pg 2.174 is active+remapped+wait_backfill, acting [8,15,16]
> > pg 2.176 is active+remapped+backfilling, acting [19,8,1]
> > pg 3.173 is active+remapped+wait_backfill, acting [8,15,16]
> > pg 2.169 is active+remapped+wait_backfill, acting [8,20,0]
> > pg 3.168 is active+remapped+wait_backfill, acting [8,20,0]
> > pg 2.161 is active+remapped+backfilling, acting [17,8,7]
> > pg 2.158 is active+remapped+backfilling, acting [15,8,9]
> > pg 2.152 is active+remapped+backfilling, acting [16,8,3]
> > pg 2.13d is active+remapped+backfilling, acting [8,16,14]
> > pg 3.13c is active+remapped+wait_backfill, acting [8,16,14]
> > pg 2.135 is active+remapped+backfilling, acting [8,11,0]
> > pg 2.127 is active+remapped+backfilling, acting [5,8,11]
> > pg 2.118 is active+remapped+wait_backfill, acting [8,19,3]
> > pg 2.115 is active+remapped+backfilling, acting [8,14,3]
> > pg 3.114 is active+remapped+wait_backfill, acting [8,14,3]
> > pg 3.117 is active+remapped+wait_backfill, acting [8,19,3]
> > pg 2.fc is active+remapped+wait_backfill, acting [8,20,3]
> > pg 3.fb is active+remapped+wait_backfill, acting [8,20,3]
> > pg 2.cc is active+clean+inconsistent, acting [20,6]
> > pg 3.c0 is active+remapped+wait_backfill, acting [8,15,13]
> > pg 2.c1 is active+remapped+wait_backfill, acting [8,15,13]
> > pg 2.ba is active+remapped+backfilling, acting [12,8,1]
> > pg 2.af is active+remapped+backfilling, acting [6,8,3]
> > pg 3.aa is active+remapped+wait_backfill, acting [8,18,17]
> > pg 2.ab is active+remapped+wait_backfill, acting [8,18,17]
> > pg 2.a2 is active+remapped+backfilling, acting [6,8,7]
> > pg 3.94 is active+remapped+wait_backfill, acting [8,13,0]
> > pg 2.95 is active+remapped+wait_backfill, acting [8,13,0]
> > pg 2.87 is active+remapped+backfilling, acting [17,8,15]
> > pg 2.6b is active+remapped+backfilling, acting [19,8,4]
> > pg 3.62 is active+remapped+wait_backfill, acting [8,16,3]
> > pg 2.63 is active+remapped+wait_backfill, acting [8,16,3]
> > pg 3.56 is active+remapped+wait_backfill, acting [6,8,0]
> > pg 2.57 is active+remapped+backfilling, acting [6,8,0]
> > pg 2.4a is active+remapped+backfilling, acting [16,8,4]
> > pg 3.2d is active+remapped+wait_backfill, acting [8,15,3]
> > pg 2.2c is active+remapped+wait_backfill, acting [8,15,9]
> > pg 2.2e is active+remapped+wait_backfill, acting [8,15,3]
> > pg 3.2b is active+remapped+wait_backfill, acting [8,15,9]
> > pg 2.21 is active+remapped+backfilling, acting [18,8,13]
> > pg 2.1d is active+remapped+wait_backfill, acting [8,16,0]
> > pg 3.1c is active+remapped+wait_backfill, acting [8,16,0]
> > pg 2.1c is active+remapped+wait_backfill, acting [8,20,7]
> > pg 3.19 is active+remapped+wait_backfill, acting [8,13,0]
> > pg 2.1b is active+remapped+wait_backfill, acting [8,10,5]
> > pg 3.1a is active+remapped+wait_backfill, acting [8,10,5]
> > pg 2.1a is active+remapped+wait_backfill, acting [8,13,0]
> > pg 3.1b is active+remapped+wait_backfill, acting [8,20,7]
> > pg 2.13 is active+remapped+wait_backfill, acting [8,17,19]
> > pg 3.12 is active+remapped+wait_backfill, acting [8,17,19]
> > pg 2.b is active+remapped+backfilling, acting [8,11,22]
> > pg 2.2 is active+remapped+backfilling, acting [6,8,3]
> > recovery 459868/11663686 objects degraded (3.943%)
> > 1 scrub errors
> > mds.picard at 10.42.6.21:6800/13626 is laggy/unresponsive
> >
> >
> >
> >
> >
> >
> >
> >
> > aaron@seven ~ $ ceph pg 2.28b query
> > { "state": "incomplete",
> > "epoch": 36361,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5],
> > "info": { "pgid": "2.28b",
> > "last_update": "35256'44286",
> > "last_complete": "35256'44286",
> > "log_tail": "34732'41286",
> > "last_user_version": 0,
> > "last_backfill":
> >
> "84ed7a8b\/rbd_data.a623c2ae8944a.0000000000052a3a\/head\/\/2",
> > "purged_snaps": "[]",
> > "history": { "epoch_created": 1,
> > "last_epoch_started": 36252,
> > "last_epoch_clean": 34760,
> > "last_epoch_split": 0,
> > "same_up_since": 35405,
> > "same_interval_since": 36276,
> > "same_primary_since": 36274,
> > "last_scrub": "34757'44284",
> > "last_scrub_stamp": "2014-02-08 11:33:51.835956",
> > "last_deep_scrub": "34757'44284",
> > "last_deep_scrub_stamp": "2014-02-08
> 11:33:45.299503",
> > "last_clean_scrub_stamp": "2014-02-08
> 11:33:51.835956"},
> > "stats": { "version": "35256'44286",
> > "reported_seq": "727",
> > "reported_epoch": "36361",
> > "state": "incomplete",
> > "last_fresh": "2014-02-10 19:35:37.361600",
> > "last_change": "2014-02-10 19:22:15.856289",
> > "last_active": "0.000000",
> > "last_clean": "0.000000",
> > "last_became_active": "0.000000",
> > "last_unstale": "2014-02-10 19:35:37.361600",
> > "mapping_epoch": 36274,
> > "log_start": "34732'41286",
> > "ondisk_log_start": "34732'41286",
> > "created": 1,
> > "last_epoch_clean": 34760,
> > "parent": "0.0",
> > "parent_split_bits": 0,
> > "last_scrub": "34757'44284",
> > "last_scrub_stamp": "2014-02-08 11:33:51.835956",
> > "last_deep_scrub": "34757'44284",
> > "last_deep_scrub_stamp": "2014-02-08
> 11:33:45.299503",
> > "last_clean_scrub_stamp": "2014-02-08
> 11:33:51.835956",
> > "log_size": 3000,
> > "ondisk_log_size": 3000,
> > "stats_invalid": "0",
> > "stat_sum": { "num_bytes": 13767208960,
> > "num_objects": 3306,
> > "num_object_clones": 0,
> > "num_object_copies": 6612,
> > "num_objects_missing_on_primary": 0,
> > "num_objects_degraded": 0,
> > "num_objects_unfound": 0,
> > "num_objects_dirty": 3300,
> > "num_whiteouts": 0,
> > "num_read": 0,
> > "num_read_kb": 0,
> > "num_write": 0,
> > "num_write_kb": 0,
> > "num_scrub_errors": 0,
> > "num_shallow_scrub_errors": 0,
> > "num_deep_scrub_errors": 0,
> > "num_objects_recovered": 0,
> > "num_bytes_recovered": 0,
> > "num_keys_recovered": 0},
> > "stat_cat_sum": {},
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > "empty": 0,
> > "dne": 0,
> > "incomplete": 1,
> > "last_epoch_started": 36252,
> > "hit_set_history": { "current_last_update": "0'0",
> > "current_last_stamp": "0.000000",
> > "current_info": { "begin": "0.000000",
> > "end": "0.000000",
> > "version": "0'0"},
> > "history": []}},
> > "peer_info": [
> > { "peer": 5,
> > "pgid": "2.28b",
> > "last_update": "34757'44284",
> > "last_complete": "34757'44284",
> > "log_tail": "34732'41284",
> > "last_user_version": 0,
> > "last_backfill":
> >
> "84ed7a8b\/rbd_data.a623c2ae8944a.0000000000052a3a\/head\/\/2",
> > "purged_snaps": "[]",
> > "history": { "epoch_created": 1,
> > "last_epoch_started": 36252,
> > "last_epoch_clean": 34760,
> > "last_epoch_split": 0,
> > "same_up_since": 35405,
> > "same_interval_since": 36276,
> > "same_primary_since": 36274,
> > "last_scrub": "34757'44284",
> > "last_scrub_stamp": "2014-02-08
> 11:33:51.835956",
> > "last_deep_scrub": "34757'44284",
> > "last_deep_scrub_stamp": "2014-02-08
> 11:33:45.299503",
> > "last_clean_scrub_stamp": "2014-02-08
> 11:33:51.835956"},
> > "stats": { "version": "34757'44284",
> > "reported_seq": "247",
> > "reported_epoch": "35404",
> > "state": "down+peering",
> > "last_fresh": "2014-02-09 21:05:56.090968",
> > "last_change": "2014-02-09 21:05:33.224591",
> > "last_active": "0.000000",
> > "last_clean": "0.000000",
> > "last_became_active": "0.000000",
> > "last_unstale": "2014-02-09 21:05:56.090968",
> > "mapping_epoch": 36274,
> > "log_start": "34732'41284",
> > "ondisk_log_start": "34732'41284",
> > "created": 1,
> > "last_epoch_clean": 34760,
> > "parent": "0.0",
> > "parent_split_bits": 0,
> > "last_scrub": "34757'44284",
> > "last_scrub_stamp": "2014-02-08
> 11:33:51.835956",
> > "last_deep_scrub": "34757'44284",
> > "last_deep_scrub_stamp": "2014-02-08
> 11:33:45.299503",
> > "last_clean_scrub_stamp": "2014-02-08
> 11:33:51.835956",
> > "log_size": 3000,
> > "ondisk_log_size": 3000,
> > "stats_invalid": "0",
> > "stat_sum": { "num_bytes": 13771403264,
> > "num_objects": 3307,
> > "num_object_clones": 0,
> > "num_object_copies": 6614,
> > "num_objects_missing_on_primary": 0,
> > "num_objects_degraded": 0,
> > "num_objects_unfound": 0,
> > "num_objects_dirty": 0,
> > "num_whiteouts": 0,
> > "num_read": 0,
> > "num_read_kb": 0,
> > "num_write": 0,
> > "num_write_kb": 0,
> > "num_scrub_errors": 0,
> > "num_shallow_scrub_errors": 0,
> > "num_deep_scrub_errors": 0,
> > "num_objects_recovered": 0,
> > "num_bytes_recovered": 0,
> > "num_keys_recovered": 0},
> > "stat_cat_sum": {},
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > "empty": 0,
> > "dne": 0,
> > "incomplete": 1,
> > "last_epoch_started": 35110,
> > "hit_set_history": { "current_last_update": "0'0",
> > "current_last_stamp": "0.000000",
> > "current_info": { "begin": "0.000000",
> > "end": "0.000000",
> > "version": "0'0"},
> > "history": []}},
> > { "peer": 8,
> > "pgid": "2.28b",
> > "last_update": "35256'44286",
> > "last_complete": "35256'44286",
> > "log_tail": "34732'41284",
> > "last_user_version": 44286,
> > "last_backfill":
> > "a8dd7a8b\/benchmark_data_seven_910_object168\/head\/\/2",
> > "purged_snaps": "[]",
> > "history": { "epoch_created": 1,
> > "last_epoch_started": 35225,
> > "last_epoch_clean": 34760,
> > "last_epoch_split": 0,
> > "same_up_since": 35405,
> > "same_interval_since": 36276,
> > "same_primary_since": 36274,
> > "last_scrub": "34757'44284",
> > "last_scrub_stamp": "2014-02-08
> 11:33:51.835956",
> > "last_deep_scrub": "34757'44284",
> > "last_deep_scrub_stamp": "2014-02-08
> 11:33:45.299503",
> > "last_clean_scrub_stamp": "2014-02-08
> 11:33:51.835956"},
> > "stats": { "version": "35256'44286",
> > "reported_seq": "109",
> > "reported_epoch": "35310",
> > "state": "peering",
> > "last_fresh": "2014-02-09 19:52:07.683337",
> > "last_change": "2014-02-09 19:52:07.683337",
> > "last_active": "0.000000",
> > "last_clean": "0.000000",
> > "last_became_active": "0.000000",
> > "last_unstale": "2014-02-09 19:52:07.683337",
> > "mapping_epoch": 36274,
> > "log_start": "34732'41284",
> > "ondisk_log_start": "34732'41284",
> > "created": 1,
> > "last_epoch_clean": 34760,
> > "parent": "0.0",
> > "parent_split_bits": 0,
> > "last_scrub": "34757'44284",
> > "last_scrub_stamp": "2014-02-08
> 11:33:51.835956",
> > "last_deep_scrub": "34757'44284",
> > "last_deep_scrub_stamp": "2014-02-08
> 11:33:45.299503",
> > "last_clean_scrub_stamp": "2014-02-08
> 11:33:51.835956",
> > "log_size": 3002,
> > "ondisk_log_size": 3002,
> > "stats_invalid": "0",
> > "stat_sum": { "num_bytes": 13763014656,
> > "num_objects": 3305,
> > "num_object_clones": 0,
> > "num_object_copies": 0,
> > "num_objects_missing_on_primary": 0,
> > "num_objects_degraded": 0,
> > "num_objects_unfound": 0,
> > "num_objects_dirty": 0,
> > "num_whiteouts": 0,
> > "num_read": 0,
> > "num_read_kb": 0,
> > "num_write": 0,
> > "num_write_kb": 0,
> > "num_scrub_errors": 0,
> > "num_shallow_scrub_errors": 0,
> > "num_deep_scrub_errors": 0,
> > "num_objects_recovered": 0,
> > "num_bytes_recovered": 0,
> > "num_keys_recovered": 0},
> > "stat_cat_sum": {},
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > "empty": 0,
> > "dne": 0,
> > "incomplete": 1,
> > "last_epoch_started": 35225,
> > "hit_set_history": { "current_last_update": "0'0",
> > "current_last_stamp": "0.000000",
> > "current_info": { "begin": "0.000000",
> > "end": "0.000000",
> > "version": "0'0"},
> > "history": []}}],
> > "recovery_state": [
> > { "name": "Started\/Primary\/Peering",
> > "enter_time": "2014-02-10 19:22:15.855010",
> > "past_intervals": [
> > { "first": 34758,
> > "last": 34796,
> > "maybe_went_rw": 1,
> > "up": [
> > 21],
> > "acting": [
> > 21]},
> > { "first": 34797,
> > "last": 34899,
> > "maybe_went_rw": 1,
> > "up": [
> > 21,
> > 5],
> > "acting": [
> > 21,
> > 5]},
> > { "first": 34900,
> > "last": 34946,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 34947,
> > "last": 34952,
> > "maybe_went_rw": 1,
> > "up": [
> > 21,
> > 5],
> > "acting": [
> > 21,
> > 5]},
> > { "first": 34953,
> > "last": 34957,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 34958,
> > "last": 34959,
> > "maybe_went_rw": 1,
> > "up": [
> > 21,
> > 5],
> > "acting": [
> > 21,
> > 5]},
> > { "first": 34960,
> > "last": 35053,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 35054,
> > "last": 35055,
> > "maybe_went_rw": 1,
> > "up": [
> > 21,
> > 5],
> > "acting": [
> > 21,
> > 5]},
> > { "first": 35056,
> > "last": 35062,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 35063,
> > "last": 35065,
> > "maybe_went_rw": 1,
> > "up": [
> > 21,
> > 5],
> > "acting": [
> > 21,
> > 5]},
> > { "first": 35066,
> > "last": 35068,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 35069,
> > "last": 35071,
> > "maybe_went_rw": 1,
> > "up": [
> > 21,
> > 5],
> > "acting": [
> > 21,
> > 5]},
> > { "first": 35072,
> > "last": 35108,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 35109,
> > "last": 35112,
> > "maybe_went_rw": 1,
> > "up": [
> > 21,
> > 5],
> > "acting": [
> > 21,
> > 5]},
> > { "first": 35113,
> > "last": 35120,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 35121,
> > "last": 35160,
> > "maybe_went_rw": 1,
> > "up": [
> > 8,
> > 5],
> > "acting": [
> > 8,
> > 5]},
> > { "first": 35161,
> > "last": 35174,
> > "maybe_went_rw": 1,
> > "up": [
> > 8],
> > "acting": [
> > 8]},
> > { "first": 35175,
> > "last": 35181,
> > "maybe_went_rw": 1,
> > "up": [
> > 8,
> > 5],
> > "acting": [
> > 8,
> > 5]},
> > { "first": 35182,
> > "last": 35194,
> > "maybe_went_rw": 0,
> > "up": [
> > 8],
> > "acting": [
> > 8]},
> > { "first": 35195,
> > "last": 35214,
> > "maybe_went_rw": 0,
> > "up": [
> > 8,
> > 5],
> > "acting": [
> > 8,
> > 5]},
> > { "first": 35215,
> > "last": 35222,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 35223,
> > "last": 35223,
> > "maybe_went_rw": 0,
> > "up": [
> > 8,
> > 5],
> > "acting": [
> > 8,
> > 5]},
> > { "first": 35224,
> > "last": 35264,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 21,
> > 8]},
> > { "first": 35265,
> > "last": 35265,
> > "maybe_went_rw": 0,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 8]},
> > { "first": 35266,
> > "last": 35287,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > { "first": 35288,
> > "last": 35299,
> > "maybe_went_rw": 1,
> > "up": [
> > 6],
> > "acting": [
> > 6]},
> > { "first": 35300,
> > "last": 35303,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > { "first": 35304,
> > "last": 35305,
> > "maybe_went_rw": 1,
> > "up": [
> > 6],
> > "acting": [
> > 6]},
> > { "first": 35306,
> > "last": 35376,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > { "first": 35377,
> > "last": 35386,
> > "maybe_went_rw": 0,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 35387,
> > "last": 35396,
> > "maybe_went_rw": 0,
> > "up": [],
> > "acting": []},
> > { "first": 35397,
> > "last": 35404,
> > "maybe_went_rw": 1,
> > "up": [
> > 5],
> > "acting": [
> > 5]},
> > { "first": 35405,
> > "last": 35407,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > { "first": 35408,
> > "last": 35616,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 21,
> > 6]},
> > { "first": 35617,
> > "last": 35618,
> > "maybe_went_rw": 0,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6]},
> > { "first": 35619,
> > "last": 36246,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > { "first": 36247,
> > "last": 36248,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 21,
> > 6]},
> > { "first": 36249,
> > "last": 36249,
> > "maybe_went_rw": 0,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6]},
> > { "first": 36250,
> > "last": 36250,
> > "maybe_went_rw": 0,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6,
> > 5]},
> > { "first": 36251,
> > "last": 36273,
> > "maybe_went_rw": 1,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 21,
> > 6]},
> > { "first": 36274,
> > "last": 36275,
> > "maybe_went_rw": 0,
> > "up": [
> > 6,
> > 5],
> > "acting": [
> > 6]}],
> > "probing_osds": [
> > 5,
> > 6],
> > "down_osds_we_would_probe": [
> > 21],
> > "peering_blocked_by": []},
> > { "name": "Started",
> > "enter_time": "2014-02-10 19:22:15.854966"}]}
> >
> >
> > --
> > Aaron Ten Clay
> > http://www.aarontc.com/
> >
> >
>
>
>
>
> --
> Aaron Ten Clay
> http://www.aarontc.com/
>
>
--
Aaron Ten Clay
http://www.aarontc.com/
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
