On Tue, 11 Feb 2014, Aaron Ten Clay wrote: > Sage, > > I changed that param and ran osd.21 again. This looks like the relevant > output: > > ==== 47+0+0 (1614986835 0 0) 0x7f11680009e0 con 0x7f12080145c0 > -1> 2014-02-11 08:19:07.954574 7f1259ff3700 10 > filestore(/var/lib/ceph/osd/ceph-21)FileStore::read(2.28b_head/c28f7a8b/rbd_data.a623c2ae8944a.00000000004ddda9 > /head//2) pread error: (5) Input/output error > > (Entire log posted at http://aarontc.com/ceph/ceph-osd.21.log) > > > I admittedly don't know how to decode the argument indicated for read() to > look on the OSD filesystem to see if it exists and is readable. 'find' > reveals a file with 'a623c2ae8944a' and '00000000004ddda9' in the name: > > 2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\udata.a623c2ae8944a.00000000004ddda9 > __head_C28F7A8B__2 > > > That file does appear to have a problem: > > riker current # stat"./2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\udata.a623c2ae8944a.00000000004dd > da9__head_C28F7A8B__2" > File:'./2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\\udata.a623c2ae8944a.00000000004d > dda9__head_C28F7A8B__2' > Size: 4194304 Blocks: 8200 IO Block: 4096 regular file > Device: fd06h/64774d Inode: 225631 Links: 1 > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2014-02-04 01:42:29.173460625 -0800 > Modify: 2014-02-04 01:42:33.006371867 -0800 > Change: 2014-02-04 01:42:33.007371844 -0800 > Birth: - > riker current # cat"./2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\udata.a623c2ae8944a.00000000004dd > da9__head_C28F7A8B__2" > /dev/null > cat:./2.28b_head/DIR_B/DIR_8/DIR_A/DIR_7/rbd\udata.a623c2ae8944a.00000000004ddd > a9__head_C28F7A8B__2: Input/output error > > > Kernel log examination leads me to believe that file is on a physically bad > part of the disk: > > Feb 11 08:29:20 riker kernel: [211279.501377] ata4.00: exception Emask 0x0 > SAct 0x1 SErr 0x0 action 0x0 > Feb 11 08:29:20 riker kernel: [211279.501379] ata4.00: irq_stat 0x40000008 > Feb 11 08:29:20 riker kernel: [211279.501381] ata4.00: failed command: READ > FPDMA QUEUED > Feb 11 08:29:20 riker kernel: [211279.501384] ata4.00: cmd > 60/08:00:58:68:da/00:00:00:00:00/40 tag 0 ncq 4096 in > Feb 11 08:29:20 riker kernel: [211279.501384] res > 41/40:08:5f:68:da/00:00:00:00:00/00 Emask 0x409 (media error) <F> > Feb 11 08:29:20 riker kernel: [211279.501386] ata4.00: status: { DRDY ERR } > Feb 11 08:29:20 riker kernel: [211279.501387] ata4.00: error: { UNC } > Feb 11 08:29:20 riker kernel: [211279.518847] xhci_hcd 0000:00:14.0: Waiting > for status stage event > Feb 11 08:29:20 riker kernel: [211279.538535] xhci_hcd 0000:00:14.0: Waiting > for status stage event > Feb 11 08:29:20 riker kernel: [211279.559359] xhci_hcd 0000:00:14.0: Waiting > for status stage event > Feb 11 08:29:20 riker kernel: [211279.580231] xhci_hcd 0000:00:14.0: Waiting > for status stage event > Feb 11 08:29:20 riker kernel: [211279.582045] ata4.00: configured for > UDMA/133 > Feb 11 08:29:20 riker kernel: [211279.582052] sd 3:0:0:0: [sdc] Unhandled > sense code > Feb 11 08:29:20 riker kernel: [211279.582053] sd 3:0:0:0: [sdc] > Feb 11 08:29:20 riker kernel: [211279.582054] Result: hostbyte=0x00 > driverbyte=0x08 > Feb 11 08:29:20 riker kernel: [211279.582055] sd 3:0:0:0: [sdc] > Feb 11 08:29:20 riker kernel: [211279.582055] Sense Key : 0x3 [current] > [descriptor] > Feb 11 08:29:20 riker kernel: [211279.582057] Descriptor sense data with > sense descriptors (in hex): > Feb 11 08:29:20 riker kernel: [211279.582058] 72 03 11 04 00 00 00 > 0c 00 0a 80 00 00 00 00 00 > Feb 11 08:29:20 riker kernel: [211279.582063] 00 da 68 5f > Feb 11 08:29:20 riker kernel: [211279.582065] sd 3:0:0:0: [sdc] > Feb 11 08:29:20 riker kernel: [211279.582065] ASC=0x11 ASCQ=0x4 > Feb 11 08:29:20 riker kernel: [211279.582066] sd 3:0:0:0: [sdc] CDB: > Feb 11 08:29:20 riker kernel: [211279.582067] cdb[0]=0x28: 28 00 00 da 68 58 > 00 00 08 00 > Feb 11 08:29:20 riker kernel: [211279.582071] end_request: I/O error, dev > sdc, sector 14313567 > Feb 11 08:29:20 riker kernel: [211279.582079] ata4: EH complete > > > Is the problem I'm having, essentially, that only OSD.21 had that file at > the time it failed, so there is no other replica to repair with? Or could I > try something to get Ceph to recover from another replica? Pretty much. If the other disk that failed has a copy of this data, then you could scrounge that up. Otherwise, I suggest overwriting the file with some zeroes (via dd; you want to preserve the xattrs) and then restarting the OSD. Hopefully there aren't others that are in a similar situation. You can later map this back to which RBD image is affected by doing 'rbd info' on each image and looking for the signature a623c2ae8944a in the block_name_prefix. In other news, 2x replication is a bit risky when you work out the odds of a double failure, IMO. :) sage > > Thanks for your help! > -Aaron > > > > On Tue, Feb 11, 2014 at 8:15 AM, Sage Weil <sage@xxxxxxxxxxx> wrote: > You are getting EIO back from the disk. Try adding > > debug filestore = 20 > > so that we can see which file(s) it is reading. If you're > lucky, the > error is with a different PG than the one(s) that need to be > recovered and > the data can be moved out of the way. > > sage > > > On Mon, 10 Feb 2014, Aaron Ten Clay wrote: > > > Sage, > > > > Thanks for the quick response! > > > > With these settings in the ceph.conf: > > > > --- snip --- > > [osd.21] > > host = > riker > > devs = > > /dev/ceph-osd-21/data > > > > debug ms = 1/5 > > debug osd = 1/5 > > debug filestore = 1/5 > > debug journal = 1 > > debug monc = 5/20 > > --- snip --- > > > > The log output is pasted at http://hastebin.com/tigimipofu to > avoid bloating > > the listserv. > > > > -Aaron > > > > > > > > > > > > > > On Mon, Feb 10, 2014 at 8:19 PM, Sage Weil <sage@xxxxxxxxxxx> > wrote: > > Hi Aaron, > > > > The key question is why osd.21 is crashing. Can you > attach the > > last few > > hundred lines of the log after the crash (which should > include a > > stack > > trace and a bit of context)? > > > > Thanks! > > sage > > > > > > On Mon, 10 Feb 2014, Aaron Ten Clay wrote: > > > > > Hi everyone, > > > > > > I've run into a problem with my cluster - 1 pg is > incomplete, > > and that is > > > blocking reads for a 100TiB RBD volume. (The VM > actually halts > > execution, > > > it's not like a normal I/O problem where the virtual > > controller times out > > > and tries to reset the bus, etc.) > > > > > > I've read several threads about the problem with > incomplete > > pgs and have dug > > > around quite a bit, but I suspect I don't quite know > where to > > look for the > > > information I need. > > > > > > The pool holding this volume has a size of 2, with > min_size of > > 2. I suspect > > > the problem began when two osds, in separate hosts, > failed > > within a short > > > time of each other. (osd.2 and osd.21 in this case.) > The > > physical disk for > > > osd.2 is dead, but 21's disk seems okay and the XFS > filesystem > > behind it > > > doesn't have any problems that xfs_repair can find. > > > > > > In attempt to resolve the issue, I've restarted the > individual > > osds, the > > > entire cluster, rebooted all the cluster hosts, and > upgraded > > to the latest > > > devel build to rule out having hit a known and fixed > bug. Most > > recently, I > > > tried marking osd 8 'out', since I believe 5 has the > data that > > is missing. I > > > can provide logfiles from the osds 5 and 8 with > increased > > debugging if > > > that'll help. > > > > > > I can restart osd.21, and that makes the "incomplete" > pg go > > away for a few > > > minutes, but osd.21 crashes within 30 seconds of being > > started, and the > > > incomplete pg comes back when that happens. > > > > > > Any suggestions on how to troubleshoot this further > would be > > helpful. I > > > thought I could chronicle my attempts to date to avoid > > duplication of effort > > > but I have tried too many things to clearly > reconstruct the > > path. > > > > > > Thanks in advance! > > > > > > > > > Here are some various stats that might help: > > > > > > aaron@seven ~ $ ceph -v > > > ceph version 0.76 > (3b990136bfab74249f166dd742fd8e61637e63d9) > > > > > > > > > aaron@seven ~ $ ceph pg stat > > > v7286520: 2200 pgs: 2048 active+clean, 78 > > active+remapped+wait_backfill, 72 > > > active+remapped+backfilling, 1 incomplete, 1 > > active+clean+inconsistent; > > > 21577 GB data, 41396 GB used, 27946 GB / 69343 GB > avail; > > 461896/11663686 > > > objects degraded (3.960%); 210 MB/s, 53 objects/s > recovering > > > > > > > > > aaron@seven ~ $ ceph health detail > > > HEALTH_ERR 78 pgs backfill; 72 pgs backfilling; 1 pgs > > incomplete; 1 pgs > > > inconsistent; 1 pgs stuck inactive; 151 pgs stuck > unclean; > > recovery > > > 459868/11663686 objects degraded (3.943%); 1 scrub > errors; mds > > picard is > > > laggy > > > pg 2.28b is stuck inactive since forever, current > state > > incomplete, last > > > acting [6,5] > > > pg 2.6b is stuck unclean for 1161.425227, current > state > > > active+remapped+backfilling, last acting [19,8,4] > > > pg 3.1e4 is stuck unclean for 888.661689, current > state > > > active+remapped+wait_backfill, last acting [8,20,22] > > > pg 2.1e5 is stuck unclean for 888.661578, current > state > > > active+remapped+wait_backfill, last acting [8,20,22] > > > pg 2.127 is stuck unclean for 889.325675, current > state > > > active+remapped+backfilling, last acting [5,8,11] > > > pg 2.63 is stuck unclean for 888.661689, current state > > > active+remapped+wait_backfill, last acting [8,16,3] > > > pg 3.62 is stuck unclean for 888.661682, current state > > > active+remapped+wait_backfill, last acting [8,16,3] > > > pg 2.1e0 is stuck unclean for 888.661675, current > state > > > active+remapped+wait_backfill, last acting [8,19,1] > > > pg 2.420 is stuck unclean for 888.662537, current > state > > > active+remapped+wait_backfill, last acting [8,17,1] > > > pg 3.1df is stuck unclean for 888.661671, current > state > > > active+remapped+wait_backfill, last acting [8,19,1] > > > pg 2.118 is stuck unclean for 888.662593, current > state > > > active+remapped+wait_backfill, last acting [8,19,3] > > > pg 2.29e is stuck unclean for 219581.715114, current > state > > > active+remapped+backfilling, last acting [14,8,7] > > > pg 3.114 is stuck unclean for 888.662518, current > state > > > active+remapped+wait_backfill, last acting [8,14,3] > > > pg 2.115 is stuck unclean for 888.662537, current > state > > > active+remapped+backfilling, last acting [8,14,3] > > > pg 2.57 is stuck unclean for 159935.373946, current > state > > > active+remapped+backfilling, last acting [6,8,0] > > > pg 3.56 is stuck unclean for 1643032.146070, current > state > > > active+remapped+wait_backfill, last acting [6,8,0] > > > pg 2.1d4 is stuck unclean for 888.661671, current > state > > > active+remapped+backfilling, last acting [8,13,3] > > > pg 3.117 is stuck unclean for 888.662503, current > state > > > active+remapped+wait_backfill, last acting [8,19,3] > > > pg 2.294 is stuck unclean for 888.662490, current > state > > > active+remapped+wait_backfill, last acting [8,17,20] > > > pg 2.41a is stuck unclean for 1161.455746, current > state > > > active+remapped+wait_backfill, last acting [20,8,0] > > > pg 2.1d0 is stuck unclean for 888.661674, current > state > > > active+remapped+wait_backfill, last acting [8,16,14] > > > pg 3.351 is stuck unclean for 888.661636, current > state > > > active+remapped+wait_backfill, last acting [8,11,17] > > > pg 3.293 is stuck unclean for 888.662448, current > state > > > active+remapped+wait_backfill, last acting [8,17,20] > > > pg 2.352 is stuck unclean for 888.661623, current > state > > > active+remapped+wait_backfill, last acting [8,11,17] > > > pg 3.1cf is stuck unclean for 888.661638, current > state > > > active+remapped+wait_backfill, last acting [8,16,14] > > > pg 2.40d is stuck unclean for 889.663590, current > state > > > active+remapped+backfilling, last acting [16,8,3] > > > pg 2.34f is stuck unclean for 888.661581, current > state > > > active+remapped+wait_backfill, last acting [8,18,9] > > > pg 3.34e is stuck unclean for 888.661644, current > state > > > active+remapped+wait_backfill, last acting [8,18,9] > > > pg 2.4a is stuck unclean for 146084.212967, current > state > > > active+remapped+backfilling, last acting [16,8,4] > > > pg 2.1cb is stuck unclean for 81410.959695, current > state > > > active+remapped+backfilling, last acting [19,8,9] > > > pg 2.1ca is stuck unclean for 160231.487694, current > state > > > active+remapped+backfilling, last acting [15,8,7] > > > pg 2.28b is stuck unclean since forever, current state > > incomplete, last > > > acting [6,5] > > > pg 2.409 is stuck unclean for 160095.566719, current > state > > > active+remapped+backfilling, last acting [4,8,1] > > > pg 2.345 is stuck unclean for 219637.448262, current > state > > > active+remapped+backfilling, last acting [6,8,15] > > > pg 2.405 is stuck unclean for 160095.566704, current > state > > > active+remapped+backfilling, last acting [14,8,6] > > > pg 2.fc is stuck unclean for 888.662298, current state > > > active+remapped+wait_backfill, last acting [8,20,3] > > > pg 2.342 is stuck unclean for 219394.652780, current > state > > > active+remapped+backfilling, last acting [6,8,22] > > > pg 2.1bf is stuck unclean for 889.325454, current > state > > > active+remapped+backfilling, last acting [11,8,7] > > > pg 2.1b8 is stuck unclean for 889.326218, current > state > > > active+remapped+backfilling, last acting [15,8,5] > > > pg 3.fb is stuck unclean for 888.662289, current state > > > active+remapped+wait_backfill, last acting [8,20,3] > > > pg 2.338 is stuck unclean for 1161.424960, current > state > > > active+remapped+backfilling, last acting [19,8,13] > > > pg 2.3f9 is stuck unclean for 889.326078, current > state > > > active+remapped+backfilling, last acting [14,8,10] > > > pg 3.1b6 is stuck unclean for 888.661612, current > state > > > active+remapped+wait_backfill, last acting [8,17,0] > > > pg 2.1b7 is stuck unclean for 888.661625, current > state > > > active+remapped+wait_backfill, last acting [8,17,0] > > > pg 2.3f7 is stuck unclean for 888.662259, current > state > > > active+remapped+backfilling, last acting [8,11,0] > > > pg 2.270 is stuck unclean for 889.669800, current > state > > > active+remapped+backfilling, last acting [6,8,7] > > > pg 2.2c is stuck unclean for 888.661599, current state > > > active+remapped+wait_backfill, last acting [8,15,9] > > > pg 3.2d is stuck unclean for 888.661593, current state > > > active+remapped+wait_backfill, last acting [8,15,3] > > > pg 2.273 is stuck unclean for 1161.773478, current > state > > > active+remapped+backfilling, last acting [18,8,1] > > > pg 2.3f0 is stuck unclean for 85410.265493, current > state > > > active+remapped+backfilling, last acting [5,8,10] > > > pg 2.2e is stuck unclean for 888.661595, current state > > > active+remapped+wait_backfill, last acting [8,15,3] > > > pg 3.2b is stuck unclean for 888.661580, current state > > > active+remapped+wait_backfill, last acting [8,15,9] > > > pg 3.1a9 is stuck unclean for 888.661574, current > state > > > active+remapped+wait_backfill, last acting [8,18,7] > > > pg 2.3ef is stuck unclean for 1161.426416, current > state > > > active+remapped+backfilling, last acting [19,8,7] > > > pg 2.1aa is stuck unclean for 888.661661, current > state > > > active+remapped+wait_backfill, last acting [8,18,7] > > > pg 2.32b is stuck unclean for 156663.886748, current > state > > > active+remapped+backfilling, last acting [14,8,17] > > > pg 2.21 is stuck unclean for 1161.769888, current > state > > > active+remapped+backfilling, last acting [18,8,13] > > > pg 3.1c is stuck unclean for 888.661636, current state > > > active+remapped+wait_backfill, last acting [8,16,0] > > > pg 2.1d is stuck unclean for 888.661648, current state > > > active+remapped+wait_backfill, last acting [8,16,0] > > > pg 2.1c is stuck unclean for 888.661632, current state > > > active+remapped+wait_backfill, last acting [8,20,7] > > > pg 2.323 is stuck unclean for 302501.945292, current > state > > > active+remapped+backfilling, last acting [13,8,10] > > > pg 2.4a0 is stuck unclean for 888.661565, current > state > > > active+remapped+wait_backfill, last acting [8,16,11] > > > pg 3.19 is stuck unclean for 888.661636, current state > > > active+remapped+wait_backfill, last acting [8,13,0] > > > pg 3.1a is stuck unclean for 888.661619, current state > > > active+remapped+wait_backfill, last acting [8,10,5] > > > pg 2.1b is stuck unclean for 888.661633, current state > > > active+remapped+wait_backfill, last acting [8,10,5] > > > pg 2.25e is stuck unclean for 888.662278, current > state > > > active+remapped+backfilling, last acting [8,14,22] > > > pg 3.1b is stuck unclean for 888.661612, current state > > > active+remapped+wait_backfill, last acting [8,20,7] > > > pg 2.1a is stuck unclean for 888.661626, current state > > > active+remapped+wait_backfill, last acting [8,13,0] > > > pg 2.3d9 is stuck unclean for 1161.773372, current > state > > > active+remapped+backfilling, last acting [18,8,16] > > > pg 3.12 is stuck unclean for 888.661578, current state > > > active+remapped+wait_backfill, last acting [8,17,19] > > > pg 2.13 is stuck unclean for 888.661591, current state > > > active+remapped+wait_backfill, last acting [8,17,19] > > > pg 2.317 is stuck unclean for 159979.814667, current > state > > > active+remapped+backfilling, last acting [4,8,1] > > > pg 2.3d7 is stuck unclean for 889.327661, current > state > > > active+remapped+backfilling, last acting [11,8,0] > > > pg 2.30c is stuck unclean for 888.663342, current > state > > > active+remapped+backfilling, last acting [8,15,0] > > > pg 2.492 is stuck unclean for 889.664908, current > state > > > active+remapped+backfilling, last acting [16,8,22] > > > pg 2.b is stuck unclean for 888.663383, current state > > > active+remapped+backfilling, last acting [8,11,22] > > > pg 2.30f is stuck unclean for 889.327852, current > state > > > active+remapped+backfilling, last acting [4,8,9] > > > pg 3.3cd is stuck unclean for 888.662282, current > state > > > active+remapped+wait_backfill, last acting [8,11,14] > > > pg 2.3ce is stuck unclean for 888.662337, current > state > > > active+remapped+wait_backfill, last acting [8,11,14] > > > pg 2.3c9 is stuck unclean for 888.662309, current > state > > > active+remapped+wait_backfill, last acting [8,19,0] > > > pg 3.3c8 is stuck unclean for 888.662329, current > state > > > active+remapped+wait_backfill, last acting [8,19,0] > > > pg 2.184 is stuck unclean for 209128.087843, current > state > > > active+remapped+backfilling, last acting [13,8,9] > > > pg 2.3cb is stuck unclean for 888.662278, current > state > > > active+remapped+wait_backfill, last acting [8,19,22] > > > pg 3.3ca is stuck unclean for 888.662310, current > state > > > active+remapped+wait_backfill, last acting [8,19,22] > > > pg 2.3ca is stuck unclean for 1161.455302, current > state > > > active+remapped+backfilling, last acting [20,8,15] > > > pg 2.c1 is stuck unclean for 888.662429, current state > > > active+remapped+wait_backfill, last acting [8,15,13] > > > pg 3.c0 is stuck unclean for 888.662398, current state > > > active+remapped+wait_backfill, last acting [8,15,13] > > > pg 2.2 is stuck unclean for 232941.977076, current > state > > > active+remapped+backfilling, last acting [6,8,3] > > > pg 2.183 is stuck unclean for 160014.724114, current > state > > > active+remapped+backfilling, last acting [5,8,1] > > > pg 3.241 is stuck unclean for 888.662292, current > state > > > active+remapped+wait_backfill, last acting [8,18,9] > > > pg 3.301 is stuck unclean for 888.663258, current > state > > > active+remapped+wait_backfill, last acting [8,19,9] > > > pg 2.242 is stuck unclean for 888.662284, current > state > > > active+remapped+wait_backfill, last acting [8,18,9] > > > pg 2.302 is stuck unclean for 888.663248, current > state > > > active+remapped+wait_backfill, last acting [8,19,9] > > > pg 2.23c is stuck unclean for 889.325420, current > state > > > active+remapped+backfilling, last acting [14,8,0] > > > pg 2.ba is stuck unclean for 889.327554, current state > > > active+remapped+backfilling, last acting [12,8,1] > > > pg 2.175 is stuck unclean for 219770.886373, current > state > > > active+remapped+backfilling, last acting [4,8,9] > > > pg 2.3b8 is stuck unclean for 888.662229, current > state > > > active+remapped+wait_backfill, last acting [8,14,22] > > > pg 2.174 is stuck unclean for 888.663214, current > state > > > active+remapped+wait_backfill, last acting [8,15,16] > > > pg 2.478 is stuck unclean for 1161.774401, current > state > > > active+remapped+backfilling, last acting [18,8,1] > > > pg 2.2f5 is stuck unclean for 888.663146, current > state > > > active+remapped+backfilling, last acting [8,12,7] > > > pg 2.176 is stuck unclean for 1161.427407, current > state > > > active+remapped+backfilling, last acting [19,8,1] > > > pg 3.3b7 is stuck unclean for 888.662341, current > state > > > active+remapped+wait_backfill, last acting [8,14,22] > > > pg 3.173 is stuck unclean for 888.663097, current > state > > > active+remapped+wait_backfill, last acting [8,15,16] > > > pg 3.232 is stuck unclean for 888.662195, current > state > > > active+remapped+wait_backfill, last acting [8,20,3] > > > pg 2.233 is stuck unclean for 888.662216, current > state > > > active+remapped+wait_backfill, last acting [8,20,3] > > > pg 2.af is stuck unclean for 85410.264417, current > state > > > active+remapped+backfilling, last acting [6,8,3] > > > pg 3.168 is stuck unclean for 888.663251, current > state > > > active+remapped+wait_backfill, last acting [8,20,0] > > > pg 2.169 is stuck unclean for 888.663291, current > state > > > active+remapped+wait_backfill, last acting [8,20,0] > > > pg 2.ab is stuck unclean for 888.662356, current state > > > active+remapped+wait_backfill, last acting [8,18,17] > > > pg 3.aa is stuck unclean for 888.662351, current state > > > active+remapped+wait_backfill, last acting [8,18,17] > > > pg 3.228 is stuck unclean for 888.662337, current > state > > > active+remapped+wait_backfill, last acting [8,14,0] > > > pg 2.229 is stuck unclean for 888.662350, current > state > > > active+remapped+wait_backfill, last acting [8,14,0] > > > pg 2.22b is stuck unclean for 1161.770522, current > state > > > active+remapped+backfilling, last acting [18,8,0] > > > pg 2.161 is stuck unclean for 1161.839067, current > state > > > active+remapped+backfilling, last acting [17,8,7] > > > pg 2.3a4 is stuck unclean for 160095.566258, current > state > > > active+remapped+backfilling, last acting [18,8,0] > > > pg 2.3a7 is stuck unclean for 888.662256, current > state > > > active+remapped+wait_backfill, last acting [8,18,22] > > > pg 3.3a6 is stuck unclean for 888.662272, current > state > > > active+remapped+wait_backfill, last acting [8,18,22] > > > pg 2.a2 is stuck unclean for 889.669714, current state > > > active+remapped+backfilling, last acting [6,8,7] > > > pg 2.467 is stuck unclean for 889.327384, current > state > > > active+remapped+backfilling, last acting [13,8,3] > > > pg 2.3a1 is stuck unclean for 159994.542551, current > state > > > active+remapped+backfilling, last acting [19,8,5] > > > pg 3.3a3 is stuck unclean for 1643146.397267, current > state > > > active+remapped+backfilling, last acting [18,8,0] > > > pg 2.158 is stuck unclean for 889.331607, current > state > > > active+remapped+backfilling, last acting [15,8,9] > > > pg 2.218 is stuck unclean for 1161.455144, current > state > > > active+remapped+backfilling, last acting [20,8,16] > > > pg 2.95 is stuck unclean for 888.662241, current state > > > active+remapped+wait_backfill, last acting [8,13,0] > > > pg 3.94 is stuck unclean for 888.662236, current state > > > active+remapped+wait_backfill, last acting [8,13,0] > > > pg 2.399 is stuck unclean for 1161.424873, current > state > > > active+remapped+backfilling, last acting [19,8,11] > > > pg 2.2da is stuck unclean for 889.325694, current > state > > > active+remapped+backfilling, last acting [10,8,22] > > > pg 2.458 is stuck unclean for 889.664588, current > state > > > active+remapped+backfilling, last acting [16,8,19] > > > pg 2.455 is stuck unclean for 888.663123, current > state > > > active+remapped+wait_backfill, last acting [8,18,17] > > > pg 2.397 is stuck unclean for 888.662229, current > state > > > active+remapped+wait_backfill, last acting [8,17,1] > > > pg 3.396 is stuck unclean for 888.662244, current > state > > > active+remapped+wait_backfill, last acting [8,17,1] > > > pg 2.152 is stuck unclean for 889.664670, current > state > > > active+remapped+backfilling, last acting [16,8,3] > > > pg 2.20c is stuck unclean for 889.325056, current > state > > > active+remapped+backfilling, last acting [14,8,20] > > > pg 2.44d is stuck unclean for 888.663004, current > state > > > active+remapped+wait_backfill, last acting [8,11,13] > > > pg 2.2ce is stuck unclean for 1161.838375, current > state > > > active+remapped+backfilling, last acting [17,8,13] > > > pg 3.20a is stuck unclean for 1161.425993, current > state > > > active+remapped+wait_backfill, last acting [19,8,0] > > > pg 2.20b is stuck unclean for 1161.424832, current > state > > > active+remapped+wait_backfill, last acting [19,8,0] > > > pg 2.2c8 is stuck unclean for 1161.427151, current > state > > > active+remapped+backfilling, last acting [19,8,9] > > > pg 2.20a is stuck unclean for 889.326613, current > state > > > active+remapped+backfilling, last acting [4,8,6] > > > pg 2.87 is stuck unclean for 1161.837786, current > state > > > active+remapped+backfilling, last acting [17,8,15] > > > pg 2.384 is stuck unclean for 889.324992, current > state > > > active+remapped+backfilling, last acting [14,8,5] > > > pg 2.2c0 is stuck unclean for 888.662914, current > state > > > active+remapped+wait_backfill, last acting [8,11,18] > > > pg 2.446 is stuck unclean for 1161.774298, current > state > > > active+remapped+backfilling, last acting [18,8,0] > > > pg 3.13c is stuck unclean for 888.662984, current > state > > > active+remapped+wait_backfill, last acting [8,16,14] > > > pg 2.13d is stuck unclean for 888.663018, current > state > > > active+remapped+backfilling, last acting [8,16,14] > > > pg 2.1fd is stuck unclean for 888.662197, current > state > > > active+remapped+backfilling, last acting [8,12,22] > > > pg 2.440 is stuck unclean for 195955.637808, current > state > > > active+remapped+backfilling, last acting [17,8,11] > > > pg 3.2bf is stuck unclean for 888.662780, current > state > > > active+remapped+wait_backfill, last acting [8,11,18] > > > pg 2.1f8 is stuck unclean for 889.326608, current > state > > > active+remapped+backfilling, last acting [11,8,22] > > > pg 2.135 is stuck unclean for 888.662853, current > state > > > active+remapped+backfilling, last acting [8,11,0] > > > pg 2.2b5 is stuck unclean for 889.325574, current > state > > > active+remapped+backfilling, last acting [10,8,7] > > > pg 2.436 is stuck unclean for 160015.918251, current > state > > > active+remapped+backfilling, last acting [17,8,3] > > > pg 2.4a0 is active+remapped+wait_backfill, acting > [8,16,11] > > > pg 2.492 is active+remapped+backfilling, acting > [16,8,22] > > > pg 2.478 is active+remapped+backfilling, acting > [18,8,1] > > > pg 2.467 is active+remapped+backfilling, acting > [13,8,3] > > > pg 2.458 is active+remapped+backfilling, acting > [16,8,19] > > > pg 2.455 is active+remapped+wait_backfill, acting > [8,18,17] > > > pg 2.44d is active+remapped+wait_backfill, acting > [8,11,13] > > > pg 2.446 is active+remapped+backfilling, acting > [18,8,0] > > > pg 2.440 is active+remapped+backfilling, acting > [17,8,11] > > > pg 2.436 is active+remapped+backfilling, acting > [17,8,3] > > > pg 2.420 is active+remapped+wait_backfill, acting > [8,17,1] > > > pg 2.41a is active+remapped+wait_backfill, acting > [20,8,0] > > > pg 2.40d is active+remapped+backfilling, acting > [16,8,3] > > > pg 2.409 is active+remapped+backfilling, acting > [4,8,1] > > > pg 2.405 is active+remapped+backfilling, acting > [14,8,6] > > > pg 2.3f9 is active+remapped+backfilling, acting > [14,8,10] > > > pg 2.3f7 is active+remapped+backfilling, acting > [8,11,0] > > > pg 2.3f0 is active+remapped+backfilling, acting > [5,8,10] > > > pg 2.3ef is active+remapped+backfilling, acting > [19,8,7] > > > pg 2.3d9 is active+remapped+backfilling, acting > [18,8,16] > > > pg 2.3d7 is active+remapped+backfilling, acting > [11,8,0] > > > pg 3.3cd is active+remapped+wait_backfill, acting > [8,11,14] > > > pg 2.3ce is active+remapped+wait_backfill, acting > [8,11,14] > > > pg 3.3c8 is active+remapped+wait_backfill, acting > [8,19,0] > > > pg 2.3c9 is active+remapped+wait_backfill, acting > [8,19,0] > > > pg 3.3ca is active+remapped+wait_backfill, acting > [8,19,22] > > > pg 2.3cb is active+remapped+wait_backfill, acting > [8,19,22] > > > pg 2.3ca is active+remapped+backfilling, acting > [20,8,15] > > > pg 2.3b8 is active+remapped+wait_backfill, acting > [8,14,22] > > > pg 3.3b7 is active+remapped+wait_backfill, acting > [8,14,22] > > > pg 2.3a4 is active+remapped+backfilling, acting > [18,8,0] > > > pg 3.3a6 is active+remapped+wait_backfill, acting > [8,18,22] > > > pg 2.3a7 is active+remapped+wait_backfill, acting > [8,18,22] > > > pg 2.3a1 is active+remapped+backfilling, acting > [19,8,5] > > > pg 3.3a3 is active+remapped+backfilling, acting > [18,8,0] > > > pg 2.399 is active+remapped+backfilling, acting > [19,8,11] > > > pg 3.396 is active+remapped+wait_backfill, acting > [8,17,1] > > > pg 2.397 is active+remapped+wait_backfill, acting > [8,17,1] > > > pg 2.384 is active+remapped+backfilling, acting > [14,8,5] > > > pg 3.351 is active+remapped+wait_backfill, acting > [8,11,17] > > > pg 2.352 is active+remapped+wait_backfill, acting > [8,11,17] > > > pg 3.34e is active+remapped+wait_backfill, acting > [8,18,9] > > > pg 2.34f is active+remapped+wait_backfill, acting > [8,18,9] > > > pg 2.345 is active+remapped+backfilling, acting > [6,8,15] > > > pg 2.342 is active+remapped+backfilling, acting > [6,8,22] > > > pg 2.338 is active+remapped+backfilling, acting > [19,8,13] > > > pg 2.32b is active+remapped+backfilling, acting > [14,8,17] > > > pg 2.323 is active+remapped+backfilling, acting > [13,8,10] > > > pg 2.317 is active+remapped+backfilling, acting > [4,8,1] > > > pg 2.30c is active+remapped+backfilling, acting > [8,15,0] > > > pg 2.30f is active+remapped+backfilling, acting > [4,8,9] > > > pg 3.301 is active+remapped+wait_backfill, acting > [8,19,9] > > > pg 2.302 is active+remapped+wait_backfill, acting > [8,19,9] > > > pg 2.2f5 is active+remapped+backfilling, acting > [8,12,7] > > > pg 2.2da is active+remapped+backfilling, acting > [10,8,22] > > > pg 2.2ce is active+remapped+backfilling, acting > [17,8,13] > > > pg 2.2c8 is active+remapped+backfilling, acting > [19,8,9] > > > pg 2.2c0 is active+remapped+wait_backfill, acting > [8,11,18] > > > pg 3.2bf is active+remapped+wait_backfill, acting > [8,11,18] > > > pg 2.2b5 is active+remapped+backfilling, acting > [10,8,7] > > > pg 2.29e is active+remapped+backfilling, acting > [14,8,7] > > > pg 2.294 is active+remapped+wait_backfill, acting > [8,17,20] > > > pg 3.293 is active+remapped+wait_backfill, acting > [8,17,20] > > > pg 2.28b is incomplete, acting [6,5] > > > pg 2.270 is active+remapped+backfilling, acting > [6,8,7] > > > pg 2.273 is active+remapped+backfilling, acting > [18,8,1] > > > pg 2.25e is active+remapped+backfilling, acting > [8,14,22] > > > pg 3.241 is active+remapped+wait_backfill, acting > [8,18,9] > > > pg 2.242 is active+remapped+wait_backfill, acting > [8,18,9] > > > pg 2.23c is active+remapped+backfilling, acting > [14,8,0] > > > pg 2.233 is active+remapped+wait_backfill, acting > [8,20,3] > > > pg 3.232 is active+remapped+wait_backfill, acting > [8,20,3] > > > pg 2.229 is active+remapped+wait_backfill, acting > [8,14,0] > > > pg 3.228 is active+remapped+wait_backfill, acting > [8,14,0] > > > pg 2.22b is active+remapped+backfilling, acting > [18,8,0] > > > pg 2.218 is active+remapped+backfilling, acting > [20,8,16] > > > pg 2.20c is active+remapped+backfilling, acting > [14,8,20] > > > pg 2.20b is active+remapped+wait_backfill, acting > [19,8,0] > > > pg 3.20a is active+remapped+wait_backfill, acting > [19,8,0] > > > pg 2.20a is active+remapped+backfilling, acting > [4,8,6] > > > pg 2.1fd is active+remapped+backfilling, acting > [8,12,22] > > > pg 2.1f8 is active+remapped+backfilling, acting > [11,8,22] > > > pg 2.1e5 is active+remapped+wait_backfill, acting > [8,20,22] > > > pg 3.1e4 is active+remapped+wait_backfill, acting > [8,20,22] > > > pg 2.1e0 is active+remapped+wait_backfill, acting > [8,19,1] > > > pg 3.1df is active+remapped+wait_backfill, acting > [8,19,1] > > > pg 2.1d4 is active+remapped+backfilling, acting > [8,13,3] > > > pg 2.1d0 is active+remapped+wait_backfill, acting > [8,16,14] > > > pg 3.1cf is active+remapped+wait_backfill, acting > [8,16,14] > > > pg 2.1cb is active+remapped+backfilling, acting > [19,8,9] > > > pg 2.1ca is active+remapped+backfilling, acting > [15,8,7] > > > pg 2.1bf is active+remapped+backfilling, acting > [11,8,7] > > > pg 2.1b8 is active+remapped+backfilling, acting > [15,8,5] > > > pg 2.1b7 is active+remapped+wait_backfill, acting > [8,17,0] > > > pg 3.1b6 is active+remapped+wait_backfill, acting > [8,17,0] > > > pg 3.1a9 is active+remapped+wait_backfill, acting > [8,18,7] > > > pg 2.1aa is active+remapped+wait_backfill, acting > [8,18,7] > > > pg 2.184 is active+remapped+backfilling, acting > [13,8,9] > > > pg 2.183 is active+remapped+backfilling, acting > [5,8,1] > > > pg 2.175 is active+remapped+backfilling, acting > [4,8,9] > > > pg 2.174 is active+remapped+wait_backfill, acting > [8,15,16] > > > pg 2.176 is active+remapped+backfilling, acting > [19,8,1] > > > pg 3.173 is active+remapped+wait_backfill, acting > [8,15,16] > > > pg 2.169 is active+remapped+wait_backfill, acting > [8,20,0] > > > pg 3.168 is active+remapped+wait_backfill, acting > [8,20,0] > > > pg 2.161 is active+remapped+backfilling, acting > [17,8,7] > > > pg 2.158 is active+remapped+backfilling, acting > [15,8,9] > > > pg 2.152 is active+remapped+backfilling, acting > [16,8,3] > > > pg 2.13d is active+remapped+backfilling, acting > [8,16,14] > > > pg 3.13c is active+remapped+wait_backfill, acting > [8,16,14] > > > pg 2.135 is active+remapped+backfilling, acting > [8,11,0] > > > pg 2.127 is active+remapped+backfilling, acting > [5,8,11] > > > pg 2.118 is active+remapped+wait_backfill, acting > [8,19,3] > > > pg 2.115 is active+remapped+backfilling, acting > [8,14,3] > > > pg 3.114 is active+remapped+wait_backfill, acting > [8,14,3] > > > pg 3.117 is active+remapped+wait_backfill, acting > [8,19,3] > > > pg 2.fc is active+remapped+wait_backfill, acting > [8,20,3] > > > pg 3.fb is active+remapped+wait_backfill, acting > [8,20,3] > > > pg 2.cc is active+clean+inconsistent, acting [20,6] > > > pg 3.c0 is active+remapped+wait_backfill, acting > [8,15,13] > > > pg 2.c1 is active+remapped+wait_backfill, acting > [8,15,13] > > > pg 2.ba is active+remapped+backfilling, acting > [12,8,1] > > > pg 2.af is active+remapped+backfilling, acting [6,8,3] > > > pg 3.aa is active+remapped+wait_backfill, acting > [8,18,17] > > > pg 2.ab is active+remapped+wait_backfill, acting > [8,18,17] > > > pg 2.a2 is active+remapped+backfilling, acting [6,8,7] > > > pg 3.94 is active+remapped+wait_backfill, acting > [8,13,0] > > > pg 2.95 is active+remapped+wait_backfill, acting > [8,13,0] > > > pg 2.87 is active+remapped+backfilling, acting > [17,8,15] > > > pg 2.6b is active+remapped+backfilling, acting > [19,8,4] > > > pg 3.62 is active+remapped+wait_backfill, acting > [8,16,3] > > > pg 2.63 is active+remapped+wait_backfill, acting > [8,16,3] > > > pg 3.56 is active+remapped+wait_backfill, acting > [6,8,0] > > > pg 2.57 is active+remapped+backfilling, acting [6,8,0] > > > pg 2.4a is active+remapped+backfilling, acting > [16,8,4] > > > pg 3.2d is active+remapped+wait_backfill, acting > [8,15,3] > > > pg 2.2c is active+remapped+wait_backfill, acting > [8,15,9] > > > pg 2.2e is active+remapped+wait_backfill, acting > [8,15,3] > > > pg 3.2b is active+remapped+wait_backfill, acting > [8,15,9] > > > pg 2.21 is active+remapped+backfilling, acting > [18,8,13] > > > pg 2.1d is active+remapped+wait_backfill, acting > [8,16,0] > > > pg 3.1c is active+remapped+wait_backfill, acting > [8,16,0] > > > pg 2.1c is active+remapped+wait_backfill, acting > [8,20,7] > > > pg 3.19 is active+remapped+wait_backfill, acting > [8,13,0] > > > pg 2.1b is active+remapped+wait_backfill, acting > [8,10,5] > > > pg 3.1a is active+remapped+wait_backfill, acting > [8,10,5] > > > pg 2.1a is active+remapped+wait_backfill, acting > [8,13,0] > > > pg 3.1b is active+remapped+wait_backfill, acting > [8,20,7] > > > pg 2.13 is active+remapped+wait_backfill, acting > [8,17,19] > > > pg 3.12 is active+remapped+wait_backfill, acting > [8,17,19] > > > pg 2.b is active+remapped+backfilling, acting > [8,11,22] > > > pg 2.2 is active+remapped+backfilling, acting [6,8,3] > > > recovery 459868/11663686 objects degraded (3.943%) > > > 1 scrub errors > > > mds.picard at 10.42.6.21:6800/13626 is > laggy/unresponsive > > > > > > > > > > > > > > > > > > > > > > > > > > > aaron@seven ~ $ ceph pg 2.28b query > > > { "state": "incomplete", > > > "epoch": 36361, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5], > > > "info": { "pgid": "2.28b", > > > "last_update": "35256'44286", > > > "last_complete": "35256'44286", > > > "log_tail": "34732'41286", > > > "last_user_version": 0, > > > "last_backfill": > > > > > > "84ed7a8b\/rbd_data.a623c2ae8944a.0000000000052a3a\/head\/\/2", > > > "purged_snaps": "[]", > > > "history": { "epoch_created": 1, > > > "last_epoch_started": 36252, > > > "last_epoch_clean": 34760, > > > "last_epoch_split": 0, > > > "same_up_since": 35405, > > > "same_interval_since": 36276, > > > "same_primary_since": 36274, > > > "last_scrub": "34757'44284", > > > "last_scrub_stamp": "2014-02-08 > 11:33:51.835956", > > > "last_deep_scrub": "34757'44284", > > > "last_deep_scrub_stamp": "2014-02-08 > > 11:33:45.299503", > > > "last_clean_scrub_stamp": "2014-02-08 > > 11:33:51.835956"}, > > > "stats": { "version": "35256'44286", > > > "reported_seq": "727", > > > "reported_epoch": "36361", > > > "state": "incomplete", > > > "last_fresh": "2014-02-10 19:35:37.361600", > > > "last_change": "2014-02-10 19:22:15.856289", > > > "last_active": "0.000000", > > > "last_clean": "0.000000", > > > "last_became_active": "0.000000", > > > "last_unstale": "2014-02-10 > 19:35:37.361600", > > > "mapping_epoch": 36274, > > > "log_start": "34732'41286", > > > "ondisk_log_start": "34732'41286", > > > "created": 1, > > > "last_epoch_clean": 34760, > > > "parent": "0.0", > > > "parent_split_bits": 0, > > > "last_scrub": "34757'44284", > > > "last_scrub_stamp": "2014-02-08 > 11:33:51.835956", > > > "last_deep_scrub": "34757'44284", > > > "last_deep_scrub_stamp": "2014-02-08 > > 11:33:45.299503", > > > "last_clean_scrub_stamp": "2014-02-08 > > 11:33:51.835956", > > > "log_size": 3000, > > > "ondisk_log_size": 3000, > > > "stats_invalid": "0", > > > "stat_sum": { "num_bytes": 13767208960, > > > "num_objects": 3306, > > > "num_object_clones": 0, > > > "num_object_copies": 6612, > > > "num_objects_missing_on_primary": 0, > > > "num_objects_degraded": 0, > > > "num_objects_unfound": 0, > > > "num_objects_dirty": 3300, > > > "num_whiteouts": 0, > > > "num_read": 0, > > > "num_read_kb": 0, > > > "num_write": 0, > > > "num_write_kb": 0, > > > "num_scrub_errors": 0, > > > "num_shallow_scrub_errors": 0, > > > "num_deep_scrub_errors": 0, > > > "num_objects_recovered": 0, > > > "num_bytes_recovered": 0, > > > "num_keys_recovered": 0}, > > > "stat_cat_sum": {}, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > "empty": 0, > > > "dne": 0, > > > "incomplete": 1, > > > "last_epoch_started": 36252, > > > "hit_set_history": { "current_last_update": > "0'0", > > > "current_last_stamp": "0.000000", > > > "current_info": { "begin": "0.000000", > > > "end": "0.000000", > > > "version": "0'0"}, > > > "history": []}}, > > > "peer_info": [ > > > { "peer": 5, > > > "pgid": "2.28b", > > > "last_update": "34757'44284", > > > "last_complete": "34757'44284", > > > "log_tail": "34732'41284", > > > "last_user_version": 0, > > > "last_backfill": > > > > > > "84ed7a8b\/rbd_data.a623c2ae8944a.0000000000052a3a\/head\/\/2", > > > "purged_snaps": "[]", > > > "history": { "epoch_created": 1, > > > "last_epoch_started": 36252, > > > "last_epoch_clean": 34760, > > > "last_epoch_split": 0, > > > "same_up_since": 35405, > > > "same_interval_since": 36276, > > > "same_primary_since": 36274, > > > "last_scrub": "34757'44284", > > > "last_scrub_stamp": "2014-02-08 > > 11:33:51.835956", > > > "last_deep_scrub": "34757'44284", > > > "last_deep_scrub_stamp": "2014-02-08 > > 11:33:45.299503", > > > "last_clean_scrub_stamp": "2014-02-08 > > 11:33:51.835956"}, > > > "stats": { "version": "34757'44284", > > > "reported_seq": "247", > > > "reported_epoch": "35404", > > > "state": "down+peering", > > > "last_fresh": "2014-02-09 > 21:05:56.090968", > > > "last_change": "2014-02-09 > 21:05:33.224591", > > > "last_active": "0.000000", > > > "last_clean": "0.000000", > > > "last_became_active": "0.000000", > > > "last_unstale": "2014-02-09 > 21:05:56.090968", > > > "mapping_epoch": 36274, > > > "log_start": "34732'41284", > > > "ondisk_log_start": "34732'41284", > > > "created": 1, > > > "last_epoch_clean": 34760, > > > "parent": "0.0", > > > "parent_split_bits": 0, > > > "last_scrub": "34757'44284", > > > "last_scrub_stamp": "2014-02-08 > > 11:33:51.835956", > > > "last_deep_scrub": "34757'44284", > > > "last_deep_scrub_stamp": "2014-02-08 > > 11:33:45.299503", > > > "last_clean_scrub_stamp": "2014-02-08 > > 11:33:51.835956", > > > "log_size": 3000, > > > "ondisk_log_size": 3000, > > > "stats_invalid": "0", > > > "stat_sum": { "num_bytes": 13771403264, > > > "num_objects": 3307, > > > "num_object_clones": 0, > > > "num_object_copies": 6614, > > > "num_objects_missing_on_primary": 0, > > > "num_objects_degraded": 0, > > > "num_objects_unfound": 0, > > > "num_objects_dirty": 0, > > > "num_whiteouts": 0, > > > "num_read": 0, > > > "num_read_kb": 0, > > > "num_write": 0, > > > "num_write_kb": 0, > > > "num_scrub_errors": 0, > > > "num_shallow_scrub_errors": 0, > > > "num_deep_scrub_errors": 0, > > > "num_objects_recovered": 0, > > > "num_bytes_recovered": 0, > > > "num_keys_recovered": 0}, > > > "stat_cat_sum": {}, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > "empty": 0, > > > "dne": 0, > > > "incomplete": 1, > > > "last_epoch_started": 35110, > > > "hit_set_history": { "current_last_update": > "0'0", > > > "current_last_stamp": "0.000000", > > > "current_info": { "begin": "0.000000", > > > "end": "0.000000", > > > "version": "0'0"}, > > > "history": []}}, > > > { "peer": 8, > > > "pgid": "2.28b", > > > "last_update": "35256'44286", > > > "last_complete": "35256'44286", > > > "log_tail": "34732'41284", > > > "last_user_version": 44286, > > > "last_backfill": > > > > "a8dd7a8b\/benchmark_data_seven_910_object168\/head\/\/2", > > > "purged_snaps": "[]", > > > "history": { "epoch_created": 1, > > > "last_epoch_started": 35225, > > > "last_epoch_clean": 34760, > > > "last_epoch_split": 0, > > > "same_up_since": 35405, > > > "same_interval_since": 36276, > > > "same_primary_since": 36274, > > > "last_scrub": "34757'44284", > > > "last_scrub_stamp": "2014-02-08 > > 11:33:51.835956", > > > "last_deep_scrub": "34757'44284", > > > "last_deep_scrub_stamp": "2014-02-08 > > 11:33:45.299503", > > > "last_clean_scrub_stamp": "2014-02-08 > > 11:33:51.835956"}, > > > "stats": { "version": "35256'44286", > > > "reported_seq": "109", > > > "reported_epoch": "35310", > > > "state": "peering", > > > "last_fresh": "2014-02-09 > 19:52:07.683337", > > > "last_change": "2014-02-09 > 19:52:07.683337", > > > "last_active": "0.000000", > > > "last_clean": "0.000000", > > > "last_became_active": "0.000000", > > > "last_unstale": "2014-02-09 > 19:52:07.683337", > > > "mapping_epoch": 36274, > > > "log_start": "34732'41284", > > > "ondisk_log_start": "34732'41284", > > > "created": 1, > > > "last_epoch_clean": 34760, > > > "parent": "0.0", > > > "parent_split_bits": 0, > > > "last_scrub": "34757'44284", > > > "last_scrub_stamp": "2014-02-08 > > 11:33:51.835956", > > > "last_deep_scrub": "34757'44284", > > > "last_deep_scrub_stamp": "2014-02-08 > > 11:33:45.299503", > > > "last_clean_scrub_stamp": "2014-02-08 > > 11:33:51.835956", > > > "log_size": 3002, > > > "ondisk_log_size": 3002, > > > "stats_invalid": "0", > > > "stat_sum": { "num_bytes": 13763014656, > > > "num_objects": 3305, > > > "num_object_clones": 0, > > > "num_object_copies": 0, > > > "num_objects_missing_on_primary": 0, > > > "num_objects_degraded": 0, > > > "num_objects_unfound": 0, > > > "num_objects_dirty": 0, > > > "num_whiteouts": 0, > > > "num_read": 0, > > > "num_read_kb": 0, > > > "num_write": 0, > > > "num_write_kb": 0, > > > "num_scrub_errors": 0, > > > "num_shallow_scrub_errors": 0, > > > "num_deep_scrub_errors": 0, > > > "num_objects_recovered": 0, > > > "num_bytes_recovered": 0, > > > "num_keys_recovered": 0}, > > > "stat_cat_sum": {}, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > "empty": 0, > > > "dne": 0, > > > "incomplete": 1, > > > "last_epoch_started": 35225, > > > "hit_set_history": { "current_last_update": > "0'0", > > > "current_last_stamp": "0.000000", > > > "current_info": { "begin": "0.000000", > > > "end": "0.000000", > > > "version": "0'0"}, > > > "history": []}}], > > > "recovery_state": [ > > > { "name": "Started\/Primary\/Peering", > > > "enter_time": "2014-02-10 19:22:15.855010", > > > "past_intervals": [ > > > { "first": 34758, > > > "last": 34796, > > > "maybe_went_rw": 1, > > > "up": [ > > > 21], > > > "acting": [ > > > 21]}, > > > { "first": 34797, > > > "last": 34899, > > > "maybe_went_rw": 1, > > > "up": [ > > > 21, > > > 5], > > > "acting": [ > > > 21, > > > 5]}, > > > { "first": 34900, > > > "last": 34946, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 34947, > > > "last": 34952, > > > "maybe_went_rw": 1, > > > "up": [ > > > 21, > > > 5], > > > "acting": [ > > > 21, > > > 5]}, > > > { "first": 34953, > > > "last": 34957, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 34958, > > > "last": 34959, > > > "maybe_went_rw": 1, > > > "up": [ > > > 21, > > > 5], > > > "acting": [ > > > 21, > > > 5]}, > > > { "first": 34960, > > > "last": 35053, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 35054, > > > "last": 35055, > > > "maybe_went_rw": 1, > > > "up": [ > > > 21, > > > 5], > > > "acting": [ > > > 21, > > > 5]}, > > > { "first": 35056, > > > "last": 35062, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 35063, > > > "last": 35065, > > > "maybe_went_rw": 1, > > > "up": [ > > > 21, > > > 5], > > > "acting": [ > > > 21, > > > 5]}, > > > { "first": 35066, > > > "last": 35068, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 35069, > > > "last": 35071, > > > "maybe_went_rw": 1, > > > "up": [ > > > 21, > > > 5], > > > "acting": [ > > > 21, > > > 5]}, > > > { "first": 35072, > > > "last": 35108, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 35109, > > > "last": 35112, > > > "maybe_went_rw": 1, > > > "up": [ > > > 21, > > > 5], > > > "acting": [ > > > 21, > > > 5]}, > > > { "first": 35113, > > > "last": 35120, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 35121, > > > "last": 35160, > > > "maybe_went_rw": 1, > > > "up": [ > > > 8, > > > 5], > > > "acting": [ > > > 8, > > > 5]}, > > > { "first": 35161, > > > "last": 35174, > > > "maybe_went_rw": 1, > > > "up": [ > > > 8], > > > "acting": [ > > > 8]}, > > > { "first": 35175, > > > "last": 35181, > > > "maybe_went_rw": 1, > > > "up": [ > > > 8, > > > 5], > > > "acting": [ > > > 8, > > > 5]}, > > > { "first": 35182, > > > "last": 35194, > > > "maybe_went_rw": 0, > > > "up": [ > > > 8], > > > "acting": [ > > > 8]}, > > > { "first": 35195, > > > "last": 35214, > > > "maybe_went_rw": 0, > > > "up": [ > > > 8, > > > 5], > > > "acting": [ > > > 8, > > > 5]}, > > > { "first": 35215, > > > "last": 35222, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 35223, > > > "last": 35223, > > > "maybe_went_rw": 0, > > > "up": [ > > > 8, > > > 5], > > > "acting": [ > > > 8, > > > 5]}, > > > { "first": 35224, > > > "last": 35264, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 21, > > > 8]}, > > > { "first": 35265, > > > "last": 35265, > > > "maybe_went_rw": 0, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 8]}, > > > { "first": 35266, > > > "last": 35287, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > { "first": 35288, > > > "last": 35299, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6], > > > "acting": [ > > > 6]}, > > > { "first": 35300, > > > "last": 35303, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > { "first": 35304, > > > "last": 35305, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6], > > > "acting": [ > > > 6]}, > > > { "first": 35306, > > > "last": 35376, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > { "first": 35377, > > > "last": 35386, > > > "maybe_went_rw": 0, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 35387, > > > "last": 35396, > > > "maybe_went_rw": 0, > > > "up": [], > > > "acting": []}, > > > { "first": 35397, > > > "last": 35404, > > > "maybe_went_rw": 1, > > > "up": [ > > > 5], > > > "acting": [ > > > 5]}, > > > { "first": 35405, > > > "last": 35407, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > { "first": 35408, > > > "last": 35616, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 21, > > > 6]}, > > > { "first": 35617, > > > "last": 35618, > > > "maybe_went_rw": 0, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6]}, > > > { "first": 35619, > > > "last": 36246, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > { "first": 36247, > > > "last": 36248, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 21, > > > 6]}, > > > { "first": 36249, > > > "last": 36249, > > > "maybe_went_rw": 0, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6]}, > > > { "first": 36250, > > > "last": 36250, > > > "maybe_went_rw": 0, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6, > > > 5]}, > > > { "first": 36251, > > > "last": 36273, > > > "maybe_went_rw": 1, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 21, > > > 6]}, > > > { "first": 36274, > > > "last": 36275, > > > "maybe_went_rw": 0, > > > "up": [ > > > 6, > > > 5], > > > "acting": [ > > > 6]}], > > > "probing_osds": [ > > > 5, > > > 6], > > > "down_osds_we_would_probe": [ > > > 21], > > > "peering_blocked_by": []}, > > > { "name": "Started", > > > "enter_time": "2014-02-10 > 19:22:15.854966"}]} > > > > > > > > > -- > > > Aaron Ten Clay > > > http://www.aarontc.com/ > > > > > > > > > > > > > > > > -- > > Aaron Ten Clay > > http://www.aarontc.com/ > > > > > > > > > -- > Aaron Ten Clay > http://www.aarontc.com/ > >
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
