Re: Cluster degraded

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Aaron,
    Thank you for the pointer, I've increased the pg_num and pgp_num and this has resolved the issue!  Spot on, I'm going to aim for 100 PG's per OSD, but I've recently expanded the cluster, and neglected to increase the numbers (hence this issue).

Sherry, Aaron, thanks again for your time!

Alex


From: "Aaron Ten Clay" <aarontc@xxxxxxxxxxx>
To: "Alex Pearson" <alex@xxxxxxxxxxx>
Cc: "Sherry Shahbazi" <shoosah@xxxxxxxxx>, ceph-users@xxxxxxxxxxxxxx
Sent: Monday, 13 January, 2014 10:25:59 PM
Subject: Re: [ceph-users] Cluster degraded

Alex,

Since it appears at first glance that all your stuck placement groups are in pools 6 and 9, perhaps those pools have too few placement groups? I imagine that's what Sherry meant to say - there shouldn't be a problem with too few OSDs as far as I am aware, but too few placement groups can cause the CRUSH algorithm to fail to locate a place for all the objects. The recommendation is "about" 100 placement groups per OSD (across all pools), but there is some realistic minimum per pool as well. Someone with more hard facts in that domain might be able to chime in here. :)

Another thing to check is that your "pg_num" and "pgp_num" match for the pool settings, since both affect the CRUSH placement rules.

HTH,
-Aaron


On Mon, Jan 13, 2014 at 3:48 AM, Alex Pearson <alex@xxxxxxxxxxx> wrote:
Sherry,
    Thank you for the quick reply, but I'm not sure that is the case?  Surely you don't need to more OSD's than pools?  Could you reference some documentation detailing that?  Also when building the cluster you have the three default pools (data, metadata and rbd) and that came up cleanly with a size of two and only two OSDs?

Thanks again

Alex


From: "Sherry Shahbazi" <shoosah@xxxxxxxxx>
To: "Alex Pearson" <alex@xxxxxxxxxxx>, ceph-users@xxxxxxxxxxxxxx
Sent: Sunday, 12 January, 2014 10:34:58 PM
Subject: Re: Cluster degraded

Hi Alex, 




I think you need to increase number of OSDs, since you have got 10 OSDs for 10 pools which is too few.

 
Best Regards
Sherry


On Monday, January 13, 2014 11:27 AM, Alex Pearson <alex@xxxxxxxxxxx> wrote:
Hello All,
I've had a look through the documentation, however I'm unable to find out how to get my PGs back to 'healthy'.  I caused the below state by issuing multiple reweight-by-utilization commands (I believe).  I have also changed the 'size' of a pool (from 2 to 3) which may also be relevant.

Everything seems to be running fine at the moment, but the cluster has a number of PGs showing unhealthy as below.  What would be the correct way to get the cluster back healthy?  I've reviewed http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/ but can't see what the correct action should be.

Thanks in advance for any help on this.

Alex



root@osh1:~# ceph --cluster apics -s
    cluster 1f8d996d-2fc1-4663-93b2-5e3155ddb3b8
    health HEALTH_WARN 49 pgs degraded; 176 pgs stuck unclean; recovery 19479/2079561 objects degraded (0.937%)
    monmap e3: 3 mons at {dog=MailScanner has detected a possible fraud attempt from "10.30.10.61:6789" claiming to be MailScanner warning: numerical links are often malicious: 10.30.10.61:6789/0,osh1=10.30.10.101:6789/0,osh2=10.30.10.102:6789/0}, election epoch 1522, quorum 0,1,2 dog,osh1,osh2
    mdsmap e294: 1/1/1 up {0=0=up:active}
    osdmap e1605: 10 osds: 10 up, 10 in
      pgmap v373901: 4528 pgs, 10 pools, 2531 GB data, 982 kobjects
            6212 GB used, 2916 GB / 9305 GB avail
            19479/2079561 objects degraded (0.937%)
                  3 active
                4329 active+clean
                147 active+remapped
                  26 active+degraded
                  23 active+clean+degraded
  client io 35388 B/s wr, 4 op/s

root@osh1:~# ceph --cluster apics health detail
HEALTH_WARN 49 pgs degraded; 176 pgs stuck unclean; recovery 19479/2079561 objects degraded (0.937%)
pg 9.63 is stuck unclean for 34587.441837, current state active+degraded, last acting [8,2]
pg 6.12d is stuck unclean for 189490.064360, current state active+remapped, last acting [2,8,3]
pg 9.61 is stuck unclean for 25930.260406, current state active+remapped, last acting [3,6,5]
pg 6.1ec is stuck unclean since forever, current state active+remapped, last acting [0,5,2]
pg 6.12e is stuck unclean for 187564.076457, current state active+remapped, last acting [5,0,8]
pg 6.1ef is stuck unclean since forever, current state active+remapped, last acting [0,3,1]
pg 6.60 is stuck unclean for 181657.261045, current state active+remapped, last acting [0,2,1]
pg 9.6f is stuck unclean for 82751.234847, current state active+degraded, last acting [2,0]
pg 6.62 is stuck unclean since forever, current state active+remapped, last acting [4,0,1]
pg 6.122 is stuck unclean for 182927.362958, current state active+remapped, last acting [6,3,1]
pg 6.64 is stuck unclean since forever, current state active, last acting [3,6,5]
pg 6.1e2 is stuck unclean for 187564.120447, current state active+remapped, last acting [5,8,9]
pg 6.67 is stuck unclean for 187564.241127, current state active+remapped, last acting [8,2,3]
pg 6.66 is stuck unclean since forever, current state active+remapped, last acting [8,2,5]
pg 6.58 is stuck unclean for 181656.731823, current state active+remapped, last acting [3,0,1]
pg 9.57 is stuck unclean for 82751.779108, current state active+degraded, last acting [0,2]
pg 6.1e6 is stuck unclean since forever, current state active+remapped, last acting [5,0,8]
pg 6.5a is stuck unclean for 182926.684982, current state active+remapped, last acting [0,2,1]
pg 6.1db is stuck unclean for 181657.277903, current state active+remapped, last acting [0,4,1]
pg 6.11c is stuck unclean for 182927.022009, current state active+remapped, last acting [5,0,1]
pg 6.1dc is stuck unclean for 187564.253602, current state active+remapped, last acting [8,3,5]
pg 9.5f is stuck unclean for 4454.417475, current state active+degraded, last acting [4,0]
pg 6.111 is stuck unclean for 189488.047268, current state active+remapped, last acting [5,6,0]
pg 6.110 is stuck unclean for 182926.646916, current state active+remapped, last acting [2,8,3]
pg 9.5d is stuck unclean for 82751.778493, current state active+degraded, last acting [0,2]
pg 6.113 is stuck unclean for 199785.387076, current state active+remapped, last acting [5,0,1]
pg 6.117 is stuck unclean for 181657.273579, current state active+remapped, last acting [0,2,1]
pg 6.109 is stuck unclean since forever, current state active+remapped, last acting [2,8,5]
pg 6.10a is stuck unclean for 181657.028166, current state active+remapped, last acting [2,8,5]
pg 6.4f is stuck unclean for 181657.256397, current state active+remapped, last acting [0,4,1]
pg 6.4e is stuck unclean since forever, current state active+remapped, last acting [3,8,1]
pg 9.41 is stuck unclean for 95426.196707, current state active+degraded, last acting [2,8]
pg 6.10f is stuck unclean since forever, current state active+remapped, last acting [6,3,1]
pg 6.1cc is stuck unclean for 189489.856555, current state active+remapped, last acting [6,2,5]
pg 6.41 is stuck unclean for 207151.663867, current state active+remapped, last acting [2,8,5]
pg 6.1ce is stuck unclean for 189490.143064, current state active+remapped, last acting [2,0,9]
pg 9.4c is stuck unclean for 4514.849807, current state active+degraded, last acting [0,4]
pg 6.103 is stuck unclean since forever, current state active+remapped, last acting [5,6,1]
pg 6.45 is stuck unclean for 189489.763856, current state active+remapped, last acting [3,8,9]
pg 6.1c3 is stuck unclean since forever, current state active+remapped, last acting [5,6,8]
pg 6.44 is stuck unclean for 189490.015875, current state active+remapped, last acting [2,8,3]
pg 6.1c2 is stuck unclean since forever, current state active+remapped, last acting [2,8,3]
pg 6.1c5 is stuck unclean for 182927.082080, current state active+remapped, last acting [5,6,1]
pg 6.1c6 is stuck unclean for 181657.041183, current state active+remapped, last acting [2,8,5]
pg 9.32 is stuck unclean for 82751.773922, current state active+remapped, last acting [0,2,3]
pg 6.1bc is stuck unclean for 181657.280292, current state active+remapped, last acting [0,2,1]
pg 6.31 is stuck unclean since forever, current state active+remapped, last acting [3,6,0]
pg 9.3e is stuck unclean for 31943.087354, current state active+degraded, last acting [2,8]
pg 6.fe is stuck unclean for 181656.557890, current state active+remapped, last acting [8,3,1]
pg 6.f0 is stuck unclean for 187564.186472, current state active+remapped, last acting [3,0,9]
pg 6.1b0 is stuck unclean for 189488.053293, current state active+remapped, last acting [5,8,9]
pg 6.35 is stuck unclean for 187086.122285, current state active+remapped, last acting [0,2,9]
pg 6.1b2 is stuck unclean for 199788.285859, current state active+remapped, last acting [5,0,6]
pg 6.29 is stuck unclean for 182927.068290, current state active+remapped, last acting [8,5,1]
pg 6.28 is stuck unclean for 181657.032858, current state active+remapped, last acting [5,0,1]
pg 6.e9 is stuck unclean since forever, current state active+remapped, last acting [2,6,3]
pg 6.e8 is stuck unclean for 182926.645708, current state active+remapped, last acting [2,6,3]
pg 6.1a8 is stuck unclean for 181657.279948, current state active+remapped, last acting [0,5,1]
pg 6.2c is stuck unclean for 182927.006868, current state active+remapped, last acting [3,0,1]
pg 9.23 is stuck unclean for 4454.412192, current state active+degraded, last acting [4,6]
pg MailScanner has detected a possible fraud attempt from "6.ec" claiming to be MailScanner warning: numerical links are often malicious: 6.ec is stuck unclean for 182927.360674, current state active+remapped, last acting [6,2,3]
pg 6.2e is stuck unclean since forever, current state active+remapped, last acting [2,6,5]
pg 6.1ae is stuck unclean for 182927.247745, current state active+remapped, last acting [4,0,1]
pg 6.1a1 is stuck unclean for 189489.814916, current state active+remapped, last acting [3,0,9]
pg 6.22 is stuck unclean for 189489.929437, current state active+remapped, last acting [2,8,4]
pg 6.24 is stuck unclean for 181656.550140, current state active+remapped, last acting [8,3,1]
pg 9.2b is stuck unclean for 173816.252908, current state active+degraded, last acting [2,6]
pg 6.1a5 is stuck unclean since forever, current state active+remapped, last acting [3,6,9]
pg 6.26 is stuck unclean for 187564.099158, current state active+remapped, last acting [4,6,8]
pg 9.17 is stuck unclean for 25930.040655, current state active+remapped, last acting [6,2,7]
pg 6.1a6 is stuck unclean for 187564.104902, current state active+remapped, last acting [5,6,9]
pg 6.199 is stuck unclean for 182927.022250, current state active+remapped, last acting [3,6,1]
pg 6.1a is stuck unclean for 25930.023296, current state active+remapped, last acting [6,2,7]
pg 6.da is stuck unclean for 187564.245796, current state active+remapped, last acting [8,5,9]
pg 6.1c is stuck unclean since forever, current state active+remapped, last acting [6,2,5]
pg 6.19d is stuck unclean since forever, current state active+remapped, last acting [8,2,3]
pg 9.1f is stuck unclean for 159167.265649, current state active+degraded, last acting [2,8]
pg 6.d1 is stuck unclean for 182927.018091, current state active+remapped, last acting [5,8,0]
pg 6.19e is stuck unclean for 182927.080993, current state active+remapped, last acting [8,4,1]
pg 6.13 is stuck unclean for 187084.294984, current state active+remapped, last acting [3,0,9]
pg 6.191 is stuck unclean for 189489.855254, current state active+remapped, last acting [6,3,9]
pg 6.d2 is stuck unclean for 182927.018037, current state active+remapped, last acting [5,0,1]
pg 6.250 is stuck unclean for 182927.034444, current state active+remapped, last acting [3,6,0]
pg 6.d4 is stuck unclean for 182927.013012, current state active+remapped, last acting [3,6,1]
pg 6.252 is stuck unclean since forever, current state active+remapped, last acting [2,0,1]
pg 9.19 is stuck unclean for 82751.368622, current state active+degraded, last acting [6,2]
pg 6.c9 is stuck unclean for 187086.138042, current state active+remapped, last acting [8,3,9]
pg 6.196 is stuck unclean for 182927.021856, current state active+remapped, last acting [3,6,1]
pg 6.257 is stuck unclean for 182926.944790, current state active+remapped, last acting [0,5,1]
pg 6.249 is stuck unclean for 182927.381620, current state active+remapped, last acting [6,2,4]
pg 6.248 is stuck unclean since forever, current state active+remapped, last acting [6,2,0]
pg 6.18a is stuck unclean since forever, current state active+remapped, last acting [3,6,1]
pg 6.24b is stuck unclean for 182926.541677, current state active+remapped, last acting [2,8,5]
pg 6.cc is stuck unclean for 182927.171805, current state active+remapped, last acting [4,0,1]
pg 9.c3 is stuck unclean for 4469.112833, current state active+degraded, last acting [4,6]
pg 6.e is stuck unclean for 181656.740682, current state active+remapped, last acting [2,6,3]
pg 9.1 is stuck unclean for 68383.647655, current state active+remapped, last acting [2,0,5]
pg 6.24c is stuck unclean for 182927.032144, current state active+remapped, last acting [3,6,0]
pg 6.c0 is stuck unclean for 187564.230695, current state active+remapped, last acting [6,5,3]
pg 6.24e is stuck unclean for 181657.258973, current state active+remapped, last acting [6,3,1]
pg 6.2 is stuck unclean for 181657.231274, current state active+remapped, last acting [6,5,1]
pg 6.4 is stuck unclean since forever, current state active, last acting [2,0,5]
pg 9.b is stuck unclean for 160369.847661, current state active+degraded, last acting [2,6]
pg 6.c5 is stuck unclean for 182927.018313, current state active+remapped, last acting [5,8,1]
pg 6.c6 is stuck unclean for 182927.171553, current state active+remapped, last acting [4,6,1]
pg 6.246 is stuck unclean for 189490.025903, current state active+remapped, last acting [0,3,9]
pg MailScanner has detected a possible fraud attempt from "6.bd" claiming to be MailScanner warning: numerical links are often malicious: 6.bd is stuck unclean since forever, current state active+remapped, last acting [2,8,4]
pg 6.23a is stuck unclean for 187086.153951, current state active+remapped, last acting [0,2,9]
pg MailScanner has detected a possible fraud attempt from "6.bf" claiming to be MailScanner warning: numerical links are often malicious: 6.bf is stuck unclean since forever, current state active+remapped, last acting [2,0,1]
pg 9.b1 is stuck unclean for 4469.112012, current state active+degraded, last acting [4,0]
pg 6.b1 is stuck unclean since forever, current state active+remapped, last acting [5,8,2]
pg 6.17e is stuck unclean for 189490.015499, current state active+remapped, last acting [8,3,9]
pg 6.23f is stuck unclean for 182927.010113, current state active+remapped, last acting [5,8,1]
pg 9.bc is stuck unclean for 178360.379416, current state active+degraded, last acting [2,0]
pg MailScanner has detected a possible fraud attempt from "9.ba" claiming to be MailScanner warning: numerical links are often malicious: 9.ba is stuck unclean for 146602.612400, current state active+degraded, last acting [2,8]
pg 6.172 is stuck unclean since forever, current state active+remapped, last acting [0,5,1]
pg 6.b4 is stuck unclean for 181657.091254, current state active+remapped, last acting [4,0,1]
pg 6.174 is stuck unclean for 187564.241472, current state active+remapped, last acting [2,8,3]
pg 6.a8 is stuck unclean for 197921.381643, current state active+remapped, last acting [3,6,0]
pg 6.169 is stuck unclean for 182926.652321, current state active+remapped, last acting [2,8,5]
pg 9.a5 is stuck unclean for 83212.585593, current state active+degraded, last acting [3,6]
pg MailScanner has detected a possible fraud attempt from "6.af" claiming to be MailScanner warning: numerical links are often malicious: 6.af is stuck unclean for 207081.360042, current state active+remapped, last acting [0,5,4]
pg 6.16c is stuck unclean since forever, current state active+remapped, last acting [2,6,5]
pg 6.22d is stuck unclean since forever, current state active+remapped, last acting [2,6,3]
pg MailScanner has detected a possible fraud attempt from "9.ae" claiming to be MailScanner warning: numerical links are often malicious: 9.ae is stuck unclean for 83238.657997, current state active+degraded, last acting [5,8]
pg 6.16e is stuck unclean for 189490.015156, current state active+remapped, last acting [8,3,9]
pg 6.223 is stuck unclean for 181657.047363, current state active+remapped, last acting [2,0,1]
pg 6.165 is stuck unclean for 182927.247437, current state active+remapped, last acting [4,6,1]
pg 6.224 is stuck unclean for 25930.448314, current state active+remapped, last acting [5,0,7]
pg 6.99 is stuck unclean since forever, current state active+remapped, last acting [3,0,1]
pg 6.9d is stuck unclean for 181656.555581, current state active+remapped, last acting [8,2,5]
pg 6.21a is stuck unclean for 189489.819499, current state active+remapped, last acting [3,6,0]
pg 6.15c is stuck unclean since forever, current state active, last acting [3,6,5]
pg 6.15f is stuck unclean since forever, current state active+remapped, last acting [0,4,1]
pg 6.21c is stuck unclean since forever, current state active+remapped, last acting [2,6,3]
pg 6.91 is stuck unclean for 182926.643616, current state active+remapped, last acting [2,0,1]
pg 6.21f is stuck unclean for 182927.089147, current state active+remapped, last acting [8,5,1]
pg 6.150 is stuck unclean since forever, current state active+remapped, last acting [5,0,1]
pg 6.210 is stuck unclean for 182927.026746, current state active+remapped, last acting [3,6,1]
pg 9.9a is stuck unclean for 82751.873851, current state active+degraded, last acting [8,2]
pg 6.152 is stuck unclean for 182927.247268, current state active+remapped, last acting [4,8,1]
pg 6.155 is stuck unclean for 189489.995299, current state active+remapped, last acting [0,4,9]
pg 6.154 is stuck unclean since forever, current state active+remapped, last acting [3,6,0]
pg 6.214 is stuck unclean since forever, current state active+remapped, last acting [6,2,5]
pg 6.149 is stuck unclean since forever, current state active+remapped, last acting [2,8,3]
pg 6.14a is stuck unclean for 182926.649578, current state active+remapped, last acting [2,6,5]
pg 6.20a is stuck unclean for 187086.145054, current state active+remapped, last acting [0,5,9]
pg 9.8e is stuck unclean for 154670.986075, current state active+degraded, last acting [2,0]
pg 6.80 is stuck unclean for 182927.071220, current state active+remapped, last acting [8,5,1]
pg 6.83 is stuck unclean for 197921.387641, current state active+remapped, last acting [3,8,1]
pg 6.82 is stuck unclean for 181656.554868, current state active+remapped, last acting [8,3,1]
pg 6.142 is stuck unclean since forever, current state active+remapped, last acting [8,2,5]
pg 9.77 is stuck unclean for 82751.780914, current state active+degraded, last acting [0,2]
pg 6.7b is stuck unclean for 182927.168302, current state active+remapped, last acting [4,8,1]
pg 6.138 is stuck unclean for 199785.426014, current state active+remapped, last acting [5,6,1]
pg 6.7a is stuck unclean for 187086.123925, current state active+remapped, last acting [0,2,5]
pg 6.13b is stuck unclean for 181657.133217, current state active+remapped, last acting [5,6,0]
pg 6.13c is stuck unclean for 181657.030277, current state active+remapped, last acting [2,6,3]
pg 6.1fd is stuck unclean for 182927.024824, current state active+remapped, last acting [3,0,1]
pg 6.7e is stuck unclean for 181656.733443, current state active+remapped, last acting [3,8,1]
pg 6.71 is stuck unclean for 182927.014811, current state active+remapped, last acting [5,6,0]
pg 6.13e is stuck unclean since forever, current state active+remapped, last acting [2,0,1]
pg 6.1fe is stuck unclean for 182927.375724, current state active+remapped, last acting [6,2,3]
pg 6.73 is stuck unclean for 182927.014794, current state active+remapped, last acting [5,6,1]
pg 6.1f1 is stuck unclean for 182927.375694, current state active+remapped, last acting [6,2,5]
pg 6.72 is stuck unclean for 181656.815359, current state active+remapped, last acting [2,0,1]
pg 6.75 is stuck unclean for 182927.070651, current state active+remapped, last acting [8,5,1]
pg 6.74 is stuck unclean since forever, current state active+remapped, last acting [8,5,0]
pg 6.77 is stuck unclean since forever, current state active+remapped, last acting [3,0,4]
pg 9.78 is stuck unclean for 4454.429800, current state active+degraded, last acting [4,8]
pg 6.1f5 is stuck unclean since forever, current state active+remapped, last acting [3,0,2]
pg 6.69 is stuck unclean for 189489.885122, current state active+remapped, last acting [4,6,9]
pg 9.66 is stuck unclean for 4454.422339, current state active+degraded, last acting [4,6]
pg 6.68 is stuck unclean for 182927.357434, current state active+remapped, last acting [6,4,1]
pg 9.64 is stuck unclean for 81616.028304, current state active+degraded, last acting [8,2]
pg 9.65 is stuck unclean for 4519.092258, current state active+degraded, last acting [6,4]
pg 9.c6 is active+clean+degraded, acting [8,3]
pg 9.c2 is active+clean+degraded, acting [5,8]
pg 9.c3 is active+degraded, acting [4,6]
pg 9.b1 is active+degraded, acting [4,0]
pg 9.bc is active+degraded, acting [2,0]
pg MailScanner has detected a possible fraud attempt from "9.bd" claiming to be MailScanner warning: numerical links are often malicious: 9.bd is active+clean+degraded, acting [6,5]
pg MailScanner has detected a possible fraud attempt from "9.ba" claiming to be MailScanner warning: numerical links are often malicious: 9.ba is active+degraded, acting [2,8]
pg 9.a5 is active+degraded, acting [3,6]
pg MailScanner has detected a possible fraud attempt from "9.ae" claiming to be MailScanner warning: numerical links are often malicious: 9.ae is active+degraded, acting [5,8]
pg MailScanner has detected a possible fraud attempt from "9.ac" claiming to be MailScanner warning: numerical links are often malicious: 9.ac is active+clean+degraded, acting [0,5]
pg 9.96 is active+clean+degraded, acting [3,0]
pg 9.9a is active+degraded, acting [8,2]
pg 9.80 is active+clean+degraded, acting [3,8]
pg 9.8e is active+degraded, acting [2,0]
pg 9.77 is active+degraded, acting [0,2]
pg 9.74 is active+clean+degraded, acting [3,0]
pg 9.72 is active+clean+degraded, acting [8,5]
pg 9.70 is active+clean+degraded, acting [5,6]
pg 9.71 is active+clean+degraded, acting [8,5]
pg 9.7f is active+clean+degraded, acting [8,3]
pg 9.7d is active+clean+degraded, acting [8,5]
pg 9.7b is active+clean+degraded, acting [3,8]
pg 9.78 is active+degraded, acting [4,8]
pg 9.66 is active+degraded, acting [4,6]
pg 9.64 is active+degraded, acting [8,2]
pg 9.65 is active+degraded, acting [6,4]
pg 9.63 is active+degraded, acting [8,2]
pg 9.6e is active+clean+degraded, acting [5,6]
pg 9.6f is active+degraded, acting [2,0]
pg 9.57 is active+degraded, acting [0,2]
pg 9.55 is active+clean+degraded, acting [3,0]
pg 9.5f is active+degraded, acting [4,0]
pg 9.5d is active+degraded, acting [0,2]
pg 9.42 is active+clean+degraded, acting [3,8]
pg 9.41 is active+degraded, acting [2,8]
pg 9.4c is active+degraded, acting [0,4]
pg 9.4b is active+clean+degraded, acting [3,8]
pg 9.3e is active+degraded, acting [2,8]
pg 9.26 is active+clean+degraded, acting [8,5]
pg 9.25 is active+clean+degraded, acting [5,0]
pg 9.23 is active+degraded, acting [4,6]
pg 9.21 is active+clean+degraded, acting [8,3]
pg 9.2e is active+clean+degraded, acting [3,6]
pg 9.2b is active+degraded, acting [2,6]
pg 9.29 is active+clean+degraded, acting [3,0]
pg 9.10 is active+clean+degraded, acting [3,0]
pg 9.1f is active+degraded, acting [2,8]
pg 9.19 is active+degraded, acting [6,2]
pg 9.b is active+degraded, acting [2,6]
recovery 19479/2079561 objects degraded (0.937%)

root@osh1:~# ceph --cluster apics pg dump_stuck
ok
pg_stat    objects    mip    degr    unf    bytes    log    disklog    state    state_stamp    v    reported    up    acting    last_scrub    scrub_stamp    last_deep_scrub    deep_scrub_stamp
9.63    519    0    519    0    1042014    3003    3003    active+degraded    2014-01-12 10:16:02.499133    1527'8548    1605:19000    [8,2]    [8,2]    615'5995    2014-01-09 12:29:18.871498    342'5982    2014-01-06 00:37:57.667418
6.12d    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:29.248336    0'0    1605:1467    [2,8]    [2,8,3]    0'0    2014-01-09 09:03:27.120625    0'0    2014-01-08 09:01:18.371990
9.61    506    0    0    0    1077912    3000    3000    active+remapped    2014-01-12 16:56:36.923766    1501'7214    1605:18207    [3,6]    [3,6,5]    1501'7214    2014-01-11 17:42:38.863720    299'6209    2014-01-06 09:26:59.628384
6.1ec    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.301048    0'0    1605:269    [0,5]    [0,5,2]    0'0    2014-01-09 12:24:21.891960    0'0    2014-01-08 11:48:09.345325
6.12e    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:35:42.537436    0'0    1605:1315    [5,0]    [5,0,8]    0'0    2014-01-09 08:41:13.417066    0'0    2014-01-08 08:40:05.539008
6.1ef    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.360917    0'0    1605:235    [0,3]    [0,3,1]    0'0    2014-01-09 12:25:01.976062    0'0    2014-01-08 12:02:39.055719
6.60    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.761453    0'0    1605:510    [0,2]    [0,2,1]    0'0    2014-01-09 10:45:18.158406    0'0    2014-01-08 10:08:18.336081
9.6f    517    0    518    0    5841250    4234    4234    active+degraded    2014-01-12 10:21:29.279125    1565'10260    1605:16979    [2,0]    [2,0]    1491'7790    2014-01-10 22:21:34.519687    600'6420    2014-01-06 21:49:48.363891
6.62    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:23:47.670890    0'0    1605:702    [4,0]    [4,0,1]    0'0    2014-01-09 16:09:40.034782    0'0    2014-01-08 13:17:31.452140
6.122    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.415398    0'0    1605:719    [6,3]    [6,3,1]    0'0    2014-01-09 08:18:09.795832    0'0    2014-01-08 08:18:06.126540
6.64    0    0    0    0    0    0    0    active    2014-01-12 16:14:33.704017    0'0    1605:33    [3,6]    [3,6,5]    0'0    2014-01-09 09:03:48.527003    0'0    2014-01-08 09:01:43.855932
6.1e2    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:29.131493    0'0    1605:1339    [5,8]    [5,8,9]    0'0    2014-01-09 09:10:43.968654    0'0    2014-01-08 09:07:03.260069
6.67    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:20:57.093502    0'0    1605:701    [8,2]    [8,2,3]    0'0    2014-01-09 15:07:17.081140    0'0    2014-01-08 15:06:59.897113
6.66    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:20:56.082554    0'0    1605:578    [8,2]    [8,2,5]    0'0    2014-01-09 12:24:21.956577    0'0    2014-01-08 11:48:32.823357
6.58    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.433527    0'0    1605:1551    [3,0]    [3,0,1]    0'0    2014-01-09 10:18:53.259457    0'0    2014-01-08 09:46:49.094908
9.57    484    0    484    0    1006483    3002    3002    active+degraded    2014-01-12 10:15:56.115195    1109'5736    1605:14446    [0,2]    [0,2]    884'5735    2014-01-09 20:55:09.218442    299'5550    2014-01-06 00:37:38.343410
6.1e6    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:35:42.536393    0'0    1605:662    [5,0]    [5,0,8]    0'0    2014-01-09 16:25:13.350510    0'0    2014-01-08 14:03:29.209531
6.5a    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.765400    0'0    1605:1825    [0,2]    [0,2,1]    0'0    2014-01-09 19:18:59.709030    0'0    2014-01-09 19:18:59.709030
6.1db    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:20.785977    0'0    1605:670    [0,4]    [0,4,1]    0'0    2014-01-10 06:37:03.579999    0'0    2014-01-10 06:37:03.579999
6.11c    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.358395    0'0    1605:1314    [5,0]    [5,0,1]    0'0    2014-01-09 09:04:45.646165    0'0    2014-01-08 09:02:38.922668
6.1dc    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:35:28.396308    0'0    1605:711    [8,3]    [8,3,5]    0'0    2014-01-09 19:56:49.359394    0'0    2014-01-09 19:56:49.359394
9.5f    514    0    514    0    726167    3002    3002    active+degraded    2014-01-12 16:23:50.792494    1491'7306    1605:16463    [4,0]    [4,0]    1491'7306    2014-01-11 18:07:46.192152    299'7125    2014-01-07 02:16:57.159837
6.111    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:07:00.617463    0'0    1605:1291    [5,6]    [5,6,0]    0'0    2014-01-09 08:40:28.369866    0'0    2014-01-08 08:39:48.560967
6.110    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:29.248910    0'0    1605:1528    [2,8]    [2,8,3]    0'0    2014-01-09 09:00:52.018913    0'0    2014-01-08 09:00:45.358662
9.5d    504    0    505    0    5894969    4241    4241    active+degraded    2014-01-12 10:16:02.372640    1586'6561    1605:15652    [0,2]    [0,2]    884'5308    2014-01-09 14:54:18.368889    299'5263    2014-01-06 14:46:01.474050
6.113    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.359390    0'0    1605:1001    [5,0]    [5,0,1]    0'0    2014-01-09 16:26:30.767291    0'0    2014-01-08 14:03:39.687286
6.117    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.570030    0'0    1605:1786    [0,2]    [0,2,1]    0'0    2014-01-09 09:06:19.414663    0'0    2014-01-08 09:06:07.543870
6.109    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:20:23.141017    0'0    1605:77    [2,8]    [2,8,5]    0'0    2014-01-09 16:54:43.106979    0'0    2014-01-08 15:33:51.778814
6.10a    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:20:23.139547    0'0    1605:1502    [2,8]    [2,8,5]    0'0    2014-01-09 09:24:12.925192    0'0    2014-01-08 09:16:57.973042
6.4f    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:20.950677    0'0    1605:517    [0,4]    [0,4,1]    0'0    2014-01-09 10:17:59.246328    0'0    2014-01-08 09:45:50.088888
6.4e    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.447561    0'0    1605:205    [3,8]    [3,8,1]    0'0    2014-01-09 10:07:42.946438    0'0    2014-01-08 09:37:30.906450
9.41    285    0    284    0    455560    3002    3002    active+degraded    2014-01-12 10:21:25.393688    1565'5155    1605:10472    [2,8]    [2,8]    615'3783    2014-01-09 10:45:50.619414    299'3781    2014-01-06 08:51:18.437773
6.10f    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.426273    0'0    1605:547    [6,3]    [6,3,1]    0'0    2014-01-09 16:54:57.680559    0'0    2014-01-08 15:08:47.013771
6.1cc    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:14:33.093331    0'0    1605:682    [6,2]    [6,2,5]    0'0    2014-01-10 02:50:32.084624    0'0    2014-01-10 02:50:32.084624
6.41    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:00.360657    0'0    1605:603    [2,8]    [2,8,5]    0'0    2014-01-09 14:30:45.305420    0'0    2014-01-08 12:43:26.784150
6.1ce    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:23.961210    0'0    1605:1495    [2,0]    [2,0,9]    0'0    2014-01-09 09:05:16.346117    0'0    2014-01-08 09:03:25.479488
9.4c    511    0    511    0    881371    3002    3002    active+degraded    2014-01-12 16:21:20.952410    1565'5905    1605:14560    [0,4]    [0,4]    615'4516    2014-01-09 08:58:52.309443    299'4492    2014-01-07 03:01:43.374142
6.103    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.369690    0'0    1605:895    [5,6]    [5,6,1]    0'0    2014-01-09 16:23:26.473540    0'0    2014-01-08 13:20:41.951785
6.45    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:56:15.121244    0'0    1605:1581    [3,8]    [3,8,9]    0'0    2014-01-09 10:16:28.274734    0'0    2014-01-08 09:45:48.564931
6.1c3    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:07:00.520696    0'0    1605:869    [5,6]    [5,6,8]    0'0    2014-01-09 16:55:27.692213    0'0    2014-01-08 15:09:34.039603
6.44    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:29.277955    0'0    1605:1494    [2,8]    [2,8,3]    0'0    2014-01-10 03:07:39.229560    0'0    2014-01-07 22:27:32.496363
6.1c2    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:29.236995    0'0    1605:73    [2,8]    [2,8,3]    0'0    2014-01-09 10:41:09.575115    0'0    2014-01-08 09:54:16.500084
6.1c5    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.270096    0'0    1605:1340    [5,6]    [5,6,1]    0'0    2014-01-09 09:08:39.895969    0'0    2014-01-08 09:06:57.244843
6.1c6    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:19:32.778447    0'0    1605:1537    [2,8]    [2,8,5]    0'0    2014-01-09 09:04:56.271867    0'0    2014-01-08 09:03:16.470590
9.32    267    0    267    0    760405    1732    1732    active+remapped    2014-01-12 10:16:08.293588    1109'1732    1605:6199    [0,2]    [0,2,3]    884'1731    2014-01-09 20:54:35.857181    299'1712    2014-01-06 00:37:16.909432
6.1bc    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.389568    0'0    1605:521    [0,2]    [0,2,1]    0'0    2014-01-09 08:49:16.759156    0'0    2014-01-08 08:49:14.982061
6.31    0    0    0    0    0    0    0    active+remapped    2014-01-10 14:48:36.832214    0'0    1605:194    [3,6]    [3,6,0]    0'0    2014-01-09 14:33:15.491643    0'0    2014-01-08 12:44:54.931011
9.3e    257    0    257    0    4863385    4266    4266    active+degraded    2014-01-12 10:19:43.301020    1527'5478    1605:10677    [2,8]    [2,8]    615'3980    2014-01-09 12:40:29.560774    299'3969    2014-01-06 17:48:58.838951
6.fe    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.384469    0'0    1605:524    [8,3]    [8,3,1]    0'0    2014-01-09 09:17:40.417863    0'0    2014-01-08 09:09:11.171327
6.f0    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:56:01.445331    0'0    1605:1576    [3,0]    [3,0,9]    0'0    2014-01-09 10:26:18.577847    0'0    2014-01-08 09:53:57.245027
6.1b0    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:29.364499    0'0    1605:1323    [5,8]    [5,8,9]    0'0    2014-01-09 08:43:40.523357    0'0    2014-01-08 08:43:22.620376
6.35    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:38.392468    0'0    1605:1801    [0,2]    [0,2,9]    0'0    2014-01-09 19:18:47.707930    0'0    2014-01-09 19:18:47.707930
6.1b2    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:07:00.543523    0'0    1605:1347    [5,0]    [5,0,6]    0'0    2014-01-09 11:12:38.981953    0'0    2014-01-08 11:11:22.856320
6.29    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.523288    0'0    1605:513    [8,5]    [8,5,1]    0'0    2014-01-09 10:44:23.127433    0'0    2014-01-08 10:07:09.308621
6.28    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.454218    0'0    1605:429    [5,0]    [5,0,1]    0'0    2014-01-09 10:44:21.126616    0'0    2014-01-08 10:05:23.287340
6.e9    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.180901    0'0    1605:78    [2,6]    [2,6,3]    0'0    2014-01-09 16:23:26.274441    0'0    2014-01-08 13:37:04.794046
6.e8    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.182154    0'0    1605:1502    [2,6]    [2,6,3]    0'0    2014-01-09 09:23:27.822206    0'0    2014-01-08 09:12:56.813109
6.1a8    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.393001    0'0    1605:673    [0,5]    [0,5,1]    0'0    2014-01-09 08:29:55.042014    0'0    2014-01-08 08:29:54.548487
6.2c    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.454337    0'0    1605:1585    [3,0]    [3,0,1]    0'0    2014-01-09 10:15:49.174461    0'0    2014-01-08 09:43:10.046785
9.23    281    0    281    0    1203992    3001    3001    active+degraded    2014-01-12 16:23:53.986197    1109'3245    1605:8663    [4,6]    [4,6]    1109'3245    2014-01-11 10:30:31.471057    299'3077    2014-01-06 14:33:55.527018
MailScanner has detected a possible fraud attempt from "6.ec" claiming to be MailScanner warning: numerical links are often malicious: 6.ec    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.267184    0'0    1605:659    [6,2]    [6,2,3]    0'0    2014-01-09 15:19:52.865780    0'0    2014-01-08 15:18:57.258859
6.2e    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:06.572899    0'0    1605:76    [2,6]    [2,6,5]    0'0    2014-01-09 08:55:56.840643    0'0    2014-01-08 08:54:48.264306
6.1ae    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:22:15.852876    0'0    1605:1363    [4,0]    [4,0,1]    0'0    2014-01-09 10:57:26.601672    0'0    2014-01-08 10:14:36.158317
6.1a1    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:30.364236    0'0    1605:1606    [3,0]    [3,0,9]    0'0    2014-01-09 09:51:37.383129    0'0    2014-01-08 09:31:18.688919
6.22    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:09.469724    0'0    1605:1429    [2,8]    [2,8,4]    0'0    2014-01-09 08:55:55.854115    0'0    2014-01-08 08:54:37.256926
6.24    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.530486    0'0    1605:702    [8,3]    [8,3,1]    0'0    2014-01-09 19:18:34.690876    0'0    2014-01-09 19:18:34.690876
9.2b    276    0    276    0    994133    3001    3001    active+degraded    2014-01-12 10:21:07.380808    1109'3229    1605:7774    [2,6]    [2,6]    615'3213    2014-01-09 10:43:26.465712    299'3202    2014-01-06 08:45:29.474538
6.1a5    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:30.365180    0'0    1605:849    [3,6]    [3,6,9]    0'0    2014-01-09 12:34:15.079349    0'0    2014-01-08 12:34:11.416407
6.26    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:23:53.989924    0'0    1605:1281    [4,6]    [4,6,8]    0'0    2014-01-09 10:43:40.110560    0'0    2014-01-08 10:05:22.274204
9.17    257    0    0    0    4933022    3001    3001    active+remapped    2014-01-12 16:14:25.437802    1527'3806    1605:10986    [6,2]    [6,2,7]    884'2624    2014-01-10 06:12:17.756516    884'2624    2014-01-10 06:12:17.756516
6.1a6    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:29.714289    0'0    1605:1340    [5,6]    [5,6,9]    0'0    2014-01-09 09:06:12.797997    0'0    2014-01-08 09:06:12.193747
6.199    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.321114    0'0    1605:1570    [3,6]    [3,6,1]    0'0    2014-01-09 10:29:18.663287    0'0    2014-01-08 09:54:08.265485
6.1a    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:14:25.438292    0'0    1605:1363    [6,2]    [6,2,7]    0'0    2014-01-09 20:36:30.994093    0'0    2014-01-09 20:36:30.994093
6.da    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:24.188309    0'0    1605:702    [8,5]    [8,5,9]    0'0    2014-01-10 06:38:52.549091    0'0    2014-01-10 06:38:52.549091
6.1c    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:06.617744    0'0    1605:748    [6,2]    [6,2,5]    0'0    2014-01-09 16:51:59.504404    0'0    2014-01-08 15:08:10.305595
6.19d    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:25.010947    0'0    1605:395    [8,2]    [8,2,3]    0'0    2014-01-09 16:55:01.114467    0'0    2014-01-08 15:34:09.794056
9.1f    256    0    256    0    505448    2911    2911    active+degraded    2014-01-12 10:15:17.820311    1491'2911    1605:7124    [2,8]    [2,8]    1109'2910    2014-01-10 21:21:06.415076    299'2831    2014-01-06 19:09:13.654085
6.d1    0    0    0    0    0    0    0    active+remapped    2014-01-10 14:48:20.302744    0'0    1605:1270    [5,8]    [5,8,0]    0'0    2014-01-09 08:40:31.389855    0'0    2014-01-08 08:39:49.560872
6.19e    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:09.549937    0'0    1605:1696    [8,4]    [8,4,1]    0'0    2014-01-09 08:48:57.237465    0'0    2014-01-08 08:48:55.964886
6.13    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:56:15.155440    0'0    1605:1565    [3,0]    [3,0,9]    0'0    2014-01-09 09:54:05.418547    0'0    2014-01-08 09:32:49.785038
6.191    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:24.020591    0'0    1605:1281    [6,3]    [6,3,9]    0'0    2014-01-10 02:50:30.042711    0'0    2014-01-10 02:50:30.042711
6.d2    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.371571    0'0    1605:1288    [5,0]    [5,0,1]    0'0    2014-01-09 09:03:58.637975    0'0    2014-01-08 09:02:33.917141
6.250    0    0    0    0    0    0    0    active+remapped    2014-01-10 14:47:51.620257    0'0    1605:1581    [3,6]    [3,6,0]    0'0    2014-01-09 10:32:08.844964    0'0    2014-01-08 09:55:38.287309
6.d4    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.347861    0'0    1605:416    [3,6]    [3,6,1]    0'0    2014-01-09 10:51:55.467448    0'0    2014-01-08 10:13:14.112677
6.252    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.297932    0'0    1605:503    [2,0]    [2,0,1]    0'0    2014-01-09 12:25:14.982234    0'0    2014-01-08 12:02:51.052423
9.19    253    0    253    0    606223    3002    3002    active+degraded    2014-01-12 10:16:02.453285    1109'3153    1605:7462    [6,2]    [6,2]    884'3152    2014-01-09 16:26:55.642238    299'3011    2014-01-07 08:37:09.343261
6.c9    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:24.189323    0'0    1605:719    [8,3]    [8,3,9]    0'0    2014-01-09 19:27:09.848901    0'0    2014-01-09 19:27:09.848901
6.196    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.323930    0'0    1605:1576    [3,6]    [3,6,1]    0'0    2014-01-09 10:27:18.612860    0'0    2014-01-08 09:54:06.250691
6.257    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.257906    0'0    1605:989    [0,5]    [0,5,1]    0'0    2014-01-09 20:06:21.532104    0'0    2014-01-09 20:06:21.532104
6.249    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:10.026851    0'0    1605:522    [6,2]    [6,2,4]    0'0    2014-01-09 09:12:36.014578    0'0    2014-01-08 09:08:16.330104
6.248    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:14:25.356629    0'0    1605:44    [6,2]    [6,2,0]    0'0    2014-01-09 16:25:16.309428    0'0    2014-01-08 14:03:38.247840
6.18a    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.324766    0'0    1605:197    [3,6]    [3,6,1]    0'0    2014-01-09 12:34:12.103024    0'0    2014-01-08 12:34:09.938504
6.24b    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:00.360878    0'0    1605:1491    [2,8]    [2,8,5]    0'0    2014-01-09 09:08:37.598426    0'0    2014-01-08 09:06:22.577679
6.cc    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:23:39.871509    0'0    1605:1311    [4,0]    [4,0,1]    0'0    2014-01-09 10:32:08.775105    0'0    2014-01-08 09:55:42.034457
9.c3    240    0    240    0    767354    2882    2882    active+degraded    2014-01-12 16:23:39.869188    884'2882    1605:6555    [4,6]    [4,6]    884'2882    2014-01-12 13:55:50.007941    299'2824    2014-01-07 05:18:32.908294
6.e    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.220860    0'0    1605:1496    [2,6]    [2,6,3]    0'0    2014-01-09 08:55:54.839006    0'0    2014-01-08 08:53:27.205348
9.1    284    0    2    0    8773472    3001    3001    active+remapped    2014-01-12 16:26:24.582499    1605'4995    1605:1908    [2,0]    [2,0,5]    1199'2589    2014-01-11 06:07:27.851789    299'2152    2014-01-07 05:12:42.379851
6.24c    0    0    0    0    0    0    0    active+remapped    2014-01-10 14:48:12.669333    0'0    1605:1604    [3,6]    [3,6,0]    0'0    2014-01-09 10:31:28.803424    0'0    2014-01-08 09:55:02.271086
6.c0    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:35:19.595209    0'0    1605:673    [6,5]    [6,5,3]    0'0    2014-01-07 15:09:56.028935    0'0    2013-12-31 19:07:00.165926
6.24e    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.182787    0'0    1605:1325    [6,3]    [6,3,1]    0'0    2014-01-10 05:43:23.038175    0'0    2014-01-10 05:43:23.038175
6.2    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.710914    0'0    1605:1214    [6,5]    [6,5,1]    0'0    2014-01-09 11:08:44.798110    0'0    2014-01-08 11:07:54.973330
6.4    0    0    0    0    0    0    0    active    2014-01-12 16:16:02.584767    0'0    1605:40    [2,0]    [2,0,5]    0'0    2014-01-09 10:39:45.988813    0'0    2014-01-08 10:03:25.712846
9.b    255    0    255    0    511815    2597    2597    active+degraded    2014-01-12 10:15:17.832137    1491'2597    1605:7550    [2,6]    [2,6]    1109'2596    2014-01-10 21:01:03.832837    299'2579    2014-01-06 19:08:34.842640
6.c5    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.373055    0'0    1605:408    [5,8]    [5,8,1]    0'0    2014-01-09 10:51:52.465758    0'0    2014-01-08 10:13:10.108756
6.c6    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:23:24.146939    0'0    1605:1320    [4,6]    [4,6,1]    0'0    2014-01-09 10:36:18.881569    0'0    2014-01-08 09:59:22.108028
6.246    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:38.136103    0'0    1605:1641    [0,3]    [0,3,9]    0'0    2014-01-09 09:23:27.018453    0'0    2014-01-08 09:17:05.086984
MailScanner has detected a possible fraud attempt from "6.bd" claiming to be MailScanner warning: numerical links are often malicious: 6.bd    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:09.458198    0'0    1605:107    [2,8]    [2,8,4]    0'0    2014-01-09 12:19:43.882949    0'0    2014-01-08 11:47:30.792530
6.23a    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:38.136777    0'0    1605:1864    [0,2]    [0,2,9]    0'0    2014-01-09 20:06:18.530400    0'0    2014-01-09 20:06:18.530400
MailScanner has detected a possible fraud attempt from "6.bf" claiming to be MailScanner warning: numerical links are often malicious: 6.bf    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.237895    0'0    1605:94    [2,0]    [2,0,1]    0'0    2014-01-09 16:49:41.912921    0'0    2014-01-08 14:40:41.355242
9.b1    278    0    278    0    1019316    3001    3001    active+degraded    2014-01-12 16:23:30.058082    1109'3071    1605:6640    [4,0]    [4,0]    1109'3071    2014-01-12 11:03:50.240907    299'2661    2014-01-07 05:18:29.080755
6.b1    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.848134    0'0    1605:238    [5,8]    [5,8,2]    0'0    2014-01-09 12:30:47.991392    0'0    2014-01-08 12:30:07.382088
6.17e    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:24.092695    0'0    1605:715    [8,3]    [8,3,9]    0'0    2014-01-10 06:39:01.563804    0'0    2014-01-10 06:39:01.563804
6.23f    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.458123    0'0    1605:1285    [5,8]    [5,8,1]    0'0    2014-01-09 08:51:50.913292    0'0    2014-01-08 08:51:29.800394
9.bc    249    0    249    0    366022    1985    1985    active+degraded    2014-01-12 10:15:17.715812    884'1985    1605:847    [2,0]    [2,0]    884'1985    2014-01-09 16:26:06.346292    299'1966    2014-01-07 08:43:56.689069
MailScanner has detected a possible fraud attempt from "9.ba" claiming to be MailScanner warning: numerical links are often malicious: 9.ba    282    0    282    0    713504    3001    3001    active+degraded    2014-01-12 10:15:45.114064    1491'3548    1605:9295    [2,8]    [2,8]    615'3148    2014-01-09 11:30:53.430338    615'3148    2014-01-08 11:14:00.957418
6.172    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.429703    0'0    1605:221    [0,5]    [0,5,1]    0'0    2014-01-09 11:18:07.081397    0'0    2014-01-08 10:24:17.153406
6.b4    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:23:30.143976    0'0    1605:1298    [4,0]    [4,0,1]    0'0    2014-01-09 10:47:47.203140    0'0    2014-01-08 10:09:23.363418
6.174    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:29.243245    0'0    1605:312    [2,8]    [2,8,3]    0'0    2014-01-09 10:53:42.519222    0'0    2014-01-08 10:13:58.149248
6.a8    0    0    0    0    0    0    0    active+remapped    2014-01-10 14:48:37.071210    0'0    1605:331    [3,6]    [3,6,0]    0'0    2014-01-09 10:31:41.761935    0'0    2014-01-08 09:55:18.027269
6.169    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:19:49.723024    0'0    1605:1501    [2,8]    [2,8,5]    0'0    2014-01-09 09:31:25.136044    0'0    2014-01-08 09:18:41.073577
9.a5    261    0    261    0    681314    3045    3045    active+degraded    2014-01-11 18:33:15.638697    1501'4656    1605:12077    [3,6]    [3,6]    884'3629    2014-01-09 15:05:31.498011    299'3603    2014-01-06 15:01:16.697258
MailScanner has detected a possible fraud attempt from "6.af" claiming to be MailScanner warning: numerical links are often malicious: 6.af    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:20.920953    0'0    1605:464    [0,5]    [0,5,4]    0'0    2014-01-09 09:00:14.242287    0'0    2014-01-08 09:00:09.061867
6.16c    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:04.813484    0'0    1605:85    [2,6]    [2,6,5]    0'0    2014-01-09 17:00:10.356004    0'0    2014-01-08 15:52:27.147650
6.22d    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.135565    0'0    1605:77    [2,6]    [2,6,3]    0'0    2014-01-09 16:08:10.003447    0'0    2014-01-08 13:16:22.419681
MailScanner has detected a possible fraud attempt from "9.ae" claiming to be MailScanner warning: numerical links are often malicious: 9.ae    279    0    280    0    4929245    4262    4262    active+degraded    2014-01-11 18:32:33.273086    1565'5245    1605:12112    [5,8]    [5,8]    615'3912    2014-01-09 12:25:39.417078    299'3902    2014-01-06 16:40:48.555270
6.16e    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:24.102547    0'0    1605:707    [8,3]    [8,3,9]    0'0    2014-01-09 21:58:00.116420    0'0    2014-01-09 21:58:00.116420
6.223    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.175408    0'0    1605:338    [2,0]    [2,0,1]    0'0    2014-01-09 10:58:10.620538    0'0    2014-01-08 10:14:45.164997
6.165    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:22:22.739013    0'0    1605:1355    [4,6]    [4,6,1]    0'0    2014-01-09 10:53:40.526426    0'0    2014-01-08 10:13:57.144933
6.224    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:16:35.166879    0'0    1605:1318    [5,0]    [5,0,7]    0'0    2014-01-11 01:55:28.428832    0'0    2014-01-08 08:51:21.796833
6.99    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.366371    0'0    1605:206    [3,0]    [3,0,1]    0'0    2014-01-09 09:39:51.857459    0'0    2014-01-08 09:27:28.474615
6.9d    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:20:48.760442    0'0    1605:712    [8,2]    [8,2,5]    0'0    2014-01-09 19:27:06.862888    0'0    2014-01-09 19:27:06.862888
6.21a    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:05:46.937303    0'0    1605:1632    [3,6]    [3,6,0]    0'0    2014-01-09 10:30:49.725194    0'0    2014-01-08 09:54:13.254937
6.15c    0    0    0    0    0    0    0    active    2014-01-12 16:14:33.270607    0'0    1605:34    [3,6]    [3,6,5]    0'0    2014-01-09 08:41:16.439686    0'0    2014-01-08 08:40:48.567231
6.15f    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:20.825238    0'0    1605:231    [0,4]    [0,4,1]    0'0    2014-01-09 16:55:22.690895    0'0    2014-01-08 15:09:02.011332
6.21c    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.140501    0'0    1605:78    [2,6]    [2,6,3]    0'0    2014-01-09 10:42:21.603282    0'0    2014-01-08 09:54:59.513208
6.91    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.239622    0'0    1605:1510    [2,0]    [2,0,1]    0'0    2014-01-09 09:21:54.744981    0'0    2014-01-08 09:11:59.770748
6.21f    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.206178    0'0    1605:506    [8,5]    [8,5,1]    0'0    2014-01-10 03:03:47.780276    0'0    2014-01-10 03:03:47.780276
6.150    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.331047    0'0    1605:206    [5,0]    [5,0,1]    0'0    2014-01-09 12:24:18.888704    0'0    2014-01-08 11:48:00.298061
6.210    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.306686    0'0    1605:456    [3,6]    [3,6,1]    0'0    2014-01-09 10:39:54.993102    0'0    2014-01-08 10:03:30.224818
9.9a    227    0    227    0    956824    3000    3000    active+degraded    2014-01-12 10:16:02.400086    884'3906    1605:9239    [8,2]    [8,2]    884'3906    2014-01-09 20:56:55.174021    299'3819    2014-01-06 00:38:22.792384
6.152    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:22:34.779934    0'0    1605:1336    [4,8]    [4,8,1]    0'0    2014-01-09 10:38:02.914887    0'0    2014-01-08 10:02:11.190879
6.155    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:38.266827    0'0    1605:670    [0,4]    [0,4,9]    0'0    2014-01-09 09:33:01.719863    0'0    2014-01-08 09:20:32.349939
6.154    0    0    0    0    0    0    0    active+remapped    2014-01-10 13:44:12.933852    0'0    1605:224    [3,6]    [3,6,0]    0'0    2014-01-09 16:55:20.691398    0'0    2014-01-08 15:09:00.010145
6.214    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:06.493689    0'0    1605:238    [6,2]    [6,2,5]    0'0    2014-01-09 10:40:00.998246    0'0    2014-01-08 10:03:35.229280
6.149    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:25.016020    0'0    1605:93    [2,8]    [2,8,3]    0'0    2014-01-09 09:03:46.143118    0'0    2014-01-08 09:01:38.386273
6.14a    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:06.551389    0'0    1605:1507    [2,6]    [2,6,5]    0'0    2014-01-09 09:26:58.010643    0'0    2014-01-08 09:17:15.010940
6.20a    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:38.145184    0'0    1605:1306    [0,5]    [0,5,9]    0'0    2014-01-09 07:55:41.720768    0'0    2014-01-08 07:54:46.098197
9.8e    281    0    281    0    597395    3003    3003    active+degraded    2014-01-12 10:21:27.943479    1109'3302    1605:7601    [2,0]    [2,0]    1109'3302    2014-01-10 22:36:07.774680    342'3053    2014-01-06 22:26:44.662514
6.80    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.444524    0'0    1605:693    [8,5]    [8,5,1]    0'0    2014-01-09 19:23:11.775546    0'0    2014-01-09 19:23:11.775546
6.83    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.393136    0'0    1605:808    [3,8]    [3,8,1]    0'0    2014-01-09 16:45:24.854229    0'0    2014-01-08 14:40:31.349407
6.82    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.460593    0'0    1605:700    [8,3]    [8,3,1]    0'0    2014-01-09 19:23:12.776633    0'0    2014-01-09 19:23:12.776633
6.142    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:20:21.411519    0'0    1605:244    [8,2]    [8,2,5]    0'0    2014-01-09 10:40:40.595621    0'0    2014-01-08 09:54:13.504283
9.77    501    0    501    0    1136032    3003    3003    active+degraded    2014-01-12 10:16:02.368324    1109'4746    1605:12686    [0,2]    [0,2]    884'4744    2014-01-09 20:56:18.077813    299'4557    2014-01-06 00:38:05.128010
6.7b    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:23:41.787528    0'0    1605:1336    [4,8]    [4,8,1]    0'0    2014-01-09 10:32:00.769992    0'0    2014-01-08 09:55:39.034384
6.138    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.343884    0'0    1605:817    [5,6]    [5,6,1]    0'0    2014-01-09 12:24:17.896516    0'0    2014-01-08 11:47:59.308707
6.7a    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:01.319589    0'0    1605:668    [0,2]    [0,2,5]    0'0    2014-01-09 19:19:09.701568    0'0    2014-01-09 19:19:09.701568
6.13b    0    0    0    0    0    0    0    active+remapped    2014-01-10 15:17:27.246371    0'0    1605:1324    [5,6]    [5,6,0]    0'0    2014-01-09 09:05:13.736448    0'0    2014-01-08 09:04:25.122441
6.13c    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.172013    0'0    1605:1557    [2,6]    [2,6,3]    0'0    2014-01-09 22:44:24.068932    0'0    2014-01-07 22:27:04.477927
6.1fd    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.312842    0'0    1605:1594    [3,0]    [3,0,1]    0'0    2014-01-09 09:56:37.535956    0'0    2014-01-08 09:34:03.817682
6.7e    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.416132    0'0    1605:1543    [3,8]    [3,8,1]    0'0    2014-01-09 10:21:38.409338    0'0    2014-01-08 09:47:43.130429
6.71    0    0    0    0    0    0    0    active+remapped    2014-01-10 14:47:48.287766    0'0    1605:1279    [5,6]    [5,6,0]    0'0    2014-01-09 09:03:49.538744    0'0    2014-01-08 09:02:17.908254
6.13e    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.208592    0'0    1605:94    [2,0]    [2,0,1]    0'0    2014-01-09 22:45:15.057054    0'0    2014-01-07 22:36:27.054235
6.1fe    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:02.181911    0'0    1605:509    [6,2]    [6,2,3]    0'0    2014-01-09 10:56:20.577888    0'0    2014-01-08 10:14:18.147755
6.73    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.410402    0'0    1605:388    [5,6]    [5,6,1]    0'0    2014-01-09 10:46:06.164304    0'0    2014-01-08 10:08:28.342536
6.1f1    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:06.503325    0'0    1605:677    [6,2]    [6,2,5]    0'0    2014-01-10 02:50:48.083370    0'0    2014-01-10 02:50:48.083370
6.72    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.245430    0'0    1605:1493    [2,0]    [2,0,1]    0'0    2014-01-09 08:57:24.880524    0'0    2014-01-08 08:55:05.238720
6.75    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:21:43.462504    0'0    1605:699    [8,5]    [8,5,1]    0'0    2014-01-09 21:20:57.327577    0'0    2014-01-09 21:20:57.327577
6.74    0    0    0    0    0    0    0    active+remapped    2014-01-10 14:47:47.808109    0'0    1605:225    [8,5]    [8,5,0]    0'0    2014-01-09 10:31:58.783857    0'0    2014-01-08 09:55:30.035789
6.77    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:24:05.866553    0'0    1605:231    [3,0]    [3,0,4]    0'0    2014-01-09 16:05:27.947031    0'0    2014-01-08 13:05:01.210834
9.78    483    0    483    0    1397009    3002    3002    active+degraded    2014-01-12 16:23:48.106924    1491'10499    1605:21936    [4,8]    [4,8]    1491'10499    2014-01-11 17:07:40.143810    299'9912    2014-01-06 14:57:57.183660
6.1f5    0    0    0    0    0    0    0    active+remapped    2014-01-12 10:16:00.112840    0'0    1605:250    [3,0]    [3,0,2]    0'0    2014-01-09 11:19:14.116060    0'0    2014-01-08 10:24:42.161664
6.69    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:55:40.680517    0'0    1605:1314    [4,6]    [4,6,9]    0'0    2014-01-09 10:31:09.756309    0'0    2014-01-08 09:54:04.000963
9.66    514    0    514    0    5941247    4254    4254    active+degraded    2014-01-12 16:23:46.541017    1527'5604    1605:14508    [4,6]    [4,6]    1199'4351    2014-01-11 17:04:53.443119    299'4300    2014-01-06 14:55:47.126133
6.68    0    0    0    0    0    0    0    active+remapped    2014-01-12 16:21:10.236438    0'0    1605:1362    [6,4]    [6,4,1]    0'0    2014-01-09 20:37:01.966498    0'0    2014-01-09 20:37:01.966498
9.64    494    0    493    0    1001770    3586    3586    active+degraded    2014-01-12 10:15:56.081499    1605'7607    1605:19399    [8,2]    [8,2]    879'6275    2014-01-09 08:51:53.242171    299'6023    2014-01-07 02:25:39.810228
9.65    509    0    509    0    1128369    3002    3002    active+degraded    2014-01-12 16:21:10.241191    1535'7350    1605:12720    [6,4]    [6,4]    884'6107    2014-01-09 18:05:34.531831    884'6107    2014-01-09 18:05:34.531831



--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Aaron Ten Clay
http://www.aarontc.com/

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux