Hey,
Cephfs has became available!
I didn't change the rules
Do you guys see something "lost" or "abolutely screwed" from these messages?
Do I have only to wait?
I see backfill_tooful: it sound strange because I have set the option
"osd backfill tooful ratio = 0.91" in conf and not one of my osd now
is over that percentage
Thanks
The health now is
HEALTH_WARN 2038 pgs backfill; 43 pgs backfill_toofull; 134 pgs
backfilling; 62 pgs degraded; 590 pgs recovery_wait; 2765 pgs stuck
unclean; recovery 2780119/22308143 degraded (12.462%); recovering 42
o/s, 197MB/s; 5 near full osd(s); noup,nodown flag(s) set
2013-04-22 11:33:31.288690 mon.0 [INF] pgmap v1459630: 17280 pgs:
14512 active+clean, 1945 active+remapped+wait_backfill, 14
active+degraded+wait_backfill, 36
active+remapped+wait_backfill+backfill_toofull, 565
active+recovery_wait, 128 active+remapped+backfilling, 4
active+remapped+backfill_toofull, 4 active+degraded+backfilling, 3
active+clean+scrubbing, 37 active+degraded+remapped+wait_backfill, 25
active+recovery_wait+remapped, 3
active+degraded+remapped+wait_backfill+backfill_toofull, 4
active+degraded+remapped+backfilling; 50432 GB data, 76416 GB used,
37015 GB / 110 TB avail; 2777977/22308143 degraded (12.453%);
recovering 15 o/s, 68099KB/s
2013/4/22 Marco Aroldi <marco.aroldi@xxxxxxxxx>:
> In the original design,
> I've change the rules since I would data placed with replica 2 in 2
> identical room (named p1 and p2)
> Now that 1 room has 4 osd out of cluster, do I have to change the
> rules and use an "type host" rule instead "type room"?
> Could this help?
>
> root default {
> id -1 # do not change unnecessarily
> # weight 122.500
> alg straw
> hash 0 # rjenkins1
> item p1 weight 57.500
> item p2 weight 65.000
> }
>
> # rules
> rule data {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type room
> step emit
> }
> rule metadata {
> ruleset 1
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type room
> step emit
> }
> rule rbd {
> ruleset 2
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type room
> step emit
> }
>
> # end crush map
>
>
> ceph health:
>
> HEALTH_WARN 2072 pgs backfill; 43 pgs backfill_toofull; 131 pgs
> backfilling; 68 pgs degraded; 594 pgs recovery_wait; 2802 pgs stuck
> unclean; recovery 2811952/22351845 degraded (12.580%); recovering 35
> o/s, 197MB/s; 4 near full osd(s); noup,nodown flag(s) set
>
>
> 2013-04-22 10:53:26.800014 mon.0 [INF] pgmap v1457213: 17280 pgs:
> 14474 active+clean, 1975 active+remapped+wait_backfill, 18
> active+degraded+wait_backfill, 37
> active+remapped+wait_backfill+backfill_toofull, 569
> active+recovery_wait, 123 active+remapped+backfilling, 3
> active+remapped+backfill_toofull, 3 active+degraded+backfilling, 6
> active+clean+scrubbing, 39 active+degraded+remapped+wait_backfill, 25
> active+recovery_wait+remapped, 3
> active+degraded+remapped+wait_backfill+backfill_toofull, 5
> active+degraded+remapped+backfilling; 50432 GB data, 76277 GB used,
> 37154 GB / 110 TB avail; 2811241/22350671 degraded (12.578%);
> recovering 29 o/s, 119MB/s
>
> 2013/4/22 Marco Aroldi <marco.aroldi@xxxxxxxxx>:
>> The rebalance is still going
>> and the mounts are still refused
>>
>> I've re-set the nodown noup flags because the osd are flapping continuously
>> and added in ceph.conf "osd backfill tooful ratio = 0.91", tryin to
>> get rid of all that "backfill_tooful"
>>
>> What I have to to now to regain access?
>>
>> I can provide you any logs or whatever you need
>> Thanks for support
>>
>> in ceph -w I see this:
>> 2013-04-22 09:25:46.601721 osd.8 [WRN] 1 slow requests, 1 included
>> below; oldest blocked for > 5404.500806 secs
>> 2013-04-22 09:25:46.601727 osd.8 [WRN] slow request 5404.500806
>> seconds old, received at 2013-04-22 07:55:42.100886:
>> osd_op(mds.0.9:177037 10000025d80.000017b3 [stat] 0.300279a9 RETRY
>> rwordered) v4 currently reached pgosd
>>
>> this is the ceph mds dump:
>>
>> dumped mdsmap epoch 52
>> epoch 52
>> flags 0
>> created 2013-03-18 14:42:29.330548
>> modified 2013-04-22 09:08:45.599613
>> tableserver 0
>> root 0
>> session_timeout 60
>> session_autoclose 300
>> last_failure 49
>> last_failure_osd_epoch 33152
>> compat compat={},rocompat={},incompat={1=base v0.20,2=client
>> writeable ranges,3=default file layouts on dirs,4=dir inode in
>> separate object}
>> max_mds 1
>> in 0
>> up {0=6957}
>> failed
>> stopped
>> data_pools [0]
>> metadata_pool 1
>> 6957: 192.168.21.11:6800/5844 'm1' mds.0.10 up:active seq 23
>> 5945: 192.168.21.13:6800/12999 'm3' mds.-1.0 up:standby seq 1
>> 5963: 192.168.21.12:6800/22454 'm2' mds.-1.0 up:standby seq 1
>>
>> ceph health:
>>
>> HEALTH_WARN 2133 pgs backfill; 47 pgs backfill_toofull; 136 pgs
>> backfilling; 74 pgs degraded; 1 pgs recovering; 599 pgs recovery_wait;
>> 2877 pgs stuck unclean; recovery 2910416/22449672 degraded (12.964%);
>> recovering 10 o/s, 48850KB/s; 7 near full osd(s); noup,nodown flag(s)
>> set
>>
>> 2013-04-22 09:34:11.436514 mon.0 [INF] pgmap v1452450: 17280 pgs:
>> 14403 active+clean, 2032 active+remapped+wait_backfill, 19
>> active+degraded+wait_backfill, 35
>> active+remapped+wait_backfill+backfill_toofull, 574
>> active+recovery_wait, 126 active+remapped+backfilling, 9
>> active+remapped+backfill_toofull, 3 active+degraded+backfilling, 2
>> active+clean+scrubbing, 41 active+degraded+remapped+wait_backfill, 25
>> active+recovery_wait+remapped, 3
>> active+degraded+remapped+wait_backfill+backfill_toofull, 8
>> active+degraded+remapped+backfilling; 50432 GB data, 76229 GB used,
>> 37202 GB / 110 TB avail; 2908837/22447349 degraded (12.958%);
>> recovering 6 o/s, 20408KB/s
>>
>> 2013/4/21 Marco Aroldi <marco.aroldi@xxxxxxxxx>:
>>> Greg, your supposition about the small amount data to be written is
>>> right but the rebalance is writing an insane amount of data to the new
>>> nodes and the mount is not working again
>>>
>>> this is the node S203 (the os is on /dev/sdl, not listed)
>>>
>>> /dev/sda1 1.9T 467G 1.4T 26% /var/lib/ceph/osd/ceph-44
>>> /dev/sdb1 1.9T 595G 1.3T 33% /var/lib/ceph/osd/ceph-45
>>> /dev/sdc1 1.9T 396G 1.5T 22% /var/lib/ceph/osd/ceph-46
>>> /dev/sdd1 1.9T 401G 1.5T 22% /var/lib/ceph/osd/ceph-47
>>> /dev/sde1 1.9T 337G 1.5T 19% /var/lib/ceph/osd/ceph-48
>>> /dev/sdf1 1.9T 441G 1.4T 24% /var/lib/ceph/osd/ceph-49
>>> /dev/sdg1 1.9T 338G 1.5T 19% /var/lib/ceph/osd/ceph-50
>>> /dev/sdh1 1.9T 359G 1.5T 20% /var/lib/ceph/osd/ceph-51
>>> /dev/sdi1 1.4T 281G 1.1T 21% /var/lib/ceph/osd/ceph-52
>>> /dev/sdj1 1.4T 423G 964G 31% /var/lib/ceph/osd/ceph-53
>>> /dev/sdk1 1.9T 421G 1.4T 23% /var/lib/ceph/osd/ceph-54
>>>
>>> 2013/4/21 Marco Aroldi <marco.aroldi@xxxxxxxxx>:
>>>> What I can try to do/delete to regain access?
>>>> Those osd are crazy, flapping up and down. I think that the situation
>>>> is without control
>>>>
>>>>
>>>> HEALTH_WARN 2735 pgs backfill; 13 pgs backfill_toofull; 157 pgs
>>>> backfilling; 188 pgs degraded; 251 pgs peering; 13 pgs recovering;
>>>> 1159 pgs recovery_wait; 159 pgs stuck inactive; 4641 pgs stuck
>>>> unclean; recovery 4007916/23007073 degraded (17.420%); recovering 4
>>>> o/s, 31927KB/s; 19 near full osd(s)
>>>>
>>>> 2013-04-21 18:56:46.839851 mon.0 [INF] pgmap v1399007: 17280 pgs: 276
>>>> active, 12791 active+clean, 2575 active+remapped+wait_backfill, 71
>>>> active+degraded+wait_backfill, 6
>>>> active+remapped+wait_backfill+backfill_toofull, 1121
>>>> active+recovery_wait, 90 peering, 3 remapped, 1 active+remapped, 127
>>>> active+remapped+backfilling, 1 active+degraded, 5
>>>> active+remapped+backfill_toofull, 19 active+degraded+backfilling, 1
>>>> active+clean+scrubbing, 79 active+degraded+remapped+wait_backfill, 36
>>>> active+recovery_wait+remapped, 1
>>>> active+degraded+remapped+wait_backfill+backfill_toofull, 46
>>>> remapped+peering, 16 active+degraded+remapped+backfilling, 1
>>>> active+recovery_wait+degraded+remapped, 14 active+recovering; 50435 GB
>>>> data, 74790 GB used, 38642 GB / 110 TB avail; 4018849/23025448
>>>> degraded (17.454%); recovering 14 o/s, 54732KB/s
>>>>
>>>> # id weight type name up/down reweight
>>>> -1 130 root default
>>>> -9 65 room p1
>>>> -3 44 rack r14
>>>> -4 22 host s101
>>>> 11 2 osd.11 up 1
>>>> 12 2 osd.12 up 1
>>>> 13 2 osd.13 up 1
>>>> 14 2 osd.14 up 1
>>>> 15 2 osd.15 up 1
>>>> 16 2 osd.16 up 1
>>>> 17 2 osd.17 up 1
>>>> 18 2 osd.18 up 1
>>>> 19 2 osd.19 up 1
>>>> 20 2 osd.20 up 1
>>>> 21 2 osd.21 up 1
>>>> -6 22 host s102
>>>> 33 2 osd.33 up 1
>>>> 34 2 osd.34 up 1
>>>> 35 2 osd.35 up 1
>>>> 36 2 osd.36 up 1
>>>> 37 2 osd.37 up 1
>>>> 38 2 osd.38 up 1
>>>> 39 2 osd.39 up 1
>>>> 40 2 osd.40 up 1
>>>> 41 2 osd.41 up 1
>>>> 42 2 osd.42 up 1
>>>> 43 2 osd.43 up 1
>>>> -13 21 rack r10
>>>> -12 21 host s103
>>>> 55 2 osd.55 up 1
>>>> 56 2 osd.56 up 1
>>>> 57 2 osd.57 up 1
>>>> 58 2 osd.58 up 1
>>>> 59 2 osd.59 down 0
>>>> 60 2 osd.60 down 0
>>>> 61 2 osd.61 down 0
>>>> 62 2 osd.62 up 1
>>>> 63 2 osd.63 up 1
>>>> 64 1.5 osd.64 up 1
>>>> 65 1.5 osd.65 down 0
>>>> -10 65 room p2
>>>> -7 22 rack r20
>>>> -5 22 host s202
>>>> 22 2 osd.22 up 1
>>>> 23 2 osd.23 up 1
>>>> 24 2 osd.24 up 1
>>>> 25 2 osd.25 up 1
>>>> 26 2 osd.26 up 1
>>>> 27 2 osd.27 up 1
>>>> 28 2 osd.28 up 1
>>>> 29 2 osd.29 up 1
>>>> 30 2 osd.30 up 1
>>>> 31 2 osd.31 up 1
>>>> 32 2 osd.32 up 1
>>>> -8 22 rack r22
>>>> -2 22 host s201
>>>> 0 2 osd.0 up 1
>>>> 1 2 osd.1 up 1
>>>> 2 2 osd.2 up 1
>>>> 3 2 osd.3 up 1
>>>> 4 2 osd.4 up 1
>>>> 5 2 osd.5 up 1
>>>> 6 2 osd.6 up 1
>>>> 7 2 osd.7 up 1
>>>> 8 2 osd.8 up 1
>>>> 9 2 osd.9 up 1
>>>> 10 2 osd.10 up 1
>>>> -14 21 rack r21
>>>> -11 21 host s203
>>>> 44 2 osd.44 up 1
>>>> 45 2 osd.45 up 1
>>>> 46 2 osd.46 up 1
>>>> 47 2 osd.47 up 1
>>>> 48 2 osd.48 up 1
>>>> 49 2 osd.49 up 1
>>>> 50 2 osd.50 up 1
>>>> 51 2 osd.51 up 1
>>>> 52 1.5 osd.52 up 1
>>>> 53 1.5 osd.53 up 1
>>>> 54 2 osd.54 up 1
>>>>
>>>>
>>>> 2013/4/21 Marco Aroldi <marco.aroldi@xxxxxxxxx>:
>>>>> So, I've restarted the new osds as many as possible and the cluster
>>>>> started to move data to the 2 new nodes overnight.
>>>>> This morning there was not netowrk traffic and the healt was
>>>>>
>>>>> HEALTH_ERR 1323 pgs backfill; 150 pgs backfill_toofull; 100 pgs
>>>>> backfilling; 114 pgs degraded; 3374 pgs peering; 36 pgs recovering;
>>>>> 949 pgs recovery_wait; 3374 pgs stuck inactive; 6289 pgs stuck
>>>>> unclean; recovery 2130652/20890113 degraded (10.199%); 58/8914654
>>>>> unfound (0.001%); 1 full osd(s); 22 near full osd(s); full,noup,nodown
>>>>> flag(s) set
>>>>>
>>>>> So I have unset the noup and nodown flags and the data started movin again
>>>>> I've increased the full ratio to 97% so now there's no "official" full
>>>>> osd and the HEALTH_ERR became HEALT_WARN
>>>>>
>>>>> However, still no access to filesystem
>>>>>
>>>>> HEALTH_WARN 1906 pgs backfill; 21 pgs backfill_toofull; 52 pgs
>>>>> backfilling; 707 pgs degraded; 371 pgs down; 97 pgs incomplete; 3385
>>>>> pgs peering; 35 pgs recovering; 1002 pgs recovery_wait; 4 pgs stale;
>>>>> 683 pgs stuck inactive; 5898 pgs stuck unclean; recovery
>>>>> 3081499/22208859 degraded (13.875%); 487/9433642 unfound (0.005%);
>>>>> recovering 11722 o/s, 57040MB/s; 17 near full osd(s)
>>>>>
>>>>> The osd are flapping in/out again...
>>>>>
>>>>> I'm disposed to start deleting some portion of data.
>>>>> What can I try to do now?
>>>>>
>>>>> 2013/4/21 Gregory Farnum <greg@xxxxxxxxxxx>:
>>>>>> It's not entirely clear from your description and the output you've
>>>>>> given us, but it looks like maybe you've managed to bring up all your
>>>>>> OSDs correctly at this point? Or are they just not reporting down
>>>>>> because you set the "no down" flag...
>>>>>>
>>>>>> In any case, CephFS isn't going to come up while the underlying RADOS
>>>>>> cluster is this unhealthy, so you're going to need to get that going
>>>>>> again. Since your OSDs have managed to get themselves so full it's
>>>>>> going to be trickier than normal, but if all the rebalancing that's
>>>>>> happening is only because you sort-of-didn't-really lose nodes, and
>>>>>> you can bring them all back up, you should be able to sort it out by
>>>>>> getting all the nodes back up, and then changing your full percentages
>>>>>> (by a *very small* amount); since you haven't been doing any writes to
>>>>>> the cluster it shouldn't take much data writes to get everything back
>>>>>> where it was, although if this has been continuing to backfill in the
>>>>>> meanwhile that will need to unwind.
>>>>>> -Greg
>>>>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>>>>>
>>>>>>
>>>>>> On Sat, Apr 20, 2013 at 12:21 PM, John Wilkins <john.wilkins@xxxxxxxxxxx> wrote:
>>>>>>> I don't see anything related to lost objects in your output. I just see
>>>>>>> waiting on backfill, backfill_toofull, remapped, and so forth. You can read
>>>>>>> a bit about what is going on here:
>>>>>>> http://ceph.com/docs/next/rados/operations/monitoring-osd-pg/
>>>>>>>
>>>>>>> Keep us posted as to the recovery, and let me know what I can do to improve
>>>>>>> the docs for scenarios like this.
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Apr 20, 2013 at 10:52 AM, Marco Aroldi <marco.aroldi@xxxxxxxxx>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> John,
>>>>>>>> thanks for the quick reply.
>>>>>>>> Below you can see my ceph osd tree
>>>>>>>> The problem is caused not by the failure itself, but by the "renamed"
>>>>>>>> bunch of devices.
>>>>>>>> It was like a deadly 15-puzzle
>>>>>>>> I think that the solution was to mount the devices in fstab using UUID
>>>>>>>> (/dev/disk/by-uuid) instead of /dev/sdX
>>>>>>>>
>>>>>>>> However, yes I have an entry in my ceph.conf (devs = /dev/sdX1 --
>>>>>>>> osd_journal = /dev/sdX2) *and* an entry in my fstab for each OSD
>>>>>>>>
>>>>>>>> The node with failed disk is s103 (osd.59)
>>>>>>>>
>>>>>>>> Now i have 5 osd from s203 up and in to try to let ceph rebalance
>>>>>>>> data... but is still a bloody mess.
>>>>>>>> Look at ceph -w output: is reported a total of 110TB: is wrong... al
>>>>>>>> drives are 2TB and i have 49 drives up and in -- total 98Tb
>>>>>>>> I think that 110TB (55 osd) was the size before cluster became
>>>>>>>> inaccessible
>>>>>>>>
>>>>>>>> # id weight type name up/down reweight
>>>>>>>> -1 130 root default
>>>>>>>> -9 65 room p1
>>>>>>>> -3 44 rack r14
>>>>>>>> -4 22 host s101
>>>>>>>> 11 2 osd.11 up 1
>>>>>>>> 12 2 osd.12 up 1
>>>>>>>> 13 2 osd.13 up 1
>>>>>>>> 14 2 osd.14 up 1
>>>>>>>> 15 2 osd.15 up 1
>>>>>>>> 16 2 osd.16 up 1
>>>>>>>> 17 2 osd.17 up 1
>>>>>>>> 18 2 osd.18 up 1
>>>>>>>> 19 2 osd.19 up 1
>>>>>>>> 20 2 osd.20 up 1
>>>>>>>> 21 2 osd.21 up 1
>>>>>>>> -6 22 host s102
>>>>>>>> 33 2 osd.33 up 1
>>>>>>>> 34 2 osd.34 up 1
>>>>>>>> 35 2 osd.35 up 1
>>>>>>>> 36 2 osd.36 up 1
>>>>>>>> 37 2 osd.37 up 1
>>>>>>>> 38 2 osd.38 up 1
>>>>>>>> 39 2 osd.39 up 1
>>>>>>>> 40 2 osd.40 up 1
>>>>>>>> 41 2 osd.41 up 1
>>>>>>>> 42 2 osd.42 up 1
>>>>>>>> 43 2 osd.43 up 1
>>>>>>>> -13 21 rack r10
>>>>>>>> -12 21 host s103
>>>>>>>> 55 2 osd.55 up 0
>>>>>>>> 56 2 osd.56 up 0
>>>>>>>> 57 2 osd.57 up 0
>>>>>>>> 58 2 osd.58 up 0
>>>>>>>> 59 2 osd.59 down 0
>>>>>>>> 60 2 osd.60 down 0
>>>>>>>> 61 2 osd.61 down 0
>>>>>>>> 62 2 osd.62 up 0
>>>>>>>> 63 2 osd.63 up 0
>>>>>>>> 64 1.5 osd.64 up 0
>>>>>>>> 65 1.5 osd.65 down 0
>>>>>>>> -10 65 room p2
>>>>>>>> -7 22 rack r20
>>>>>>>> -5 22 host s202
>>>>>>>> 22 2 osd.22 up 1
>>>>>>>> 23 2 osd.23 up 1
>>>>>>>> 24 2 osd.24 up 1
>>>>>>>> 25 2 osd.25 up 1
>>>>>>>> 26 2 osd.26 up 1
>>>>>>>> 27 2 osd.27 up 1
>>>>>>>> 28 2 osd.28 up 1
>>>>>>>> 29 2 osd.29 up 1
>>>>>>>> 30 2 osd.30 up 1
>>>>>>>> 31 2 osd.31 up 1
>>>>>>>> 32 2 osd.32 up 1
>>>>>>>> -8 22 rack r22
>>>>>>>> -2 22 host s201
>>>>>>>> 0 2 osd.0 up 1
>>>>>>>> 1 2 osd.1 up 1
>>>>>>>> 2 2 osd.2 up 1
>>>>>>>> 3 2 osd.3 up 1
>>>>>>>> 4 2 osd.4 up 1
>>>>>>>> 5 2 osd.5 up 1
>>>>>>>> 6 2 osd.6 up 1
>>>>>>>> 7 2 osd.7 up 1
>>>>>>>> 8 2 osd.8 up 1
>>>>>>>> 9 2 osd.9 up 1
>>>>>>>> 10 2 osd.10 up 1
>>>>>>>> -14 21 rack r21
>>>>>>>> -11 21 host s203
>>>>>>>> 44 2 osd.44 up 1
>>>>>>>> 45 2 osd.45 up 1
>>>>>>>> 46 2 osd.46 up 1
>>>>>>>> 47 2 osd.47 up 1
>>>>>>>> 48 2 osd.48 up 1
>>>>>>>> 49 2 osd.49 up 0
>>>>>>>> 50 2 osd.50 up 0
>>>>>>>> 51 2 osd.51 up 0
>>>>>>>> 52 1.5 osd.52 up 0
>>>>>>>> 53 1.5 osd.53 up 0
>>>>>>>> 54 2 osd.54 up 0
>>>>>>>>
>>>>>>>>
>>>>>>>> ceph -w
>>>>>>>>
>>>>>>>> 2013-04-20 19:46:48.608988 mon.0 [INF] pgmap v1352767: 17280 pgs: 58
>>>>>>>> active, 12581 active+clean, 1686 active+remapped+wait_backfill, 24
>>>>>>>> active+degraded+wait_backfill, 224
>>>>>>>> active+remapped+wait_backfill+backfill_toofull, 1061
>>>>>>>> active+recovery_wait, 4
>>>>>>>> active+degraded+wait_backfill+backfill_toofull, 629 peering, 626
>>>>>>>> active+remapped, 72 active+remapped+backfilling, 89 active+degraded,
>>>>>>>> 14 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 8
>>>>>>>> active+degraded+remapped+wait_backfill, 20
>>>>>>>> active+recovery_wait+remapped, 5
>>>>>>>> active+degraded+remapped+wait_backfill+backfill_toofull, 162
>>>>>>>> remapped+peering, 1 active+degraded+remapped+backfilling, 2
>>>>>>>> active+degraded+remapped+backfill_toofull, 13 active+recovering; 49777
>>>>>>>> GB data, 72863 GB used, 40568 GB / 110 TB avail; 2965687/21848501
>>>>>>>> degraded (13.574%); recovering 5 o/s, 16363B/s
>>>>>>>>
>>>>>>>> 2013/4/20 John Wilkins <john.wilkins@xxxxxxxxxxx>:
>>>>>>>> > Marco,
>>>>>>>> >
>>>>>>>> > If you do a "ceph tree" can you see if your OSDs are all up? You seem to
>>>>>>>> > have at least one problem related to the backfill OSDs being too full,
>>>>>>>> > and
>>>>>>>> > some which are near full or full for the purposes of storage. See the
>>>>>>>> > following in the documentation to see if this helps:
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > http://ceph.com/docs/master/rados/configuration/mon-config-ref/#storage-capacity
>>>>>>>> >
>>>>>>>> > http://ceph.com/docs/master/rados/configuration/osd-config-ref/#backfilling
>>>>>>>> >
>>>>>>>> > http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#no-free-drive-space
>>>>>>>> >
>>>>>>>> > Before you start deleting data as a remedy, you'd want to at least try
>>>>>>>> > to
>>>>>>>> > get the OSDs back up and running first.
>>>>>>>> >
>>>>>>>> > If rebooting changed the drive names, you might look here:
>>>>>>>> >
>>>>>>>> > http://ceph.com/docs/master/rados/configuration/osd-config-ref/#general-settings
>>>>>>>> >
>>>>>>>> > We have default settings for OSD and journal paths, which you could
>>>>>>>> > override
>>>>>>>> > if you can locate the data and journal sources on the renamed drives. If
>>>>>>>> > you
>>>>>>>> > mounted them, but didn't add them to the fstab, that might be the source
>>>>>>>> > of
>>>>>>>> > the problem. I'd rather see you use the default paths, as it would be
>>>>>>>> > easier
>>>>>>>> > to troubleshoot later. So did you mount the drives, but not add the
>>>>>>>> > mount
>>>>>>>> > points to fstab?
>>>>>>>> >
>>>>>>>> > John
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Sat, Apr 20, 2013 at 8:46 AM, Marco Aroldi <marco.aroldi@xxxxxxxxx>
>>>>>>>> > wrote:
>>>>>>>> >>
>>>>>>>> >> Hi,
>>>>>>>> >> due a harware failure during expanding ceph, I'm in big trouble
>>>>>>>> >> because the cephfs doesn't mount anymore.
>>>>>>>> >> I was adding a couple storage nodes, but a disk has failed and after a
>>>>>>>> >> reboot the OS (ubuntu 12.04) renamed the remaining devices, so the
>>>>>>>> >> entire node has been screwed out.
>>>>>>>> >>
>>>>>>>> >> Now, from the "sane new node", I'm taking some new osd up and in
>>>>>>>> >> because the cluster is near full and I can't revert completely the
>>>>>>>> >> situation as before
>>>>>>>> >>
>>>>>>>> >> *I can* afford data loss, but i need to regain access to the filesystem
>>>>>>>> >>
>>>>>>>> >> My setup:
>>>>>>>> >> 3 mon + 3 mds
>>>>>>>> >> 4 storage nodes (i was adding no. 5 and 6)
>>>>>>>> >>
>>>>>>>> >> Ceph 0.56.4
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> ceph health:
>>>>>>>> >> HEALTH_ERR 2008 pgs backfill; 246 pgs backfill_toofull; 74 pgs
>>>>>>>> >> backfilling; 134 pgs degraded; 790 pgs peering; 10 pgs recovering;
>>>>>>>> >> 1116 pgs recovery_wait; 790 pgs stuck inactive; 4782 pgs stuck
>>>>>>>> >> unclean; recovery 3049459/21926624 degraded (13.908%); recovering 6
>>>>>>>> >> o/s, 16316KB/s; 4 full osd(s); 30 near full osd(s); full,noup,nodown
>>>>>>>> >> flag(s) set
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> ceph mds dump:
>>>>>>>> >> dumped mdsmap epoch 44
>>>>>>>> >> epoch 44
>>>>>>>> >> flags 0
>>>>>>>> >> created 2013-03-18 14:42:29.330548
>>>>>>>> >> modified 2013-04-20 17:14:32.969332
>>>>>>>> >> tableserver 0
>>>>>>>> >> root 0
>>>>>>>> >> session_timeout 60
>>>>>>>> >> session_autoclose 300
>>>>>>>> >> last_failure 43
>>>>>>>> >> last_failure_osd_epoch 18160
>>>>>>>> >> compat compat={},rocompat={},incompat={1=base v0.20,2=client
>>>>>>>> >> writeable ranges,3=default file layouts on dirs,4=dir inode in
>>>>>>>> >> separate object}
>>>>>>>> >> max_mds 1
>>>>>>>> >> in 0
>>>>>>>> >> up {0=6376}
>>>>>>>> >> failed
>>>>>>>> >> stopped
>>>>>>>> >> data_pools [0]
>>>>>>>> >> metadata_pool 1
>>>>>>>> >> 6376: 192.168.21.11:6800/13457 'm1' mds.0.9 up:replay seq 1
>>>>>>>> >> 5945: 192.168.21.13:6800/12999 'm3' mds.-1.0 up:standby seq 1
>>>>>>>> >> 5963: 192.168.21.12:6800/22454 'm2' mds.-1.0 up:standby seq 1
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> ceph mon dump:
>>>>>>>> >> epoch 1
>>>>>>>> >> fsid d634f7b3-8a8a-4893-bdfb-a95ccca7fddd
>>>>>>>> >> last_changed 2013-03-18 14:39:42.253923
>>>>>>>> >> created 2013-03-18 14:39:42.253923
>>>>>>>> >> 0: 192.168.21.11:6789/0 mon.m1
>>>>>>>> >> 1: 192.168.21.12:6789/0 mon.m2
>>>>>>>> >> 2: 192.168.21.13:6789/0 mon.m3
>>>>>>>> >> _______________________________________________
>>>>>>>> >> ceph-users mailing list
>>>>>>>> >> ceph-users@xxxxxxxxxxxxxx
>>>>>>>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > John Wilkins
>>>>>>>> > Senior Technical Writer
>>>>>>>> > Intank
>>>>>>>> > john.wilkins@xxxxxxxxxxx
>>>>>>>> > (415) 425-9599
>>>>>>>> > http://inktank.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> John Wilkins
>>>>>>> Senior Technical Writer
>>>>>>> Intank
>>>>>>> john.wilkins@xxxxxxxxxxx
>>>>>>> (415) 425-9599
>>>>>>> http://inktank.com
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
